Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
159175 stories
·
33 followers

Building safe MCP servers for your PostgreSQL database

1 Share

Model Context Protocol (MCP) is an open protocol that describes how agents can connect to external tools and data sources, and is now widely supported by the most popular coding agents (like GitHub Copilot, Claude Code, and Codex) and agent frameworks (like LangChain and Pydantic AI). If you want to give agents a standard way to access the data in a database, you can build your own MCP server and expose tools for the agent to query or even modify data. But you need to design your MCP server carefully, to ensure that agents can do everything that users want - but nothing that you don't want them to do!

In this blog post, we'll walk through the range of ways to build MCP servers on top of a PostgreSQL database, since PostgreSQL is the most popular open source database and is production-ready with hosted offerings like Azure Database for PostgreSQL. You can apply these same principles to any database, however.

There's a spectrum of ways to build MCP servers on top of a database. We'll start with the most flexible option, exploratory servers that allow the agent to generate full SQL queries, conclude with the strictest option, fully typed tools for templated queries, and explore options in the middle too.

Spectrum of MCP database designs from exploratory free-form SQL to operational, fully typed tools.

Free-form SQL

Let's take a look at a simple MCP server that gives the agent as much information and control as possible. For all of our examples, we use the Python language and the FastMCP package, but SDKs are available in multiple languages. All code is available in the GitHub repository.

We start off by giving the server a name, which the agent will see and consider when deciding which MCP server to invoke for a given user query:

mcp = FastMCP("Bees database MCP server")

For this example, my database stores observations of bees, so I name it accordingly.

We then define an execute_sql tool that accepts any SQL string, executes it against the database, and returns the rows.

@mcp.tool()
async def execute_sql(sql: str) -> str:
  """Execute a SQL query against the database and return results."""
  engine = await _get_engine()
  async with engine.connect() as conn:
    result = await conn.execute(text(sql))
    if result.returns_rows:
      columns = list(result.keys())
      rows = result.fetchall()
      return {"columns": columns, "rows": [[str(v) for v in row] for row in rows]}
    await conn.commit()
    return f"Statement executed. Rows affected: {result.rowcount}"

How will the agent know what SQL can be passed into that tool, however? We need to give it a way to discover the schema, so we also define a get_db_schema tool that dumps out the entire schema with table names, columns, and data types.

@mcp.tool()
async def get_db_schema() -> str:
  """Return the database schema for all public tables."""
  engine = await _get_engine()
  return await get_db_schema_text(engine)

We can test this MCP server out with a coding agent like GitHub Copilot. When we ask the agent "Which bees are active in El Cerrito in April?", the agent realizes that the Bees MCP server has relevant tools for the task, first calls get_db_schema, then calls execute_sql with a SELECT query. The database returns the results and the agent formats them into a Markdown table.

GitHub Copilot answering a question about bees in El Cerrito after calling the get_db_schema and execute_sql MCP tools.

This MCP server works - we got the answer we wanted - but as you may have already noticed, there are multiple problems and risks to this approach.


Problem: Schema bloat

Let's tackle the problem with the get_db_schema tool first - it dumps everything! My observations database has only 5 tables and 60 columns, but a production database may have hundreds of tables and thousands of columns. Dumping the entire schema can confuse the LLM with irrelevant information, and unnecessarily fill up its context window.

What can we do instead? Progressive schema discovery. We provide two tools: list_tables that only returns table names, and describe_table that returns the columns only for the given table.

@mcp.tool()
async def list_tables() -> str:
  """List all tables in the public schema. Call this first to discover available tables."""
  async with engine.connect() as conn:
    result = await conn.execute(text(
        "SELECT table_name FROM information_schema.tables "
        "WHERE table_schema = 'public' AND table_type = 'BASE TABLE'"))
    return {"tables": [row[0] for row in result.fetchall()]}

@mcp.tool()
async def describe_table(table_name: str) -> str:
  """Describe the columns of a specific table. Call list_tables() first to see available tables."""
  async with engine.connect() as conn:
    result = await conn.execute(text(
        "SELECT column_name, data_type, is_nullable FROM information_schema.columns "
        "WHERE table_schema = 'public' AND table_name = :table_name "),
        {"table_name": table_name})
    rows = result.fetchall()
  columns = [{"name": col, "type": dt, "nullable": n == "YES"} for col, dt, n in rows]
  return {"table": table_name, "columns": columns}

When we expose these tools to GitHub Copilot, the agent first calls list_tables, then makes two calls to describe_table, one for each relevant table. The agent requires 3 tool calls for schema discovery instead of the single call required before, so this server design can increase latency. However, for databases with large schemas, it prevents context bloat. You can decide based on schema size whether the tradeoff is worth it.

GitHub Copilot progressively discovering the database schema by calling list_tables followed by describe_table.

Problem: Mutations without guardrails

Now let's tackle the destructive elephant in the room: execute_sql can execute any valid SQL, including updates and deletions. If a user asks the agent, "How many bee observations have quality grade 'needs_id'? Might want to delete those", it might just delete thousands of rows with a single DELETE statement. If that's okay with you, great, but for many scenarios, you'll want to either completely prevent mutation or at least require user confirmation first.


Read-only SQL tool

Let's start by making a read-only version of the SQL execution tool. The execute_readonly_sql tool below includes multiple guardrails: a verification that the SQL contains only SELECT, a 30-second timeout to prevent expensive queries, and a maximum of 100 rows:

@mcp.tool(annotations=ToolAnnotations(readOnlyHint=True), timeout=30.0)
async def execute_readonly_sql(sql: str) -> dict:
  """Execute a read-only SQL query against the database.
  Only SELECT statements are allowed. Non-SELECT statements are rejected.
  Results are capped at 100 rows."""
  try:
    validated_sql = validate_readonly_sql(sql)
  except ValueError as e:
    raise ToolError(str(e))

  async with engine.connect() as conn:
    result = await conn.execute(text(validated_sql))
    columns = list(result.keys())
    rows = result.fetchmany(MAX_LIMIT) # Cap rows regardless of LIMIT
    return {"columns": columns, "rows": [[str(v) for v in row] for row in rows]}

Notice the tool is annotated with readOnlyHint=True, one of the allowed annotations from the MCP specification. When we set that read-only hint on a tool, we're sending a signal to the MCP client that this is a tool that does not modify data, which may affect how the client renders the tool or handles approvals. But it is only a hint, not a contract. A server could lie about it, or even unintentionally report it incorrectly. As the server developer, we must enforce actual read-only operations inside the tool logic itself.

That's the goal of validate_readonly_sql: a programmatic guarantee that the provided SQL string is a SELECT statement and nothing more. In Python, I implemented that check using the pglast package for parsing the Abstract Syntax Tree (AST) of the SQL string, confirming that it contained a single statement, and confirming that the single statement is specifically a SELECT statement:

def validate_readonly_sql(sql: str) -> str:
  try:
    stmts = pglast.parse_sql(sql)
  except pglast.parser.ParseError as e:
    raise ValueError(f"SQL parse error: {e}")

  if len(stmts) != 1:
    raise ValueError("Only one statement is allowed")
  if (stmt_type := type(stmts[0].stmt).__name__) != "SelectStmt":
    raise ValueError(f"Only SELECT statements are allowed, got {stmt_type}")
  return sql

That will block the majority of destructive SQL calls, such as:

Input Result
NOT VALID SQL!!! ❌ SQL parse error: syntax error
SELECT 1; DELETE FROM observations ❌ Only one statement is allowed
DELETE FROM observations ❌ Only SELECT statements are allowed, got DeleteStmt

We're not safe yet! There are still a few tricky destructive SQL statements that can pass that check. We could extend the AST-based parsing to try to block those, but PostgreSQL offers a better way: read-only enforcement at the database level.

When we connect to the database, we run this SET command to enforce read-only transactions only:

SET default_transaction_read_only = ON

That blocks these CTEs that start with WITH and hide mutations inside:

  • WITH d as (DELETE ...) SELECT * FROM d
  • WITH u as (UPDATE ...) SELECT * FROM d

We can go even further and create a dedicated PostgreSQL role for the MCP server that only has the ability to issue SELECT queries on a given schema:

CREATE ROLE mcp_readonly;
GRANT CONNECT ON DATABASE bees TO mcp_readonly;
GRANT USAGE ON SCHEMA public TO mcp_readonly;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO mcp_readonly;

That blocks these SELECT statements that call potentially destructive built-in SQL functions:

  • SELECT pg_terminate_backend(pid)
  • SELECT pg_read_file('/etc/passwd')
  • SELECT pg_reload_conf()

We could choose to enforce read-only access only via the least-privilege role, but that means your server has no layers of protection if that role isn't set properly for some reason. Just in case, it's best to employ all four layers of protection.

Could a malicious user or a capricious agent still find a way to slip a dangerous query through? If you need a 100% guarantee, the best option is to not expose SQL at all.


Templated query tools

In this approach, we define tools specific to common user needs, and those tools accept values that get safely merged into a templated SQL query - or passed to an ORM call.

For example, the search_species tool below accepts a search query string and an integer limit, and executes a templated SQL query on a hard-coded table:

@mcp.tool(annotations=ToolAnnotations(readOnlyHint=True))
async def search_species(q: str, limit: int = 10) -> list[SpeciesResults]:
  """Search bee species by scientific or common name.
  Use to resolve a name to a taxon_id before calling other tools."""
  sql = text("""
    SELECT taxon_id, scientific_name, common_name, family, genus FROM species
    WHERE to_tsvector('simple',
        coalesce(scientific_name, '') || ' ' || coalesce(common_name, ''))
        @@ plainto_tsquery('simple', :q)
    ORDER BY scientific_name ASC LIMIT :limit""")
  async with engine.connect() as conn:
    result = await conn.execute(sql, {"q": q, "limit": min(limit, 50)})
    return [SpeciesResult(...) for row in result.fetchall()]

We need to define additional tools for every SQL query that might be needed to answer user questions, like a search_observations_tool that accepts latitude, longitude, date, and species parameters.

When we provide GitHub Copilot with those tools and ask the question "Are there any carpenter bees around Berkeley?", the agent first calls search_species with a query of "carpenter bee" to get names and scientific metadata for matching bees, then calls search_observations with the latitude and longitude for Berkeley.

GitHub Copilot searching for carpenter bees near Berkeley by calling the typed search_species and search_observations tools.

The obvious advantage of this approach is that the agent never writes the SQL statements themselves, so it can't accidentally issue a destructive, expensive, or slow query.

There's a massive drawback: the agent can only answer the subset of user questions that you've anticipated. If you decide to go with this approach, try to find a way to monitor which of your users' questions can't be answered, perhaps by exposing a give_feedback tool on the server that encourages feature requests.


Elicitation for destructive actions

If you are developing an MCP server that basically serves as an administration tool (versus a data analysis and exploration tool), then you likely do want to allow deletion - but with caution. In a database admin UI, a delete button is typically bright red and pops up a dialog to confirm deletion before proceeding:

Database administration dialog requiring confirmation before permanently deleting selected rows.

We can achieve a similar UI for our MCP server, thanks to form-based elicitation, a relatively recent addition to the MCP spec. In the MCP clients that support elicitations, the client will pop up a form with our desired question and options. We can then change what our tool does, depending on what the user selects.

For example, this delete_observation tool uses an elicitation to confirm the user really wants to delete the row that it found in the database:

@mcp.tool(annotations=ToolAnnotations(destructiveHint=True))
async def delete_observation(ctx: Context, observation_id: int) -> str:
  """Delete a bee observation."""
  row = ... # look up the record
  result = await ctx.elicit(
    f"Permanently delete observation #{row.observation_id}?\n"
    f".  {row.scientific_name} on {row.observed_data}\n",
    response_type=["yes, delete it", "no, keep it"])
  if result.action == "cancel" or result.data == "no, keep it":
    return "Deletion cancelled."
  await session.execute(
    text("DELETE FROM observations WHERE observation_id = :oid"),
    {"oid": observation_id}
  )
  await session.commit()
  return f"Deleted observation #{observation_id}"

When we ask GitHub Copilot to delete an observation, the agent runs that delete_observation tool and the elicitation dialog pops up. The user has to explicitly click to confirm deletion.

GitHub Copilot displaying an MCP elicitation form that asks the user to confirm permanent deletion of a bee observation.

Elicitation is also useful beyond destructive operations. You can use it for resolving ambiguity in user queries ("Did you mean...?") or suggesting alternative queries when a request would be too expensive (like narrowing a 200 km search radius to 50 km).


Which approach should you use?

We've explored a range of options for exposing your database as an MCP server:

Comparison of MCP database designs: free-form SQL for prototyping, read-only SQL for analytics, and templated query tools for production, all protected by database-level permissions.

Free-form SQL is a good fit for internal prototyping where you need maximum flexibility. Read-only SQL works well for data analytics use cases, to allow arbitrary analysis. Templated queries are the safest bet for production and user-facing scenarios. Across all approaches, always enforce DB-level permissions to reduce risk.

Building MCP servers for your database is a great way to empower users to interact with data through natural language, but you should design your tools with safety in mind.

To learn more, explore the complete source code on GitHub which contains four MCP servers demonstrating each of the techniques, and can be run either locally on on Azure.

Read the whole story
alvinashcraft
just a second ago
reply
Pennsylvania, USA
Share this story
Delete

From Sync APIs to support for the GPT-5 model series and agentic workflows: What’s new in Azure Content Understanding – August 2026

1 Share

Enterprise content is no longer just something people read. AI apps and agents are only as useful as the information they can understand, yet much of the world’s enterprise knowledge is locked in documents, forms, tables, images, audio, and video. The latest Azure Content Understanding updates help developers turn that content into structured, grounded data with less custom processing. This release brings broader support for the GPT-5 model series, lower token usage, improved confidence scoring, new synchronous APIs for Read and Layout, advanced contextualization for higher-quality extraction, new tax-focused prebuilt analyzers, semantic chunking for retrieval workflows, and agentic document reasoning for more complex extraction scenarios.

Specifically, we’re announcing two updates to Azure Content Understanding in Foundry Tools:

  • A refreshed CU 1.0 API, now generally available for production workloads. CU 1.0 adds broader GPT-5 series support, lower token use, and improved grounding and confidence scoring.
  • A new CU 2.0 public preview for developers exploring next-generation document understanding. CU 2.0 adds synchronous Read and Layout APIs, advanced contextualization, semantic chunking, improved classification, new prebuilt analyzers, and agentic document reasoning.

Together, these updates make Content Understanding (CU) more efficient for production workloads today while expanding the range of document automation, retrieval, and reasoning scenarios developers can build tomorrow.

Overview of new CU features
Overview of new Azure Content Understanding features.

CU 1.0: Better economics and reliability for production workloads

We’ve made significant enhancements to the CU 1.0 GA API (2025-11-01), allowing customers to take advantage of broader GPT-5 series support, improved grounding efficiency, and a refreshed confidence model in their existing production workflows. Each of the three improvements below targets a different part of that production path: the model you run, the tokens it costs, and the confidence you can place in the result.

Expanded support for GPT-5.5 and lower GPT-5 series model support

Content Understanding analyzers can now use the GPT-5 model series, including GPT-5.5, GPT-5.4, GPT-5.3, GPT-5.2, GPT-5.1, and GPT-5 series models, across standard, mini, and nano variants.

This gives developers more flexibility to choose the right model deployment for their workload. Some scenarios prioritize maximum accuracy. Others prioritize latency or cost. By expanding GPT-5 series support, CU enables customers to evaluate these tradeoffs on their own data while continuing to use the same analyzer model and API patterns. Customers can use existing PTU commitments.

Improved grounding efficiency

The GA refresh also improves grounding efficiency by merging extraction and grounding more effectively in the processing flow. In internal evaluations, this reduced average inference token usage by up to 28 percent for GPT-4.1 and GPT-5.2, while also improving average accuracy by up to 3 percent.

For customers building high-volume extraction workflows, this matters in two ways. First, lower token usage can reduce the cost of running LLM-backed extraction. Second, better grounding efficiency helps preserve traceability to the source content without requiring developers to add extra post-processing logic.

Refreshed confidence model

The GA refresh includes a refreshed confidence model. In internal evaluations, accuracy measured by AUROC improved by up to 14 percent for GPT-4.1 and GPT-5.2.

Confidence scores are a critical part of production automation. They help applications decide when to move straight through, when to route to human review, and when to apply additional validation. A stronger confidence model makes it easier for teams to build automated pipelines that are both efficient and auditable.

For guidance on how to choose the right model for specific tasks and more details on our quality benchmarks see Azure Content Understanding GPT-5 Series Guide: Model Selection, Grounding Improvements, and Confidence Enhancements.

CU 2.0 preview: New building blocks for AI apps and agents

In addition to the enhancements available in the refreshed GA API, we are introducing the new CU 2.0 Preview (2026-06-01-preview).

CU 2.0 preview advances two developer priorities:

  1. Improving the quality and efficiency of content pipelines, and;
  2. Enabling new low-latency and reasoning-intensive scenarios.

The capabilities below build on the same GPT-5 series foundation as the refreshed GA API.

GPT-5.5 and lower models supported in preview

The CU 2.0 public preview also supports the GPT-5 model series. This means customers can test new preview capabilities using the same model series direction as the refreshed GA path.

This is important for migration planning. Customers can evaluate accuracy, latency, and cost on representative data before deciding which model deployment and API version best fit their production needs.

Improving quality and efficiency for production workloads

The first goal of the new CU 2.0 preview version is to improve the work teams already run in production: raising extraction quality, lowering token cost, and making retrieval and routing more dependable. Five preview capabilities move in that direction, starting with Advanced Contextualization, which underpins several of the new features.

Advanced Contextualization

Advanced Contextualization helps custom analyzers use labeled examples and document knowledge more efficiently. In internal evaluations, it improved average accuracy by up to 3.5 percent while reducing average LLM token usage by up to 22 percent. Training data remains in the customer’s Azure Storage account and is used as a knowledge source rather than copied into the analyzer, preserving customer-controlled storage. The result is higher-quality structured extraction with fewer tokens and less data-management overhead.

For more details, see analyzer improvements for more information on training data management.

The same approach is used in five new prebuilt analyzers. For these prebuilt analyzers, Advanced Contextualization reduces LLM token consumption by up to 99 percent. That makes the new prebuilt analyzers more practical for high-volume scenarios where both quality and cost matter.

The takeaway is simple: Advanced Contextualization is not just a quality feature. It is a production efficiency feature. It helps customers get better structured extraction while using fewer LLM tokens and keeping training inputs under their own storage governance.

New prebuilt analyzers for tax and other document types

This preview introduces new prebuilt tax analyzers powered by Advanced Contextualization, extending support beyond individual tax forms to enterprise and state-level tax workflows: 1065, 1120-S, 8865, 1041 Schedule K-1 and Minnesota State M1.

The new analyzers support complex, multi-page layouts and use advanced contextualization to achieve higher extraction quality, lower latency, and competitive pricing, dramatically reducing LLM token consumption, with some extraction scenarios requiring no LLM tokens at all. These prebuilt analyzers are also available in the Content Understanding Studio and Microsoft Foundry, allowing developers to visually verify values, review bounding boxes for in-document grounding, and inspect confidence scores before deploying to production. For the complete list of available prebuilt analyzers and guidance on how to use and customize them, see prebuilt analyzers documentation.

K 1 Tax form in CU Studio image
The Schedule K-1 prebuilt analyzer (prebuilt-tax.us.1065ScheduleK1) converts complex tax forms into structured JSON for downstream processing.

Semantic chunking in prebuilt-documentSearch

Retrieval quality often depends on how content is chunked. Fixed-size chunking can split a table from its heading, separate related paragraphs, or break a multi-page structure into fragments that are hard for retrieval systems to use.

The CU 2.0 public preview adds semantic chunking in prebuilt-documentSearch. Instead of splitting only by character count or page boundary, semantic chunking uses document structure to create more meaningful retrieval units. Semantic chunking preserves context and semantic relationships across sentences and paragraphs, and it is especially useful for RAG and agentic retrieval scenarios where the quality of what gets retrieved directly impacts the quality of reasoning outcome.

Classification: in-page splitting and confidence

Real enterprise submissions do not always align cleanly with page boundaries. A loan package, tax submission, case file, or scanned packet may contain multiple logical documents, and a single physical page can include the end of one section and the beginning of another.

The CU 2.0 public preview improves classification with in-page splitting. This allows classification to identify document segments at finer granularity than whole pages.

The preview also adds confidence for splitting and classification. Applications can use those signals to decide when to route a segment automatically, when to invoke a downstream analyzer, and when to send a result for human review.

This moves classification from a best-effort routing step toward a more operationally useful control point in document automation pipelines. For details, see classification enhancements.

Signature detection and metadata extraction in Layout

Classification decides what a document is; the Layout analyzer captures more of what is inside it. It now adds signature detection and document metadata extraction.

Signature detection helps identify signature regions and their locations in documents such as contracts, forms, invoices, and signed submissions. Metadata extraction surfaces available document properties such as author, title, creation date, content type, and language.

Combined with Layout analyzer’s existing ability to extract document structure and visual elements, including sections, headings, formatting, tables, figures, hyperlinks, annotations, and other layout elements, these new capabilities provide a more complete representation of document content and context from a single analyzer, making it easier to build intelligent document processing and agentic workflows.

To learn more, see the Layout analyzer documentation.

Try out signature detection and metadata extraction in the Content Understanding Studio.

Signature feature in CU Studio image
Signature detection in Azure Content Understanding Studio, showing detected signature regions alongside the source document.

Expanding what Content Understanding can solve

The second goal of this preview API is to reach new scenarios beyond the standard extraction. Two capabilities open that door: synchronous processing and agentic reasoning for the hardest extractions.

Read and Layout APIs

Document workflows such as grounding an AI agent during a customer interaction, validating an identity document, or triggering a workflow when a file is submitted require an immediate response. CU 2.0 preview adds synchronous operations for the Read (prebuilt-read) and Layout (prebuilt-layout) analyzers to support these low-latency scenarios. The operations return structured results directly in the response. Documents can be submitted as binary content or by URL and are processed without temporary service-side storage. To learn more, see Azure Content Understanding announces Synchronous Operations.

Agentic mode for complex field extraction

If synchronous APIs are about responding faster, agentic mode is about reasoning harder. Some document extraction tasks require more than a single pass over the content. The answer may depend on evidence spread across a long document, values may need to be compared or validated, or a field may require reasoning over intermediate results before producing a final output.

For these scenarios, the CU 2.0 preview introduces agentic mode.

Agentic mode applies an iterative extraction workflow for complex document understanding. It is designed for harder extraction scenarios where standard extraction may not be sufficient, such as long legal agreements, financial filings, insurance records, or other documents where the relevant evidence is distributed across multiple sections.

Agentic mode works with an analyzer schema and uses additional reasoning to identify relevant content, extract values, evaluate intermediate results, and refine the final output. Because it performs additional reasoning, it can increase latency and token consumption compared with standard extraction. Customers should evaluate agentic mode on representative documents and use it when the expected quality gain justifies the additional cost and processing time. Learn more about agentic mode.

How to get started

The two updates are designed to be used together: one for production today, and one for evaluating what comes next. Choose the path that matches your workload:

  • Use the refreshed CU 1.0 GA API to improve existing workloads, including GPT-5 series support, grounding efficiency improvements, and the refreshed confidence model.
  • Use the CU 2.0 public preview to evaluate the next generation of CU capabilities, including synchronous Read and Layout APIs, Advanced Contextualization, semantic chunking, new prebuilt analyzers, improved classification, and agentic mode.

To explore prebuilt analyzers, custom analyzers, and structured outputs, start in Content Understanding Studio or Microsoft Foundry.

The post From Sync APIs to support for the GPT-5 model series and agentic workflows: What’s new in Azure Content Understanding – August 2026 appeared first on Microsoft Foundry Blog.

Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

Azure Content Understanding GPT-5 Series Guide: Model Selection, Grounding Improvements, and Confidence Enhancements

1 Share

Enterprise content is no longer just something people consume. As organizations increasingly rely on AI to extract and act on information from documents, images, audio, and video, Azure Content Understanding is expanding support for the GPT-5 series and improving grounding and confidence to deliver greater flexibility, efficiency, and quality.

This expanded model catalog enables organizations to choose the right level of intelligence for each workload, helping reduce costs for high-volume processing while preserving access to advanced reasoning capabilities where needed. It also provides optimized pipelines tuned for each model. Preprocessing allows the models to support larger files and higher quality than the simple LLM document pipelines. Generating grounding and confidence scores enables automated validation and higher straight-through processing rates. It also provides a clear path forward as older foundation models retire, allowing customers to transition to newer generations of models without redesigning their Content Understanding workflows.

What’s New

This release expands Content Understanding support to the GPT-5, GPT-5.1, GPT-5.2, GPT-5.4, and GPT-5.5 series including standard, mini, and nano models across document, image, video, and speech analysis.

Just as important, the release introduces an updated grounding and confidence scoring method that generates higher quality outputs and reduces overall cost. In our tested configurations, it consumed up to 28% fewer total inference tokens and the full-inference LLM cost decrease by up to 25% while improving confidence scores accuracy and grounding accuracy by up to 14% and 3% respectively (measured by AUROC and grounding exact match).

Choosing the Best Model for Your Task

Think of model selection as a mixing board, with quality on your content and end-to-end cost as the two faders to adjust. A model that excels on forms may not lead on other tasks such as video segmentation, speech classification, or image generation. As organizations increasingly leverage AI to extract information from content selecting the right model is critical for balancing accuracy, cost, latency, throughput, compliance, and regional deployment requirements.

While we cannot benchmark every combination of input types, schema definitions, and deployment topologies, below is a set of starting points and recommendations based on our testing of the most common scenarios.

We encourage customers to leverage the table above to choose a model short list, and then evaluate on your own data to make the decision. Meanwhile, you should consider other factors such as your budget, quality bar, regional availability, throughput target, and existing capacity.

Table 1: General Model Selection Guidelines

Modality Balanced recommendation Best quality Lower-cost choice
Document GPT-5.1 or GPT-5.2 GPT-5.5 about +2% better quality at about 101% higher cost GPT-5.4 Mini costs about 50% less with an average –2% lower quality on our answer match metric than balanced GPT-5.2
Video GPT-5 or GPT-5.1 Same as balanced for this use case GPT-5 Mini costs about 28% less than GPT-5 with about –7% lower overall Generation F1
Speech1 GPT-5.1 or GPT-5.2 GPT-5.5 about +2% better quality than GPT-5.2 at about 101% higher cost GPT-5.4 Mini costs about 48% less with about –2 Answer Match points lower quality than balanced GPT-5.2
Image GPT-5.1 GPT-5.5 delivers about +3% better quality for about 130% higher cost For classification, GPT-5 Mini costs about 52% less than GPT-5.1 with about –12% lower Classify F1

1 Note: Speech and document extraction are similar tasks, so we currently recommend the same models for both.

Detailed model quality and cost analysis is included in the Appendix.

Grounding and Confidence improvements

In this release, we also improved grounding efficiency and refreshed the underlying confidence scoring method, so customers can get more useful evidence and ranking signals without building complex post-processing systems.

Grounding: Fewer Tokens Consumed

The updated grounding system more efficiently identifies the source for the extracted data. Across all model types, we observed 20-30% fewer input tokens and 18-28% fewer total inference tokens per document.

Bar chart comparing input and output tokens per document for GPT-4.1 and GPT-5.2 with previous and updated grounding methods. Updated grounding reduces tokens by 28% for GPT-4.1 and 23% for GPT-5.2, highlighted with red bars and specific token counts.
Updated grounding reduced zero-shot inference tokens by 28% for GPT-4.1 and 23% for GPT-5.2

Those reductions lowered the full-inference LLM bill by 11-25% across the tested model-and-labeled-sample configurations and reduced P50 and P95 latency. Grounding accuracy remains similar to the previous method.

Confidence: Improved Accuracy and Generalizability

We crafted a new confidence scoring method that is applicable to a broader set of models. Measured by AUROC, which measures how reliable the confidence scores rank correct fields above incorrect fields across various threshold, the new confidence scoring system improved by about 9% for GPT-4.1 and 14% for GPT-5.2 against the previous method.

Conf ranking image
Confidence-sensitive workloads: We observed a significant confidence-model quality drop with GPT-5 Mini and GPT-5 Nano. Avoid these models when confidence quality is important to your workflow.

A field’s baseline confidence score combines several inputs, and score distributions can differ by field type. For straight-through processing, set acceptance thresholds field by field and recalibrate them whenever you switch models.

Appendix: Detailed Model Quality/Cost Analysis

In this Appendix, we present more details on the model quality and cost analysis conducted in-house. These results may not be representative for every production workload, but it should be helpful to the reader as a guidance for model selection.

Documents/Speech: Finding the Right Balance

Dataset

The evaluation dataset span from structured to semi-structured to unstructured documents, with a total of 71 document types. Since Content Understanding supports field extraction with and without labeled samples, we tested configurations where there are zero, one, five, and all available labeled samples, and average across them to calculate the average accuracy of a given model.

Quality is reported as a macro average across leaf fields and analyzers, excluding container fields such as arrays and objects. The evaluated documents averaged 3.35 pages. Note workloads with substantially longer files, different schemas, or different training-example strategies may see a different cost and quality frontier.

We use these document results to guide the current speech recommendation because both tasks extract structured fields from source content. We are not publishing a separate speech accuracy graph.

Results

In the figure below, GPT-5.1 and GPT-5.2 stand out as well-balanced models between AI quality and cost, and they differ by about 1% in Answer Match score. At the top of the quality range, GPT-5.5 delivers about 2% more Answer Match than GPT-5.2, but increases estimated average cost by about 101%.

Compared with GPT-5.2, GPT-5.4 Mini reduces estimated cost by about 48% while giving up about 2% in Answer Match. We recommend customers to start with GPT-5.1/5.2 or GPT-5.4 mini, then add GPT-5.5 when its quality improvement can justify the premium.

docs image
GPT-5.1 and GPT-5.2 form the balanced document cost-quality frontier

Video: the Biggest Model is not the Best Model

Dataset

The video evaluation combined two distinct workload shapes: a segmentation-focused, 60-minute video dataset, and a short whole-video dataset averaging just under one minute. We compute generation F1 score covering both scalar answer fields and timestamped custom segments, e.g., semantically relevant time windows such as when a logo appears on screen, the duration of a news segment, or an ad break.

This benchmark is most relevant to workloads that extract both video-level facts and segments from long media. Short clips, different frame density, or schemas without segmentation may produce a different ranking.

Results

As shown in the figure below, GPT-5 reached 89.0% overall Generation F1, GPT-5.1 reached 88.6%, and GPT-5.4 reached 87.5%. GPT-5 and GPT-5.1 form both the balanced and best-quality pair for this use case; GPT-5 also cost about 18% less than the GPT-4.1 baseline in the release summary.

For a lower-cost video option, we recommend customers start with GPT-5 Mini. It cost about 28% less than GPT-5 on the segmentation-focused benchmark while giving up about 7% overall Generation F1.

GPT-5 matches baseline video segmentation quality at 18% lower benchmark cost

video image
GPT-5 matches baseline video segmentation quality at 18% lower benchmark cost

Images: Select Different Models for Classify and Generate tasks

Dataset

The image evaluation covered 17 zero-shot datasets with five repeats. The set spanned classification and generative tasks across product, apparel, industrial-defect, scene, and object-oriented content.

Classification used F1 as metrics. Generative fields used a 1-7 rubric score, so the two quality axes answer different questions and should not be collapsed into one winner.

Results

As shown in the figure below, GPT-5.1 led classification at 69.5% Classify F1, making it an attractive choice for that use case. For generation, GPT-5.5 led at 5.0 out of 7, about 3% higher than GPT-5 Mini at about 375% higher cost.

img image

For a lower-cost classification option, GPT-5 Mini cost about 52% less than GPT-5.1 while giving up about 12% Classify F1. We recommend customers to start with GPT-5.1 for classification-led workloads, and GPT-5 Mini for generation-led workloads. If necessary, test if GPT-5.5’s quality gain could justify the premium.

img 2 image
GPT-5.1 leads image classification while costing less than the GPT-4.1 baseline

So what should teams do next?

The expanded catalog gives every channel on the mixing board a wider range. The next step is to choose two or three candidates for your workload and test which combination of quality, cost, availability, and throughput deserves the final setting.

Keep the comparison simple: use the same analyzer, schema, representative input set, and labeled examples for every run. Change only the model deployment, then compare output quality, latency, token usage, and failure rate. Before testing, check the analyzer’s supportedModels response and confirm that each candidate is available in your region.

Start in Content Understanding Studio

  1. In Content Understanding Studio, open Settings, add your Foundry resource, and configure its default model deployments. Studio can deploy required models automatically when no suitable default exists.
  2. Follow the Content Understanding Studio quickstart to select an analyzer and run it on your own representative content.
  3. Test each candidate against the same files. Review the extracted fields and raw response, and record the quality, latency, and usage that matter to your workload.

Or compare deployments through the REST API

For a repeatable evaluation, pass a different modelDeployments mapping in each analyze request. A request-level mapping overrides the resource defaults, so you can keep the analyzer and inputs unchanged while swapping the completion deployment.

For example, you can call:

POST /contentunderstanding/analyzers/myInvoice:analyze

{
  "inputs": [
    {
      "url": "<representative-input-url>"
    }
  ],
  "modelDeployments": {
    "prebuilt-analyzer-completion": "<candidate-deployment-name>",
    "prebuilt-analyzer-embedding": "<embedding-deployment-name>"
  }
}

Run the same request once per candidate, then compare the extracted results and the response’s usage data. Start with the balanced recommendation, add the lower-cost option, and include the best-quality model only when the remaining accuracy gap matters to the workflow.

Additional Links

The post Azure Content Understanding GPT-5 Series Guide: Model Selection, Grounding Improvements, and Confidence Enhancements appeared first on Microsoft Foundry Blog.

Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

From assistance to execution: How enterprises put AI to work

1 Share
OpenAI research reveals how enterprises are adopting agentic AI, using ChatGPT and Codex, and how frontier firms are pulling ahead in AI adoption.
Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

The Claude in Chrome side panel is now Claude Cowork

1 Share
The Claude in Chrome side panel is now Claude Cowork
Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

Introducing Grok 4.6

1 Share
Built for long-running agents and more ambitious interactive and visual work.
Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories