Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
158385 stories
·
33 followers

Fast Web Services with Marten and Polecat

1 Share

In many .NET systems, writing a web service that returns query results means some combination of:

  1. Query data from EF Core — which is going to do who knows what to build up SQL, execute that, then spend some time materializing the raw database results into .NET objects
  2. Since we’ve all been taught for years that it’s harmful to expose our internal entity shapes to the outside world, maybe you’re running the results through some kind of object to object mapping to a different DTO shape
  3. Finally, after all the database querying and object mapping, you’ll finally use a JSON serializer to write results to the HTTP response stream

Whew. That’s a non-trivial amount of your time (or AI tokens) and a significant amount of runtime overhead with all the transformations and thrashing your memory with all the object allocations involved.

Now let’s talk about some capabilities in Marten and Polecat to sidestep the mass majority of that overhead in some cases — but first, I do need to say that if you’re using Event Sourcing, the persisted data in a Marten or Polecat database for query models is purpose built for clients as it is. No extra mapping necessary. In a way, the “AutoMapper” activity happens directly in projections for a system using Event Sourcing.

If you are building HTTP services on top of Marten or Polecat, both of these tools have a “JSON Streaming” feature that can be used to build very fast web services by writing the raw JSON stored in PostgreSQL or SQL Server directly to the HTTP response for the most efficient possible HTTP web services in the read side of a CQRS architecture.

Core team member Anne Erdtsieck just made some a bunch of extensions to Marten and Polecat‘s ability to stream the raw, persisted JSON data stored in the database straight to HTTP responses, and that makes now a good time to show off what we have.

For Minimal API endpoints (and for frameworks like Wolverine.Http that dispatch any IResult return value), Marten.AspNetCore (Polecat.AspNetCore has similar support) ships seven typed result wrappers that carry the streaming behavior above as endpoint return values while also contributing correct OpenAPI metadata:

TypeSourceResponse shape404 on miss?
StreamOne<T>IQueryable<T> — regular Marten document querySingle Tyes
StreamMany<T>IQueryable<T> — regular Marten document queryJSON array T[]no (empty array = 200)
StreamAggregate<T>IDocumentSession + stream id — event-sourcedSingle Tyes
StreamPaged<T>IQueryable<T> — regular Marten document queryPaged JSON envelopeno (empty page = 200)
StreamPagedByCursor<T>IQueryable<T> (with OrderBy/ThenBy)no (empty array = 200)
StreamEventStateIQuerySession + stream id — event streamSingle StreamStateResponseyes
StreamEventsIQuerySession + stream id — event streamJSON array EventResponse[]yes (configurable)

Each type implements both IResult (so ASP.NET Minimal API dispatches it via ExecuteAsync) and IEndpointMetadataProvider (so Swashbuckle, NSwag, and the built-in OpenAPI generator see the right response shape), while delegating the actual body write to WriteSingle/WriteArray/WriteLatest/WriteStreamState/WriteEvents. Returning one from an endpoint is a concise, typed alternative to writing the HTTP handshake manually.

StreamOne<T> — single document with 404 on miss

app.MapGet("/issues/{id:guid}",
(Guid id, IQuerySession session) =>
new StreamOne<Issue>(session.Query<Issue>().Where(x => x.Id == id)));

Returns 200 application/json with the document JSON on a hit, 404 on a miss. Content-Length and Content-Type are set automatically, matching the behavior of WriteSingle<T>.

StreamMany<T> — JSON array

app.MapGet("/issues/open",
(IQuerySession session) =>
new StreamMany<Issue>(session.Query<Issue>().Where(x => x.Open)));

Returns 200 application/json with a JSON array body. An empty result set yields [], not a 404 — matching the behavior of WriteArray<T>.

StreamPaged<T> — paged JSON envelope (single round trip)

app.MapGet("/issues/paged/{pageNumber:int}/{pageSize:int}",
(int pageNumber, int pageSize, IQuerySession session) =>
new StreamPaged<Issue>(session.Query<Issue>().OrderBy(x => x.Description), pageNumber, pageSize));

Returns 200 application/json with a single JSON envelope combining paging metadata and the matching documents for that page:

{"pageNumber":3,"pageSize":25,"totalItemCount":1207,"pageCount":49,"hasNextPage":true,"hasPreviousPage":true,"items":[...]}

pageNumber is 1-based. totalItemCount and pageCount are computed from a count(*) OVER() window function added to the same SQL query that fetches the page, so the whole response — count and documents both — comes from a single database round trip. Documents inside items are streamed as raw, already-persisted JSON, without a deserialize/serialize step. An empty page still returns 200 with totalItemCount: 0pageCount: 0, and an empty items array — never a 404.

Internally, StreamPaged<T> delegates to the IQueryable<T>.StreamPagedJsonArray() extension method described in the Paging docs, which can also be used directly (e.g. from an MVC controller action) instead of through the IResult wrapper.

StreamAggregate<T> — event-sourced aggregate (latest)

app.MapGet("/orders/{id:guid}",
(Guid id, IDocumentSession session) =>
new StreamAggregate<Order>(session, id));

Returns 200 application/json with the JSON of the latest projected aggregate state, or 404 if no stream exists. A constructor overload accepts string ids for stores configured with string-keyed streams.

StreamEventState — event stream metadata

Writes the high level metadata of a single event stream — Marten’s StreamState — as JSON, or 404 when the stream does not exist:

app.MapGet("/minimal/order/{id:guid}/state",
(Guid id, IQuerySession session)
=> new StreamEventState(session, id));

A constructor overload accepts a string stream key for stores configured with string-keyed streams.

The response body is a StreamStateResponse, not StreamState itself. StreamState.AggregateType is a System.Type, and System.Text.Json refuses to serialize those outright (Serialization and deserialization of 'System.Type' instances is not supported), so the aggregate type is projected down to its simple name in AggregateTypeName:

{
"id": "0198e1b4-5b1c-7a1e-9a3f-2f2f5b6c7d8e",
"key": null,
"version": 2,
"aggregateTypeName": "Order",
"lastTimestamp": "2026-07-26T09:41:02.113Z",
"created": "2026-07-26T09:41:02.098Z",
"isArchived": false
}

StreamEvents — raw events of a stream 9.20

Writes the raw events of a single event stream as a JSON array:

app.MapGet("/minimal/order/{id:guid}/events",
(Guid id, IQuerySession session)
=> new StreamEvents(session, id));

StreamEvents carries the same optional versiontimestamp, and fromVersion filters as FetchStreamAsync(), and there is a string stream key overload as well.

Elements are EventResponse, not IEvent itself — IEvent.EventType is a System.Type and hits the same System.Text.Json wall as above. Use eventTypeName, Marten’s stable event type alias, to discriminate event types on the client. The assembly qualified .NET type name (DotNetTypeName) is deliberately left off the wire:

[
{
"id": "0198e1b4-5b1c-7a1e-9a3f-2f2f5b6c7d8e",
"version": 1,
"sequence": 41,
"streamId": "0198e1b4-5b1c-7a1e-9a3f-2f2f5b6c7d8e",
"streamKey": null,
"eventTypeName": "order_placed",
"timestamp": "2026-07-26T09:41:02.098Z",
"tenantId": "*DEFAULT*",
"isArchived": false,
"causationId": null,
"correlationId": null,
"headers": null,
"data": { "description": "Widget", "amount": 99.95 }
}
]

Empty streams: 404 or an empty array?

FetchStream yields an empty list both for a stream that does not exist and for a filter that excludes every event, and the two cannot be told apart. StreamEvents therefore exposes an OnEmptyStatus that defaults to 404, matching the other single-resource results. Set it to 200 when running off the end of a stream is expected rather than exceptional — paging forward with fromVersion, for example:

// Paging forward through a stream: running off the end is expected, not a 404
app.MapGet("/minimal/order/{id:guid}/events/from/{fromVersion:long}",
(Guid id, long fromVersion, IQuerySession session)
=> new StreamEvents(session, id, fromVersion: fromVersion)
{
OnEmptyStatus = StatusCodes.Status200OK
});


Read the whole story
alvinashcraft
43 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

Tool search: Finding the right tool at the right time

1 Share

Every tool you give an agent is both a capability and a distraction. Five tools make an agent feel capable. Fifty tools make it feel prepared. A hundred tools can make every turn start with thousands of tokens of names, descriptions, JSON schemas, argument definitions, and nested parameters before you’ve asked anything useful. The agent looks more powerful, but first it has to read a menu it may not need.

This is one of the tensions Toolboxes in Microsoft Foundry is designed to solve at enterprise scale. A single toolbox can front Microsoft IQ, Work IQ, OpenAPI tools, A2A integrations, remote MCP servers, and several native Azure capabilities. Agent builders should be able to scale heterogeneous tool catalogs without rebuilding integrations or loading every tool on every turn. The old line about great power going hand-in-hand with great responsibility applies here, but the responsibility lives at the context layer: if the platform makes it easy to connect everything, it also needs a way to keep the model focused on what matters now.

Tool search capability in Toolbox emerged as we worked backwards from customer experience. Large tool catalogs were becoming too expensive to send to the model on every turn, and the model didn’t need most of them for most tasks. Our initial experiment of deferring all the tools and letting the model search for tools based on user query did what we hoped: it made tool-using agents cheaper, made the prompt smaller, and kept the system prompt stable enough to work well with prompt caching.

Then the real story emerged. We thought we were solving a token-cost problem; we were also building a search product. At small scale, a catalog can look like schema management: register the tool, validate the JSON, expose it to the model, dispatch the call. At larger scale, tool names and descriptions become ranking features. That’s the shift from tool-maxxing to tool relevance-maxxing.

The default agent tax 

The problem isn’t just cost, though cost is the easiest part to measure. A full manifest also fills the context window with definitions unrelated to the current task. The model has to choose from an overcrowded menu, and the prompt prefix becomes larger and more fragile.

Prompt caching is enabled by default in Azure OpenAI and is the recommended behavior, so our baseline had to include it. But caching isn’t the same as not loading. Cached tokens are roughly 90% cheaper than regular input tokens, not free, and cached context still competes for the model’s attention. The obvious move was to stop loading everything upfront.

Two tools instead of a hundred 

Tool search changes the initial contract between the toolbox and the model. Instead of exposing every toolbox tool in the first tools/list, Foundry can expose two meta-tools: tool_search(query, limit) and call_tool(name, arguments). The model describes the capability it needs, Foundry searches the toolbox, and the model receives a small set of matching tool definitions before calling the chosen tool with name and arguments.

The rest of the catalog stays hidden from the initial tool list. That’s why the second proxy exists: if a tool wasn’t registered in the original tools/list, many runtimes will guard against the model calling it directly as an unknown tool. call_tool gives the framework a registered, policy-aware dispatch path.

An architectural diagram showing how tool search surfaces two tools rather than 100 tools

That sounds almost too small to be an architecture, which is partly why we liked it. It works inside today’s tool-calling contract and doesn’t depend on a model-specific feature. Strategically placing this at the Toolbox layer is critical because remote MCP servers, OpenAPI tools, A2A integrations, native Azure tools, and other entries can be discovered through one mechanism. Foundry indexes the tool name, description, argument names, and argument descriptions up to three levels of nesting.

That minimal shape was deliberate. We didn’t want builders to tune a retrieval system before they had shipped an agent. The default should be sane enough to try immediately: attach a toolbox, enable search, and let the platform choose the initial indexing and ranking behavior. The surface also stays model-agnostic: no special model family, new MCP primitive, or shared ranking semantics across every provider. Builders only need to tell the agent to call tool_search before concluding a capability is missing, and they can then steer the shortlist size with limit: five results by default, more for ambiguous workflows, fewer for narrow ones.

The savings were real

We evaluated tool search on ToolRet, a large-scale open benchmark with more than 44,000 tools and 7,000 queries. The goal was to measure how token savings change as the catalog grows, so we ran an ablation that increased the number of tools available to the model and compared token consumption with and without tool search. The full tradeoff is shown in Figure 2 below.

A chart showing tokens saved using tool search compared to baseline

​ 
The savings scaled with toolbox size. With 50 tools in the toolbox, tool search reduced token use by more than 60%. With 1,000 tools, the savings rose above 97%. In this baseline, every tool was provided to the model upfront, and the model chose from the full catalog.

Retrieval quality was the real test

Retrieval quality was the harder question. We measured it with Recall@10 and compared tool search against BM25s and BGE-reranker-v2-gemma, as shown in Figure 3. For this run, we used only the user query and left out the benchmark’s instruction string. Generating that instruction at runtime would require another LLM call, adding both cost and latency, so the test reflected the cheaper path we would want in production. Tool search uses an enhanced sparse-retrieval pipeline with lexical similarity matching, and its Recall@10 was comparable to GPU-based reranking without depending on expensive cross-encoder reranking at serving time.

WebCodeCustomized
Tool search45.99%39.56%41.36%
BM25s24.62%28.23%32.39%
BGE-reranker-v2-gemma45.94%38.23%49.43%

Figure 3: Comparing Recall@10 across various methods. BM25s, BGE-reranker-v2-gemma results are from [1].

The table shows tool search improvement over BM25s in all three categories: web, code, and customized. It is nearly tied with BGE-reranker-v2-gemma on web, slightly ahead on code, and behind on customized. Clearly, in two of the three categories, tool search is competitive with the GPU-based reranker, without the GPU cost.

These results show that, in addition to cost optimization, tool search can preserve strong retrieval quality while shrinking the tool context the model has to carry. As tool catalogs grow, the advantage comes from making the right tools discoverable at the right moment without paying the full cost of exposing everything upfront.

Tuning the search space 

Benchmark testing revealed that tool search failed when tool descriptions were uneven, sometimes capturing implementation detail instead of user intent vocabulary. Some descriptions were too generic: “get,” “create,” “manage,” “REST API.” The terms had to be reflective of actual user queries.

Toolbox allows developers to configure an optional search-only text field, additional_search_text, for every tool. This additional text is indexed and helps discover a particular tool with higher accuracy. This additional text isn’t visible to models in MCP responses, thus helping you keep all the token savings. No changes are made to the original tool schema of the source MCP server. The returned schema stays clean while the search index learns aliases, domain terms, internal names, and user vocabulary.

For example, a database tool called execute_query might need to be found when a user says “analytics,” “dashboard,” “SQL,” “reporting,” or “warehouse.” A description like “runs a query against the configured database” may be accurate, but it isn’t very searchable. Search-only text can add terms users and models reach for: “analytics query, dashboard data, SQL report, warehouse lookup, inspect tables.” With tuned metadata, retrieval hit rate improved by about 56%, and end-to-end accuracy improved by about 55%, recovering to within about 4% of the full-catalog baseline.

This is where the work became more interesting than simply “saving tokens.” Tool search turned tool curation into an information-retrieval discipline. Adding a tool now raises different questions: what words will a user use, is the description specific enough to beat nearby tools, and is the tool too important to rely on retrieval?

Search is for the long tail

Most tool use has a Pareto shape. A small fraction of tools handles most tasks, but the long tail still matters because rarely used tools are often exactly the ones you need in high-stakes moments: rotate a credential, recover a failed deployment, apply a compliance exception, inspect an audit trail. Search is a good default for that long tail. It is not a good default for tools the model constantly needs.

Several tools are part of the agent’s core contract: policy tools, frequently used data access tools, or capabilities the model should never have to rediscover. Toolbox auto-pins frequently used tools based on usage at a per user level. Auto-pinned tools will be visible in tools/list call after a warmup period, with stale entries aging out as usage changes. Developers can also manually pin tools on top of this. Deterministic pinning keeps the prompt prefix stable, which preserves prompt-cache behavior.

When we would use tool search

If your toolbox has more than 10–15 tools, different tasks need different subsets, or one agent serves many workflows, tool search is worth testing. It’s most useful when the manifest is becoming a material part of cost, when the catalog changes often, or when no fixed tool subset is right.

Tool search is less compelling for tiny catalogs, agents that almost always use the same few tools, or catalogs whose descriptions are too vague to improve. Smaller is not better if the right capability disappears. Tool search changes the agent from “choose from everything” to “ask for a shortlist.” The shortlist has to be good.

Try it

Tool search is in preview for Toolboxes in Foundry. Start with the Microsoft Learn tool search docs, enable toolbox_search on a versioned toolbox, and test before promoting. Then inspect the misses. The first useful tuning pass probably won’t be algorithmic. It will be editorial: improve descriptions, add additional_search_text for domain vocabulary, and pin the tools that are part of the agent’s core contract.

1. import os 2. from azure.identity import DefaultAzureCredential 3. from azure.ai.projects import AIProjectClient 4. from azure.ai.projects.models import MCPTool, ToolSearchToolboxTool 5.   6. client = AIProjectClient( 7. endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], 8. credential=DefaultAzureCredential(), 9. ) 10.   11. # ToolboxSearchToolType() enables tool search — other tools in the toolbox are discovered on 12. # demand through tool_search instead of being listed up front. Add as many MCP servers as you need; 13. # tool search keeps the agent's initial tool surface small regardless of toolbox size. 14. toolbox_version = client.toolboxes.create_version( 15. name="my-toolbox", 16. description="Large toolbox with tool search enabled", 17. tools=[ ToolboxSearchToolboxTool(), 18. { 19. "type": "mcp", 20. "server_label": "analytics", 21. "server_url": "https://db-mcp.internal/sse", 22. "tool_configs": { 23. "execute_query": { 24. "pin": True, 25. "additional_search_text": "SQL database analytics reporting dashboard queries", 26. }, 27. "list_tables": { 28. "additional_search_text": "schema columns metadata table structure discover", 29. }, 30. }, 31. }], 32. ) 33. print(f"Created toolbox `{toolbox_version.name}` (version {toolbox_version.version})") 34.  

For more detailed steps on integration with the agent framework of your choice, click here.

Enable tool search with one click in the Foundry Portal:

The post Tool search: Finding the right tool at the right time appeared first on Command Line.

Read the whole story
alvinashcraft
53 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

In AI-Native Delivery, Architecture Has to Say Where to Stop

1 Share
When you heat metal and cool it slowly, its internal stresses are relieved. The internal structure settles into a more stable state. This is annealing, and I think it is a useful way to think about software architecture when AI agents are doing part of the delivery work. Hot and cold are not just descriptions. […]



Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

Version 0.1.7

1 Share

What's Changed

  • fix: avoid O(n^2) value lookups in PPTX chart conversion by @kz-000 in #2227
  • Fix invalid LaTeX macros for mu, nu, tau, and down-arrow in equation conversion by @ksprajapatii in #2228
  • Fix typos and formatting in comments, docstrings, and markdown by @chienyuanchang in #2223
  • fix: handle PPTX SVG images without a rasterized fallback by @guoyu-wang in #2233
  • Fix omml template bugs. by @afourney in #2257

New Contributors

Full Changelog: v0.1.6...v0.1.7

Read the whole story
alvinashcraft
2 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Adding a custom MCP server to Claude and ChatGPT

1 Share

TIL: Adding a custom MCP server to Claude and ChatGPT

Connecting a custom MCP server to Claude and ChatGPT's standard chat interfaces is possible, but can take quite a few steps.

Tags: ai, generative-ai, chatgpt, llms, claude, model-context-protocol

Read the whole story
alvinashcraft
3 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

How GPT-5.6 fuses frontier intelligence with frontier efficiency

1 Share
GPT-5.6 improves AI efficiency across models, inference, and agentic workflows, helping deliver more useful intelligence per dollar.
Read the whole story
alvinashcraft
3 minutes ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories