Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
159485 stories
·
33 followers

Nothing informs the writing process like writing

1 Share

I took this with the custard Cream Camera over at Ross’s place. I added the colours later.

I’m putting together an article about the custard cream camera and I’m grappling with how I’m going to structure it, and what I’m going to have space to talk about. I’ve had this problem lots of times, and I’ve discovered that the best way to structure an article is to start writing stuff that you think might make sense. After I’ve written a bit, walked away from it, come back to it and read it a few times a structure emerges that works. Trying to build the thing in my head just doesn’t work. I have to have something to play with first.

This might mean that I throw away quite a bit of the stuff that I wrote, but I never actually discard any of it. Instead, everything I throw away is stored in a “scrap” folder so that I can go back to it later, perhaps for another article or a blog post. In fact, I only really consider myself to be making progress on something when I have a few pages in the scrap folder…

Read the whole story
alvinashcraft
2 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

The AI Agent Ate My Budget! Here Is What I Should Have Built Instead

1 Share

Why deterministic automation may be the wisest investment you make before the token economy catches up with your credit card.

In this post, I want to talk about a choice most teams are making without realizing it: delegating work to AI agents when they should be automating it deterministically, and the financial cliff that choice creates.

AI Ate My Budget - money flying away
Image generated with AI

We are leaving the golden age of cheap tokens. The era where companies handed AI a blank check and called it innovation is ending. Reports across the industry tell the same story: budgets consumed in months instead of years, teams scrambling to explain runaway costs and executives quietly asking whether the return ever justified the spending. AI agents, the ones that loop, retry, reason and call themselves back, turned out to be remarkably efficient at one thing: burning through tokens.

We talk about the current AI cycle resembling a bubble. And although it has not fully burst yet, the air is leaving. What remains will be expensive. The professionals who built their entire workflow on token-hungry agents will be the first to feel the squeeze.

The Difference Nobody Talks About

There is a fundamental distinction between the two approaches that the industry keeps conflating into a single term.

An AI agent is a system that receives a goal, reasons about it, makes decisions at runtime and consumes tokens with every step. It is powerful. It is also unpredictable in cost because you are paying for the machine to think every time it runs.

Deterministic automation is a system that analyzes a known structure, applies fixed rules and produces a predictable output, without tokens or runtime reasoning. That will reduce the surprises on your invoice.

Both use intelligence. One front-loads it. The intelligence lives in the design, not in the execution. I call it deterministic AI: a system shaped by years of architectural thinking, where the machine does not need to reason because the reasoning already happened when you built it.

The most intelligent system is the one that does not need to think at runtime.

What I Built Instead

I spent years building a platform that generates entire SaaS applications from a data structure. Not a prototype, and not a scaffold. A complete, production-grade system: backend in .NET, frontend in React, data-first, mobile-first, unit test coverage on both sides, observability baked in with Prometheus, Loki and Tempo, and ready-made Grafana dashboard templates. Roughly 60 files per entity, generated deterministically from the shape of your data.

When a client asked me for a Micro CRM to be delivered overnight, I did not open an AI chat and start prompting. I ran my automation. In under eight hours, including three hours of database design, I made it myself. It was done. Backend, frontend, tests, monitoring. Everything in less than eight hours.

And with zero tokens consumed. (That also cuts unpredictable costs.) The result was identical to what I would have delivered if I had spent a week writing it by hand, because the automation encodes years of decisions I already made.

This is not anti-AI. This is anti-waste.

Where AI Still Earns Its Place

I am not arguing against AI. I use it. But I use it surgically, not as a crutch.

My automation already had a structure designed around C# interfaces, built specifically so that an AI model could understand and navigate the codebase. When I needed to expand unit test coverage, I pointed an AI coding assistant at the existing tests and had it generate new ones that followed the same patterns. I ran it for about six hours over two days. The cost was around $20 in tokens.

Twenty dollars. A manageable, traceable cost for a well-scoped task. But here is the part that matters: I had set a spending limit. Without it, that same task could have spiraled into hundreds, maybe thousands. And that, I believe, is exactly where companies bleeding money on AI are failing: not in choosing to use it, but in using it without boundaries.

I also explored running a custom LLM locally using Ollama and integrating it with my coding environment. It offers unlimited token usage at zero marginal cost, but it demands serious hardware: a GPU with at least 16 or 20 GB of VRAM. For teams that can invest in the infrastructure, it is a path worth exploring, one that decouples your productivity from someone else’s pricing model. Here is a tutorial about my journey implementing custom LLM on Cursor.

Use AI where it multiplies your intelligence. Automate where it would only repeat it.

Build the Machine That Does Not Need the Machine

There is a deeper lesson here, and it goes beyond cost optimization.

If your entire delivery pipeline depends on an external AI service to function, you have a single point of failure priced by someone else. Token prices go up. APIs go down. Models get deprecated. Rate limits tighten. And when any of that happens, your ability to deliver stops with it.

Deterministic automation is yours. You own it. It runs on your terms. It does not get more expensive overnight. And it does not hallucinate.

The architect who invests in automation today is building a shelter for when the token economy changes, and it will change. The one who delegates everything to agents is building on rented land.

Automate with AI. But never depend on AI to execute.

Conclusion

We are at an inflection point. The decisions architects and developers make right now, about what to automate deterministically and what to delegate to AI agents, will define the resilience and economics of their products for years to come.

The question is not whether AI is useful. It is. The question is whether you are building something that survives when the cost of thinking doubles.

So before you spin up another agent, ask yourself: could this be a machine that already knows the answer?

Read the whole story
alvinashcraft
2 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Useful JavaScript Additions in ECMAScript 2026

1 Share

The useful part of the ECMAScript 2026 standard for JavaScript developers is not a dramatic change to the language. It is that several common use cases now have direct names.

JavaScript gets a new language standard every year. Some editions introduce syntax that changes how programs are written or how they perform. The ECMAScript 2026 (ES17) version brings a handful of useful features which we will look at. These features add focused APIs for tasks that JavaScript developers already handle with small helpers, repeated checks or easy-to-miss workarounds.

In this article, I will focus on additions that can remove code, make intent clearer or prevent subtle errors.

Runtime support: These APIs did not arrive in every runtime at the same time. Check the Browser compatibility table on each linked MDN page for the browsers and runtimes you support. Use a tested fallback when an API is unavailable.

Useful JavaScript Additions in ECMAScript 2026
Image generated with AI

Build a Map as Values Arrive

In my article about array grouping, I showed a simple way to group related things using Object.groupBy() and Map.groupBy(). Both methods begin with a collection that already exists, but what if you want to do the same for streaming data?

For example, this code groups employees from an asynchronous stream:

const employeesByDepartment = new Map();

for await (const employee of employeeStream) {
  if (!employeesByDepartment.has(employee.department)) {
    employeesByDepartment.set(employee.department, []);
  }

  employeesByDepartment
    .get(employee.department)
    .push(employee);
}

There is nothing complicated here, but the has(), set() and get() sequence is boilerplate code around the operation we actually care about—adding a new employee to a department.

ECMAScript 2026 adds Map.prototype.getOrInsert() and Map.prototype.getOrInsertComputed(). These methods return the value corresponding to the specified key. If not present, it inserts a new entry with the key and a given default value, and returns the inserted value.

You should use getOrInsertComputed() whenever the default should be created lazily, particularly when creating it is expensive, has side effects or allocates a mutable object such as an array. With the computed version, the code from before becomes:

const employeesByDepartment = new Map();

for await (const employee of employeeStream) {
  employeesByDepartment
    .getOrInsertComputed(employee.department, () => [])
    .push(employee);
}

The result is simple and compact.

Collect an Asynchronous Iterable into an Array

An asynchronous generator is a useful way to hide pagination. Its caller can consume one sequence without knowing where one response page ends and the next begins. For example:

async function* fetchAllIssues(url) {
  while (url) {
    const response = await fetch(url);

    if (!response.ok) {
      throw new Error(`Request failed: ${response.status}`);
    }

    const page = await response.json();

    yield* page.items;
    url = page.next;
  }
}

To collect every result into an array, you would normally write a loop:

const issues = [];

for await (const issue of fetchAllIssues("/api/issues")) {
  issues.push(issue);
}

The Array.fromAsync() method can be used to perform that collection directly:

const issues = await Array.fromAsync(
  fetchAllIssues("/api/issues"),
);

Isn’t that shorter and simpler?

Despite its name, the method also accepts synchronous iterables and array-like objects. It awaits values from those sources one at a time.

Array.fromAsync() also accepts a mapping function, and the runtime waits for the mapped result before reading the next value. For example, the same code can be adjusted to return only the issue title using the map function:

const issueTitles = await Array.fromAsync(
  fetchAllIssues("/api/issues"),
  (issue) => issue.title,
);

Given that example, it might be misleading to think that Array.fromAsync() runs independent operations concurrently. For example:

// The requests start one after another.
const sequential = await Array.fromAsync(urls, fetchJson);

// All requests are initiated before any of them is awaited.
const concurrent = await Promise.all(urls.map(fetchJson));

You should use Promise.all() when independent operations can run at the same time. And reach for Array.fromAsync() when the source or mapping step may be asynchronous, and you want lazy, ordered consumption. Also remember what the method returns—an array containing every result. Keep the for await...of loop when you want to process a large stream incrementally and avoid collecting a stream that may never end.

Combine Iterables into One Lazy Sequence

Sometimes several iterable sources should behave like one continuous sequence. Suppose an application checks its built-in routes first, then registered routes by plugins and finally a catch-all route. You could spread everything into a new array, but that would eagerly consume each iterable and allocate another collection.

A generator can keep the sequence lazy:

function* allRoutes() {
  yield* builtInRoutes;
  yield* pluginRoutes;
  yield fallbackRoute;
}

Iterator.concat(), on the other hand, expresses the same operation without the custom generator:

const allRoutes = Iterator.concat(
  builtInRoutes,
  pluginRoutes,
  [fallbackRoute],
);

The result yields the built-in routes first, followed by the plugin routes and then the fallback. Values are pulled only as the consumer advances the iterator; Iterator.concat() does not collect them into a new array first. Each argument must be an iterable object, which is why the example wraps fallbackRoute in an array.

The Iterator.concat()method works with synchronous iterables only. It does not combine asynchronous iterables.

Preserve Large Integers in JSON

Chat platforms often hand out snowflake IDs as part of a message or other kinds of data. These are commonly 64-bit integers and can exceed Number.MAX_SAFE_INTEGER, so parsing them as JSON numbers can silently lose precision. Consider this response:

const payload = `{
  "messageId": 1183028002140618753,
  "channel": "general"
}`;

const event = JSON.parse(payload);

console.log(event.messageId);
// 1183028002140618800

The logged value isn’t the value in the JSON text.

ECMAScript 2026 gives the JSON.parse() reviver function a third argument, named context. When the value is an unmodified primitive from the parser, context.source contains the original JSON text. We can use that to parse and convert the text to the proper type, in this case BigInt.

Here’s the sample from earlier, rewritten using the reviver function:

const event = JSON.parse(
  payload,
  (key, value, context) => {
    if (key === "messageId") {
      return BigInt(context.source);
    }

    return value;
  },
);

console.log(event.messageId);
// 1183028002140618753n

By the time the reviver receives value, the Number has already lost precision. However, context.source lets the code ignore that damaged value and build a BigInt from the original digits instead.

Serialization has the opposite problem: JSON.stringify() throws when it reaches a BigInt, unless you handle the value. The new JSON.rawJSON() method lets a replacer function for JSON.stringify provide valid JSON text for a primitive value. The JSON.rawJSON() method creates a “raw JSON” object containing JSON text.

Here’s an example using JSON.stringify() and JSON.rawJSON() together:

const json = JSON.stringify(
  event,
  (key, value) =>
    typeof value === "bigint"
      ? JSON.rawJSON(value.toString())
      : value,
);

console.log(json);
// {"messageId":1183028002140618753,"channel":"general"}

Used together, the two APIs let this program recover the original digits as a BigInt and serialize those digits back into JSON without rounding them.

Using those APIs doesn’t mean you should turn every integer into a BigInt. A count, price and database identifier may all appear as JSON numbers, but they do not necessarily belong in the same underlying JavaScript type. That said, it is important to remember that the parser exposes context.source only for unmodified primitive values, and JSON.rawJSON() accepts only valid JSON text representing a primitive value.

Convert Bytes to and from Base64 or Hex

Binary data has often taken an unnecessary detour through strings. Converting to and from bytes wasn’t a natural interface in the language. I think Bun was the first JS runtime I used that had a built-in API for converting bytes to various data types. Fortunately, the 2026 ECMAScript standard release brings the following functions for converting bytes:

  • Uint8Array.prototype.toBase64
  • Uint8Array.prototype.toHex
  • Uint8Array.prototype.setFromHex and its static form Uint8Array.fromHex
  • Uint8Array.prototype.setFromBase64 and its static form Uint8Array.fromBase64

How are they useful, you may ask?

Imagine you want to create a URL-safe token. The code might look like this:

const bytes = crypto.getRandomValues(new Uint8Array(32));

const token = btoa(String.fromCharCode(...bytes))
  .replaceAll("+", "-")
  .replaceAll("/", "_")
  .replace(/=+$/, "");

The program starts with bytes, turns them into a temporary string, encodes that string and then adjusts the alphabet and padding. It works, but none of those intermediate steps express the real task: encode these bytes as base64url.

We can use the Uint8Array.prototype.toBase64() method to do the conversion instead:

const bytes = crypto.getRandomValues(new Uint8Array(32));

const token = bytes.toBase64({
  alphabet: "base64url",
  omitPadding: true,
});

The reverse conversion is also as simple as:

const decoded = Uint8Array.fromBase64(token, {
  alphabet: "base64url",
});

The setFromBase64() and setFromHex() methods write into an existing array and return an object with read and written counts. Unlike fromBase64() and fromHex(), they are useful when you need to control memory allocation, decode into a preallocated buffer or track how much input fits. See the docs for Uint8Array.fromBase64() to learn about the available input options and runtime support.

These methods do not replace TextEncoder or TextDecoder. Use those APIs to convert between text and bytes; use the new Uint8Array methods to convert between bytes and base64 or hexadecimal representations.

Recognize Error Objects Across Realms

The instanceof Error looks like the obvious way to check if an object is an Error. It stops being reliable when the value comes from another JavaScript realm, e.g., an iframe or the Node.js vm context. Each realm has its own Error constructor, so a genuine error from another realm can fail an instanceof check.

Try this in the browser console:

const iframe = document.createElement("iframe");
document.body.append(iframe);

const otherError = new iframe.contentWindow.Error("Failure");

console.log(otherError instanceof Error);
// false

This may not surprise many experienced JavaScript programmers who have been deceived by some JavaScript quirkiness.

The solution is to use the new Error.isError(). Error.isError() performs a built-in check by testing for the internal [[ErrorData]] slot instead of relying on the current realm’s prototype chain. This makes it analogous to Array.isArray() as a reliable cross-realm check.

If you append console.log(Error.isError(otherError)) to the previous code snippet you ran in your browser console, you should see the correct result.

The method is also useful in a catch block because JavaScript allows any value to be thrown:

try {
  await runPlugin();
} catch (value) {
  const error = Error.isError(value)
    ? value
    : new Error(String(value), { cause: value });

  reportError(error);
}

You should use Error.isError() when you need to know whether a value is a real Error object. It deliberately does not treat a plain object with name and message properties as one.

Sum Floating-point Values More Accurately

A straightforward reduce() can lose information while adding floating-point values, and the failure mode is sneakier than you’d expect, because the result doesn’t always look obviously wrong. Consider a motion-sensor library that applies a large per-device calibration offset, adds a small reading, then removes the offset again:

const readings = [1e16, 3.5, -1e16];

const total = readings.reduce(
  (sum, value) => sum + value,
  0,
);

console.log(total);
// 4

The correct answer is 3.5—that’s the actual reading once the offset cancels out. Near 1e16, adjacent representable numbers are two units apart. The exact value 1e16 + 3.5 therefore rounds to 1e16 + 4, and subtracting the offset leaves 4 instead of 3.5. The result isn’t a crash but a plausible-looking wrong number, which is what makes this type of bug easy to miss in code review.

That’s where Math.sumPrecise() comes in. It uses a more accurate summation algorithm, so switching from .reduce() to Math.sumPrecise() gets you the correct answer.

Here’s an accurate way to rewrite it:

const total = Math.sumPrecise(readings);

console.log(total);
// 3.5

The Math.sumPrecise() method accepts an iterable of numbers. It does not coerce strings or BigInt values into numbers. An empty iterable, or one containing only -0, returns -0. The name deserves one warning though, because Precise does not mean decimal arithmetic precision. This familiar result does not change:

console.log(Math.sumPrecise([0.1, 0.2]));
// 0.30000000000000004

Both inputs are already binary floating-point approximations. Math.sumPrecise() reduces the additional error introduced while summing them; it does not change how JavaScript represents numbers. That makes it useful for numerical aggregation, but not a complete solution for money. Use an appropriate decimal type or an integer representation for financial values.

That’s a Wrap

The useful part of the ECMAScript 2026 standard is not a dramatic change to the language. It is that several common use cases now have direct names: get a Map value or create it, preserve the original digits from JSON, encode bytes without pretending they are text, collect an asynchronous sequence, recognize a real Error, sum numbers with less loss and join iterables lazily. None of these APIs will transform an application on its own. They can, however, replace code that is easy to repeat (boilerplate code), easy to get slightly wrong or harder to understand than the operation it performs.

That is why JavaScript has become nicer to use!

Make sure to check runtime support before using these additions in production. The standard defines the language, but every runtime follows its own release schedule. You can get the source code for some of the examples on GitHub.

Further Reading

Read the whole story
alvinashcraft
3 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Microsoft Releases Open Source Test Unit Agent

1 Share
Read the whole story
alvinashcraft
3 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Making LLM Features Cheap Enough to Actually Ship

1 Share

Part 7 of 8: Durable AI for InterlinedList


Six posts in, and I’ve been dancing around a number. Every proposal I’ve sketched (durable generate, the calendar planner, the link crawler, batch enrichment) carries a cost, and the expensive part is never Temporal. It’s the tokens. A worker box and a managed Temporal namespace run me on the order of tens of dollars a month (obviously at this time), flat, whether they process ten jobs or ten thousand. The model bill is the one that scales with usage. It’s also the one that decides whether an AI feature is worth shipping or is a slow-motion budget fire.

So this post is the payoff. Durability isn’t just nice, durability is money. The mechanics I keep reaching for aren’t in the plan because they’re clever. They’re there because each one structurally drives the token bill down, and does it in a way I can measure instead of squint at and hope for. Keeping this cheap relative to the Claude and ChatGPT APIs was never a footnote I tacked onto the series. It’s been the point the whole way through.

None of this is live yet. AI assistance in the product is still marked Coming Soon. This is me brainstorming the cost layer I’d build, and how I’d hand it to Claude to build with me.

The Single Biggest Saver Is Not Re-Paying for Work You Already Did

Start with the one that matters most. Durable memoization.

When a Temporal workflow runs an activity (say, an llmCall that generates a document section) the result gets persisted the moment it completes. If the workflow later fails and retries, it does not re-run that activity. It replays the history, sees the completed result, and moves on.

Think about what that means on a six-step generation pipeline that dies at step five. Without durability, a retry starts over: steps one through four re-invoke the model, and you pay for all four again just to get back to where you already were. With Temporal, the retry re-runs step five and only step five. Steps one through four are already-billed calls that never happen twice.

That’s the difference between retries as a cost multiplier and retries that are free. On a batch enrichment job that fails at item 47 of 50, you re-bill item 47, not items 1 through 46. I wrote a whole post about that in Part 6, and it’s the same mechanic every proposal here leans on. Memoization is why a flaky provider or a worker restart doesn’t torch the budget.

Cheapest Model First, Escalate Only on Demand

The next lever is refusing to reach for the expensive tier by default. The llmCall activity would run a model cascade: cheapest capable tier first (Haiku-class, Flash-class), escalating only when the task actually needs it.

Most of what these features do is mechanical. Tagging a message. Summarizing a row. Extracting a field. That work does not need a frontier model, and paying frontier rates for it is just lighting money on fire. So the cascade defaults to cheap, and it reserves the Opus-class tier for the thing the user explicitly asked for: long-form synthesis, a research draft, a document they told me they want written well.

Paired with that: per-target max_tokens ceilings, so a runaway generation can’t balloon. A message caps around 1k, a list around 4k, a document around 16k. The ceiling isn’t a polite suggestion the model is vaguely aware of. The activity enforces it as a hard cap. So you get the cheap tier by default, a small ceiling per target, and the expensive model only shows up when you asked for it.

A Gate That Throws Before It Spends

Cascades and ceilings keep individual calls cheap. But the scenario that scares me is unbounded spend: a runaway agent loop, or a leaked user key someone else is now happily burning. So the plan puts a budget gate before every call, not after.

An assertBudget activity reads accumulated spend from the AiGeneration ledger and throws a non-retryable BudgetExceeded when the user is over their ceiling. Non-retryable is the important word. A normal failure retries; this one halts the workflow cold, because retrying a budget breach just tries again to spend money you’ve already said no to.

Roughly what I’d hand to Claude as the shape:

// activity: runs on the worker, before any billable llmCall
export async function assertBudget(userId: string, estimate: TokenEstimate): Promise<void> {
  const spent = await getSpendThisPeriod(userId); // counts only, from AiGeneration
  const remaining = budgetFor(userId) - spent;

  if (estimate.maxTokens > remaining) {
    // non-retryable: a budget breach must not loop and re-attempt the spend
    throw ApplicationFailure.nonRetryable(
      `Budget exceeded: need ${estimate.maxTokens}, have ${remaining}`,
      "BudgetExceeded",
    );
  }
}

A workflow calls assertBudget and then llmCall, in that order, every time. Worst case is a bounded overspend of one call, never an open tap.

Never Bill the Same Request Twice

Then there’s the plain-dumb-obvious saver: caching. A response cache keyed by hash(system + prompt + model). If an identical request comes through (same system prompt, same user prompt, same model) it’s served from cache and never re-billed. Regenerate the same summary twice, pay once.

Same idea one layer down for link work: URL-hash dedup. When the crawler from Part 5 fetches and embeds a link, it keys on a hash of the URL. A link shared across ten messages, three docs, and a list row gets fetched, extracted, and embedded exactly once, ever. Embeddings are already orders of magnitude cheaper than generation, so computing them once and caching forever makes the cheapest part of the pipeline round down to nothing.

And to keep from wasting tokens on retries you caused yourself: worker concurrency and task-queue rate caps tuned to stay under provider limits. Blow past a rate limit and you get a 429 storm, and every retry in that storm is wall-clock and, on some providers, tokens down the drain. Obeying the limit is cheaper than fighting it.

One more, because it’s the most satisfying: confirm-before-spend. For the prompt-to-list flow, the workflow generates the schema, then pauses on a Signal and waits. It does not spend a single row-generation token until the user confirms the schema is right. Get the schema wrong and you’ve spent one cheap schema call, not fifty row calls against a shape nobody wanted. That pause is impossible in a serverless handler and native in Temporal.

The Ledger Counts, It Doesn’t Read

Every one of these levers needs a source of truth, and that’s the AiGeneration ledger. Critical design decision: it records counts and status only. Tokens in, tokens out, which model, succeeded or failed. It never stores the prompt text or the model output.

That’s not laziness. One table does three jobs. Privacy, because I’m not warehousing what people wrote or what the model said back. The quota counter, because assertBudget reads its sums off it. And the cost dashboard, because those same rows drive a per-user “AI usage and spend” view: tokens, calls, cache-hit rate, budget remaining. When something looks expensive, the ledger doubles as the debugging trail. You can see that a workflow made forty calls without ever seeing what it said.

Build the Guardrails With Claude, Then Make It Prove Them

Now the part I’ve been wanting to get to, because this is where working with an LLM changed how I’d approach any of it.

Memoization and caching only save money if they’re in the right places. Put a cache breakpoint one step too early and you re-bill everything after it. Miss a memoization boundary and a retry quietly re-runs a paid call. The savings are entirely a function of where the boundaries land, and eyeballing a six-step pipeline for those boundaries is the kind of thing I get wrong.

So the first move is a diagnostic question, not a code request. I’d point Claude at the actual workflow and ask: “walk this pipeline and tell me exactly where a retry would re-bill the model.” Then let its answer drive placement. It reads the activity boundaries, traces the retry path, and tells me which calls are already durably memoized and which ones a mid-pipeline failure would re-invoke. That answer is the map. The caching and memoization go where the map says, not where I guessed.

The second move matters more: make it prove the savings. Memoization you can’t prove is just a hope with good intentions. So I’d have Claude write a Temporal replay test (using TestWorkflowEnvironment) that fails an activity partway through the pipeline and then asserts the already-completed LLM activities are not invoked again on retry.

it("does not re-bill completed LLM activities on retry", async () => {
  const llmCall = vi.fn()
    .mockResolvedValueOnce("step-1 result")
    .mockResolvedValueOnce("step-2 result")
    .mockRejectedValueOnce(new Error("provider blip at step 3")) // fail mid-pipeline
    .mockResolvedValue("step-3 result (retry)");

  await worker.runUntil(client.workflow.execute(generatePipeline, { /* ... */ }));

  // steps 1 and 2 completed before the failure, so Temporal replays their
  // memoized results and MUST NOT call the model for them again.
  const stepsCalled = llmCall.mock.calls.map((c) => c[0].step);
  expect(stepsCalled.filter((s) => s === 1)).toHaveLength(1);
  expect(stepsCalled.filter((s) => s === 2)).toHaveLength(1);
  expect(stepsCalled.filter((s) => s === 3)).toHaveLength(2); // failed once, retried
});

That test is the proof. It fails loudly the day someone refactors the pipeline in a way that breaks memoization and starts silently re-billing steps one and two. The savings stop being a story I tell about the architecture and start being a property CI enforces. Same loop for the response cache: a test that fires two identical requests and asserts the second one hits cache and never reaches the model.

Ask the agent where the money leaks. Let its answer place the guardrails. Then make it write the test that locks them in. That loop works because a cost regression is invisible until the bill shows up weeks later, and a test drags it into the light on the exact commit that caused it, while you can still git blame your way back to it.

Where This Lands

Part 8 is the capstone: the agentic research-to-draft flow, the single most token-hungry thing in the whole set. It’s the reason every governor in this post has to exist before it ships. I’ll also lay out how I’d sequence the entire build, starting from the foundation phase (the Temporal wiring and the shared cost primitives everything else reuses) and working all the way up to the agent, and how I’d verify each piece as it lands.

Adron brainstorming and working on InterlinedList.

Read the whole story
alvinashcraft
3 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

How do functions like alloca allocate memory from the stack?

1 Share

A little while ago, I talked about how compilers ensure that large stack allocations do not skip over the guard page. Shawn Van Ness was curious how this works with _alloca. “Does it do the necessary _chkstk() probing?”

Yes, the _alloca() function calls the same _chkstk() function to probe the stack before adjusting the stack pointer for the allocated memory.

Here’s an artificial example:

#include <malloc.h>

void consume(void*,void*);

void f(int n)
{
    char buffer[16384];
    consume(alloca(n), buffer);
}

On x86-64, this results in

        push    rbp
        mov     eax, 16416          ; probe for local frame
        call    __chkstk
        sub     rsp, rax            ; create local frame

        lea     rbp, [rsp+32]

        movsxd  rax, ecx            ; n
        lea     rcx, [rax+15]       ; round up to multiple of 16
        and     rcx, -16

        mov     rax, rcx            ; special __chkstk calling convention
        call    __chkstk
        sub     rsp, rcx            ; allocate n bytes

        lea     rdx, [rbp]          ; rdx -> buffer
        lea     rcx, [rsp+32]       ; rcx -> alloca'd memory
        call    consume

        lea     rsp, [rbp+16384]    ; clean up local frame
        pop     rbp
        ret     0

Observe that the same __chkstk function is used both for performing the initial stack probe when creating the local frame as well as for the alloca().

The post How do functions like <CODE>alloca</CODE> allocate memory from the stack? appeared first on The Old New Thing.

Read the whole story
alvinashcraft
3 minutes ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories