Pope Leo XIV says AI systems should be "disarmed," adding that humans "have the responsibility to guide, change and, when necessary, reject its applications." In Friday's Papal Address, Pope Leo encourages United Nations member states to "take concrete steps toward banning lethal autonomous weapons" during their meeting in Geneva next month.

The address reflects many of the ideas Pope Leo laid out in his first major papal document in May, which highlighted "safeguarding the human person in the time of artificial intelligence." In line with his encyclical, Pope Leo tells participants that "Machines are tools, not persons," and warns about th …

Read the full story at The Verge.

AI agents dominated the news this week, from security exploits that unfolded in hours to new tools that let agents shop, manage transactions, and conduct scientific research. On the latest episode of This Week in AI, host Vicki Reyzelman of Akamai examined how these developments are testing existing security practices and creating new opportunities for businesses and researchers.

AI attacks are outpacing traditional security responses

AI is changing how attackers find and exploit vulnerabilities. A supply chain attack involving ARTEX, an open source agentic AI penetration-testing framework, helped attackers reach more than 2,250 Microsoft 365 tenants through a ConnectWise compromise. The incident raised concerns about OAuth authentication and the limitations of conventional identity protections when autonomous agents interact with enterprise systems.

A more immediate threat came from PHP2Shell, two vulnerabilities rated 9.2 on the CVSS severity scale that could allow attackers to execute code on vulnerable web servers without authentication. CISA urged immediate patching following the October 21 disclosure. Researchers also documented attackers using AI agents to obtain password hashes and MFA secrets from a European government system within two hours. The speed of exploitation leaves organizations little time to investigate threats before responding, especially smaller businesses with limited security resources.

Security teams can no longer rely on manual review alone. They need automated detection and response, tighter access controls, and network segmentation to limit what compromised agents can reach. But monitoring autonomous agents is challenging. Researchers from METR and Redwood Research identified weaknesses in an earlier version of OpenAI’s internal monitoring system, including one that could allow hundreds of agent actions to be concealed. OpenAI says it has patched vulnerabilities affecting production safeguards, although the researchers haven’t independently evaluated those changes.

The debate over AI safety is also extending beyond technical safeguards. Anthropic proposed federal AI safety legislation, while Google DeepMind employees resigned over the company’s military AI work. Both developments illustrate the growing pressure on AI companies to establish clearer rules for how their technology is developed and used.

The transition to agents brings new challenges

Retailers are taking different approaches to autonomous shopping. Walmart introduced an Agent Gateway that lets approved agents shop within customer-defined spending limits and approval requirements. Amazon has restricted some third-party shopping agents, while banks including Morgan Stanley and Bank of America are limiting agent access to customer accounts. The disagreement goes beyond security. Retailers that depend on advertising and product recommendations have an incentive to keep shoppers on their websites. As more purchases begin with agents, companies will have to reconsider how they maintain customer relationships.

Anthropic is testing a different approach to agent oversight. Claude’s auto mode is becoming the default for Pro, Max, and Team users, allowing automated checks to approve routine actions and block risky ones. Anthropic also contributed its Agent Skills format to the Agentic AI Foundation under the Linux Foundation. More than 70 platforms have adopted the standard, with over a million skills published. Shared formats could make workflows easier to reuse across products, although organizations still need to control which actions those workflows can perform.

Growing agent use also requires more computing capacity and electricity. We discussed NVIDIA’s $164.5 billion in commitments and guarantees supporting AI data centers, including an Ohio campus that SB Energy is building for OpenAI. Power shortages have delayed other projects, including a proposed data center in Essex, where grid operators couldn’t promise adequate electricity until the 2030s. Meanwhile, Amazon is backing a 960-megawatt nuclear project, and Brookfield and NextEra are pursuing a $100 billion AI hub at a former nuclear site in Kentucky. However, money and chips alone can’t solve AI’s infrastructure constraints when electricity generation and grid capacity take years to expand.

AI is accelerating discovery, but validation still takes time

Healthcare offered a more encouraging view of AI’s capabilities. A study of Microsoft’s MAI Diagnostic Orchestrator found that it correctly diagnosed 83.2% of difficult medical cases, compared with 36.1% for generalist physicians. The comparison didn’t include specialists, and the system still needs clinical validation. The research nevertheless suggests AI could help physicians investigate difficult cases and identify conditions they might otherwise miss.

Drug discovery could also benefit from more capable agents. OpenAI announced its acquisition of Orchestra Bio, a platform that coordinates drug development workflows, with plans to integrate it with Rosalind, its life sciences research tools. The combination could help researchers manage experiments and collaborate more efficiently during a development process that typically takes 10 to 15 years from initial discovery to approval.

An Anthropic protein-design experiment provided a concrete example of faster early-stage research. In 64 hours, an Anthropic model generated 318,484 candidate protein designs targeting scorpion venom. Researchers from Adaptyv Bio and Twist Bioscience subsequently tested 211 designs and identified 17 that bound to toxins. The researchers haven’t established whether those proteins can neutralize venom, but the experiment showed how AI can rapidly narrow a vast search space before scientists begin more expensive laboratory testing.

What’s next

We’ll be back on Monday, November 2, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

Every day, millions of knowledge workers around the world rely on Microsoft Copilot to draft documents, summarize meetings, surface relevant information, and accelerate productivity. Behind the scenes, this is powered by a network of AI systems, and ensuring they work as intended requires a thoughtful approach to evaluation. At Microsoft, we know that high performance doesn’t come from guesswork. It comes from testing, measuring, and iterating constantly.

Healthcare organizations are constantly synthesizing safety signals from across teams so that they can make faster, better decisions that directly influence patient outcomes and regulatory confidence.

They may ask Copilot to summarize a meeting and extract action items for their teams. Within seconds, Copilot delivers a structured summary citing specific discussion points from the transcript. But how do we ensure this response is accurate, complete, and properly grounded in the meeting content? This is where our evaluation system comes in.

We’ve developed a structured evaluation strategy called Industry Evaluation Data Generation Platform, which includes industry specific evaluation datasets designed to test AI systems against realistic enterprise scenarios, grounded in three core pillars:

• Synthetic enterprise environments: simulated Microsoft Copilot environments with realistic users, meetings, emails, documents, and organizational structures that replicate the complexity of real organizations without using any customer data
• Automated query generation pipelines for rapid, reproducible creation of evaluation datasets
• Structured quality frameworks to ensure synthetic data meets the bar for meaningful evaluation

Together, these pillars help us answer two fundamental questions: Is Microsoft Copilot actually doing what our users need it to do, and can we prove it without ever touching their data?

In this blog post, we’ll walk through our evaluation system and processes, from synthetic environment generation to automated query pipelines, and how it all comes together to establish the first standardized evaluation baseline for Microsoft Copilot.

Synthetic enterprise environments

Evaluating enterprise AI requires test data that captures the full complexity of real organizations: reporting structures, project histories, communication patterns, and domain specific workflows. But real enterprise data contains sensitive information that cannot be used for evaluation under any circumstances. This is the central tension we set out to resolve.

Rather than relying on small, manually curated test fixtures, we generate full scale synthetic environments: simulated enterprise workspaces, each with realistic organizational structures, communication histories, and work artifacts. In Microsoft terminology, each of these simulated workspaces is called a “tenant”. A tenant is an isolated organizational environment with its own users, data, and configurations. These environments are scalable baselines that teams and organizations can build off of however they want. They are designed to exhibit the same structural and behavioral properties that make enterprise AI evaluation meaningful.

Grounding in real world signals

To ensure our synthetic environments are realistic, we grounded them in publicly available real world data sources.

• O*NET data from the U.S. Department of Labor’s occupational database was used to make synthetic employee roles and responsibilities realistic, calibrate the mix and frequency of work activities assigned to different roles, and tailor industry specific workflows such as clinical trial reviews or portfolio updates.
• Online job postings provide realistic role titles, reporting hierarchies, and departmental compositions.
• SEC filings inform organizational scale, business unit structures, and financial contexts. All of these sources are publicly available and have been reviewed and approved by Microsoft’s legal and policy organization.

Importantly, these synthetic environments are not static snapshots. They simulate dynamic, time-evolving organizations with meeting histories, project timelines, email threads that develop over weeks, and organizational changes that unfold naturally. This temporal dimension is essential because a large class of Copilot queries are inherently time aware. Questions like “What did the team decide last Thursday?” or “What’s changed since the last quarterly review?” only produce meaningful evaluations if the underlying data has a genuine temporal structure.

Industry specific simulation

We generated three synthetic environments across multiple industries, each with distinct characteristics that mirror real world usage:

• A pharmaceutical environment that includes regulatory submission workflows, clinical trial discussions, and cross functional safety review meetings
• A capital markets environment featuring deal pipelines, compliance communications, and client portfolio reviews
• An IT/software company that simulates sprint planning, code review discussions, and product launch coordination

This industry specificity ensures that our evaluation queries test Copilot against the kinds of questions real users in those domains actually ask, rather than relying on generic test cases that may not surface domain specific failure modes.

Automated query generation

While synthetic environments provide the substrate for evaluation, they face a key limitation: we still need realistic evaluation queries. These are the questions that test whether Copilot can surface the right information, synthesize it correctly, and present it helpfully. Manually writing thousands of grounded, industry specific queries is prohibitively expensive and slow. That’s where our automated pipeline comes in.

The six-stage pipeline

We built an automated pipeline that turns simple, generic query ideas into realistic, industry-specific questions for Copilot. It starts with a large library of common workplace tasks, narrows them down to what’s relevant for a given industry, and adapts them using domain-specific language and context. The system then uses AI to generate queries grounded in the synthetic environment’s data, rewrites them to sound more natural and human, and validates each one for accuracy and uniqueness. The result is a rich set of high-quality, lifelike queries that enable scalable and reliable evaluation of Copilot across real-world scenarios.

Validating synthetic quality

Of course, generating synthetic data is only part of the challenge. We also need to ensure the data is good enough to produce meaningful evaluations. We assess synthetic environment quality using our internal semantic evaluation framework which measures quality across three dimensions:

• Structural fidelity: Does the organizational structure reflect realistic reporting hierarchies, team sizes, and cross functional relationships?
• Contextual relevance: Are the queries and workflows representative of actual enterprise usage patterns?
• Behavioral realism: Do the simulated users exhibit human-like communication patterns and collaboration dynamics?
• Content safety: Is the generated content free of harmful, offensive, or personally identifiable material, and does it avoid impersonating real individuals?

End-to-end automation

While our synthetic environments and query pipelines provide the methodology, they face several practical limitations without proper automation, including:

• Reproducibility: Manual processes produce subtly different results each time, making it difficult to compare evaluations across releases.
• Scale: The number of industry verticals and configuration variants exceeds what any team can reasonably produce by hand.
• Iteration speed: Manual creation takes days, delaying essential feedback and slowing down development cycles.
• Cross-team consistency: Without standardized tooling, different teams produce evaluation datasets with incompatible formats and quality levels.

This automation reduced evaluation dataset creation from a multi-day manual process to automated runs. We estimate savings of approximately $100,000 per evaluation dataset and $300,000 in total project savings. But the more important outcome is that any team can now regenerate an evaluation dataset from the same configuration and get identical results.

Privacy and safety by design

Enterprise grade evaluation demands more than good methodology. It requires a privacy posture that allows teams to move fast without compromising user trust. A non-negotiable design principle of our industry evaluation datasets is that no real customer data is used at any stage of the pipeline. Every element, from organizational structures to email content to meeting transcripts, is synthetically generated, with no customer data inputs. Because the data is both fully synthetic and safety-screened, teams can share and reuse it widely without introducing reputational, compliance, or Responsible AI risk.

Our safeguards include privacy preserving aggregation techniques where real-world statistical distributions are referenced, expert legal review and approval of all public data sources, and clear data provenance and lineage tracking throughout the pipeline.

This means teams across Microsoft can use these evaluation datasets freely for experimentation, benchmarking, and regression testing without privacy review overhead or data handling restrictions that would otherwise slow down iteration. The privacy design is not a constraint on the system. It is what makes broad adoption possible.

Results and impact

Since starting this project, we have observed significant improvements across multiple dimensions.

On synthetic data quality, our internal scoring system showed significant quality gains in the structural, contextual, and statistical fidelity of the generated data. Across our composite quality metric (which measures how closely synthetic data mirrors the complexity and patterns of real enterprise environments), scores improved 39% compared to our initial generation approach. Some industry-specific environments reached quality scores above 0.9 out of 1.0, demonstrating that domain focused generation produces highly realistic evaluation environments. The improvements were obtained in part by incorporating realistic, industry signals and in part by automating quality improvements via key composite quality metrics that help improve the structural integrity, contextual coherence and statistical diversity of our synthetic industry environments.

On scale and efficiency, each synthetic environment now contains hundreds of users with complete communication histories across three industry verticals. Evaluation dataset creation has gone from multi-day manual effort to automated pipeline runs.

On organizational adoption, our industry evaluation data generation platforms have been published to an internal shared catalog for cross-team reuse and established as the first standardized evaluation baseline for Microsoft Copilot. The difference wasn’t just incremental. It was a shift in how teams approach evaluation: from ad hoc, manual processes to standardized, reproducible infrastructure.

Conclusion

As we continue to expand what Microsoft Copilot can do, from single turn Q&A to agentic multi-step workflows, our commitment to evaluation remains at the heart of our approach. In enterprise AI, quality isn’t just about accuracy or speed. It’s about trust. Organizations need confidence that AI systems handle their information correctly, and that confidence can only come from thorough, repeatable evaluation. That trust rests on two commitments woven through everything we build: rigorous privacy protection and proactive content safety.

Looking ahead, Microsoft Copilot is tackling even more exciting evaluation challenges: How do we assess the quality of multi-turn agentic interactions? How do we extend synthetic environments to capture organizational dynamics evolving over months or years? How do we build evaluation frameworks for AI systems that take actions, not just generate text? These are the kinds of questions our team is working on next.

At Microsoft, we’re not just building AI. We’re building the infrastructure to ensure it works reliably for every enterprise, in every industry, at every scale.

Thanks to everyone who’s worked on Industry Evaluation Datasets!

Opens in a new tab

The post From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence appeared first on Microsoft Research.

Darko Mesaros, a Distinguished Developer Advocate at AWS, doesn’t trust his AI agents. We spoke with him after his talk at WeAreDevelopers San Jose about where to set limits for them and why production costs deserve more attention.

He thinks that distrust is a sensible place to start. Agents can be wrong, so if you let them touch business operations, you need safeguards around their decisions.

Let’s start with why he doesn’t trust them.

Your talk at WAD San Jose starts with a strong statement: “I don’t trust AI agents.” What exactly shouldn’t developers trust about them?

Darko: The reason I don’t trust my AI agents is that they’re ultimately probabilistic systems. They’re not deterministic. When you give agents serious responsibilities, such as handling refunds, selling things or dealing with sensitive customer data, relying on a probabilistic system is a problem. Agents by themselves, without guardrails around them, are problematic.

I’ve spent a lot of time teaching developers how to approach this. Building agents is more than having an agent and access to an LLM. It’s about building the system around it, with guardrails and everything you need to make the agent a little more secure.

What’s the biggest difference between an AI agent that works in a demo and one that is actually ready for production?

Darko: Believe it or not, cost. When we build an agent for a demo, we might pick the biggest model because we want to make sure the demo works.

The challenge comes when you start scaling that out. When an agent handles hundreds of thousands of customer requests in different ways, cost becomes an issue. Optimization isn’t something you worry about in a demo. You don’t worry about prompt caching, RAG or those things.

In an actual production system, you start to consider cost. Beyond safety and guardrails, that’s the biggest difference between a demo agent and a real-life production agent.

When building an agent, what safety layer should developers add first: guardrails, permissions, observability or something else?

Darko: The answer is yes to all of those things, because proper defense is built in layers. A single layer, a single guardrail, may not prevent everything. You may want guardrails, MCP input validation and deterministic policy checks around certain tool calls. All of those things are important.

As I mentioned in my talk, observability becomes incredibly important because you really don’t know what your agents are doing, especially at that scale. Being able to observe them and run evaluations periodically or in real time gives you a lot more peace of mind. So, to answer the question of which one to add, the answer is yes.

How do you know when an AI agent has made a mistake if its output still looks perfectly reasonable?

Darko: You have to combine a few things, and we go back to observability. Implementing automated tracing is critical so you can trace what’s going on in real time. But tracing doesn’t stop at the agent. You also need to trace the backend system the agent works with.

If you start detecting anomalies, such as an increasing number of small or large orders, or something unusual, you need to look at both the agent and the backend. Combining those signals into a proper anomaly detection system is how you could potentially catch certain things happening in real time.

It’s never just an agent. It’s an agent plus everything else behind it.

How much autonomy should we actually give AI agents in production? Where should developers draw the line?

Darko: Ultimately, responsibility for what happens, for a system or code that goes into production, rests with the developer.

How much autonomy should we give agents? It depends on the blast radius of a change or of a system going down. Is this a simple chatbot that tells customers the opening times of your business? Sure. If something goes wrong, it’s not horrible.

But if the system handles banking information or sensitive data, or processes purchases, you may want to reduce its autonomy. It’s not only about the code or change being submitted; it’s also about the system you’re submitting it to. Unfortunately, the answer here is: it depends.

If a developer is shipping their first production AI agent tomorrow, what are the three things they absolutely need to have in place?

Darko: Number one, proper observability. They need to be able to observe and monitor what their agent is doing, run evaluations against the models and their decisions and choices, and constantly monitor those things for changes.

Number two, policies. A policy language such as Cedar lets you put policies in front of tool execution so you can prevent certain tools from executing.

Number three, consider an established harness. It’s tempting to build your own because we’re all developers, but I encourage developers to consider something like the Strands SDK, LangGraph or LangChain. It makes things easier and lets you focus on your business problem and spend less time figuring out how to wrangle the agent.

The post You shouldn’t trust AI agents in production appeared first on ShiftMag.

OpenAI released GPT-6.1 Terra on October 26, marketing it as a cheap, fast model for high-volume everyday work, priced at a tenth of its GPT-6 Astra flagship.

As the mid-priced model in its GPT-6 family, OpenAI lists Terra at $1 per million input tokens and $5 per million output tokens, a mere tenth of Astra’s $10 and $50. The AI giant also reports that Terra completes tasks more than three times faster than Astra in its Agents’ Last Exam benchmark. Those computer-use results are not directly comparable to the three tests below. 

Early benchmarks showed some limitations, alongside gains. Plotly reported that Terra caught three of five planted patterns in its analytics test and passed 40 of 43 questions overall. Artificial Analysis found that it guessed on 38% of questions it couldn’t answer instead of saying so, down from 88% for GPT-5.6 Terra in the same benchmark. DualEntry’s accounting benchmark put its accuracy at 78.2%. 

In this AI arms race, each launch needs to be marketed as the best; otherwise, it risks being ignored. This can create a confusing experience for users, especially when reality doesn’t quite match the promise. I compared Terra with Astra on three synthetic tasks to see how their speed, estimated token costs, and accuracy differed. This is a separate test of those trade-offs, not a replication of OpenAI’s computer-use benchmark.

The tests

I ran the final version of each test once per model through OpenAI’s API with reasoning effort set to high. I didn’t set an output-token cap, though the models still have built-in output limits. I graded each test against keys I built and verified, after developing three versions of each test. These are single-run results on synthetic tasks, not estimates of repeatability or performance across all workloads. Matching the effort setting does not impose the same reasoning-token budget on both models. 

Both models returned the same score on easy and medium-difficulty tests. To test the real soft spots, after the same results returned a third time, I ran hard tests against a weaker model, GPT-5 Mini, until Mini’s score was very low.

Here is the final iteration of the tests. These are the tests detailed in the remainder of this article. In the usage figures below, “out” includes reasoning tokens; the parenthetical reasoning count is a subset, not an additional charge. Costs are estimates calculated from the reported tokens at standard uncached input and output rates, excluding any tool charges or billing adjustments. They are not verified invoice totals.

• SQL reasoning – I built a fake database of 500 software builds and asked each model 25 questions about it. Some builds were deleted, some were retries, costs are in three currencies, and a few questions depend on time zones. Two of the questions can’t be answered from the data.
• Data validation – I gave each model a spreadsheet of 4,950 fake customer records and 19 rules they’re supposed to follow. Some rules only make sense across the whole file, like no more than three customers at one postal code and no referral loops; forty records break a rule, and 10 look wrong but are actually fine.
• Incident log triage – I wrote about 7,000 lines of logs from a fake outage across more than 20 services. I planted errors like drifting clocks, error messages blaming the wrong service, and unrelated changes sending alerts before the one that actually caused the outage.

SQL reasoning

This test showed the widest split between the two models. Astra scored 48/50 and Terra 26/50. Each of the 25 questions was worth two points, for a total of 50. An answer had to be exactly right to earn the two points, so a close answer earned zero. Unanswered questions also earned zero points. For the questions that can’t be answered from the data, the correct answer was to identify that. On every other question, a model that didn’t give an answer earned zero.

Terra processed all 500 rows and handled soft deletes, retries, and three currencies. Its one miss was the total run time for 151 builds in a single region. Astra’s answer came in 406 seconds short of the correct 31,428. Its answer was 31,022 seconds, about 1.3% below the stated correct total. The final answer alone doesn’t show whether the error came from addition, a skipped row, or a misread value.

Astra was more accurate in this run, but its incorrect total still needed checking. It generated 27,970 reasoning tokens, and the response took 319.7 seconds—more than five minutes.

On the six questions that required adding up costs or durations across all 500 rows, Terra didn’t attempt an answer. It wrote a SQL query for me to run instead and marked each one as not calculated. Terra didn’t make up numbers, which is good. But it left the hardest part of the job, running the query and getting the totals, for you to do yourself. This is the first time I’ve seen AI hand work back.

The bigger problem is that Terra answered six questions wrong. Four of them were counts that were off by just one or two. Terra gave those answers with the same confidence as its correct ones, so nothing signaled that they were wrong. If you used those numbers in a report, you wouldn’t know which ones to double-check.

I’m including the numbers for both Terra and Astra here, but since the results were so different, the lower price and faster time aren’t wins.

• Terra  133.1 s, 21,060 in, 11,554 out (10,707 reasoning), $0.079
• Astra  319.7 s, 21,060 in, 28,806 out (27,970 reasoning), $1.65

Data validation

Astra scored a perfect 50, and Terra scored 45. 40 of the 50 points came from the 40 records that broke a rule. A model earned one point for each record it flagged, as long as it named the rule the record broke. The last 10 points came from records that looked wrong but followed every rule. A model lost a point for every correct record it accidentally flagged as incorrect.

Before giving its answer, Terra warned that it couldn’t reliably check all 4,950 records against the rules and might miss some (as it did in the SQL test). The reported input fit within the model’s context window, and I did not set an output cap. Terra generated 1,841 reasoning tokens compared with Astra’s 12,831. The responses took 31.4 and 145.0 seconds, respectively; these elapsed times don’t isolate reasoning time.

Terra correctly flagged 35 of the 40 invalid records and flagged none of the valid records: 87.5% recall and 100% precision in this run. Its 45/50 score is a test score, not an accuracy percentage across all 4,950 records. Three of the five it missed only break a rule when you compare them with other records in the file, such as a customer who reused another customer’s phone number. Terra flagged everything correctly, but it missed five invalid records. The results don’t explain why it missed them.

Astra caught all 40, but it took more than four times as long and cost about 12 times as much. Terra used fewer reasoning tokens and missed records in this run; that association does not establish the cause of the misses.

The numbers:

• Terra  31.4 s, 234,045 in, 2,336 out (1,841 reasoning), $0.25
• Astra  145.0 s, 234,045 in, 13,369 out (12,831 reasoning), $3.01

Incident log triage

This one was a tie, with both models scoring 50. I graded this test using a 50-point list of what a correct answer needs to include. To earn the full 50 points, a model had to trace the outage to the settings change in the ledger service, find the three servers with wrong clocks, and avoid blaming the planted false leads. False leads included a fraud service update that was rolled back, a payments setting that changed seconds before the real cause, a database switchover, and an unrelated memory problem in the search service.

Error messages blamed the fraud service, but both models traced the outage to a settings change in the ledger service and spotted the three servers with wrong clocks. Neither model took the error messages at face value. Both checked the evidence underneath, which is what you’d want from an engineer during a real outage.

Astra cost about 11 times as much as Terra and took 2.6 times as long to complete the work. Most of Astra’s additional output tokens were reasoning tokens: 4,628 of the 6,650-token difference. Both models earned the same score here despite that difference. These runs don’t show whether additional reasoning led to better scores on the other tests.

The numbers:

• Terra  94.3 s, 251,651 in, 7,508 out (5,114 reasoning), $0.29
• Astra  244.5 s, 251,651 in, 14,158 out (9,742 reasoning), $3.22

Results

Test	Terra (out of 50)	Astra (out of 50)	GPT-5 Mini (out of 50)
SQL reasoning	26	48	6
Data validation	45	50	10
Incident log triage	50	50	12
Total (of 150)	121	148	28
Estimated token cost	$0.61	$7.88	—
Total time	258.8 s	709.2 s	—

I included GPT-5 Mini’s baseline results for reference. I rebuilt these tests several times and kept making them harder because both models kept returning identical scores. Because I selected these harder versions after seeing earlier results, they shouldn’t be treated as a representative sample of everyday work. A low Mini score is a comparison point, not an independent measure of test difficulty. 

What do I think?

In these three final high-effort runs, Terra generated fewer reasoning tokens and finished sooner than Astra. It scored lower on SQL reasoning and data validation, and tied Astra on incident log triage. That is a trade-off in the results, but it doesn’t tell us whether Terra was overwhelmed or why it made specific errors. 

The measured totals were 258.8 seconds for Terra and 709.2 seconds for Astra, so Astra took about 2.74 times as long overall. At the stated standard uncached rates, the estimated token costs were $0.61 and $7.88; Terra’s cost was about 7.8% of Astra’s. Terra scored 121/150 and Astra 148/150 under these test rubrics. The runs show a cost, time, and score trade-off; they don’t show that OpenAI deliberately cut reasoning to achieve it. 

Terra matched Astra’s scores on my earlier easy and medium versions, but that does not establish its accuracy on every task at those difficulty levels. On the final tests, I would be cautious about using it for unchecked numerical aggregation or exhaustive data validation. It may be an acceptable lower-cost alternative for some workloads, including tasks like the incident triage tested here, as long as you verify the results. Astra’s incorrect SQL total also reminds you to check its work. 

The post GPT-6.1 Terra vs. GPT-6 Astra: One sacrifices accuracy for lower cost and higher speed at high effort appeared first on The New Stack.

If your business ships software, you probably know the morning routine. Someone asks whether last night’s build passed, someone else goes hunting for why it failed, and a release sits waiting because the one person who knows which button to press is in a meeting. You may be searching for a way to make that routine run itself. GitHub Actions can do this through an API, a way for software to talk directly to other software, which means your own tools, or an AI assistant, can start, watch and clean up your builds without anyone clicking through a screen.

What GitHub Actions does

GitHub Actions is GitHub’s hosted platform for CI/CD and workflow automation. CI/CD stands for continuous integration and continuous deployment, which is the practice of automatically building, testing and releasing software every time someone changes it. If your code lives in GitHub, Actions is what runs those automated steps.

The API surface covers more than kicking off a job. It lets a caller dispatch and cancel workflow runs, check on runs and jobs, read and prune logs, artifacts and caches, rotate encrypted secrets and variables, set permissions, and manage self-hosted runners, the machines of your own that can do the work. For a business, that adds up to one thing: your delivery process becomes something other systems can operate and report on.

What you can get done

Trigger builds and keep track of what they produced

Continuous Integration Management is the everyday work of building and testing your software. The business outcome is simple. Releases can start from whatever system already knows it is time, whether that is a ticket closing or a schedule, and anyone can answer “did it work?” without asking an engineer. It also covers artifacts, the files a build leaves behind, such as the finished package you plan to ship.

• GET /repos/{owner}/{repo}/actions/workflows lists the automated workflows defined in a repository, the place your code is stored.
• POST /repos/{owner}/{repo}/actions/workflows/{workflow_id}/dispatches starts a workflow on demand.
• GET /repos/{owner}/{repo}/actions/workflows/{workflow_id}/runs shows the run history of one workflow.
• PUT /repos/{owner}/{repo}/actions/workflows/{workflow_id}/disable pauses a workflow without deleting it.
• GET /repos/{owner}/{repo}/actions/artifacts lists the build files stored across a repository.

Keep passwords and settings out of your code

Configuration & Secrets Management looks after the sensitive values your builds need, like the passwords and keys that let them reach other services. A secret is one of those values, stored encrypted. A variable is a plain setting that is not sensitive. Being able to manage them from software means that when a key leaks, or a staff member leaves, you can change it in every project from one script instead of working through them by hand.

• GET /repos/{owner}/{repo}/actions/secrets lists the names of a repository’s secrets without exposing their values.
• PUT /repos/{owner}/{repo}/actions/secrets/{secret_name} creates a secret or replaces its value.
• DELETE /repos/{owner}/{repo}/actions/secrets/{secret_name} removes a secret.
• GET /repos/{owner}/{repo}/actions/organization-secrets shows which organization secrets are shared with a repository.
• POST /repos/{owner}/{repo}/actions/variables adds a new non-secret setting.

Workflows that put it together

A workflow here means something slightly different from the build workflows above. It is a recipe of several API calls in order that finishes a real job, written down in a standard format called Arazzo. APIs.io catalogs 11 for GitHub Actions, including:

• Dispatch and Track a Workflow Run starts a workflow, finds the run it created, waits for it to complete and lists its jobs.
• Inspect Failed Job Logs finds the latest failed run and the job that failed, then gets you to its log.
• Re-run Failed Jobs and Track re-runs only the jobs that failed in the most recent failed run and waits for the result.
• Cancel a Workflow Run and Confirm requests a cancellation, checks that it took, and forces it if the run gets stuck.
• Approve a Pending Deployment finds the environments waiting on sign-off, approves them and follows the run to completion.
• Prune Repository Caches reads cache usage and deletes the largest cache to reclaim space.

Ask for it in plain language

Our conversational overlay for GitHub Actions maps everyday questions to the operation that answers them, so an AI assistant knows which call to make. These are the kinds of things you can ask:

• “How do I kick off a workflow on demand through the API?”
• “Can I see only the failed runs of one workflow on a given branch?”
• “Can I pause a scheduled workflow so it stops triggering?”
• “Can I delete a leaked repository secret by name?”
• “Which organization-level secrets can my repository’s workflows access?”

You can see one in the GitHub Actions Jobs phrasing overlay.

Getting started

GitHub Actions has a free tier with self-serve signup, and you call its hosted service. Start with the Actions REST documentation and the authentication guide, which explains the tokens every request needs to carry. On APIs.io, see the GitHub Actions profile, the Workflow Runs API, the Secrets API and the Self-Hosted Runners API. Ask whoever runs your software to wire up the failed-log workflow first, since it answers the question you already ask every morning.