Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
154428 stories
·
33 followers

Who Authorized That? The Delegation Problem in Multi-Agent AI

1 Share

Your AI agent booked a meeting, summarized a financial report, and emailed the highlights to three stakeholders. To do this, it called a calendar agent, a document analysis agent, and an email agent. Each accessed internal systems, made decisions about what to include, and acted on your behalf.

Here’s the question your security team can’t answer: Who authorized the email agent to read that financial report?

In most current architectures, the honest answer is no one explicitly. The logs may show that a service called another service. But they can’t show that the delegation itself was authorized. The authorization didn’t fail loudly. It leaked silently through the chain.

This is the delegation problem in multi-agent AI. As enterprises connect agents through protocols such as MCP and A2A, they’re solving the connectivity problem faster than they’re solving the authority problem. The result is a new security boundary that most enterprise architectures have not yet modeled, precisely because most organizations still treat it as orchestration rather than authorization.

Agents are connecting faster than authorization is adapting

The agent ecosystem has moved fast over the past two years. Anthropic’s MCP gave model-powered applications a standard way to connect to tools, data sources, and services. Google’s A2A protocol gave agents a standard way to communicate and coordinate across systems. Frameworks and SDKs such as LangChain, CrewAI, and Google’s ADK made it easier to build multi-agent workflows where one agent orchestrates several others.

What these protocols don’t yet provide, at least not as a mature common layer, is a delegation-aware authorization model.

MCP describes a protected server as an OAuth 2.1 resource server, with the MCP client acting as an OAuth client making requests on behalf of a resource owner. That’s a familiar and well-understood pattern, but it was designed for a world where a human clicks “Allow” and a single client gets a scoped token. It doesn’t address what happens when Agent A receives that token, delegates a subtask to Agent B, and Agent B spawns Agent C to handle part of it. Each hop in that chain either reuses the original token (overprivileged) or has no token at all (untracked).

A2A was built for interoperability: independent, potentially opaque agent systems communicating and coordinating actions across enterprise platforms. That’s the right problem to solve. But communication and delegation governance are different layers. A2A helps agents discover, describe, and communicate with one another. This is necessary infrastructure, but it isn’t the same as delegated authority. It doesn’t tell you whether a specific downstream action was legitimately derived from an upstream instruction.

Static API keys are even weaker for this problem. A key grants access to a service. It says nothing about who is using it, what they’re using it for, or whether the entity presenting it is the same one it was issued to. Service accounts identify a workload, not an intent. When three agents share a service account, every action looks the same in your logs.

None of these tools are broken. They solve different problems. The gap is structural. Authentication answers which agent is calling. Authorization defines what that agent may access. The harder question, and the one most enterprise architectures are not yet designed to answer, is whether a specific downstream action was legitimately derived from an upstream instruction, under narrowed constraints, with a verifiable chain back to a human decision. That’s the delegation question, and it sits in a layer that today’s stack doesn’t really have.

In a clean version of this picture, privilege should sit only with the agent that touches the outside world. If a payer (A) asks a bookkeeper agent (B) to make a payment, and the bookkeeper asks a banking agent (C) to execute the transfer, only the banking agent needs banking authority. The bookkeeper doesn’t need to move money. It only needs to know the request came from an authorized payer. The banking agent only needs to know the request came from an authorized bookkeeper. This is the principle of least privilege, a concept the security community has lived with for decades, applied to delegation chains. The difficulty is that today’s agent stacks make it hard to enforce.

What breaks in the chain

Consider a treasury reporting workflow in a regulated bank. A planning agent is allowed to read liquidity projections and produce a daily summary for senior finance users. To complete the task, it delegates chart generation to a visualization agent and narrative review to a communications agent. The visualization agent doesn’t need access to raw account-level data. The communications agent doesn’t need access to the underlying liquidity model. Yet unless the delegation layer attenuates permissions, both may receive more context than their task requires. The result isn’t a dramatic breach, but it is a quiet expansion of access that the access-control model never explicitly approved.

The risk isn’t limited to internet-facing agents. Many delegation failures happen entirely inside the enterprise boundary. An internal agent may call another internal agent, which calls an internal tool, which sends data to an approved SaaS service. Every individual step may look acceptable. The risk appears in the composition: The final data movement or action may exceed the intent of the original authorization.

This pattern creates three categories of failure that enterprises may have to explain to regulators, auditors, or customers.

Ghost permissions. A finance analyst assistant has been given access to a customer transactions database to support quarterly reporting. It calls a summarization agent: “summarize recent transactions for these accounts.” The summarization agent now operates against customer records, even though no policy engine granted it that access. The analyst assistant’s privileges effectively traveled with the request. The permission is a ghost. It exists in practice but not in any authorization system.

Scope drift. Even when an agent starts with narrow permissions, delegation tends to widen scope rather than narrow it. An agent authorized to read Q1 revenue data delegates to a charting agent, which calls an external rendering API, which now has the revenue figures. The data left the organization through three hops of implicit trust. Each agent acted within what it understood as its scope. The aggregate result exceeded what any human would have approved.

Broken audit trails. Regulated industries require the ability to answer “who did what and why” for any consequential action. In a single-agent system, this is manageable. In a multi-agent chain, the audit trail fragments across agents, protocols, and services. When a compliance team asks why a particular customer communication was sent, the answer might involve four agents across two protocols, none of which logged the delegation chain. The action is traceable to a system but not to a decision.

These aren’t edge cases. They’re a common outcome when delegation isn’t modeled explicitly. The delegation problem isn’t a bug in any particular framework. It’s a gap in the layer between them.

What a delegation-aware model requires

A delegation-aware authorization model has to solve four things at once, which is part of why no existing layer covers it cleanly.

The first is identity. The downstream agent needs a cryptographic credential that the receiving system can verify independently, not just a hostname or an API key. Hostnames lie. API keys travel. A real identity is one the calling system cannot fabricate.

The second is attenuation. When an agent delegates a task, the subagent should receive strictly fewer permissions than the parent—never the same set, and certainly never more. This is the principle of least privilege applied to delegation chains, and almost no current tooling enforces it by default.

The third is purpose. “Read this report to summarize liquidity exposure for the CFO” is a different authorization from “read this report and send selected figures to an external charting service.” It may be the same data and the same agent, but it’s two very different risk profiles. Without a purpose binding, the authorization layer has no way to distinguish them.

The fourth is audit. The organization should be able to reconstruct, after the fact, who delegated what, under which constraints, and what evidence each agent produced at completion. Not just which systems were called but which decisions were made and on whose authority.

It’s possible for agents to authenticate successfully even when they don’t have accountable authority. They can prove who they are and still execute actions that no human ever authorized.

Emerging approaches

Several efforts address parts of this problem: workload identity standards, agent metadata in tokens, OAuth-based MCP authorization, A2A authentication patterns, and agent identity frameworks. These are useful building blocks, but identity is not the same as delegated authority. A signed agent card can help establish an agent’s declared identity and capabilities. An OAuth token can tell you what a client may access. Neither, by itself, proves that a specific downstream action was authorized by a specific upstream decision under narrowed constraints.

One emerging pattern is delegation-bound capability tokens: short-lived credentials that bind an invocation to an agent identity, a constrained permission set, and a provenance record. One example is the Agent Identity Protocol (AIP), which I’ve been working on as an Internet-Draft and open source implementation. AIP is still early, but it illustrates the shape of one possible answer: invocation-bound tokens that carry identity, attenuated permissions, and provenance through a delegation chain. The token chain itself becomes part of the audit evidence rather than something reconstructed after the fact from fragmented logs.

Complementary approaches are also emerging. Behavioral credentials, the idea that agents should be continuously reauthorized based on runtime behavior rather than just initial permissions, address a related but distinct problem. Delegation tokens tell you who authorized what. Behavioral monitoring tells you whether the agent is still acting within its authorized profile. A complete solution will likely need both.

None of these approaches have reached mainstream adoption. But the fact that they are emerging simultaneously, from different corners of the industry, signals that the delegation gap is real and recognized.

What enterprise teams should do now

You don’t need to wait for standards to mature before addressing the delegation problem. There are concrete steps that security, platform, and architecture teams can take today.

Map your delegation chains. Most teams deploying multi-agent workflows haven’t documented which agents call which other agents, with what permissions, through which protocols. Start there. If you can’t draw the graph, you can’t secure it.

Audit implicit permissions. For every agent-to-agent interaction, ask: Was this access explicitly granted, or is the downstream agent inheriting permissions by proximity? If the answer is inheritance, you have a ghost permission that needs a policy decision.

Require scope attenuation. Establish an architectural rule: When an agent delegates a task, the subagent must receive fewer permissions than the parent, never more. Current tooling doesn’t enforce this automatically, but you can enforce it in your orchestration layer.

Build the audit trail before the auditor asks. If your organization is in a regulated industry, the question “Who authorized this agent action?” will eventually be asked. The time to instrument delegation logging is before that question arrives, not after. Log the full chain: which agent initiated the task, what permissions were passed, which subagents were invoked, and what each one accessed.

Test with real tooling. Delegation-aware approaches, including capability-token designs, workload identity standards, and agent identity frameworks, are early but functional. Running one in a nonproduction environment will expose gaps in your current authorization model that architecture review alone will not surface.

Delegation is the security boundary

The first phase of enterprise agent adoption was about connectivity: Can the agent reach the tool, the API, the database, or the other agent? The next phase will be about accountable delegation: Should this agent be allowed to ask that agent to do this specific thing, with this data, under these constraints?

That question won’t be answered by prompt engineering. It belongs in the authorization layer, the platform layer, and the audit trail.

Enterprises don’t need to solve the entire standards problem today. But they do need to stop treating delegation as an implementation detail. In multi-agent systems, delegation is the security boundary.



Read the whole story
alvinashcraft
16 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

I’m a Professional Fact-Checker. AI Is Wrong More Often Than You Think

1 Share
Can AI do fact-checking? A WIRED fact-checker fact-checks.
Read the whole story
alvinashcraft
17 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Building fun and creative messaging experiences on WhatsApp - Michelle “MishManners” Duke

1 Share
From: NDC
Duration: 1:00:39
Views: 60

This talk was recorded at NDC Sydney in Sydney, Australia. #ndcsydney #ndcconferences #developer #softwaredeveloper

Attend the next NDC conference near you:
https://ndcconferences.com
https://ndcsydney.com/

Subscribe to our YouTube channel and learn every day:
/ @NDC

Follow our Social Media!

https://www.facebook.com/ndcconferences
https://twitter.com/NDC_Conferences
https://www.instagram.com/ndc_conferences/

#ai #machinelearning #tools #api #whatsapp #openai #twilio

In this talk, I’ll show how to turn a simple WhatsApp message into an AI-powered creative experience. Using Twilio, WhatsApp, and OpenAI’s image generation APIs, we’ll build a workflow where users can send images via WhatsApp; think selfies, pet pics, and photos from the conference, and receive a cartoon, sketch, or fun version of your photo to share on your socials, group chats, and company message boards.

In this talk we'll cover:
- Why should we consider creating fun and engaging experiences via messaging
- How OpenAI's API works
- How to set up Twilio’s WhatsApp sandbox to send and receive incoming media
- How to process and transform images with OpenAI’s latest APIs
- Utilising existing libraries to transform images
- Sending images through WhatsApp
- Best practices for handling image input, latency, and user experience
- Other cool things you can build once you have images

This demo is practical, fun, and showcases how teams can blend communication APIs with AI to create delightful, interactive messaging experiences.

Read the whole story
alvinashcraft
17 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Why the Product Trio Breaks the Hand-Off Mentality That Kills Team Engagement | Njegos Ilic

1 Share

Njegos Ilic: Why the Product Trio Breaks the Hand-Off Mentality That Kills Team Engagement

Read the full Show Notes and search through the world's largest audio library on Agile and Scrum directly on the Scrum Master Toolbox Podcast website: http://bit.ly/SMTP_ShowNotes.

 

"I can't change people, but I can definitely involve them." - Njegos Ilic

 

Njegos describes a pattern he's encountered multiple times as a product owner: teams where engagement is almost nonexistent. He walks into a refinement session, presents ideas, asks for feedback — and gets crickets. Nobody pushes back, nobody asks questions, nobody challenges the assumptions. The result is a product owner working in isolation, defining everything alone, only to discover gaps during development that could have been caught early with a single conversation. Njegos is honest about the limits of what any one person can do — you can't change people's personalities, and expecting a Scrum Master to do so is unrealistic. But what you can do is involve people. His approach when joining a new team: don't come in announcing how things will work. Instead, learn how the team already works, meet them where they are, and then find ways to fit new concepts into their existing rhythm. For the non-negotiable things — the red lines — he's precise, open, and always provides an alternative rather than just pushing his way.

 

In this segment, we talk about Discovery and Delivery and the Product Trio concept.

 

Self-reflection Question: When you join a team meeting and get silence instead of feedback, do you assume agreement — or do you treat it as a signal that something deeper needs to change?

Featured Book of the Week: Inspired by Marty Cagan

Njegos recommends Inspired by Marty Cagan as the book that most shaped his approach to product ownership. He highlights the entire SVPG series — including Empowered and Transformed (available as the Product is Hard SVPG Box Set) — but points to the Product Trio concept as especially powerful. As Njegos puts it, the Product Trio — bringing together a product manager, a tech lead, and a designer — removes the hand-off mentality where each discipline works in isolation. Instead of the product owner defining everything alone and handing it to the team, the trio shapes problems together during discovery, so that by the time work reaches the team, there's shared understanding of why they're building something, not just what to build.

 

[The Scrum Master Toolbox Podcast Recommends]

🔥In the ruthless world of fintech, success isn't just about innovation—it's about coaching!🔥

Angela thought she was just there to coach a team. But now, she's caught in the middle of a corporate espionage drama that could make or break the future of digital banking. Can she help the team regain their mojo and outwit their rivals, or will the competition crush their ambitions? As alliances shift and the pressure builds, one thing becomes clear: this isn't just about the product—it's about the people.

 

🚨 Will Angela's coaching be enough? Find out in Shift: From Product to People—the gripping story of high-stakes innovation and corporate intrigue.

 

Buy Now on Amazon

 

[The Scrum Master Toolbox Podcast Recommends]

 

About Njegos Ilic

 

Njegos is a motivated and forward-thinking Product Manager and Agile Project Manager with experience in fast-paced SaaS environments. He empowers teams through leadership and guidance across product development. With a Lean mindset, he simplifies complexity, delivers in small, testable increments, and leverages rapid feedback loops to prioritize outcomes over output.

 

You can link with Njegos Ilic on LinkedIn.





Download audio: https://traffic.libsyn.com/secure/scrummastertoolbox/20260526_Njegos_Ilic_Tue.mp3?dest-id=246429
Read the whole story
alvinashcraft
17 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Lean Into the Ambiguity Before the Ambiguity Leans Into You

1 Share

Bob and Josh sit with a hard truth this week. The change rolling through software and leadership isn't optional anymore. It's not Agile in 2007, where you could adopt it or sit it out. AI is rewriting what application development looks like, what team size means, and what kind of leader your organization actually needs.

The good news is you're standing in a greenfield. Not your codebase. You. Your hiring approach, your operating model, the way you lead, all of it is open for redesign right now.

Bob shares the cartoon that captures the whole problem (everyone lines up to change others, nobody lines up to change themselves), walks through reframing as a daily leadership tool, and gets honest about applying it in his own caregiving life with his wife.

Josh challenges leaders to stop letting ambiguity happen to them and start steering into the maelstrom on purpose. If you've been waiting for things to settle before you make a move, this is the nudge to stop waiting.

Stay Connected and Informed with Our Newsletters

Josh Anderson's "Leadership Lighthouse"

Dive deeper into the world of Agile leadership and management with Josh Anderson's "Leadership Lighthouse." This bi-weekly newsletter offers insights, tips, and personal stories to help you navigate the complexities of leadership in today's fast-paced tech environment. Whether you're a new manager or a seasoned leader, you'll find valuable guidance and practical advice to enhance your leadership skills. Subscribe to "Leadership Lighthouse" for the latest articles and exclusive content right to your inbox.

Subscribe here

Bob Galen's "Agile Moose"

Bob Galen's "Agile Moose" is a must-read for anyone interested in Agile practices, team dynamics, and personal growth within the tech industry. The newsletter features in-depth analysis, case studies, and actionable tips to help you excel in your Agile journey. Bob brings his extensive experience and thoughtful perspectives directly to you, covering everything from foundational Agile concepts to advanced techniques. Join a community of Agile enthusiasts and practitioners by subscribing to "Agile Moose."

Subscribe here

Do More Than Listen:

We publish video versions of every episode and post them on our YouTube page.

Help Us Spread The Word: 

Love our content? Help us out by sharing on social media, rating our podcast/episodes on iTunes, or by giving to our Patreon campaign. Every time you give, in any way, you empower our mission of helping as many agilists as possible. Thanks for sharing!





Download audio: https://episodes.captivate.fm/episode/9e0c4dd4-f38f-41f5-8cb5-f16335dd9d58.mp3
Read the whole story
alvinashcraft
18 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Bringing LLMs to the edge

1 Share

This Maker Monday, the Raspberry Pi AI Camera meets large language models. This tutorial is courtesy of the editor of Raspberry Pi Official Magazine, Lucy Hattersley.

(Yes, we know it’s Tuesday, but we enjoyed a glorious and uncharacteristically sunny Bank Holiday Monday in the UK yesterday.)

Large language models (LLMs) offer new intuitive ways to interact with technology. From natural conversations with chatbots to summarising long documents, LLMs excel at understanding and generating human‑like text. 

The Raspberry Pi AI Camera detects objects in real time while the LLM interprets the results, combining vision data with language-based insight

What happens when we combine the power of LLMs with the Raspberry Pi AI Camera? This pairing opens up new ways to connect the physical world of vision recognition to intelligent language-driven systems. 

These powerful new systems are being called vision-language models (VLMs). This approach lets you build systems that describe and reason about the physical world using natural language. All without streaming video to the cloud, helping to keep your capture private and reduce the burden of GDPR compliance.

Figure 1: Constant data flow from the AI camera to the user

In this tutorial we will consider one way to do this using the Raspberry Pi AI Camera. Our approach will be where the Raspberry Pi AI Camera constantly sends prompts containing the metadata to the LLM. This approach can be seen in Figure 1.

Set up AI Camera

Ensure your Raspberry Pi AI Camera is connected to Raspberry Pi. Before we start, ensure that your Raspberry Pi runs the latest software. Run the following command to update:

$ sudo apt update && sudo apt full-upgrade

The AI Camera must download runtime firmware onto the IMX500 sensor during startup. To install these firmware files onto your Raspberry Pi, run the following command:

$ sudo apt install imx500-all

Raspberry Pi’s AI Camera does the heavy lifting with the AI model detecting objects, recognising patterns, and generating metadata on the sensor like {Cat (0.76), Box (0.81)}.

Instead of streaming raw video to the cloud, the system can output the inference results as metadata, drastically reducing the amount of data transmitted to the cloud or to other systems. This is particularly beneficial in environments with limited bandwidth or expensive data costs. This means the camera provides structured insights as inference results; for example, labels, bounding boxes, and confidence scores. These are then passed to an LLM, which turns structured detection data into human-readable summaries and contextual insights.

The code snippet (01_aicam_to_llm.py) at the end of this article can be adapted to your own situations. This sends the metadata from the Raspberry Pi AI Camera to an LLM using OpenAI. To run it, you will need to install modlib and the OpenAI library, then get your own API key for OpenAI.

Let’s set up the code. First, clone all the files from our GitHub account.

$ git clone https://github.com/lucyhattersley/aicam_llm.git

Take a look inside with ls and you will see example code for all our projects. Many code files contain the same code with different prompts. We expect you to finally use one of the original code files with your own prompt. 

We will need to create a virtual environment so we can add the OpenAI and Application Module Library (modlib) packages.

$ python -m venv env

And activate our virtual environment:

$ source env/bin/activate

Use pip to install modlib and openai:

$ pip install modlib openai

Now edit the file and add your API key. We are going to use the Thonny IDE to do this:

$ thonny 01_aicam_to_llm.py

Add your API key to line 8, replacing <OPENAI_API_KEY> with the key inside straight quotes so it looks like:

client = OpenAI(api_key="abcde012345")

Save the file and exit Thonny.
Now run the file with:

$ python 01_aicam_to_llm.py

The first time you do this, it will perform a Network Firmware Upload. Wait for the file to upload (around 30 seconds). After this, the terminal will display a text description of what is in the viewfinder:

LLM summary: At 16:33:29,
The camera detected several objects with their respective confidence scores.
The detected objects include:
**Persons**: 3 instances with confidence scores of 0.44, 0.38, and 0.32.
**Books**: 2 instances with confidence scores of 0.44 and 0.32.
**Potted plant**: 1 instance with a confidence score of 0.38.
**Dining table**: 1 instance with a confidence score of 0.38.
**Cup**: 1 instance with a confidence score of 0.32.
**Bowl**: 1 instance with a confidence score of 0.32.
This suggests a setting likely involving people, reading materials, and dining or relaxation items.

We can adjust this program to identify different things by adjusting the prompt on line 23 of our code. The subsequent programs adjust this prompt to perform different tasks.

  • 01a_smart_home.py
  • 01b_retail_shelf.py
  • 01c_factory_floor.py

Inspect these programs with Thonny or an IDE of your choice and look at the prompt on line 23.

Smart Home Observer

On the Raspberry Pi AI Camera, we run an object detection model to detect objects of interest like people and pets, producing results with data containing the class and confidence like:

{"detections": ["Person (0.92)", "Cat (0.87)", "Box (0.82)"]}

Then the Raspberry Pi AI Camera sends this information to the LLM, which processes the results. The prompt on line 23 is:

prompt = f"You have access to a smart camera in the living room of my home. At {time.strftime('%H:%M:%S')}, the camera detected: {labels}"

When run, the code produces a friendly update:

At 14:23, one person is in the living room with the cat. A box is in the room as well.
The smart home observer in action, showing person and cat detections with LLM summary

Retail Shelf Monitor

With a Raspberry Pi AI Camera monitoring a shelf, vending machine, or a fridge, we can use an object detection model to detect the items we wish to monitor. Then we can add functionality to check what shelf or row the items are on. We send the LLM the detections with a prompt:

prompt = f"You have access to a smart camera in a vending machine. At {time.strftime('%H:%M:%S')}, the camera detected: {labels} Provide information on the stock levels of the vending machine."

And the LLM generates a report:

"Four soda bottles are left in row three — stock may need replenishing soon."
Retail shelf monitor detecting bottles in row three

Factory Floor Watcher

Raspberry Pi AI Camera checks if workers are wearing safety gear. In this situation, we can add some more application logic to match people with high-vis jackets to make sure they are wearing one. The prompt on line 23 of our code is:

prompt = f"You have access to a smart camera in a warehouse. At {time.strftime('%H:%M:%S')}, the camera detected: {labels} Provide information if people are wearing highvis jackets."

Then the metadata is forwarded to an LLM, which produces a natural alert:

Warning: one worker is not wearing a high-vis.

As we can see, the prompt on line 23 of our code can be adjusted to a wide variety of tasks using natural language. 

Factory floor watcher detecting compliant and non-compliant workers

The post Bringing LLMs to the edge appeared first on Raspberry Pi.

Read the whole story
alvinashcraft
18 minutes ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories