Pankaj Kumar: The Scrum Master Success Metric of Team Independence
Read the full Show Notes and search through the world's largest audio library on Agile and Scrum directly on the Scrum Master Toolbox Podcast website: http://bit.ly/SMTP_ShowNotes.
"The best thing to measure the success of a Scrum Master is that the team is not dependent on the Scrum Master." - Pankaj Kumar
For Pankaj, Scrum Master success has a clear test: can the team work, decide, inspect, and improve without depending on the Scrum Master for every step? He looks for self-organization, confidence in Scrum events, and a team that can continue improving without constant intervention. But he does not rely only on impressions. Pankaj uses team metrics such as velocity, backlog health, lead time, cycle time, and Cumulative Flow Diagrams when Kanban is in use. He also pays attention to team satisfaction and regular feedback. The frequency of that reflection depends on team maturity. Mature teams may need a two-week check-in rhythm, while newer teams need closer weekly attention. His maturity model is deliberately adapted team by team, because every team's product, skills, technical context, and backlog are different. The bigger point for Scrum Masters is useful: success is not being needed in every conversation. Success is seeing the team grow enough that your presence becomes lighter.
Self-reflection Question: What would your team do this week if you were not available to guide the Scrum events?
Featured Retrospective Format for the Week: Anonymous Action-Tracking Spreadsheet
Pankaj keeps retrospectives simple and practical. Before the retrospective, he shares a spreadsheet where team members can add what went well, what needs improvement, the action needed, who is accountable, the deadline, and the status of previous actions. He also allows anonymous input, which helps quieter team members raise points they may not want to voice live. During the meeting, the team reviews previous retrospective actions and then discusses the current sprint. Pankaj is careful about who is in the room. Sometimes supervisors are needed, but often their presence changes what people are willing to say. For him, facilitation starts with creating the right audience for the conversation.
[The Scrum Master Toolbox Podcast Recommends]
🔥In the ruthless world of fintech, success isn't just about innovation—it's about coaching!🔥
Angela thought she was just there to coach a team. But now, she's caught in the middle of a corporate espionage drama that could make or break the future of digital banking. Can she help the team regain their mojo and outwit their rivals, or will the competition crush their ambitions? As alliances shift and the pressure builds, one thing becomes clear: this isn't just about the product—it's about the people.
🚨 Will Angela's coaching be enough? Find out in Shift: From Product to People—the gripping story of high-stakes innovation and corporate intrigue.
Pankaj Kumar is an experienced Scrum Master and Agile Coach with over 18 years in IT. He helps teams adopt Agile ways of working, improve collaboration, remove impediments, and deliver value consistently. He facilitates Scrum ceremonies, coaches teams and stakeholders, and supports Agile transformation across organizations.
1225. This week, we talk to lexicographer Steve Johnson from Dictionary.com. We look at how slang like "no cap" and "throw hands" works its way up from a separate slang dictionary, why one lexicographer fought against adding "rizz," and where "lost in the sauce" comes from. Then we look at what "overprompting" says about how we talk to AI, why food words like "quesabirria" and "elote" can take years to reach the dictionary, and how the team is starting to size up contenders for word of the year.
🔗 Share your familect recording in Speakpipe or by leaving a voicemail at 833-214-GIRL (833-214-4475)
As AI moves beyond the cloud and into robots, vehicles, and other physical systems, new approaches to models, simulation, and inference are emerging. Daniel and Chris are joined by Ming-Yu Liu, Vice President of Cosmos Lab at NVIDIA, who shares his perspective on open models, world models, and the future of physical AI. They explore why open models matter for innovation, how world models can help AI understand and simulate the physical world, and what it could take to build increasingly capable physical AI systems.
Midwest AI Summit: Join AI practitioners on October 15 in Indianapolis for practical sessions, hands-on discussions, and real-world AI solutions. Use code PracticalAI20 to save 20% on your registration. https://midwestaisummit.com/#tickets
Prediction Guard: A self-hosted AI control plane for running agents in high impact environments. predictionguard.com/practicalai
PreEmptive has provided application shielding and hardening solutions for over 20 years, protecting .NET, MAUI, Java, Android, and JavaScript applications from reverse engineering, tampering, malware insertion, debugging, and runtime attacks. Dotfuscator for .NET and MAUI, DashO for Java and Android, and JSDefender for JavaScript integrate into development workflows to apply layered protection such as code obfuscation, string encryption, tamper detection, root and jailbreak detection, and runtime checks.
Mobile apps give users on-the-go access to banking services, email, the internet, games, healthcare portals, and more. They’re a must-have for any smartphone owner. However, they’re also attractive targets for attackers seeking to steal sensitive information and intellectual property, modify code, spy on users, or bypass business logic.
To protect businesses and consumers from mobile app attacks, many developers incorporate app shielding. This security method uses a combination of structural and runtime techniques to make applications harder to reverse engineer, tamper with, debug, or exploit. Organizations that develop mobile, desktop, web, and enterprise applications can use app shielding as part of a broader application protection strategy.
TL;DR
App shielding protects applications from reverse engineering, tampering, debugging, malware insertion, code injection, and runtime attacks. It combines techniques such as code obfuscation, string encryption, tamper detection, integrity checks, RASP, anti-debugging, and environment checks. PreEmptive supports app shielding through Dotfuscator for .NET and MAUI, DashO for Java and Android, and JSDefender for JavaScript, helping teams add layered protection directly into their build process.
What is app shielding?
App shielding is a cybersecurity defense that protects applications from threats such as data theft, reverse engineering, code tampering, debugging, malware insertion, and runtime manipulation. It’s commonly used in apps with valuable intellectual property, sensitive data, compliance requirements, or high fraud risk, such as banking apps, healthcare portals, payment apps, enterprise software, gaming apps, and government applications.
Unlike security methods that focus only on network or perimeter defense, app shielding protects the application itself. It does this by combining static and dynamic protection mechanisms, including code obfuscation, encryption, tamper detection, root and jailbreak detection, anti-debugging, and runtime application self-protection.
PreEmptive’s app shielding approach protects applications at build time by applying layered defenses directly into the application. This helps development teams make deployed binaries and scripts more resistant to reverse engineering, tampering, and runtime analysis without relying on a separate external agent.
In-app protection vs. application hardening vs. app shielding: what’s the difference?
Several cybersecurity methods are used to protect applications from attacks, including in-app protection, application hardening, and app shielding. Each method supports application security, but they emphasize slightly different techniques.
Protect the app across build-time and runtime attack scenarios
In-app protection
In-app protection uses built-in security controls to deter specific threats. Methods like code obfuscation make it harder for attackers to read or understand an application’s code, while runtime and environment checks can detect conditions such as debugging, tampering, rooting, or jailbreaking.
PreEmptive’s in-app protection capabilities are designed to be infused into the application itself, helping teams detect and respond to attacks without requiring a separate external monitoring agent.
Application hardening
Application hardening uses a range of techniques to safeguard an application’s code and make it harder to reverse engineer, modify, or exploit. Hardening often includes obfuscation, string encryption, control flow transformation, anti-debugging, tamper detection, and runtime checks.
PreEmptive’s hardening tools help teams apply these protections to .NET, MAUI, Java, Android, and JavaScript applications as part of the software build process.
App shielding
App shielding is the broader application protection strategy. It incorporates multiple structural and dynamic techniques to prevent unauthorized access, secure sensitive data, protect intellectual property, detect runtime attacks, and deter code injection or tampering.
PreEmptive supports app shielding through a product suite built for different application stacks: Dotfuscator for .NET and MAUI, DashO for Java and Android, and JSDefender for JavaScript.
Why is app shielding important? Key benefits explained
App shielding protects organizations and app users from cybersecurity threats. When implemented correctly, it reassures users that an app is safer to use and helps teams reduce the risk of reverse engineering, tampering, fraud, and data exposure. App shielding can also support compliance and risk reduction efforts in regulated industries such as healthcare, finance, e-commerce, government, and defense.
Protect against application-level risks
App shielding defends applications against threats such as:
Reverse engineering: Code obfuscation makes the application codebase harder to interpret, reducing the risk of intellectual property theft or exploit discovery.
App tampering: Anti-tamper and integrity checks help detect attempts to modify application code or alter app behavior.
Debugging and dynamic analysis: Anti-debugging and runtime checks help detect tools attackers use to inspect application logic.
Malware insertion: Embedded protections can help make it harder for attackers to modify or repackage applications with malicious code.
Unauthorized access: Environment and integrity checks can help detect risky runtime conditions before they lead to data or logic exposure.
Build security into the application
App shielding supports DevSecOps by making protection part of the application itself. Instead of relying only on perimeter defense or post-deployment monitoring, shielding embeds protective controls into the application during development and build workflows.
PreEmptive’s build-time approach allows teams to apply app shielding protections during the same process they already use to build and release software.
Improve user experience
Strong app shielding can protect applications without interrupting legitimate users. For example, an app may detect a rooted or jailbroken device, debugging tools, or tampering attempts and respond according to the organization’s risk policy.
The best approach depends on the application’s risk profile. A banking app may need stricter responses than a lower-risk consumer app, while an enterprise or healthcare app may prioritize data protection, regulatory expectations, and user trust.
Add layered protection
App shielding goes beyond one security control. It combines multiple layers, including obfuscation, encryption, anti-tampering, integrity checks, anti-debugging, root and jailbreak detection, and runtime protection. This makes the application harder to analyze, modify, or exploit.
PreEmptive’s layered application protection uses a combination of obfuscation, encryption, shielding, root detection, and tamper proofing to make apps more resistant to exploitation.
Maintain regulatory readiness
Some apps, especially those used in healthcare, banking, finance, e-commerce, government, and defense, are subject to strict data protection and security expectations. App shielding techniques can help teams demonstrate stronger application protection practices as part of a broader compliance and risk management program.
Protect data and privacy
Many apps handle sensitive user data, including payment details, health information, personal identifiers, credentials, or confidential business information. App shielding helps protect that data by making it harder for attackers to reverse engineer sensitive logic, extract secrets, modify code, or intercept behavior at runtime.
Build user confidence
Customers are wary of sharing personal information with applications they do not trust. A reputation for weak security can hurt engagement and retention. By applying app shielding, teams can help demonstrate that application protection is part of the product’s design, not an afterthought.
What are the two types of app shielding mechanisms?
App shielding typically uses two categories of protection: structural protection and dynamic protection. Structural defenses protect the application at the code or binary level. Dynamic defenses provide runtime safeguards when the application is in use.
Structural protection
Structural, or static, protection is integrated into the application before it runs. Common techniques include:
Code obfuscation: Makes code harder to understand or reverse engineer.
String encryption: Protects sensitive strings and values from easy extraction.
Control flow transformation: Makes program logic harder to follow.
Integrity checks: Help detect whether the application has been modified.
Resource encryption: Helps protect embedded resources or sensitive assets.
Structural protection is useful for defending against reverse engineering, data extraction, code theft, and intellectual property exposure.
PreEmptive’s Dotfuscator, DashO, and JSDefender all support structural protection by transforming code during the build or packaging process.
Dynamic protection
Dynamic protection uses runtime security checks to detect risky conditions while the application is running. Common techniques include:
Runtime application self-protection: Observes application behavior and responds to suspicious activity.
Anti-debugging: Detects attempts to attach debuggers or analysis tools.
Root or jailbreak detection: Identifies compromised mobile environments.
Tamper detection: Detects whether the application has been modified.
Environment checks: Identifies unsafe or unexpected runtime conditions.
Dynamic protection is useful against tampering, runtime analysis, emulator abuse, debugging, code injection, and manipulation after deployment.
Why use both types of protection?
Combining structural and dynamic protection gives teams stronger coverage. Structural protection makes the application harder to understand and modify. Dynamic protection helps detect and respond to attacks while the application runs.
Together, these layers make the attacker’s job more expensive, slower, and less reliable.
What are the top app shielding techniques?
App shielding is not a single security protection. It combines multiple security strategies that work together to prevent attackers from exploiting applications.
Technique
Protection type
Primary threat addressed
Where PreEmptive fits
Code obfuscation
Structural
Reverse engineering and IP theft
Dotfuscator, DashO, JSDefender
String encryption
Structural
Extraction of sensitive strings or values
Dotfuscator, DashO, JSDefender
Control flow transformation
Structural
Static analysis and reverse engineering
Dotfuscator, DashO, JSDefender
Anti-tampering
Dynamic
Code modification and repackaging
Dotfuscator, DashO
RASP and runtime checks
Dynamic
Runtime attacks and suspicious behavior
Dotfuscator, DashO
Integrity checking
Dynamic
Binary or package modification
Dotfuscator, DashO
Anti-debugging
Dynamic
Debugger attachment and dynamic analysis
Dotfuscator, DashO
Root or jailbreak detection
Dynamic
Compromised mobile environments
DashO
Environment checks
Dynamic
Unsafe or unexpected runtime conditions
DashO and platform-specific runtime checks
1. Code obfuscation
Attackers may try to access an application’s codebase to steal intellectual property, understand business logic, or identify weaknesses. Code obfuscation makes this harder by transforming code into a form that is difficult to understand while preserving functionality.
Common code obfuscation techniques include:
Renaming: Replaces meaningful variable names, methods, and classes with less meaningful names.
Control flow obfuscation: Adjusts program logic flow so it is harder to reverse engineer.
String encryption: Encrypts sensitive strings in the code and decrypts them only when needed.
Data obfuscation: Disguises data structures and constants to reduce easy extraction.
PreEmptive’s Dotfuscator applies symbol renaming, control flow obfuscation, string encryption, and related protections to .NET and MAUI applications as part of the build process. DashO applies similar protections to Java and Android applications, while JSDefender protects JavaScript applications from static analysis and reverse engineering.
2. White-box cryptography
Attackers may attempt to steal or misuse cryptographic keys stored in an application. White-box cryptography helps protect keys and cryptographic operations in hostile environments where attackers may be able to inspect the app.
White-box cryptography can be especially relevant for payment apps, media apps, and applications that need to protect sensitive keys even when the app is deployed to user-controlled devices.
3. Anti-tampering
Attackers may attempt to modify an application’s code to steal data, bypass controls, unlock paid features, or compromise functionality. Anti-tampering techniques detect or block attempts to change application code.
Common anti-tampering measures include:
Checksum or signature verification
Runtime integrity checks
Anti-debugging
Root or jailbreak detection
Responses such as logging, alerts, shutdown, or disabled functionality
PreEmptive’s tamper detection injects runtime integrity checks directly into the protected application at build time. This helps teams detect binary modification or unauthorized changes without requiring a separate external agent.
4. Runtime application self-protection
Structural techniques protect apps at the code level, but RASP elevates protection while the application is running. RASP observes the application’s runtime behavior and can detect suspicious activity such as injection attempts, debugger attachment, tampering, or execution in compromised environments.
PreEmptive’s runtime checks can detect debugger attachment, binary modification, and root or jailbreak conditions in process. This helps applications respond to threats even when they are running outside the organization’s direct control.
5. Encryption
Encryption converts sensitive data into unreadable text so unauthorized parties cannot interpret it without the proper key. In app shielding, encryption can protect strings, resources, configuration values, credentials, and other sensitive assets embedded in the application.
PreEmptive’s app protection tools include string encryption and related techniques designed to make sensitive values harder to extract from protected applications.
6. Integrity checking
Integrity checking verifies that the application has not been modified. It may calculate a value when the app starts or during execution and compare it against an expected value. If the check fails, the app can log the event, alert an administrator, disable functionality, or take another configured action.
PreEmptive’s integrity and tamper detection capabilities are designed to help teams detect unauthorized modification of protected binaries.
7. Runtime protection
Runtime protection scans for suspicious activity when an application is running. It can help detect abnormal behaviors that put the application or user data at risk, such as tampering, debugger attachment, or execution in unsafe environments.
PreEmptive supports runtime protection through in-app checks that can be added during the build process.
8. Secure communication protocols
App shielding can be paired with secure communication protocols, such as HTTPS and TLS, to help ensure data access and transmission are protected. Secure communication helps reduce the risk of interception, but it does not replace code obfuscation, runtime checks, or tamper protection.
9. Environment checks
Environment checks inspect the platform or device where an application is running. These checks can identify conditions such as rooted or jailbroken devices, emulators, debuggers, or other signs of compromise.
PreEmptive’s DashO supports Java and Android application protection, including protections that help teams detect risky runtime environments and respond according to application policy.
Why PreEmptive for application shielding
PreEmptive is a trusted application protection provider for teams that need to protect applications across .NET, MAUI, Java, Android, and JavaScript environments. PreEmptive’s product suite includes:
Dotfuscator: Application protection for .NET and MAUI.
DashO: Application protection for Java and Android.
JSDefender: Application protection for JavaScript.
Dotfuscator has been embedded into Visual Studio since 2003 and is subject to Microsoft’s regression tests, code audits, and security reviews. PreEmptive says this makes its technology the only third-party technology with that level of Visual Studio integration and validation.
PreEmptive’s build-time approach means protection can be configured once and applied during the build process. Teams can add obfuscation, control flow transformation, string encryption, tamper detection, runtime checks, root detection, and related protections without changing the way end users install or run the application.
PreEmptive also brings breadth across application stacks. Dotfuscator, DashO, and JSDefender help teams apply layered application protection across desktop, mobile, web, cloud, and enterprise environments.
Application stack
PreEmptive product
Common protections
.NET and MAUI
Dotfuscator
Renaming, control flow obfuscation, string encryption, tamper detection, runtime checks
Java and Android
DashO
Obfuscation, control flow transformation, string encryption, tamper detection, root and jailbreak detection
JavaScript
JSDefender
JavaScript obfuscation, string and literal transformation, control flow protection, runtime protection techniques
Organizations including Charles Schwab, Citibank, IBM, FedEx, and ADP are listed among PreEmptive’s clients. PreEmptive also states that more than 5,000 companies and 20,000 users worldwide rely on its application protection tools.
App shielding examples: how real apps stay secure
In practice, app shielding techniques work together to safeguard applications from intrusion, analysis, and tampering. Here’s how organizations may apply them.
White-box cryptography for payment apps
Payment apps may use white-box cryptography to protect encryption keys during financial transactions. Even if an attacker has access to the app’s underlying code, white-box cryptography can make it harder to locate or extract the keys needed to misuse payment data.
Anti-tampering for streaming services
Streaming service apps may use anti-tampering techniques to prevent attackers from modifying the app to bypass access controls, copy content, or disable business rules. Anti-tampering tools can detect unauthorized modification and trigger a configured response.
Code obfuscation to protect health information
Health-oriented apps may use code obfuscation to make application logic harder to reverse engineer. Anyone attempting to inspect the app would encounter transformed code that is harder to read, making it more difficult to identify sensitive logic, extract intellectual property, or understand how protected workflows operate.
Runtime checks for financial services apps
Financial services apps may use runtime checks to detect debugging tools, rooted devices, tampering attempts, or suspicious environments. These checks help reduce the risk of fraud, credential theft, and unauthorized access.
JavaScript shielding for web applications
JavaScript apps expose client-side logic directly to users and attackers. JSDefender helps protect JavaScript applications by making source-distributed code harder to understand, modify, or exploit after deployment.
How to choose the right app shielding approach
The right app shielding approach depends on your application stack, threat model, compliance requirements, and business risk.
If you are protecting intellectual property
Prioritize code obfuscation, control flow transformation, string encryption, and anti-tamper checks. These techniques make proprietary logic harder to reverse engineer or reuse.
If your app handles payments or sensitive data
Prioritize layered protection that includes code obfuscation, encryption, tamper detection, runtime checks, and environment detection. Payment and banking apps may also need white-box cryptography and stricter runtime responses.
If your app runs on mobile devices
Prioritize root and jailbreak detection, anti-debugging, tamper detection, integrity checks, and runtime protection. Mobile apps often run in environments the organization does not control, so in-app protection becomes especially important.
If your app uses JavaScript
Prioritize JavaScript obfuscation, control flow protection, string transformation, and runtime checks. Client-side and Node.js JavaScript can be easier to inspect than compiled binaries, so shielding should be part of the build process.
If your team needs protection without slowing development
Choose tools that integrate into your existing build pipeline and automate protection at every build. PreEmptive’s Dotfuscator, DashO, and JSDefender are designed to fit into development workflows so teams can apply protection without making security a separate manual step.
Bottom line
App shielding protects applications from reverse engineering, tampering, debugging, malware insertion, and runtime attacks by combining structural and dynamic protection techniques. The strongest strategies use multiple layers, including code obfuscation, string encryption, tamper detection, integrity checks, RASP, anti-debugging, root detection, and environment checks.
PreEmptive’s application shielding tools, Dotfuscator for .NET and MAUI, DashO for Java and Android, and JSDefender for JavaScript, integrate into development workflows and apply layered protection during the build process. This helps teams protect deployed applications without relying on a separate external agent or waiting until after release to think about application protection.
Ready to evaluate app shielding for your application stack? Start a free trial or contact PreEmptive to see how Dotfuscator, DashO, and JSDefender can help protect your applications.
App shielding FAQs
What is app shielding?
App shielding is a set of application protection techniques that defend software from reverse engineering, tampering, debugging, malware insertion, and runtime attacks. It commonly includes code obfuscation, encryption, anti-tampering, integrity checks, RASP, and environment checks.
How does app shielding work?
App shielding works by adding protective controls directly into the application. Static protections make the code harder to inspect or modify, while runtime protections detect suspicious behavior, tampering, debugging, rooted devices, or other risky conditions while the app is running.
What is the difference between app shielding and application hardening?
Application hardening strengthens an application against analysis and unauthorized modification, often through obfuscation, encryption, and tamper detection. App shielding is a broader strategy that combines hardening with runtime checks, RASP, environment detection, and other protections.
What is the difference between app shielding and RASP?
RASP is one app shielding technique. It detects and responds to suspicious activity while an application is running. App shielding is broader and may include RASP, code obfuscation, string encryption, anti-debugging, tamper detection, root detection, and integrity checks.
How does PreEmptive support app shielding?
PreEmptive supports app shielding through Dotfuscator for .NET and MAUI, DashO for Java and Android, and JSDefender for JavaScript. These tools apply protections such as obfuscation, string encryption, control flow transformation, tamper detection, runtime checks, and environment checks.
Which PreEmptive product should I use?
Use Dotfuscator for .NET and MAUI applications, DashO for Java and Android applications, and JSDefender for JavaScript applications. Teams with multiple application stacks may use more than one PreEmptive product to apply layered protection across their portfolio.
Does app shielding replace secure coding?
No. App shielding complements secure coding, SAST, SCA, penetration testing, and runtime monitoring. It helps protect deployed applications from analysis and tampering, but teams should still build secure code and test for vulnerabilities throughout the SDLC.
This is part two of my series on the agent orchestration framework I'm building in .NET. You really should read the first part, Introduction, before this one, so if you haven't already, take the time to do so now!
In this post I'm going to talk about the orchestrator, the central piece of the framework. I'm going to describe its contract and how we should use it. Some related parts will be covered in other posts.
The IOrchestrator Interface
The orchestrator is main part of the framework, as I said before. It is the one that coordinates - with some assistance - the different agents, distributes the work to be done, enforces reviews and revision processes, then gathers all responses and summarises them. The interface that describes the orchestrator is IOrchestrator (surprise, surprise), and this is what it looks like:
I took out all comments and some less-important stuff. In a nutshell, we have:
The IOrchestrator interface implements IDisposable, this is essentially to clear the event handlers, as the orchestrator itself is pretty much stateless - it does contain references to a number of services, which may or may not be stateless (all of the built-in services described in this post are stateless)
It exposes a number of events, for the different stages of the orchestration process (more on this in a moment)
The most important method, and the only one shown here, is OrchestrateAsync. This is the method that starts the orchestration process, it is asynchronous and returns an OperationResult
A property that indicates whether or not the orchestrator is currently processing (IsProcessing); it is enabled while the OrchestrateAsync method is running
The only class that implements IOrchestrator is Orchestrator, and it is sealed, meaning, not meant to be derived from.
There is a configuration class, OrchestratorOptions, which can be provided to the Orchestrator constructor. This is what it contains:
public sealed class OrchestratorOptions
{ public bool EnableEvents { get; set; }
public bool EnableInternetSearch { get; set; }
public bool EnablePeerReview { get; init; }
public int MaxReviseAttempts { get; init; }
public int? MaxTokens { get; init; }
}
The properties are:
EnableEvents: controls whether or not the orchestrator raises events for each stage of the processing; the default is true
EnableInternetSearch: controls whether Internet searches are allowed; the default is false
EnablePeerReview: whether or not to enforce review of agent responses; the default is true
MaxReviseAttempts: the maximum number of revise attempts before the agent response is considered final; the default is 5
MaxTokens: the maximum number of tokens that can be consumed for the processing of an orchestration, including all agents that are involved; null, meaning, no limitation will be enforced
The Orchestration Process
The orchestrator deals with agents (IAgent instances, of which I won't talk much here), so it must receive a non-empty collection of agents, with distinct names. It also receives a prompt and, somehow, distributes that prompt - or, actually, a specific prompt that is derived from the original one - to one or some of the agents it knows.
The process starts with the OrchestrateAsync method, which is responsible for creating a session id, and starting the flow for a given prompt. This prompt is first sanitised before usage, so as to prevent unwanted contents. There are many stages in the agent processing, and some events are raised. This flow consists of:
Filter the original prompt before using it (IPromptFilter)
Plan what to do, meaning, what are the sub-tasks and the required specialisations (IPlanner)
Assign each plan assignment to a concrete agent (IAgentSelector)
Execute each agent with its own assignment (IAgentScheduler)
For each finished agent (author), select another agent to review its work (IAgentSelector)
Ask the review agent to review and produce feedback (*)
Ask the author to revise based on the feedback (*)
When all reviews are finished, create a summary of all the final responses (ISynthesizer)
(*) these steps may occur multiple times
When the Orchestrator is processing, its IsProcessing property will be true, and no other orchestration operation can be started. The orchestration execution can be cancelled at any time through the CancellationToken parameter.
A session is essentially a call to OrchestrateAsync, an orchestration. Its id should be unique and is present as a property in all classes that are produced or consumed by the orchestration process, its purpose is to add a correlation identifier.
The following events are raised during the orchestration process, which should be self-explanatory:
OrchestrationStarting: raised once when the orchestration starts (OrchestrateAsync)
OrchestrationStopped: raised when OrchestrateAsync finishes
AgentTrainingStarting: raised for each agent when it is being trained
AgentTrainingStopped: raised when an agent has finished training
AgentExecutionStarting: raised when an agent's execution is starting
AgentExecutionStopped: raised when the agent execution finished
AgentReviewStarting: raised when a review process is starting
AgentReviewStopped: raised when a review process finished
AgentReviseStarting: raised when an agent is beginning to revise its work based on a review
AgentReviseStopping: raised when an agent's revise process has finished
One note: events are only raised if OrchestratorOptions.EnableEvents is set to true, which is the default.
Some events - AgentTraining*, AgentExecution*, AgentReview*, AgentRevise* - can, of course, be raised multiple times, one for each agent - AgentTraining*, AgentExecution* - or maybe multiple times for a single agent - AgentReview*, AgentRevise*. The Orchestration* events only fire once per orchestration. All these events are raised from another thread, so they do not block the working of the orchestrator, and throwing an exception from an handler will be ignored silently. Each event has an argument that is specific to the event and adds context properties including a SessionId property.
The orchestrator also produces many logs, using the Information, Warning, Debug, or Error, severity levels. These can be enabled or disabled by the usual ways. All operations and their results are logged. Traces are also produced, including the following activities:
orchestrator.orchestrate: when the orchestrator starts
orchestrator.plan: when the orchestrator is planning
orchestrator.synthesize: when the orchestrator is synthesising the responses
The following tags are included:
orchestrator.session_id: the session id
orchestrator.prompt_length: the prompt length
orchestrator.max_tokens: the maximum configured number of tokens to consume
orchestrator.enable_peer_review: whether peer review is enabled
orchestrator.max_revise_attempts: the maximum configured revise attempts
orchestrator.assignments_count: the number of assignments after the plan
orchestrator.tokens_consumed: the total number of tokens consumed at the end of the orchestration
At the end of the OrchestrateAsync method the orchestrator returns an OrchestrationResult instance. It contains these properties (simplified):
public sealed record OrchestrationResult {
public string SessionId { get; }
public string FinalResponse { get; }
public IReadOnlyList<AgentResponse> AgentResponses { get; }
public IReadOnlyList<ReviewResult> Reviews { get; }
public bool WasShortCircuited { get; }
public bool Cancelled { get; }
public long TotalTokens { get; }
public bool Success { get; }
public string? Error { get; }
}
Where:
AgentResponses contains the final responses, one for each agent that was involved in the orchestration
Cancelled is set to true if the operation was cancelled through the CancellationToken parameter
Error contains an error message if the orchestration was not finished successfully or was cancelled
FinalResponse property contains the synthesised responses from all the agents
Reviews contains all the final reviews
SessionId is the id of the current session
Success means that the orchestration finished successfully
TotalTokens contains the ever-increasing number of tokens used by the processing
WasShortCircuited is flagged if the work was handled by a single agent
So, it is safe to look at FinalResponse if Success is true, or to Error otherwise.
Service Dependencies
The Orchestrator class, besides the IAgent collection, makes use of the following services, which actually implement the important parts of the flow:
Selects the appropriate agent the execution or review according to some algorithm
DefaultAgentSelector
ISharedStateManager
Manages the shared state which may be used to pass information between agents
InMemorySharedStateManager
IPlanner
Creates the execution plan
DefaultPlanner
IPromptFilter
Filters the prompt before using it
DefaultPromptFilter
ISynthesizer
Summarises all the final agent responses
DefaultSynthesizer
All of these services are injected through the Orchestrator constructor, and are all optional, meaning, the default implementation will be used if one is not provided.
There are two included implementations of IAgentScheduler: one that processes each agent sequentially (SequentialAgentScheduler), and another one that executes all in "parallel" (ParallelAgentScheduler), which is the default. I will dwell in this in a future post.
DefaultAgentSelector uses one algorithm for selecting the executing agent and another for the reviewer, I will also cover them shortly.
Both DefaultPlanner and DefaultSynthesizer implementations depend on the existence of an IAgent with a specialization of Planning and Synthesis, respectively. If one does not exist, the implementation will be sub-optimal.
The InMemorySharedStateManager implementation of ISharedStateManager manages state in memory, essentially a collection of key-value pairs. It is provided as a sample, more robust implementations might include out-of-process storage.
The included implementation of IPromptFilter, DefaultPromptFilter, just returns the original prompt, makes no attempt to filter or change it. It is provided just as a sample.
All of these services are required for the orchestration to work, but different implementations can lead to very different results. It is interesting to notice that the Orchestrator class itself knows nothing about AI, LLMs, or Microsoft.Extensions.AI: this knowledge exists in some of these services.
Conclusion
We are just starting to get into the technical aspects of my orchestration framework, here I just covered one of the most important parts of it; I certainly did not cover all of the orchestrator, but I hope you can get a good picture of it. In the next post I will cover the agents, which are also very important, as they wrap the connection to an LLM provider and model.
I hope you find this interesting, feel free to ask or comment whatever you like!
Agents write most of my code these days. That leaves me with this question: what do I do with all this free time?
I considered learning a new framework, but that sounds suspiciously like work. Luckily, the VS Code team has a better idea. They gave me something to take care of.
What is the VS Code pet?
The pet is an interactive character that sits above the chat input box. It reacts to chat activity and to whatever you do to it.
So while the agent works, the pet watches. Before, I was the only one staring at the progress indicator while pretending to supervise. Now I have company.
Remark: The pet is marked as experimental, so it might change or be removed. Don't get too attached. (I got attached within five minutes.)
Adopting your pet
Type /vscode-pet in the chat input to show or hide it.
In the new-session view of the Agents window, you can also right-click outside the input box and select Pet (/vscode-pet).
Show or hide. That's the entire commitment. As far as I can tell, there is no feeding schedule, no vet, and no 3 AM walks. Compared to my Kubernetes cluster, it's remarkably low maintenance.
Spending quality time
My calendar is suddenly wide open, so let's see what we can do together:
Select it to trigger a reaction. With the keyboard, press Tab to focus it, then Enter or Space. Yes, the pet is accessible. Your accessibility audit will be thrilled.
Drag it around the chat and drop it wherever you like. It's like rearranging the furniture, except the furniture has opinions.
Flick it to throw it. Don't worry, the pet comes back.
Hop it with the Left and Right arrow keys while it has focus.
Throw it at a wall by holding Shift with an arrow key.
That last one is officially documented. I'm not sure what it says about the creators, but I hope they are feeling well.
Right-click the pet (or press Shift+F10 when it has focus) and you get a context menu where you can:
view its achievements
send it on the run
resize it
switch between Stable and Insiders colors
Achievements are the interesting part. For the first time, there is a way to measure how well I'm doing as a pet owner. Finally, a metric I can bring to my next performance review. "Delivered 14 microservices" doesn't land like "Unlocked all pet achievements."
One pet, many sessions
Only one pet appears at a time in the active chat surface. Its position and size are shared across chats and windows, and they persist after you restart VS Code.
So, the pet remembers where you left it. That's more than I can say for most of my agents, who forget everything the moment the context window fills up. 😁
Remark: VS Code supports running multiple agent sessions in parallel. Parallel pets are not supported. I think the priorities are clear.
So, what now?
Does this mean I should spend my newly found free time playing with a pet in my editor? Probably not.
But right now, my agent is refactoring a legacy module, my pet is hopping around, and I'm drinking coffee. One of us is being productive, and I have a strong suspicion it's not me.
That's it. Back to work. Or at least back to the pet.
Log in
