Git worktrees have been around for over ten years, but if you’re like me, you might not have heard of them until recently. AI coding tools now regularly use worktrees to make local changes in parallel, allowing multiple coding agents to work autonomously without interfering with each other’s work. Once complete, you can then merge those worktrees back into your repository. This allows you to avoid sending code to the cloud while still parallelizing as much work as possible.

Worktrees are an elegant solution to the parallelization problem, although they are frequently misunderstood and, more importantly, poorly explained. This post demystifies worktrees so you can get up and running quickly.

Creating branches

You can think of a worktree as a branch that exists in a different location from the project root. It still maintains all of the commit history from the root, it just exists in a different space. You can push and pull from remotes, too. For all intents and purposes, it functions like any other branch in your repository. The only difference is where the files live.

Without worktrees, you’d create a branch like this:

git checkout -b feature/name main

This creates a new branch off of main called feature/name in your project root. For example, if your project exists in ~/projects/my-project, that’s also where the branch is created. (The project root is also called the main worktree.)

If instead you want to create a worktree with that branch name, you can do so with this command:

git worktree add ../projects/my-project.worktrees/feature-name -b feature/name main

This creates the directory project.worktrees/feature-name and creates a branch off of main called feature/name. All of your files from main now exist in this separate directory, and you can cd into that directory to continue work as usual, with a couple of caveats:

1. Dependencies are not shared. If you are working on a Node.js project, you’ll need to run npm install again inside the worktree directory. This directory doesn’t have access to the main worktree’s node_modules directory, so it needs its own copy. This is true for any projects requiring installation of dependencies to work.
2. The worktree owns the branch. Back in your main worktree, you’ll get an error if you try git checkout feature/name because every branch can only be checked out to one directory. The worktree directory currently has that branch checked out so no other directory can do so.

Otherwise, you can treat a worktree directory like any other directory with a clone of your repository.

Directory conventions

There are two conventions for creating worktree directories:

1. Direct sibling. A lot of tutorials describe creating worktree directories as direct siblings of the main worktree. So if your main worktree is ~/projects/my-project, you might create a worktree directory of ~/projects/my-project-feature-name.
2. Shared sibling ancestor. Some coding agents create a single sibling directory within which all worktree directories are created. For example, ~/projects/my-project.worktrees contains all of the worktree directories related to ~/projects/my-project. I prefer this approach and use it in this post.

Merging changes

How you merge changes from a worktree back into your main worktree is a matter of preference.

Pull requests

If you’re working in a collaborative environment, you might just push your worktree branch to origin and open a pull request. That pull request can then be merged directly into main just like any other pull request. Because a worktree contains a complete copy of your repository, the relationship between the worktree branch and the remote persists.

Merge locally

If you’d rather merge changes from your worktree manually, you can do that much the same way you would with any other branch. First, be sure you have all of your changes committed in your worktree. Then, you can merge into the main worktree:

# Go into worktree
cd ../my-project.worktrees/feature-name

# Rebase on top of main to ensure a clean merge
git rebase main

# Go to main worktree
cd ../my-project

# Merge in your changes
git merge feature/name

At this point, you can decide whether you want to keep the branch and worktree around.

Cleaning up

When you’re done with a worktree, you can remove it:

# Deletes the directory but not the branch
git worktree remove ../my-project.worktrees/feature-name

This physically removes the worktree directory but does not remove the branch. At this point, with the branch no longer checked out in a worktree, you can once again check it out in the main worktree if you’d like.

When you’re sure you no longer need the branch, you can delete it as usual:

# Delete the branch
git branch -d feature/name

Other tips

There are a few other useful things to know about worktrees.

List all worktrees

To see all worktrees that currently exist (including the main worktree):

git worktree list

This is especially helpful when using AI coding agents who may be creating worktrees without your direct knowledge.

Fewer characters to type

If you’d like to type fewer characters, you can set up an alias for worktree:

# Set wt as an alias for worktree
git config --global alias.wt worktree

# Much shorter commands
git wt add ../myproject.worktrees/ -b feature/name main
git wt list
git wt remove ../myproject.worktrees/

Conclusion

Git worktrees are a powerful, underappreciated feature that fit naturally into modern development workflows, especially as AI coding agents become a bigger part of the picture. Once you understand that a worktree is really just a branch checked out to a different directory, the mental model clicks quickly. You get all the benefits of parallel development without shipping your code to a remote service to make it happen.

The workflow is straightforward: create a worktree for each task, do your work, merge or open a pull request when done, and clean up. Commands like git worktree list give you visibility into what’s checked out where, and a simple alias like wt keeps the overhead low. Give worktrees a try next time you need to juggle multiple tasks in the same repository.

AI prompts live or die on specificity. “Help me debug this auth issue” gets a different response than “I’m getting 401 errors on this collection; help me debug the auth, sync my environment variables, and validate OAuth 2.0 token expiration.” Same intent, very different output.

Postman’s prompting research calls this out: vague instructions are the single biggest reason AI output disappoints developers. The fix isn’t a smarter model. It’s a better prompt.

This is the problem Postman Agent Mode recipes solve. Recipes are pre-written prompt templates for common API tasks, covering everything from auth debugging to compliance audits. You drop one into the chat, swap in your collection or specification, and Agent Mode runs the task using your actual workspace context: your requests, environment variables, response bodies, and API specifications.

In this post, I’ll walk through four recipes I’ve found genuinely useful, explain the prompt patterns behind them, and show you how to write your own.

What Agent Mode does

Before we get to recipes, a quick mental model. Agent Mode isn’t a chatbot that happens to know about HTTP. It runs inside your Postman workspace and has access to:

• Your collections and the requests in them
• Environment variables (secret values stay redacted)
• Live response bodies from requests you’ve sent
• API specifications in Spec Hub
• Native Git history when connected

When you ask Agent Mode to “fix the auth on this collection,” it can inspect the headers, see which requests are failing, and propose changes. That’s the part recipes lean on: they assume the agent can read what you’ve built and act on it.

Recipe 1: Debug auth errors

Auth is where most API integrations fall apart, and 401 errors are the loudest signal that something is off. The official Agent Mode Debug Auth Errors recipe starts with this prompt:

I'm getting 401 errors on this collection; help me debug the auth, sync my
environment variables, and validate OAuth 2.0.

What I like about this prompt is how it stacks three diagnostic steps in one go. Most 401 issues come from one of a handful of root causes: an expired access token, a missing Authorization header, a Bearer prefix typo, or an environment variable pointing at the wrong host. Asking the agent to walk all three explicitly means you don’t end up with a generic “check your credentials” response.

When I run this against a collection where the access_token variable has gone stale, Agent Mode typically:

1. Checks the token expiration claim in any JWT it can find
2. Compares the Authorization header format across requests
3. Suggests refreshing the token using the OAuth 2.0 authorization helper

Worth noting: the recipe assumes you’ve set OAuth client credentials in an environment, not hardcoded them in a request. If you’ve stored secrets directly in a collection, the Postman Secret Scanner will catch it before Agent Mode does.

Recipe 2: Collection cleanup

Collections get messy fast. Three engineers, two months, and you’ve got duplicate endpoints, half-renamed folders, and three different ways to call /users. The Collection Cleanup recipe uses this prompt:

Find duplicate requests, identify redundant endpoints, and reorganize this
collection by grouping related requests into folders with descriptive names.

What the agent does here is interesting. It looks beyond identical URLs and matches on semantic similarity, so two requests hitting /api/v1/users/{id} and /api/v1/users/:id get flagged even though the path templating differs. It also reads request descriptions and method types to group related calls.

I ran this on a 60-request collection I’d inherited and ended up with:

Proposed changes:
  - Merge 4 duplicate POST /users requests into 1 canonical version
  - Group 12 requests into new folder "User management"
  - Group 8 requests into new folder "Authentication"
  - Mark 3 requests as deprecated (no longer in OpenAPI spec)

Apply changes? [y/N]

Always review the diff before applying. I once accepted changes that merged two requests that looked identical but had different test scripts. The cleanup removed scripts I needed.

Recipe 3: Broken documentation refresh

API docs go stale faster than anything else in a codebase. The Broken Documentation Refresh recipe uses recent test runs as the source of truth:

Analyze my latest test results and schemas to generate comprehensive API
documentation with interactive examples and current response shapes.

This is a meaningfully different approach from generating docs out of a static specification. Instead of trusting the OpenAPI file, the agent reads recent responses from your collection runs and writes the docs around what the API returns in practice. If your spec says one thing and the response says another, the docs end up matching the response.

The output goes into a markdown file you can publish through Postman Docs or commit to your repo. The interactive examples are real requests with sanitized response bodies, not placeholder JSON.

Recipe 4: Compliance audit

This one is the most opinionated and the most useful for anyone shipping to a regulated industry. The Compliance Audit recipe:

Implement security testing and compliance checks for GDPR and HIPAA; validate
role-based access control, scan for exposed secrets, and check sensitive
data handling in request and response bodies.

The recipe scans for a few patterns:

1. Hardcoded secrets in request URLs, headers, or bodies (anything matching a known token format)
2. Personally identifiable information in test data, like email addresses or phone numbers that aren’t redacted
3. Missing RBAC tests for endpoints that should enforce role boundaries
4. Sensitive fields returned in response bodies without redaction

This is where the Postman API Governance rules come in. If your team has custom governance rules defined, the audit checks against those too. I’ve used this to find an internal API where the staging environment had a real customer email in the test body. Embarrassing, but useful to catch before it hit production logs.

Write your own recipe

The official recipes are starting points. Once you see the pattern, you can write your own.

A good recipe has three parts:

1. The task, stated specifically. Not “test my API” but “test all GET endpoints in this collection for 200/404/500 responses and check that response times stay under 250ms.”
2. The context, scoped to what the agent should look at. Mention the collection name, the environment, or specific folders. Use @ in the chat to reference them directly.
3. The output format, when it matters. “Return a markdown table with endpoint, expected status, actual status, and response time.”

Here’s one I use for a personal project:

@MyAPI Run a load test against the /search endpoint at 50 requests per second
for 30 seconds. Use the queries in test-queries.csv. Report p50, p95, and p99
response times in a markdown table. Flag any 5xx responses with the full
request and response body.

The result is specific, scoped, and clear about what to return. The agent has everything it needs to give a useful answer the first time.

Gotchas I’ve hit

A few things I’ve learned the hard way running these:

Agent Mode uses AI credits. Every recipe consumes Postman AI credits, and the more complex recipes (compliance audits, server scaffolding) can use a lot. Check your usage before running batch operations on large collections.

The agent will modify your collection if you let it. Always review proposed changes before approving. Collections are versioned in Native Git, so you can roll back, but it’s faster to read the diff first.

Secrets stay safe, but test data doesn’t. Environment variables marked as secrets are redacted from prompts. Test data in your requests is not. If you have customer PII in test bodies, the agent sees it.

Try it yourself

The fastest way to get a feel for this is to run a recipe against an existing collection. If you don’t have one handy, fork the Postman Echo collection into your workspace.

Then open Agent Mode in Postman (the chat icon in the workbench), paste in one of the recipe prompts from the official recipes page, and see what it does.

Worth running the cleanup recipe first on a low-stakes collection. It gives you a feel for the diff review flow without risking anything important. Once you trust the output on a sandbox collection, the auth debugging and compliance audit recipes are worth running on something you actually ship.

Resources

• Postman Agent Mode overview
• Agent Mode recipes catalog
• Seven best practices for writing high quality prompts
• Postman AI credits and billing
• Postman API Governance
• Native Git in Postman
• OAuth 2.0 RFC 6749
• OpenAPI specification

The post Postman Agent Mode Recipes for Common API Tasks appeared first on Postman Blog.

Over the past few years, we’ve been on a journey to modernise how we run Amazon Elastic Compute Cloud (EC2) instances at Slack.

In our first post, Advancing Our Chef Infrastructure, we shared how we moved from a single Chef stack to a resilient, multi-stack setup with versioned cookbook deployments and safer promotion workflows. This afforded us far more reliability and operational control across tens of thousands of EC2 instances.

Subsequently, in Advancing Our Chef Infrastructure: Safety Without Disruption, we tackled deployment risk without the need for teams to rewrite their cookbooks. By introducing split production environments, signal-based Chef runs, and smarter rollout mechanisms, we dramatically reduced the impact radius of failures while keeping our legacy platform stable. These changes allowed us to safely operate our EC2 ecosystem at scale while we plan the future at a relaxed pace.

But as we kept improving, a bigger truth became clear.

Even with safer rollouts, better orchestration, and stronger guardrails, the old model—continuously updating long-lived EC2 instances—was hitting its limits. Service-level deployments were tricky, infrastructure drift was inevitable, and coordinating changes across multiple layers added complexity. Containers solved this for some classes of workloads, but not everything could migrate easily.

We needed a new approach—one that brought modern deployment practices like immutability, progressive rollouts, and automated safety directly to EC2 instances.

Enter Shipyard.

Shipyard is Slack’s next-generation EC2 platform. It treats infrastructure as deployable artifacts rather than endlessly mutable instances. It gives teams service-level deployment primitives, tight integration with our build and orchestration systems, and the confidence to update infrastructure with the same safety and predictability we expect from modern app delivery platforms.

In this post—the third chapter of our journey—we’ll explain why we built Shipyard, the principles behind its design, and how it represents a fundamental shift in how we think about running EC2 instances at Slack.

What Shipyard Provides

Shipyard is designed to bring modern infrastructure principles to EC2 instances while preserving the flexibility teams expect from EC2. Rather than focusing on configuration management as the center of the system, Shipyard shifts responsibility toward build pipelines, deployable artifacts, and automated safety mechanisms.

Some of the key capabilities of the platform include:

Multi-Architecture and Multi-OS Support

The platform is designed from the start to support multiple CPU architectures, including both AMD64 and ARM-based Graviton instances, with support for multiple operating systems such as Ubuntu, RHEL, and Amazon Linux. This flexibility allows teams to optimize for cost, performance, or compatibility without needing separate platform implementations.

Shipyard is particularly valuable for workloads that cannot migrate to containers, such as infrastructure components, Kubernetes worker nodes, and our egress network stacks.

Metrics-Driven Deployments with Safety Controls

Each service integrates with our deployment orchestration system called Gondola to enable progressive rollouts with metric based automated safety checks. Deployments can automatically halt based on service health signals, or automated rollback to previous known good versions.

Fast and Predictable Provisioning

Shipyard uses a layered image approach, inspired by how containers work. A shared “golden” base image provides common infrastructure components, and service-specific images are built on top of that foundation. This minimizes work at launch time, so instances can come online quickly and predictably across regions.

Simplified Configuration Management

One of the biggest architectural shifts is how we use configuration management.
Previously, instances would run scheduled Chef jobs that periodically checked and reapplied configuration, so any manual or unexpected changes would be reverted back to the desired state.

In the new model, configuration is applied during well-defined lifecycle phases like image baking and initial provisioning, rather than being continuously enforced in the background. Configuration tools are mainly used to deploy services, not to constantly modify the entire system. This reduces background load, avoids unintended overwrites, and makes system behavior much easier to reason about, since instances aren’t continuously changing over time.

Real-Time Inventory and Fleet Visibility

Shipyard comes with a new inventory system called Peekaboo, giving us near real-time visibility into the state of our EC2 fleet. Instead of relying on Chef Server as the source of truth, Peekaboo taps directly into cloud events and instance metadata, providing better telemetry across environments. It can even track instances from non-Shipyard deployments, giving us a complete view of the entire fleet in one place.

We built Peekaboo using AWS EventBridge, OpenSearch, and Lambda. It has everything teams need: a UI to explore the fleet, an API for integrations, and a command-line interface (CLI) for quick command-line checks. By centralising this information, it removes the guesswork and gives us a single place to view and manage EC2 instances.

Short-Lived, Continuously Refreshed Instances

To keep our EC2 instances secure and truly immutable, each instance has a limited lifespan and is automatically rotated on a regular schedule. This means our fleets are always fresh; potential vulnerabilities have less time to cause issues, and teams focus on replacing instances rather than making in-place changes.

Golden Base Images: Introducing slack-zero

At the foundation of the Shipyard is a shared base image called “slack-zero.” This is the core machine image built by Slack’s Compute Platform Team and maintained collaboratively with our security and monitoring teams.

The slack-zero image contains:

• Operating system baseline and hardening
• Networking and service discovery configuration
• Monitoring and security agents
• Common tooling and foundational system configuration

You can think of slack-zero similarly to how teams use a base Docker image or how we start from a vendor-supplied Ubuntu image and layer additional components on top. It provides a standardised, trusted foundation that every service inherits, while still allowing teams to customize their own runtime environment on top of it.

Base images are treated as immutable but ephemeral. When foundational components need to change—such as a security patch, monitoring update, or networking improvement—a new slack-zero image is produced. Downstream service images can then rebuild on top of the updated base to inherit the latest fixes and improvements.

To build slack-zero, we use AWS Image Builder rather than Packer. Image Builder provides several built-in advantages over our previous approach with Packer, including:

• Lifecycle Management: Old AMIs are automatically cleaned up using lifecycle policies, helping reduce storage costs.
• AWS System Manager (SSM) Parameter Publishing: Each new slack-zero image updates an SSM parameter that indicates the latest available AMI for an account. Service pipelines read this parameter to ensure they always build on the most up-to-date base.
• Event-Driven Automation: When a slack-zero image finishes baking successfully, EventBridge and Lambda automatically trigger downstream pipelines in service owner accounts so dependent images can be rebuilt.
• Built-in Testing: Before an AMI is published, Image Builder launches temporary instances and runs validation tests. This ensures every image is verified before distribution, reducing risk and increasing confidence in production rollouts.

Together, these capabilities allow us to continuously evolve the platform foundation while keeping adoption friction low for service teams.

Service Images

Each service team builds its own AMIs using slack-zero as the foundation. This allows teams to control their runtime environment while inheriting standardized platform components maintained by the platform organization.

Service image pipelines define:

• What software is installed
• How the service is configured
• What happens during instance initialisation for this service

Because most configuration is baked directly into the image, instances launch quickly and consistently, minimizing configuration drift and ensuring predictable behavior across the fleet.

By combining the immutable slack-zero base with service-specific layers, Shipyard provides both platform stability and team-level flexibility, enabling services to innovate safely without sacrificing operational consistency.

Baking and Provisioning

Shipyard separates instance preparation into two distinct phases: baking and provisioning.

During the bake phase, we install packages and include configuration that is consistent across environments. This ensures every instance starts from a fully prepared, known-good state with the majority of work already completed before launch.

Environment-specific settings—such as secrets, regional configuration, or deployment metadata—are applied during the provisioning phase when the instance boots. This step is intentionally lightweight and typically involves only dropping configuration, retrieving secrets, and starting services.

By moving heavy operations like package installation into the bake phase, instances can become operational in seconds rather than minutes. This fast startup time is critical for scaling events, rolling deployments, and automated instance replacement.

This provisioning model provides a strong balance of consistency, speed, and flexibility: images deliver a stable baseline, while minimal provisioning adapts instances to their runtime environment without introducing drift.

Deployments and Fleet Updates

When teams need to roll out changes, they build a new Amazon Machine Instance (AMI) and then run their deployment pipeline to roll it out. Instead of patching existing instances, fleets are updated through controlled replacements, keeping everything consistent and predictable.
For Auto Scaling Groups (ASGs), we use AWS Instance Refresh, and Kubernetes worker fleets use Karpenter for lifecycle-driven updates. Services with special deployment needs can use alternative rollout executors. Our global deployment orchestrator, Gondola, supports these patterns, giving teams flexibility while keeping a consistent deployment experience.

Emergency Fixes and Rapid Deployment Pathways

For urgent situations, the platform allows targeted configuration changes on running instances, but these are meant for emergencies only. Affected instances are expected to be replaced afterward via regular deployment pipelines.
Our emergency workflows use AWS Systems Manager with a predefined document to run selected Chef recipes, letting teams quickly apply critical fixes. Once stable, instances are cycled to return to the intended immutable state.

What Do Customer Pipelines Look Like?

Customer pipelines in Shipyard can have multiple stages, giving teams flexibility to design them around their service and operational needs. Each stage in Gondola represents a deployable unit, such as an ASG, a Kubernetes cluster, or a group of EC2 instances.
For example, the Egress Team runs separate canary and production ASGs in each availability zone, with deployment stages ordered so updates flow sequentially. Gondola updates each stage, monitors key metrics, and rolls back automatically if problems are detected, preventing issues from spreading.
This staged approach, combined with Shipyard’s fast provisioning and immutable AMIs, lets teams safely deploy complex updates at scale while maintaining observability and control.

What Happens in the Gondola Stage?

When Gondola builds an artifact, it produces a deployable package for a service with two main parts:

• The AMI to roll out across the fleet
• The Chef artifact containing versioned recipes associated with a Git commit

Gondola treats these together as a single deployable unit. Each stage uses a service-defined executor to perform the rollout:

• ASG-based deployments: The executor updates the launch template with the new AMI and configuration. Chef code is packaged to Amazon Simple Storage Service (S3), and new instances use a baked-in bootstrapper to fetch the correct artifact and run the relevant recipes. Configuration metadata ensures only the right configuration is applied.
• Kubernetes worker fleets: The executor tells Karpenter which AMI to use and provides the same configuration metadata. Nodes bootstrap in the same way, ensuring consistent provisioning.

For services with special deployment needs, Gondola makes it easy to add new executors. This lets Shipyard support different deployment models while keeping artifact handling and provisioning consistent and predictable.

By combining AMI updates, versioned configuration artifacts, and metadata-driven bootstrapping, Gondola ensures every instance gets the correct software and configuration for its role, no matter how it’s deployed.

Simple Pipeline

Shared Responsibility

Shipyard’s layered image model is built on a clear shared responsibility between platform teams and service teams. The Compute, Security, and Monitoring teams manage the base layer, making sure it includes all global infrastructure components, security patches, and essential configurations. Service teams then build their own AMIs on top of this base, adding the software and settings specific to their service.

Whenever the Compute team rolls out a fix or update—whether it’s a security patch, a monitoring agent update, or a networking change—service teams are responsible for incorporating the updated base into their own AMIs. This way, every service image automatically benefits from the latest improvements and security fixes from the shared base.

This approach lets each team focus on what they do best while keeping the fleet consistent, secure, and reliable. The diagram below illustrates how the base and service layers work together and highlights where each team’s responsibilities lie.

A Caveat on Immutability

While Shipyard instances are mostly immutable, there is an important exception: secrets. Each instance runs the Consul Template service, which allows us to roll out updated secrets from Vault without cycling the fleet. This means that, although packages and configurations are fixed at bake time, sensitive data like credentials or certificates can still be updated dynamically. Our infrastructure is semi-immutable: the core system and service layers remain consistent, but critical runtime secrets can be refreshed safely as needed. This approach balances stability, predictability, and security across the fleet.

The Reaper

The Reaper evaluates two primary inputs. First, it consumes signals from external systems—such as security tooling or AWS EC2 events—that indicate an instance may no longer be in its desired state and should be considered “tainted.” Second, The Reaper performs periodic checks to identify instances that have been running for longer than their allowed lifespan. When either condition is met, the instance is scheduled for replacement according to its service policies.

This approach helps reduce configuration drift, limit exposure to vulnerabilities, and reinforce the principle that infrastructure should be updated by redeploying rather than modifying systems in place. The result is a platform that is more reliable, auditable, and predictable.

For example, while manual remote access remains available for emergency scenarios, manually accessing a production-class node will generate a signal that marks the instance for eventual replacement, supporting our immutable infrastructure goals.

The Reaper also integrates with Peekaboo to track instance age across the fleet. Once a node reaches its maximum lifespan, it follows the same graceful replacement workflow.

Looking ahead, we plan to make the system more context-aware so that only meaningful changes—such as software updates or configuration drift—trigger replacement, while read-only or low-risk actions do not create unnecessary churn.

Taming the Reaper

The Reaper is designed not only to enforce lifecycle policies, but also to give teams control over how replacements occur. Built-in rate limiting allows service owners to define how many instances can be replaced at a time, scoped by service, region, or availability zone, preventing sudden capacity impacts.

For emergency situations, we provide a global pause mechanism—the “big red button.” By placing a control object in S3, teams can temporarily halt all Reaper activity across the fleet. This provides a safe and immediate way to stop instance cycling during incidents or periods of elevated risk.

We also provide a CLI that allows service owners to manage rate limits, inspect configuration, and activate or release the global pause when needed.

In addition, controlled access mechanisms such as short-lived SSH certificate workflows can be used for break-glass scenarios where deeper investigation is required, while still maintaining overall lifecycle safety commitments.

Together, these capabilities make the Reaper both predictable and controllable—enforcing instance immutability by default while giving teams the visibility and safeguards they need to operate confidently.

How Do We Test Changes?

Both platform teams and service owners need a safe way to test cookbook changes before merging a pull request, so we built a system called Ship Quick—a developer workflow that runs a realistic bake-and-provision test on real infrastructure.
Developers run a CLI command from their cookbook repo, where a YAML file defines the test cases. Ship Quick packages the cookbook, uploads it to S3, and sends a workflow message to a queue. A fleet of worker instances—managed by a lightweight process called Longshoremen—picks up the job, detaches from its Auto Scaling Group, runs the Chef workflow, streams logs back to the CLI, and then terminates (unless the developer chooses to keep it for debugging).
We run Longshorem in two separate worker fleets because of how our bootstrapping works. The vanilla Ubuntu fleet is used to bake and test the base slack-zero image—it can’t build on top of itself, so starting from a clean Ubuntu AMI is required. The slack-zero fleet is for service team cookbooks, which depend on the pre-baked slack-zero AMI. Running tests from this fleet ensures provisioning is validated against the same foundation used in production.
Splitting the fleets this way helps ensure each layer is tested against the correct base. Both fleets scale automatically with demand, and slack-zero workers are continuously updated to the latest images so tests always reflect the current production environment.
Teams that build images in their own AWS accounts can also provision dedicated worker fleets and route Ship Quick jobs to them, ensuring isolation while keeping the same workflow.

What’s Next?

So far, Shipyard has been working really well for short-lived services, and we’re actively onboarding teams from the legacy EC2 platform. Our next challenge is long-lived instances—things like Slack’s data nodes, singleton services like GitHub Enterprise, or third-party business technology instances such as Atlassian JIRA. These can’t be cycled quickly, so we need ways to patch and update them safely while ensuring the Reaper handles them correctly.

We’re collaborating closely with service teams to develop new deploy executors in Gondola for these longer-lived workloads. As more teams adopt Shipyard, we’ll keep iterating on tooling, developer workflows, and the overall deployment experience to meet the platform’s diverse needs.

Future posts in this series will cover the challenges we encounter as Shipyard evolves and take a closer look at its components—including the Shipyard API, image pipelines, developer workflows, and our inventory system. Stay tuned!