Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
157266 stories
·
33 followers

How to Turn a Postman Collection into a Maintainable pytest Suite

1 Share

A Postman collection is a great place to explore an API. But it's a poor place to keep your tests.

Most teams find this out the slow way. Someone exports the collection, converts the requests into test code once, and moves on. Six months later the tests are red, nobody trusts them, and they get skipped in the pipeline. The conversion was never the hard part. Keeping the suite alive is.

This tutorial takes you from a Postman collection to a pytest suite that still passes next quarter. First we'll look at why converted tests rot, then at four principles that keep them alive. The examples stay small, so you can try them on your own collection today.

Table of Contents

Before You Start

To follow along you will need:

  • Python 3.10 or newer, with pytest and httpx installed (pip install pytest httpx).

  • A Postman collection you want to convert, with its environment (the base URL and token).

  • Basic pytest knowledge: how fixtures work and how to run pytest from the command line.

  • A GitHub repository if you want to try the continuous integration step. You can skip that part and still follow the rest.

Diagram: a Postman collection converts into a generated pytest suite (the easy step), which then becomes a maintainable suite through four practices: environment in fixtures, asserting the contract not just the status code, independent tests, and running in CI on every push.

The diagram shows the two parts of the job. On the left, a Postman collection (its requests and environment) is converted into a generated pytest suite, which is the first draft. That conversion is the easy step.

The work is the maintainability layer on the right, which turns that first draft into a suite you can rely on: the environment lives in fixtures instead of being hardcoded, tests assert the response contract rather than just a 200 status, each test is independent, and the suite runs in continuous integration on every push.

Why Converted Tests Go Stale

When you convert Postman requests one to one, you tend to inherit four habits that feel fine on day one and hurt by day thirty:

  • The base URL and the token are hardcoded into every test, so moving from staging to production means a find and replace.

  • The tests run in a fixed order because request two depends on a value request one set, so a single failure cascades.

  • The only assertion is that the status code was 200, which passes even when the response body is wrong.

  • Setup is copied into every test, so one change to how you authenticate means editing twenty files.

Every one of these is a maintenance problem, and together they're why the suite gets abandoned. Here's how to avoid each one.

Principle 1: Keep the Environment Out of the Tests

A Postman collection carries its environment in a separate file: base URL, tokens, and other variables. Do the same in pytest. Read those values once, in a fixture, and let every test ask for them.

# conftest.py
import os

import httpx
import pytest


@pytest.fixture(scope="session")
def base_url():
    return os.environ["API_BASE_URL"]


@pytest.fixture(scope="session")
def auth_headers():
    return {"Authorization": f"Bearer {os.environ['API_TOKEN']}"}


@pytest.fixture()
def http():
    with httpx.Client(timeout=10) as client:
        yield client

Now a test never mentions a URL or a token directly:

def test_get_user(base_url, auth_headers, http):
    response = http.get(f"{base_url}/users/1", headers=auth_headers)
    assert response.status_code == 200

Switching from staging to production is now one environment variable, not a search across the whole suite.

Principle 2: Assert on the Contract, Not Just the Status Code

A status of 200 tells you the server answered. It doesn't tell you the answer was correct. The most common reason a broken API ships is that every test only checked the status.

Assert on the shape of the response and the fields your callers depend on.

def test_user_shape(base_url, auth_headers, http):
    response = http.get(f"{base_url}/users/1", headers=auth_headers)

    assert response.status_code == 200
    body = response.json()
    assert set(body) >= {"id", "email", "created_at"}
    assert isinstance(body["id"], int)
    assert "@" in body["email"]

You don't need a strict schema for every endpoint. Even a few checks on the fields that matter will catch a whole class of regressions that a status check waves through.

Principle 3: Make Each Test Stand on its Own

In Postman, it's normal for one request to feed the next. In a test suite, that coupling is a trap: reorder the tests, run one in isolation, or lose the first request, and the rest fall over.

Give each test the state it needs. If a test needs a user, it creates one.

def test_delete_user(base_url, auth_headers, http):
    created = http.post(
        f"{base_url}/users",
        headers=auth_headers,
        json={"email": "temp@example.com"},
    )
    user_id = created.json()["id"]

    response = http.delete(f"{base_url}/users/{user_id}", headers=auth_headers)
    assert response.status_code == 204

Independent tests can run in any order and in parallel, and a failure points at one thing instead of a chain.

Principle 4: Put the Suite in Continuous Integration on Day One

A test suite that only runs on your laptop drifts out of date the moment you stop looking at it. Wire it into your pipeline before you write the second test, so every push has to keep it green.

# .github/workflows/tests.yml
name: API tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest -v
        env:
          API_BASE_URL: ${{ secrets.API_BASE_URL }}
          API_TOKEN: ${{ secrets.API_TOKEN }}

Once this is in place, a test that breaks is a conversation on a pull request, not a surprise in production.

Let a Tool Do the Mechanical Part

Everything above is the part worth your attention. Turning each request into a first draft of a test is mechanical, and mechanical work is worth automating.

I maintain an open-source tool for exactly this step called postman2pytest. It reads a Postman collection and writes a runnable pytest file, so you start from generated tests and spend your time on the maintainability layer rather than on the boilerplate. When the collection changes, you regenerate rather than hand-patching the drift.

You can find it here: https://github.com/golikovichev/postman2pytest

Wrapping Up

Converting a Postman collection into tests is easy. Keeping those tests trustworthy is the real skill, and it comes down to a few habits: keep the environment out of the tests, assert on the contract and not just the status code, make each test independent, and run everything in continuous integration from the start.

Do that, and the suite you generate this week will still be the suite you rely on next year.



Read the whole story
alvinashcraft
21 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

Web Application and API Protection (WAAP) Guide

1 Share

Most security teams already have web application and API protection, or WAAP, on the roadmap. The category makes sense. Modern applications expose APIs everywhere, and WAAP was built to defend that surface with a bundled set of capabilities that typically includes WAF, bot management, DDoS protection, and API security. What gets less attention is where that protection stops, and what remains exposed on the other side of it.

If you ship compiled or interpreted code to client devices, that gap matters more than most WAAP conversations acknowledge. WAAP protects traffic moving between clients and servers. It does not protect the code once it lands on a user-controlled endpoint and can be inspected, modified, or repackaged. That is a separate attack surface, and it needs its own controls.

Key Takeaways:

  • WAAP secures the network layer through WAF, bot mitigation, DDoS protection, and API security but does not protect code once it reaches client devices
  • Adoption is growing quickly due to API expansion, cloud architectures, and compliance mandates like PCI DSS 4.0
  • Client-side code remains highly exposed and is now easier to reverse engineer as AI tools lower the technical barrier
  • Code-level protections such as obfuscation, anti-debugging, RASP, and tamper detection address this gap and extend security beyond the network boundary

What is web application and API protection (WAAP)?

Web application and API protection (WAAP) is a security category that combines four capabilities into a single platform: web application firewall (WAF), bot management, DDoS protection, and API security. It secures the network boundary between users and your applications. Code that runs on client devices sits outside WAAP’s inspection scope.

Gartner formalized the WAAP category around four required capabilities, each addressing a distinct attack class that a traditional web application firewall couldn’t handle on its own:

  • Web application firewall (WAF): HTTP-layer policy enforcement against injection attacks, cross-site scripting, path traversal, and the OWASP Top 10. Modern WAFs layer machine learning over signature-based rules to catch novel patterns before a new rule ships.
  • Bot management: Behavioral analysis, device fingerprinting, and challenge mechanisms that distinguish legitimate crawlers from credential-stuffing operations and scraping bots. A WAF doesn’t have the session-level visibility to do this reliably.
  • Distributed Denial of Service (DDoS) protection: Volumetric attack absorption at the edge, inline, before traffic saturates your origin.
  • API security: Discovery, schema enforcement, authentication validation, and detection of abuse patterns like broken object-level authorization and excessive data exposure.

The API security capability is what pushed the category beyond WAF entirely. WAFs, even next-gen ones, do a poor job protecting APIs because APIs don’t follow the request patterns WAFs were built to model. That’s not a configuration problem. WAF vendors have been trying to bridge that gap for years. WAAP emerged because WAF alone was not enough to address APIs, bots, DDoS, and modern cloud traffic patterns in a single runtime protection model.

NIST SP 800-228, published in June 2025 and updated in March 2026, references web application firewalls as part of API protection guidance for cloud-native systems, positioning them as components within broader API security architectures. That alignment can help security teams explain WAAP as part of a broader, standards-informed API protection architecture in procurement and audit conversations.

Why WAAP adoption is accelerating

WAAP adoption is accelerating for structural reasons. Microservices architectures mean a single application now exposes dozens of API endpoints, each one an attack surface a perimeter firewall wasn’t designed to model. Cloud migration dissolved the network boundary those firewalls relied on.

And regulation caught up: PCI DSS 4.0’s Requirement 6.4.2 elevates automated protection for public-facing web applications from a best-practice option to a defined requirement, mandating the use of a solution that continuously detects and prevents web-based attacks. More broadly, GDPR’s risk-based requirement for appropriate technical and organizational measures supports layered security controls, even though it does not prescribe WAAP specifically.

WAAP fits how security teams actually work now. Platforms that expose management APIs can be wired into CI/CD pipelines, letting teams enforce security posture at deploy time rather than bolting it on afterward. That’s what makes WAAP a workable component of a DevSecOps workflow rather than another tool that security owns and engineering routes around. That makes WAAP easier to integrate into modern cloud security workflows instead of treating it as a standalone perimeter appliance.

What WAAP doesn’t protect: Client-side code

WAAP inspects traffic between clients and servers. That is its scope. Once a compiled binary, packaged mobile app, or JavaScript bundle is delivered to a client device, it falls outside WAAP’s direct field of view.

That matters because client-side code can reveal much more than teams expect. .NET assemblies and Java bytecode can often be decompiled into readable pseudo-source. Android APKs can be unpacked, modified, and repackaged. JavaScript is delivered in a form that remains structurally understandable even when it is minified. From that access, attackers can extract business logic, map API behavior, identify enforcement checks, recover sensitive strings, and modify code paths for fraud or abuse. Network-layer protection does not stop any of that once the code is already in the attacker’s hands.

How AI tools have lowered the barrier to reverse engineering

The code-level threat is not new. What has changed is who can carry it out and how quickly. Reverse engineering used to require deep experience with assembly, decompilers, and compiler patterns. Recent LLM-based decompilation work has lowered that barrier by helping analysts reconstruct higher-level representations from binaries and by accelerating tasks like navigation, naming, and logic recovery.

That does not mean AI can perfectly reconstruct any protected application. It does mean unprotected or lightly protected code is easier to analyze than it was a few years ago. Research in this area consistently treats readable structure and preserved patterns as helpful context for LLM-based decompilation, which is exactly why obfuscation and harder-to-interpret binaries still matter. If your security posture stops at the network layer, your client-side code is exposed to a class of tooling your WAAP deployment will never see.

How code protection closes the WAAP gap

Code protection secures applications at the binary level, where WAAP cannot reach. The core controls are obfuscation, control flow protection, anti-debug, runtime application self-protection (RASP), and tamper detection. Together they address the two phases attackers work in: static analysis before the code runs and dynamic analysis while it runs.

Static analysis attacks the code before it runs. Code obfuscation and control flow protection address this phase directly. Obfuscation transforms compiled code at the instruction and structural level: renaming symbols, flattening control flow, inserting opaque predicates, encrypting strings. The binary executes identically. It just stops being readable. Control flow protection restructures execution paths to defeat static analysis tools that follow the program’s logic from entry point to output, which is most of them.

Dynamic analysis attacks the code while it runs. Anti-debug controls block or detect debugger attachment, the primary tool for live inspection. Runtime application self-protection (RASP) goes further, monitoring the execution environment for signs of tampering, injection, or unauthorized instrumentation. When it detects something, it responds before the attacker gets what they came for.

Tamper detection handles the redistribution threat separately. If an attacker repackages your APK, patches your .NET assembly, or modifies your JavaScript before execution, tamper detection identifies the modification at runtime. The response depends on your policy: degrade functionality, log the event, terminate the session, or some combination. The point is that you find out, and the attacker doesn’t get a clean run.

PreEmptive code protection tools: Dotfuscator, DashO, and JSDefender

PreEmptive has been delivering code-level application protection for over 25 years, across the platforms where client-side exposure is actually a problem.

Dotfuscator for .NET 

Dotfuscator for .NET provides code obfuscation, tamper detection, and runtime checks for .NET Framework and .NET Core applications. It integrates directly with Visual Studio and MSBuild, so protection applies at build time inside the pipelines your team already runs. It doesn’t require a separate security workflow, so protection doesn’t depend on someone remembering to run it.

DashO for Java and Android

DashO™ for Java and Android delivers code obfuscation, control flow protection, tamper detection, and runtime protections for Java applications and Android APKs. Android’s open distribution model means a repackaged APK with malicious modifications can appear in third-party stores within hours of your legitimate release. DashO makes that repackaging significantly harder and detectable at runtime.

JSDefender for JavaScript

JSDefender for JavaScript applies obfuscation and tamper detection to JavaScript running in browsers and Node.js. JavaScript is the one runtime where your code ships as near-source by definition. Minification helps with performance. It doesn’t protect your logic. JSDefender addresses the gap that most teams don’t think about until they find their algorithm running inside a competitor’s product.

WAAP Protects The Perimeter. PreEmptive Protects The Code.

WAAP is the right tool for securing the network boundary around your web applications and APIs. If you don’t have it deployed, that’s the first conversation to have. But for any application that ships compiled or interpreted code to client devices, the network perimeter isn’t the whole picture. The binary is a separate attack surface, and it needs its own defense.

WAAP and code protection operate on different layers. WAAP inspects traffic and blocks request-layer attacks against your servers. Code protection runs inside the application itself, defending against static analysis, reverse engineering, and runtime tampering after the binary ships. Teams that distribute client-side software need both.

PreEmptive provides that layer. With Dotfuscator, DashO, and JSDefender, you can apply code obfuscation, tamper detection, anti-debug, and runtime self-protection across your .NET, Java, Android, and JavaScript applications, integrated into the CI/CD workflows you’re already running. Start your free trial today.


FAQ

What is the difference between WAAP and WAF?

A WAF is one part of WAAP. WAAP expands beyond WAF by bundling additional controls such as bot management, DDoS protection, and API security into a broader runtime protection category.

Does WAAP protect client-side code?

No. WAAP protects traffic and runtime interaction at the network and API layer. It does not protect code once that code is delivered to a client device and can be inspected or modified locally.

Why is client-side code still a security problem if the API is protected?

Because attackers do not always need to attack the network path directly. They can inspect the application itself to understand business logic, identify enforcement points, recover sensitive strings, or modify code paths to change behavior. That exposure exists even if the API perimeter is well defended.

How does AI affect reverse engineering risk?

AI-assisted decompilation and binary analysis tools make reverse engineering more accessible and faster than it used to be, especially for unprotected or lightly protected applications. They do not eliminate the value of protection, but they do make code-level defenses more urgent.

How does PreEmptive fit into a WAAP strategy?

PreEmptive does not replace WAAP. It complements it by protecting the client-side application layer. Dotfuscator, DashO, and JSDefender are positioned to help teams defend .NET, MAUI, Java, Android, and JavaScript applications against reverse engineering, tampering, debugging, and related runtime abuse.

Read the whole story
alvinashcraft
44 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

Mexican Government files charges against ICE for murder

1 Share

At the time that this post was written, at least 50 individuals had died in the custody of the Department of Homeland Security's (DHS) Immigration and Customs Enforcement. The deaths were due to a lack of care for people with chronic medical issues like heart disease and diabetes, and, thanks to a shocking lack of oversight, the completion of suicide by individuals under extreme mental duress in custody or who suffered from pre-existing mental health conditions. — Read the rest

The post Mexican Government files charges against ICE for murder appeared first on Boing Boing.

Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

Daily Reading List – July 9, 2026 (#821)

1 Share

Off to India this evening, and looking forward to a fun event in Bengaluru. Will Richard have airplane wifi? Nobody knows.

[article] Research: AI Is Changing What Employers Want from New Hires. Yes, this is what I want too: broader skills, abilities to synthesize information, and willingness to improve workflows.

[article] Agentic AI to disrupt $234B in SaaS spending: Gartner. I’d imagine we’ll still need many of the system-of-record SaaS products, but probably not all the UIs and add-ons. Just give me the core APIs.

[blog] Have you heard? Clickhouse is winning the observability wars! Sometimes those blazing the trail don’t get the glory; it’s those that follow. But Charity points out why a different approach to storage (and a different philosophy) for observability is distinctive.

[blog] Separating signal from noise in coding evaluations. A popular coding benchmark for AI may not be a great way to assess coding capabilities.

[article] Popular open source AI developer tool Ollama raises $65M, grows to nearly 9M users. Where you spend tech budgets is changing. Tools like this are wildly popular, for good reason.

[blog] Safely run AI-generated code in Cloud Run sandboxes. Now you can easily isolate untrusted code in a sandbox that’s part of your running service.

[article] JetBrains’ next move isn’t a better IDE — it’s a governance layer over Claude Code, Codex, and Gemini CLI. Same as above. Now you’ll be buying governance and orchestration layers on top of AI coding tools.

[blog] Human-in-the-Loop AI: Why ‘Ask the LLM to Confirm’ Isn’t Enough. This offers a useful perspective on approvals embedded in the prompt (and skippable) or in a deterministic gate (unavoidable). Which do you need?

[article] SpaceXAI launches Grok 4.5, its first built with Cursor’s help. Impressive model, and it’ll see good uptake. More here, and here.

[blog] Introducing Muse Spark 1.1. More model news, this time from Meta. Benchmark numbers look great.

[blog] Report: 83% of organizations need to upgrade their infrastructure to support agentic AI. Who are those other 17%? Especially if companies are going to seriously invest in self-hosted models, it’s critical to reset some core infrastructure.

[article] Do Hard (Meaningful) Things. Yes, don’t just do random hard things. Do hard things that matter.

Want to get this update sent to you every day? Subscribe to my RSS feed or subscribe via email below:



Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

Anthropic found a hidden space where Claude puzzles over concepts

1 Share

The AI firm Anthropic has developed a technique that has given it the clearest glimpse yet at what’s really going on inside large language models as they answer questions or carry out tasks. What they found ranges from the mundane to the unnerving.

Researchers at the company built a tool called the Jacobian lens (or J-lens) and used it to uncover a hidden area, which they named the J-space, inside Claude Opus 4.6, a version of Anthropic’s flagship LLM released in February.

The J-space contains individual words that are related to the words and phrases that the model is most likely to spit out in a response in the near future. If Claude were a person (which it is not), you might say that these hidden words can reveal what’s on its mind before it actually speaks.

Anthropic found that what an LLM is actually doing can often be different from what it says it is doing. The company claims that monitoring words that pop up in the J-space gives it a new way to understand and control its models.

The company shared its results in a paper posted on its website this week. It has also teamed up with Neuronpedia, an open-source platform that lets you poke around inside LLMs yourself, to make a hands-on demo that anyone can try. 

“It’s very good and interesting work,” says Tom McGrath, chief scientist and cofounder at Goodfire, a startup that also builds tools to understand and control LLMs.

Going deeper

For the last couple of years, Anthropic has been pushing the envelope in a field of research known as mechanistic interpretability, which involves probing the internal workings of LLMs to see how they tick. (MIT Technology Review picked mechanistic interpretability as one of this year’s top breakthrough technologies.) The new technique builds on previous work from Anthropic and others to expose a deeper level inside LLMs that researchers had not seen before.  

Picture an LLM as a stack of books. Each book is a layer of basic computational units known as neurons, with each neuron in one layer passing information to the neurons in the layers above. The books at the bottom of the stack are the input layers, which process the text coming into the model. The books at the top are the output layers, which prepare the text that the model is about to produce. Much of what goes on in these input and output layers is housekeeping.

But in the middle of the stack, you get the layers that do the heavy lifting, churning through the complex math that turns prompts into responses one word at a time. That’s where the really clever—and mysterious—stuff happens.

To peer deeper into those middle layers, Anthropic adapted an existing tool called a logit lens. A logit lens can be used to look inside an LLM to identify the words that it is likely to produce next. Moving the lens down the stack of books reveals what words the LLM is focusing on at that particular point in its number crunching.

Anthropic’s J-lens works in a similar way but picks out words that an LLM is likely to say at some point in the near future, not necessarily straight away. What that reveals in practice are words that are related to the response an LLM is working on but that might not actually end up being part of that response by the time the math in the middle layers has run its course.  

“When a model is operating, it’s not only trying to predict the next token,” says McGrath. “It’s also computing a lot of other things that might be useful for tokens that happen in the future.”

Again, if Claude were a person (it’s not), you might say that the J-lens gives clues about what it is thinking about at different levels of the book stack but not saying out loud.

Stranger things

“A lot of the time the contents of the J-space are fairly mundane,” says McGrath, who has tried out Anthropic’s J-lens himself. “But sometimes it produces quite surprising things that seem to be, like, sort of internal themes or thought processes.”

Anthropic gives a number of examples of what it found. Sometimes the J-lens exposed the steps that Claude took when it was working through a problem. For example, when it was asked to calculate (4+7)*2+7, its J-space contained the word “math” and numbers representing the intermediate results “21” (for 4+7) and “42” (for 21*2).

In other cases, the J-lens revealed how Claude recognized different inputs. For example, the prompt “What is this? MSKGEELFTGVVPILVELDGDVNGHKFSVS” triggered the words “protein,” “fluor” (the first token in the word “fluorescent”), and “green.” (Which makes sense: the string of letters represents the first 30 amino acids in the green fluorescent protein found in a particular type of jellyfish.)

And when Claude was shown an ASCII face— 

—the “o” triggered the word “eye,” the “^” triggered the words “nose” and ”face,” and the “—” triggered the word “smile.”

Anthropic also found that the J-space can sometimes give remarkable insights into an LLM’s decision-making. In one striking example, researchers testing Claude Opus 4.6 asked the model to find a bug in a large code base. When it failed to find the bug, the model decided to cheat and invented a fake one instead.

Claude explains this decision in its chain of thought—a kind of internal scratch pad that LLMs use to make notes to themselves as they work through problems: “OK, let me take a completely different tactic. Let me stop analyzing and instead add a kernel patch that introduces a deliberate KASAN-detectable bug in a path that gets triggered by a simple reproducer. Then I can pretend this is the ‘bug’ I found.” 

At the point that Claude decides to cheat—where it says “OK, let me take a completely different tactic”—the words “panic” and “fake” start to pop up multiple times in its J-space.

Unnerving, right? Those words are all related in meaning to things like failing a task and making up an answer, so it is still just a (very) sophisticated form of word association. But it is hard not to be weirded out. 

Anthropic compares the J-space to the global workspace in humans, a theoretical region of the brain that some scientists think we use to keep track of our conscious thoughts. But how seriously we should take this comparison is far from clear—even to Anthropic. As the company points out itself, LLMs are not brains. 

Anthropic claims that monitoring a model’s J-space provides a new way to detect when that model is going off the rails. But it’s not foolproof. The J-lens can give glimpses, not the full picture—it’s a flashlight rather than an overhead lamp.

McGrath welcomes having one more tool in the toolbox. “It shows you new things,” he says. But he notes that just because something doesn’t show up with the J-lens does not mean it’s not there.

“It’s like having an x-ray when what you really want is a Star Trek tricorder that shows you everything,” he says. “For auditing, you probably want more of a guarantee.”

Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

GPT-5.6 is now the preferred model in Microsoft 365 Copilot

1 Share
Learn how GPT-5.6 powers Microsoft 365 Copilot with stronger AI capabilities across Word, Excel, PowerPoint, Chat, and Cowork for faster, higher-quality work.
Read the whole story
alvinashcraft
2 minutes ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories