I once chased a bug. A very special case of a flaky test. It was a mix of a race condition and memory allocation. A real doozy.
I had a test – a bunch, really – ready to catch it when it appeared. With logs, and dumps. A wide net was cast. And you can’t build a whole test suite without thinking about the conditions of when the bug appears and when it won’t. It’s a whole model I was keeping in my head.
Then the bug reared its ugly head. And it didn’t fit the model. Tests were changed. Model replaced. The bug reappeared.
And the cycle began again. And again. I was replacing models. I was really losing my grip on reality. Or how I interpreted it.
Those tests were traps. But flaky tests have the same effect on us. We feel that our tests – which are supposed to model real use – aren’t doing their job.
But sometimes it’s much easier to fall into the rabbit hole, without race conditions. Sometimes we plant the traps we fall into.
Sometimes we don’t even notice we plant them.
Our brains are magnificent. But when we write a test, we focus on it only. We build our model of how it works, what makes it fail, and we imagine how, at the right time, we’d break down the door, shine the flashlight on the culprit and say – “stop right there.”
That mental model is built on things we know. What will happen when the test runs. How the system will behave. But really, these are just assumptions. We don’t really know.
One of the assumptions is how the system state will be when the test starts running.
Here’s an example.
In our Bigger Better Bookstore app, we have books. So if you want to check the “Create Book” feature, you’d write something like this:
def test_create_and_get_book():
book_id = "book-123"
book_data = {
"unique_id": book_id,
"title": "The Lord Of The Rings: The Fellowship Of The Ring",
"author": "J. R. R. Tolkien",
"price": 12.99
}
response = requests.post(f"{BASE_URL}/books", json=book_data)
assert response.status_code == 201
assert response.json()["title"] == "The Lord Of The Rings"
response = requests.get(f"{BASE_URL}/books/{book_id}")
assert response.status_code == 200
assert response.json()["title"] == "The Lord Of The Rings"
Make a POST call to create it, then a GET to make sure it was created. Of course, our book IDs are unique – otherwise we won’t be able to tell all the versions of Lord of the Rings apart.
And the test passes!
If you run it once. Run it twice and it fails. The book ID is already taken. Nobody cleaned up the book created in the first run.
That’s a very simple case of State Pollution. The test changes the initial state of the system – in time for the next time.
Of course, once you understand what happened, you’ll find a fix. But what led us down the path of fragility?
The assumption that the system will always start out the same way. Maybe we thought we’d take care of the cleanup. Or someone else would. Or we didn’t think about it at all.
Which happens a lot. We can’t think of everything, all the time.
State Pollution is just one flavor of assumption. There are a lot more. Every flaky test is built on assumptions. Most of them are true most of the time. Except the ones that fail us.
The fix in this case? Clean up after yourself, or use a unique ID every run. But more importantly, how do we avoid it next time?
You need to ask yourself: when this runs again, is it starting always from the same point?
By the way, if you generate tests in bulk, the genie brings its own assumptions. Get ready for a surprise or ten.
On June 3rd, I’m running the next webinar: “Modern API Testing: Fighting Flakiness.”
We’ll go through a whole lineup of suspects — with live examples. Each one has a signature. Each one hides in a different blind spot.
I’ll talk about AI test generation and where it comes up with its own assumptions.
Plus — because I’m lazy and can’t trust myself, I built a tool. It reads my test code and tells me what was I thinking. Or implicitly assuming. I’ll show it too.
June 3rd, 3PM CET / 9AM EDT. Free.
Register Now
The API Testing Strategist Masterclass
Flakiness deserves a lot more than a webinar. A whole module in the Masterclass will teach you how to fight flakiness.
Seven weeks. Live. With me.
Twenty seats. Early bird is $599 until May 4th.
See The Masterclass
P.S. I never caught that bug. Probably moved to another app.
The post
The Flaky Test Files: The Case of the State Pollution first appeared on
TestinGil.
Log in
