|
Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
|
Every data team eventually hits the same wall. A number on a dashboard gets questioned, nobody can prove who is right, and trust quietly leaks out of the whole system. Good data quality and observability is what keeps that from happening, and over six posts I worked through how to build it, one hard part at a time.
Read on its own, each post solves one problem. Read in order, they follow the path a data team walks as it matures: understand the problem, assess the risk, define the fix, build the controls, monitor for what slips through, and respond calmly when something breaks anyway.
Here is the whole set, with the single idea behind each one and the one thing to do about it.
“The dashboard is wrong” is not a diagnosis. It is a feeling, and it hides three different problems: a value that is wrong (quality), a value that is correct but late (reliability), and a change upstream that no rule was watching for (observability). Each one has a different owner and a different fix.
So when a number is disputed, ask three questions before you open a query. Is a value wrong or missing? That is quality. Are the values fine but late? That is reliability. Did something change that no rule saw? That is observability. The label is the fastest way to stop three teams from investigating the same ghost.
The takeaway: name the problem before you touch it, and the fix is already half done.
Read the full post: Data Quality, Data Reliability, and Data Observability: Telling the Three Apart
Profile a messy table and the errors with the biggest counts grab all the attention. They are almost never the ones that matter. Forty duplicate invoices feeding the revenue number will hurt you far more than twelve thousand harmless trailing spaces.
The fix is two columns, not a new tool. Next to every finding, write who feels it downstream and how often it happens, then sort by those instead of by count. Impact times likelihood is the whole model, and it quietly floats your real priorities to the top while the loud, harmless piles sink.
The takeaway: the most dangerous issue is rarely the one with the biggest number.
Read the full post: Count Is Not Risk: Ranking Data Issues by What They Actually Cost
“Just make the data clean” cannot be tested, so it can never be met. A usable requirement has three parts: a rule a check can answer yes or no, a threshold that says how much deviation is acceptable, and a named owner accountable for the definition. That is the difference between a wish and a data contract.
Say each expectation out loud and hold it against three questions. Can I write it as a check that returns a number? Have I decided what number passes? Can I name the person who owns it? Three yeses and you have a requirement. A shrug on any of them and you have a preference.
The takeaway: if it cannot fail a test, it is not a requirement.
Read the full post: “Clean Data” Is Not a Requirement: Writing Rules People Can Act On
You can have the right check and still catch the problem too late. A validation that runs after publication is a smoke detector wired to go off the morning after the fire.
There are only three places a control can live, and each catches something different. Ingestion stops bad rows at the front door. Transformation catches the errors your own joins and calculations create. The release gate is the last block before anyone downstream reads the data. Put the control at the earliest point that can still stop the problem, decide whether it blocks or only warns, and quarantine what fails so the good data keeps moving.
The takeaway: a check that cannot stop the flow is just a log line.
Read the full post: Where Should a Data Quality Check Live? Gates, Controls, and Quarantine
Row-level rules only catch the failures you already imagined. They cannot see a feed that arrived stale, a row count that collapsed, a column that changed type, or a value mix that quietly shifted. Those four signals, freshness, volume, schema, and distribution, are where the surprises live.
You cannot spot abnormal without first defining normal. For your most important table, write down four things: when it should arrive, how many rows is normal, what its schema should be, and the usual mix of values in the column you care about. That is a baseline, and a baseline is most of monitoring. Then alert on the size of the change, not the fact of it, so a two percent wobble stays quiet and a sixty percent drop wakes someone.
The takeaway: rules protect you from the mistakes you imagined, monitoring protects you from the ones you did not.
Read the full post: It Passed Every Test and Still Broke: Four Signals Validation Never Catches
A wrong number in a live meeting makes everyone want to start changing things immediately. That instinct is how a one-hour fix becomes a three-day cleanup.
Triage first, because you cannot size a response until you know the blast radius. Contain before you fix, because cleaning while the tap runs is endless. Trace to the source instead of patching the number, or it comes back next week. Verify the fix, tell the people who trusted the bad value, and close with a short blameless review so the same break does not return.
The takeaway: changing the data is the middle of an incident, not the end.
Read the full post: You Just Found Bad Data in Production. Now What?
Read as a set, they build on each other, because you cannot rank a problem you have not named, define a fix for one you have not ranked, or control one you have not defined. Understanding comes first, response comes last, and monitoring is the quiet layer that keeps the whole thing honest between incidents.
Here is the entire guide in one line each, if you ever need to find the right idea fast.
| When you hit this | Do this |
|---|---|
| The dashboard is “wrong” | Name it: quality, reliability, or observability |
| A long list of data errors | Rank by impact and likelihood, not by count |
| “Just make it clean” | Write a rule, a threshold, and a named owner |
| A check that fires too late | Put controls at ingestion, transformation, and the gate |
| Every test passed, it still broke | Monitor freshness, volume, schema, and distribution |
| A wrong number in production | Triage, contain, trace, verify, tell, review |
Start at the top, or jump to whichever problem is on fire today. Each post also carries a link to its companion Pluralsight course if you would rather see the idea built hands-on.
Data quality and observability is not one skill but six connected moves. Name the problem before you chase it, because “the dashboard is wrong” hides three different failures: a wrong value, a late value, and a silent change. Rank issues by impact and likelihood, not by row count. Turn vague requests like “make it clean” into a rule, a threshold, and an owner. Place controls where they can stop bad data, not merely log it. Monitor the signals row-level tests miss: freshness, volume, schema, and distribution. And when something breaks anyway, respond calmly rather than fast. Read the six posts in order and you have a working playbook for trustworthy data, from the first disputed number to the last blameless review.
Reference: Pinal Dave (https://blog.sqlauthority.com/), X
First appeared on Data Quality and Observability: A Field Guide in Six Parts
Get caught up on the latest technology and startup news from the past week. Here are the most popular stories on GeekWire for the week of July 12, 2026.
Sign up to receive these updates every Sunday in your inbox by subscribing to our GeekWire Weekly email newsletter.
Stardew Valley, the blockbuster farming sim from solo Seattle developer Eric “ConcernedApe” Barone, is coming to Magic: The Gathering via a three-part Secret Lair drop from Renton-based Wizards of the Coast — with original pixel art from Barone himself. … Read More
F5 names former Amazon executive as its chief people officer; Rudra Mitra departs as a Microsoft CVP after more than 27 years; and Qualtrics adds leaders. … Read More
Dave Brown is leaving Amazon Web Services after nearly 19 years, departing at the end of July for a new role outside the company. … Read More
The AI-generated video is an interesting study in how technology that’s very much being built and hyped in Seattle can be used to illustrate what the city sort of looked like more than three decades ago. … Read More
Apptio co-founders Sunny Gupta and Kurt Shintaffer launched Thira, a Bellevue-based enterprise AI startup, with $21 million in seed funding led by Madrona. … Read More
JPMorgan Chase is building out a new AI software infrastructure team, with a major presence in Seattle, focused on running AI across its data centers and outside providers in a way that controls costs, protects its intellectual property, and avoids tying its fortunes to any one vendor. … Read More
Rina Hahn leaves role as Remitly’s CMO; Temporal promotes Preeti Somal to EVP amid reorganization; and Veeam and Qualtrics make leadership changes. … Read More
Dave Brown, the senior AWS executive whose departure Amazon announced this week, is joining Meta to work on its data center build-out, according to a Wall Street Journal report. … Read More
General Fusion’s stock is trading up after it became the first fusion energy company to go public on a major exchange, debuting Monday on Nasdaq. … Read More
Seattle-area startups raised $2.7 billion in the first half of 2026, down 40% from a year ago, even as U.S. … Read More
Elizabeth Stone is the Chief Product and Technology Officer (CPTO) at Netflix, where she oversees Engineering, Product, and Design. Since her first appearance on the podcast two years ago—which remained my second-most-popular episode for more than a year—she has expanded her role to lead product, in addition to engineering. Before Netflix, Elizabeth was VP of Science at Lyft, Chief Operating Officer at Nuna, an economist at Analysis Group, and a trader at Merrill Lynch.
In our in-depth conversation, we discuss:
1. Why “systems thinking” is now the most important skill she looks for
2. How to manage the flood of AI-generated output without losing quality or signal
3. How Netflix thinks about AI fluency as a universal expectation rather than a level-specific skill
4. What “excellence as an operating system” means
—
Brought to you by:
WorkOS—Make your app enterprise-ready, with SSO, SCIM, RBAC, and more: https://workos.com/lenny
Mercury—Radically different banking, now with Command: https://mercury.com/command?utm_source=lennys&utm_medium=sponsored_newsletter&utm_campaign=26q3_brand_campaign
—
Episode transcript: https://www.lennysnewsletter.com/p/netflix-cpto-on-ai-and-the-future
—
Archive of all Lenny's Podcast transcripts: https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0
—
Where to find Elizabeth Stone:
• LinkedIn: https://www.linkedin.com/in/elizabeth-stone-608a754
—
Where to find Lenny:
• Newsletter: https://www.lennysnewsletter.com
• X: https://twitter.com/lennysan
• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/
—
In this episode, we cover:
(00:00) Introduction
(02:25) AI and role confusion: the storming phase before the forming phase
(07:36) How roles have changed in the past two and a half years
(11:55) Will functions survive? The case for craft specialism
(13:26) What Netflix is hiring more of—and less of
(17:22) Why systems thinking is the rising skill across every function
(20:20) Is the design process dead?
(22:08) Skills trending down
(28:33) AI fluency and Netflix’s career ladder overlay
(31:00) AI use cases beyond coding
(35:12) Netflix’s AI history
(38:36) Excellence as an operating system
(41:11) The pillars of the excellence OS
(46:41) The keeper’s test—and why it’s mostly a positive conversation
(50:21) Attracting top talent in the age of frontier AI labs
(52:54) Junior talent, craft mastery, and the mentorship question
(56:25) Where engineering goes in 5 to 10 years
(59:45) The future of entertainment: beyond film and TV
(1:02:18) AI in Hollywood: Netflix’s creator-enablement position
(1:06:15) Lightning round and final thoughts
—
Referenced:
• How Netflix builds a culture of excellence | Elizabeth Stone (CTO): https://www.lennysnewsletter.com/p/how-netflix-builds-a-culture-of-excellence
• Brian Chesky’s new playbook: https://www.lennysnewsletter.com/p/brian-cheskys-contrarian-approach
• The design process is dead. Here’s what’s replacing it. | Jenny Wen (head of design at Claude): https://www.lennysnewsletter.com/p/the-design-process-is-dead
• Claude Code: https://www.anthropic.com/product/claude-code
• Claude Cowork: https://www.anthropic.com/product/claude-cowork
• Netflix’s “Keeper Test” and Why You Need It | Lorne Rubis: https://www.highlights.lornerubis.com/2015/08/the-netflix-keeper-test-and-the-courage-to-take-it
• Innovation for Filmmaking, By Filmmakers: Why InterPositive Is Joining Netflix: https://about.netflix.com/en/news/why-interpositive-is-joining-netflix
• InterPositive: https://weareinterpositive.com
• Netflix Prize: https://en.wikipedia.org/wiki/Netflix_Prize
• Quarterback on Netflix: https://www.netflix.com/title/81482895
• The Bill Simmons Podcast on Netflix: https://www.netflix.com/title/82186214
• Spencer Pratt on Instagram: https://www.instagram.com/spencerpratt
• Salman Rushdie’s Substack: https://salmanrushdie.substack.com
• Remarkably Bright Creatures on Netflix: https://www.netflix.com/title/81911351
• Eight Sleep: https://www.eightsleep.com
• Tour de France: https://www.letour.fr/en
—
Recommended books:
• Thinking in Systems: https://www.amazon.com/Thinking-Systems-Donella-H-Meadows/dp/1603580557
• Into Thin Air: A Personal Account of the Mt. Everest Disaster: https://www.amazon.com/Into-Thin-Air-Personal-Disaster/dp/0385494785
• Liar’s Poker: https://www.amazon.com/Liars-Poker-Norton-Paperback-Michael/dp/039333869X
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.
—
Lenny may be an investor in the companies discussed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Log in
