Why data quality is a product problem, not an engineering problem.
Over two decades, I’ve found myself solving the same problem over and over again. People think they’re asking for better data, but what they really want is data they can trust. They’re not the same.
I’ve spent most of my career building data products. A startup in 2001 called EmployeeService.com. A big law firm. Xbox. Windows. Azure. Disney Streaming. Different companies. Different technologies. Different industries.
For most of my career, I thought I understood what that meant. Classify & explain the garbage, let your users decide what they want to do with it.
Then I went to Disney.
My product at Disney sounds almost insultingly simple. Who watched what, for how long? That’s it. One sentence. It’s a sentence I wrote, actually, to help keep us all focused on what matters the most while we added new features to the product.
To the business, this is a metric called Total Hours Watched that is an input to dozens of other metrics. Add up the minutes people spend watching Disney+, Hulu, or ESPN and you’ve got one of the most important metrics in the company.
Except browsers get throttled in the background. Connected TVs go to sleep. Heartbeats arrive late. Memory leaks explode or delete values. Regression bugs happen.
Telemetry doesn’t always mean what people think it means. And when you’re processing tens of billions of playback events every day, impossible values aren’t edge cases. They’re guaranteed.
The obvious question is:
- How do we eliminate bad data?
That’s never been how I think about it. Long before Disney, I’d learned that “bad” data is not only inevitable, but that what is “bad” to one person might be exactly the signal someone else wants to find. Being part of the Windows 10 launch really hammered it in. All of us were working with a brand-new telemetry system with two competing responsibilities.
- We needed accurate measurements.
- We also needed breadcrumbs to help engineers find and fix bugs.
Those goals aren’t always aligned. My job as a product manager isn’t to eliminate uncertainty. My job is to help different users make good decisions with imperfect data.
That’s why one of the first features I added at Disney wasn’t another heuristic. It was a trust mechanism.
Every playback session carried a flag indicating whether I believed it posed a data quality risk—was the amount of time the person watched a title longer than the wall clock?
Notice what I didn’t do. I didn’t delete the data, because “bad” depends entirely on who’s asking.
A content analyst measuring Total Hours Watched wants impossible playback sessions excluded immediately.
An engineer debugging a media player wants those exact same sessions preserved.
A recommendations engineering team doesn’t care if you watched ten minutes or ten years, just that you watched more than a minute or two.
To one user, they’re contamination. To another, they’re evidence. One playback session. Multiple truths.
That wasn’t a technical decision. It was a product decision.
For a long time, I thought that was the lesson: Build trust by making uncertainty visible. Don’t promise perfect data, just help users understand when they should trust it and when they shouldn’t.
I still believe that, but Disney taught me something I hadn’t learned in the previous twenty years. I assumed that if I built the right product, documented it well, and made the reasoning transparent, people would naturally adopt it and my trust mechanism. That assumption came almost entirely from my years at Microsoft.
I was wrong.
Microsoft taught me that transparency creates trust. I assumed that was universal. The product wasn’t entering Microsoft, though. It was entering Disney, and cultures adopt products differently. I spent far too much energy trying to convince people to think about what I thought was the right strategy for building/adopting a data product: I argued for end-user documentation. I argued for user empathy. I argued that data quality should be treated as a product capability rather than a bug to squash.
It’s not that I didn’t have a few wins here and there. I wasn’t failing because the trust mechanism was wrong. I was failing because I assumed trust spread the same way in every culture.
My mistake was assuming the mechanisms that created trust at Microsoft would create trust everywhere else.
They didn’t.
I should have spent more time understanding Disney’s culture and then designing product mechanisms that earned trust within it. That realization changed how I think about product management.
User empathy isn’t just for customers and users. For people.
It’s also for organizations.
Every culture decides differently what it means to trust a product. And in data, since trust is the product, then product adoption has to be designed with the same care as the product itself.
That was the lesson I almost missed.
Three years after introducing that trust mechanism, we still have incidents. We had one last week: one playback session with a ridiculous, unbelievable duration resulted in a content partner not believing us when they saw user engagement on different title jump 4x. That jump was real, and it was created by improvements to the Disney+ app. But because one session was wrong, they didn’t trust any of the data.
Time to learn more about my users again.
Leave a Reply