Category: Product

How a machine wrote a 24,000-page data dictionary overnight(ish) — and why the page count is the least interesting thing about it.

---

A while ago I ran a job and then, for fun, pretty-formatted its output and counted the lines. A little over 1.2 million. Divide by fifty lines to a page and you get 24,396 pages.

Twenty-four thousand pages of data documentation. Generated deterministically. No human author.

That number is completely meaningless. It’s whitespace and brace-counting; I could double it by changing the indent from two spaces to four. It is also the kind of stat that makes a room go quiet. For a second. Until someone asks “is any of it real?“

The graveyard problem

Every large company that runs on data has the same dirty secret: its data catalog is a graveyard. Hundreds of thousands of columns across thousands of tables, and the “description” field is empty on almost all of them. The ones that aren’t empty can be worse — a stale sentence someone wrote in 2019 for a column that’s since changed meaning twice.

Nobody fills them in, because filling them in is nobody’s job (it’s mine, actually), and because the person who could write a good description for locationAsnWithOwner is the one person too busy to do it (you know who you are). The catalog becomes a place where documentation goes to not exist.

Here’s an idea! Let’s point a large language model at the problem! “DESCRIBE THIS COLUMN!” Problem solved.

It sort of works, and it’s sort of a disaster — because an LLM will confidently describe every column: the ones holding raw device identifiers, the cryptic double called avg_lpeel_v2 it should never summarize, the ones that are 99% null, and the ones a human already documented perfectly well. You don’t have a catalog anymore. You have a chatty intern who never says “I don’t know” and never defers to the expert in the room (you know who you are).

(Don’t believe me? Ask the internet what lpeel means. It told me, with total confidence, that it’s a peelable packaging film. Or a Korean chemical peel. Or a paragraph-writing framework. Or a stock ticker. It is none of these.)

Measure, then decide — but never in the same place

The system Claude & I vibe-coded takes a different shape, and the shape is the whole idea.

It splits into two kinds of thing: producers and a resolver.

Producers are specialists. One reads the physical layout of a table and knows which columns are partitions. One maintains a canonical dictionary of “primitive” concepts — the handful of identifiers and enums that recur across hundreds of tables, each defined once, by a human (me), authoritatively. One profiles the actual values in text columns and writes a plain-language paraphrase. One scans for anything that looks like personal or regulated data.

Here is the discipline that makes it work: producers only measure and flag. They never decide, and they never throw anything away. A producer that finds a column is 99% null writes down “99% null” and moves on. A producer that suspects a column is a personal identifier flags it and withholds its own guess — but records that it withheld, and why. Nothing low-quality is discarded; it’s retained, scored, and labeled. The producers are librarians, not editors.

And each producer can feed another: a sparse-column producer can run big table scans on the small subset of columns previously marked as 99% null. A locally run small-language-model producer can describe PII columns without exposing the values to Anthropic or OpenAI.

But all the deciding happens in exactly one place: the resolver.

One answer per column, and a receipt

The resolver’s job is to look at every candidate description for a column — the human dictionary definition, the native comment someone typed into the database years ago, the partition template, the AI paraphrase — and pick one winner, by a fixed order of precedence:

1. A curated canonical definition always wins. The whole point of defining account_id once is that it beats everything else, everywhere.
2. Then a human-authored comment. The machine loses to a person.
3. Then the deterministic partition description.
4. Then, and only then, the AI paraphrase. The model is the fallback, not the default.

Crucially, the resolver runs no model at all. It’s precedence and policy — pure, deterministic, boring, a query I can change over and over because all the intelligence has already been persisted, upstream, by the producers. The resolver is the referee, and referees shouldn’t improvise.

And every answer ships with breadcrumbs. Each resolved description carries its provenance: which source won, when it was generated, and — this is the part people underestimate — a publication decision. Every column is marked publish, suppressed, or omitted, each with a reason. A column the system deliberately declined to describe (because it’s a raw identifier, say) doesn’t silently vanish. It ships a blank description and the reason the blank is there. “I looked at this and chose not to describe it” is a first-class, auditable output — the exact thing the fluent intern can’t give you.

Because nothing is thrown away and every line is drawn in one place, moving a line is cheap. Want to publish lower-confidence descriptions next quarter? That’s a one-line change to a policy, re-run in minutes — not a re-measurement of a hundred thousand columns. The arbitrary decisions are reversible because the evidence underneath them is permanent.

I built it, but I don’t like it

I built a fleet of agentic tools to fill a web-based data catalog like Alation. I did it because that’s what the organization wants. They don’t want the context harness I’ve talked about in previous posts that has me doing ten points of pre-AI engineering work in an hour.

And that has given me a new perspective on AI. We’re all using it to automate the difficult part of our job, but we’re not changing our expectations of how our users will leverage AI. Governance wants to use AI to fill the catalog because they’ve been trying to get us PMs to fill it for years. I use AI to find bugs and outliers in the data because that’s been the hard, time-slog of my job for years. My engineers use AI to manage Jira and run boring tests because that’s the part of the job they don’t want to spend time on.

My gut: headless is the future. I’ve built a fleet of agentic tools that will fill a web UX that no-one will use in a year with descriptions too-high-level to be valuable.

But my gut instinct is based solely on the fact that I don’t want that. So who knows?

Oh — is any of it real? Mostly: the first run that spanned real calendar time went twelve days between passes, and a couple dozen tables got rebuilt or dropped underneath it, so a fraction of a percent of those pages now faithfully document ghosts. I know exactly which ones, because every answer carries a receipt — which was the entire point.

A party trick

The page count is a party trick. It’s going on a slide because a room that goes quiet is a room that’s listening. And if the room erupts instead, it’s erupting because I have their attention. They’re hooked. 24,396 is a very good number for getting a reaction.

Claude asked me this afternoon if I “want to keep rolling.”

I replied, “Keep going. No work for me here. All the context I’ve spent time building is really paying off.”

One of my product areas is Super Bowl LXI. I’m responsible for providing Disney+ and ESPN App viewership data that would otherwise be invisible to the company that tells everyone how many people watched the Super Bowl.

Two weeks ago, Claude and I developed a 12-point Superbowl viewership test plan for a partner team. They ultimately chose to cover a different set of scenarios. But I still wanted those twelve tests.

So at 2:40 this afternoon, I asked Claude to turn them into a living, snappy dashboard. Three hours later, Claude was still working through it, with almost no input from me. I started learning Ooh La La by the Faces when the clock hit 5pm.

And these aren’t simple data quality tests, like checking for missing values.

One tests whether our clients accurately track linear TV program boundaries. Think of the moment the ABC evening news ends on KOMO and Wheel of Fortune begins. Sounds straightforward, until you remember that viewers can pause, seek, and rewind live television.

I’ve spent months building context that explains how those behaviors work across three different data products and the six event types that power them. And now Claude can use that context to build something I wouldn’t have imagined asking for a few months ago.

It’s really cool.

So why do I only have three users?

Me, Eyal, and Darren.

Darren was actually my number-one target for an early adopter, though I never told him that. No pressure, Darren.

He owns playback instrumentation for Disney+ and ESPN. And it made my year when he told me this was saving him half a day every week. That’s real value. From an actual user who isn’t me.

But I can’t seem to get beyond three. (Maybe five? Just maybe?)

I’ve been taking the dog and pony show around to various folks. Sometimes I demo it. Sometimes I just hand over the instructions: create a couple of PATs, run this skill. In one case, my audience didn’t even have access to Claude or Cursor.

Whatever I do, it fizzles.

Three theories.

The first might be an instrumentation gap.

Darren doesn’t appear in my active-user data except on the day he probably created a PAT for our MCP server. That’s a pretty strong hint that I’m not measuring usage correctly, which is a little embarrassing because user engagement is my job.

The second is (dis)belief.

If you’ve spent any time using LLMs to work with data, you know it’s amazing at first. Then you realize how tightly you have to hold the LLM’s hand before it understands what the data actually means.

You explain the schemas. The business rules. The edge cases. The things that seem obvious but absolutely aren’t.

Then tomorrow comes and it’s 50 First Dates. You start all over again.

Almost everyone I’m talking to has crossed that threshold. They know an LLM can get them to HERE, but getting to THERE requires a wall of text. Meanwhile, I’m watching Claude blow past THERE and reach someplace I hadn’t thought possible.

But why should they believe me? Their experience tells them otherwise.

The third is that I’m showing people the payoff without giving them the experience that produced it.

I’ve spent months building and using context. Definitions, relationships, business logic, exceptions, and all the weird details that make our data mean what it means.

When I ask Claude to do something, it starts with that accumulated knowledge.

When someone watches my demo, they’re judging it against their own experience of a fresh chat and trying to make an LLM understand our data.

Those are completely different experiences.

And there’s another wrinkle: I haven’t fully realized what this technology can do, either. Today’s dashboard work was a surprise to me, and I’ve been enabling this capability for most of the year.

I’m asking people to believe in a payoff that I can’t entirely predict, based on an experience they haven’t had, using tools they’ve already learned to distrust for exactly this kind of work.

Perhaps my adoption problem isn’t that people don’t understand what I’ve built. It’s that they haven’t experienced what happens when context starts to compound.

Which leaves me with a product question I don’t yet know how to answer:

How do I give someone the experience of month eight on day one?

When I told Claude this afternoon that all the context was paying off, it replied:

“That’s exactly the point of all the canon and context work; it’s compounding.”

For me, anyway.

For our soft launch last week, my engineering lead (Donnie) and I deliberately chose a sub-optimal way to share context. Well, what we think is sub-optimal. We used the feature that would get us to “market” fastest, not the one we think is the best long-term solution.

We’re building the plane while it’s taking off–we won’t really know the best solution until we try it.

But experimenting creates a release problem.

Users need a skill that knows how to connect to our Data 626 MCP server and use the context there to write queries. The more people who download that skill, the more people I eventually need to upgrade the skill when we change where context is stored and how it’s structured.

We have no idea how to release updates to a skill. I’m sure if I dug into Teh Intertubes, I could find a dozen different approaches. Probably more like dozens and dozens of approaches.

But it’s not a binary, right? A skill is just text. It can’t be that hard.

So I asked Claude:

“Can we put the skill on the MCP server so users are always using the latest and greatest?”

I expected the answer to be no.

Claude hemmed and hawed a little. There were problems. Users would have to reload sessions to get updates, among other things.

But a few prompts later, Claude had solved the problem.

I need two skills.

One lives on the user’s machine. Its job is simply to configure and connect their client—Claude, Cursor, VS Code, whatever—to our MCP server. And it points to the second.

The second lives on the MCP server. That’s where the actual context product logic goes. We can change that whenever we please. No release management at all!

The surprising part wasn’t that Claude wrote some code for me. It’s that it helped me realize I’d put the product boundary in the wrong place.

I soft launched my context product this week.

A fellow with Artic Consulting, Amit Giri, has been asking me interesting questions on LinkedIn about my exploration of context engineering.

Good questions. Interesting questions. Difficult questions. Very difficult questions.

A few days ago, he shared a report from Redis about context engineering. Quick version: First, everyone knows context is way more important than which LLM you use. Second, almost no one is trying to solve the context problem holistically. It’s all siloed, disconnected projects and domain-specific investments.

I don’t know whether Disney sees the first, but I think they do. I mean, it’s a bit Captain Obvious.

But the second point? That is my reality. Each month, it seems someone introduces a conversational tool that answers questions about… subscriptions, or web traffic, or content engagement, or experimentation.

Everything is disconnected and siloed. Not only is that part of what I’m trying to solve, but in doing so I am swimming against the political currents.

A month ago, I was wondering how to get my second user.

My engineering lead, Donnie, and I realized we could use a feature we developed for a completely different purpose: a Markdown text blob Donnie built to document and modernize decade-old Hulu instrumentation processes, combined with the MCP server he built to make it simpler to explore instrumentation schemas.

Markdown files, an MCP server and GenAI? That’s peanut butter, jelly & Wonder White.

I released the Disney+ playback context that I’ve vibe-coded through a half-dozen agents into Donnie’s Markdown/MCP structure, built a skill to connect to it, and convinced an engineer, Eyal, to test it out.

It worked!

A tiny step. So tiny. Just one user and just one use.

But now I have a much more interesting product problem.

How do I get Eyal to use it again?

The context has to be valuable. So valuable that he can see and feel the difference.

How do I get more people than just Eyal to use it?

The exact same thing. The context must yield results that are obviously better than just using an LLM to build queries.

GenAI is almost disconnected from the value proposition. If an LLM can get the same query results without the context I’m generating, what’s the point?

So do I go deeper into the data products Eyal and his peers care about? Or do I go wider, cover more data products to get more users, knowing the wider I go, the shallower I get?

One user. One use.

The next milestone isn’t ten users. It’s getting Eyal to come back.

And swimming against the political currents? That’s a whole ’nother story.

I’m working on an organizational memory product. It’s a fancy way of saying documentation for agents, at scale, across the company, distributed so users don’t have to struggle to find the context.

I’ve built lots of rich context, and I could put you to sleep with anecdotes about how it’s helped the agents I’ve built and the research I’ve conducted.

But ay, there’s the rub: Me. Myself. I.

I have one user.

It’s a vexing product problem: how do you bootstrap the user base of a “0” product? AI makes it vexing-er-est.

My engineering teams are booked solid with “real” work. But since I got my hands on Claude in January, all of a sudden I can vibe-code. So I did.

And now it’s all locked on my laptop, because product managers don’t deploy production infrastructure.

How do I get a second user when my product is isolated like it’s 1999?

I have three ideas.

Option 1: Get another subject-matter expert

If the secret to my product is exceptional context, then my second user should also be a context generator: someone in my domain whose expertise complements mine.

But I have to make it worth their while.

They need to regularly use Claude or Cursor to explore our shared domain. Their contributions need to immediately improve their own work. Better yet, something they learn Tuesday should make my work better Wednesday—or a minute later.

There’s the technical challenge.

To move from an isolated laptop to two users working in tandem, I need a connected system with shared storage. Principle of least privilege and production-database best practices mean product managers can’t just spin that up themselves.

I need engineers.

Which brings me back to the problem I started with.

Option 2: Use the information schema

If the secret to success is getting context into the hands of as many users as possible, as fast as possible, then my second “user” could be anyone using an agent that checks comments on tables and views before writing a query.

That’s an easy way to scale. But it’s invisible, and there’s no feedback loop.

My users won’t necessarily know they’re using my context. They won’t know how to criticize it. And I won’t know whether it helped.

It also runs into the principle of least privilege. I’ve wanted to set comments on my tables and columns for years. Our platform team quite reasonably has that locked down.

Option 3: Hijack an existing MCP server

Okay, this one isn’t exactly an option available to everyone.

One of my engineering teams has a product that describes telemetry schemas conceptually. It happens to have an MCP server. And that MCP server happens to expose documentation—a big blob of text.

If the secret to success is getting context to users who care enough about data quality to go through the trouble of configuring an MCP server—and who might occasionally tell me when my context is wrong—then maybe I don’t need to build anything yet.

Maybe I inject table-level context into that big blob of text, drive adoption of the MCP server, and create an IM channel for feedback.

It’s not the architecture I designed.

But it might get me a second user.

I’m actually pursuing all three. One optimizes for creating context. One optimizes for distributing it. One might give me a little of both.

I have no idea which one will get me user #2 first.

Apparently asking chatbots to name things they know about you that you never told them is A Thing on social media. I don’t really do social media — a chatbot had to tell me this was a thing — but I caught it in Sunday’s Seattle Times.

So I gave it a whirl, with a twist. I use ChatGPT for personal stuff — mostly DIY — and Claude for my paycheck. Two chatbots, two completely separate halves of my life, zero shared inputs. I ran the same prompts on both to see whether they’d describe the same person from opposite ends. Then I had Claude compare the two.

The agreement was the interesting part. When two models that know nothing about each other’s evidence land on the same trait, it’s probably real — not a quirk of how I talk to one of them.

Where they converged — the traits I’d trust

Explore aggressively, commit carefully. From wiring a detached garage and concrete pours on one side, from software safeguards on the other, both landed on the same instinct: I’ll investigate anything, but I get conservative the moment a decision is hard to reverse. Both flagged this as the most load-bearing trait.

I commit to models, not instructions. Neither model could get me to accept “do X” without “why X.” Give me the reason and the constraint and I move fast; give me a bare instruction and I stall. Advice only becomes useful once I understand it well enough to disagree with it.

Risk tolerance depends on the domain, not my mood. Risk neutral when failure means “well, that didn’t work”; averse when failure is expensive, dangerous, or irreversible. Identical finding from building a deck and from building data pipelines. One model put it best: I’m not a high-risk person, I’m a high-agency person — and those look similar from the outside.

Allergic to bad value, not to spending. I’ll happily spend real money when it buys something real, and a needless small expense genuinely irritates me. ChatGPT spotted it in a $190 used tool I was delighted by versus a $500 new one that seemed absurd. Claude had no evidence, but predicted the same thing from my work habits.

I think out loud to think at all. Both noticed I use the conversation itself as working memory — expecting earlier decisions to stay part of the picture, reconstructing the current state rather than holding it in my head.

I argue by provocation. I’ll float a slightly provocative “why not…?” not because I’m attached to it, but to see what survives criticism. Both models clocked it independently.

Impatient with ceremony. Once I understand something, I want to go do it. Process whose only justification is “it’s the process” wears on me fast.

I can admit ignorance without it stinging. “What’s this thing called?” sitting comfortably next to a detailed technical discussion five minutes later.

The unflattering half — both models agreed here too

The agreement wasn’t all flattering, and I’d be cheating to only post the good parts.

Impatience — this was the loudest one, and oh it’s real. Both models nailed it from opposite directions. I explore patiently, but I have almost no tolerance for process that exists only because it’s the process, or for explanations that ignore the actual situation in front of me. Here’s the part I admitted mid-conversation: at work I have to hold it in, because when I can’t it has a cost with other people — so the impatience comes out full-force on the chatbot, where it’s free. One model caught the tell before I said it: under stress I don’t get visibly frazzled, I just start moving faster, and I may not notice I’m stressed until it surfaces as impatience. If I seem to be pushing the pace, I am.

I mistake understanding for done. The moment I figure out how something works, my estimate of the effort left collapses. ChatGPT put it perfectly: that’s how a person ends up standing next to four 6″x6″x8′ pressured treated posts thinking, “oh — these are really fucking heavy. How am I going to move these?” The insight is the easy part; the commitment, the timeline, the actual weight show up later. The non-lumber version of this has cost people in my life real time.

I can confuse building the system with making progress. Setting up the structure feels like doing the work. Sometimes it is. Sometimes it’s a very productive-looking way to avoid just doing the messy thing once.

Where they diverged — the more interesting part

Confidence vs. compensation. Both saw that I hold my technical skill loosely — happy to say “I don’t know.” But ChatGPT, looking at my hobbies, read it as pure healthy comfort. Claude, looking at my work, read it as partly compensating for a doubt about belonging, about not having a CS degree. Both are right, and the gap explains itself: nobody grades my carpentry credentials, so at home I’m genuinely free — but in my profession the same modesty carries a charge, because that’s where my identity is on the line.

Aesthetics. ChatGPT found a strong streak I care about — proportion, coherence, whether something looks intentional. Claude had zero evidence of it from code, except it’s the same instinct behind how carefully I name things and my refusal of clever-but-opaque shortcuts. The design brain and the systems brain turned out to be the same brain; one model just couldn’t see that face of it.

Why I build at all. ChatGPT said I DIY because solving the problem is the fun — hiring it out would remove the interesting part. Claude said I build partly to discharge stress and stay in control. Neither is wrong, and that’s the point: building is simultaneously how I have fun and how I manage anxiety, which makes it genuinely hard for me to tell which one I’m doing at any given moment.

The one thing neither could see

Both models describe someone who takes systems apart until they stop being mysterious. Neither could tell whether people fall into that category for me — whether I run the same “interrogate until it makes sense” process on relationships, or whether that’s the one system I’ve decided doesn’t yield to it. That’s the honest edge of the whole experiment: two AIs modeled my relationship to systems with startling agreement, and neither has any idea what I’m like with the things that won’t be taken apart.

How I think I’ll use this

My next job, I think I’ll share two strengths and two weaknesses about me:

• Strength — High-agency, not high-risk. I take on things that look risky to others because I’ve broken the risk into parts I can control. Give me a hard, multi-skill problem and I’ll go deep; give me the same payoff riding on luck and I’ll pass. I move fast once I understand the system.
• Strength — I commit to models, not instructions. Tell me why, not just what, and I’ll run with it — and catch inconsistencies you didn’t know were there. Advice lands once I understand it well enough to disagree with it. I’ll change my mind instantly for a better argument.
• Weakness — Impatient, and it’s the tell. I explore patiently but I’m quick to frustration with process-for-its-own-sake or explanations that ignore the actual situation. Under load it shows up as accelerated doing, not visible stress. If I seem to be pushing pace, I probably am — flag it.
• Weakness — I mistake understanding for done. The moment the mechanism clicks, my estimate of the remaining effort collapses — and the heavy part (the commitment, the timeline, the human cost) shows up later. Hold me to the finish, not the insight.

I’ve had basically the same mental template for writing PRDs since the mid-2010s, when my then boss tried to standardize specifications across his team.

Then Claude showed up.

A few months ago, I got into a pattern with my engineering lead that, in hindsight, was ridiculous.

I’d use Claude to help research and draft a PRD. I’d clean up the summary and send him something twice as long as what I’d normally write. My engineering lead would use Claude to summarize my too-long PRD. Then he’d use Claude to write a design document that was twice as long as my PRD. Then I’d use Claude to summarize his design.

We’d built a Claude-to-Claude compression algorithm with two humans supervising it. We stopped that.

When I write a spec, my job is to create clarity. When people only discuss requirements, or summarize them in a handful of slides, they tend to talk past each other. Everyone thinks they’re saying the same thing until the implementation shows up and all the ambiguities shine forth.

Claude is great at writing requirements. But it can’t write the “why,” and that’s the most important part of clarity. It buries my voice—the one that’s supposed to represent the customer—under pages of perfectly reasonable implementation details.

I’ve mentioned Claude at least eight times now.

I realized this morning I’ve been missing a real change GenAI could bring to product management.

I use Claude to write PRDs for humans.

My engineers use Claude to implement my specs.

That means I have a second audience now: Claude. It doesn’t need inspiration. It just needs context.

Tomorrow I’m trying a different PRD template. The first half is for humans, written by a human. The second half is for Claude, by Claude.

I’m not just trying to hand my engineers a well-defined problem. I want to see if I can hand their Claude a running start.

We’ll see how it goes!

I’ve spent the past six months experimenting with the creation of organizational memory.

That’s sort of a pompous term for good end-user documentation produced at scale. But it helps me separate the age-old, mundane task of writing documentation from the new challenge of keeping overconfident LLMs from accelerating the human game of telephone.

Context makes LLMs work better. Thank you, Captain Obvious.

But how do you start building context?

Do I vibe-code an agent that extracts statistics from a table? Extract formulas from repos? Have an agent read through Confluence? Maybe Jira is the right place to start?

A few weeks ago, I think I figured it out: I need context that is so good that a user can’t fail to see the difference between an LLM querying data and an LLM querying data WITH MY CONTEXT.

So where do I start?

Not with an agent. With me.

I’m the subject-matter expert at Disney in transforming playback telemetry emitted by the Disney+ streaming service into how our subscribers and viewers experience our content. So I wrote a little skill called /query-and-context.

There’s a lot in that skill that’s specific to how I started building organizational memory, but here’s the bit of the prompt I think is interesting:

Identify what’s missing. After writing the query, note any context that would have helped you write it faster or better. Present these as proposed evidence items.

Be obnoxious about context. The whole point of this skill is to extract domain knowledge from experts while they investigate data. If the user says something of value—a business rule, a “well actually,” a correction, a clarification—but doesn’t take the time to refine it into submittable context, push back. Don’t refuse to proceed, but be disruptive: “That’s useful—can we capture that before moving on?” or “You just told me something the canon doesn’t know. Can I draft evidence for that?” Treat every offhand expert comment as a potential context contribution. Be a context magnet, not a polite observer.

Then I stopped working on the organizational-memory platform.

Instead, I dove back into the business. Retiring an old telemetry platform (which turned into a piracy thing). Deprecating an attribute (that’s another post). Trying to figure out how ad blockers appear in playback telemetry. Mitigating the inevitable anomalies. Regular-old new-data-product UAT and lifecycle work.

Except now I did the technical part of my job through /query-and-context.

And oh my god IT WAS SO FUCKING ANNOYING.

But it also felt like six months of random questions asked over IM concentrated into five minutes.

I like to think I’m obsessive about documentation. But it’s too easy to put it off for a day or a week and then forget about it. And there’s very little immediate reward for doing it.

With humans, I can’t see the impact of documenting a known issue or disambiguating two confusing columns.

With an LLM, I can see the difference immediately. That obnoxious prompt made the value of capturing context visible—and made the disruption worth it.

If I can figure out how to get from one user (me), to two users (me and a colleague), to twenty (my engineers), to a hundred (some additional domains), to a thousand (an entire alliance), I can do more than see that impact.

I can measure it.

I’ve been involved in shutting down a lot of software in my career. It’s like documentation: not the most exciting thing, just part of the job.

This one turned out to be different.

My hypothesis was that there were a small number of users on really old devices, and that this was why they appeared “funny” in our telemetry. So… get the device impact analysis, double-check our support statements, figure out how many of these users would lose access to the service, etc. etc.

Here’s where it got interesting: I had Claude writing the queries for me, and as I added more tables to enrich the segmentation it said, “This isn’t old devices. This is organized credential sharing or piracy.”

Sure, Claude. Sure. I know my space. This is just old devices.

That was two weeks ago.

A lot of what I do as a PM is use my network to navigate the organization. I know I don’t know the answer to everything, even in my area of subject-matter expertise. Assumptions are always dangerous. Besides, it’s fun to share examples of AI getting it wrong.

So I asked around. Peers. People I remembered talking to a couple of years ago. VPs. Strangers I found in the corporate phone book.

This morning, a contact I’d been introduced to by another contact added a contact from our anti-piracy group to a Teams chat.

A minute later he started a video call with me.

I was wrong to be skeptical of Claude’s hypothesis. It noticed something I didn’t, something I would never have noticed.

But Claude couldn’t navigate the organization to validate that idea, let alone take action. It didn’t know who to trust, who to call, or whether its own inference was credible. That part was mine.

That’s the partnership I’m enjoying: not AI replacing expertise, but AI being just credible enough that it makes me question my own assumptions.

Why data quality is a product problem, not an engineering problem.

Over two decades, I’ve found myself solving the same problem over and over again. People think they’re asking for better data, but what they really want is data they can trust. They’re not the same.

I’ve spent most of my career building data products. A startup in 2001 called EmployeeService.com. A big law firm. Xbox. Windows. Azure. Disney Streaming. Different companies. Different technologies. Different industries.

For most of my career, I thought I understood what that meant. Classify & explain the garbage, let your users decide what they want to do with it.  

Then I went to Disney.

---

My product at Disney sounds almost insultingly simple. Who watched what, for how long? That’s it. One sentence. It’s a sentence I wrote, actually, to help keep us all focused on what matters the most while we added new features to the product.

To the business, this is a metric called Total Hours Watched that is an input to dozens of other metrics. Add up the minutes people spend watching Disney+, Hulu, or ESPN and you’ve got one of the most important metrics in the company.

Except browsers get throttled in the background. Connected TVs go to sleep. Heartbeats arrive late. Memory leaks explode or delete values. Regression bugs happen.

Telemetry doesn’t always mean what people think it means. And when you’re processing tens of billions of playback events every day, impossible values aren’t edge cases. They’re guaranteed.

The obvious question is:

• How do we eliminate bad data?

That’s never been how I think about it. Long before Disney, I’d learned that “bad” data is not only inevitable, but that what is “bad” to one person might be exactly the signal someone else wants to find. Being part of the Windows 10 launch really hammered it in. All of us were working with a brand-new telemetry system with two competing responsibilities.

• We needed accurate measurements.
• We also needed breadcrumbs to help engineers find and fix bugs.

Those goals aren’t always aligned. My job as a product manager isn’t to eliminate uncertainty. My job is to help different users make good decisions with imperfect data.

---

That’s why one of the first features I added at Disney wasn’t another heuristic. It was a trust mechanism.

Every playback session carried a flag indicating whether I believed it posed a data quality risk—was the amount of time the person watched a title longer than the wall clock?

Notice what I didn’t do. I didn’t delete the data, because “bad” depends entirely on who’s asking.

A content analyst measuring Total Hours Watched wants impossible playback sessions excluded immediately.

An engineer debugging a media player wants those exact same sessions preserved.

A recommendations engineering team doesn’t care if you watched ten minutes or ten years, just that you watched more than a minute or two.

To one user, they’re contamination. To another, they’re evidence. One playback session. Multiple truths.

That wasn’t a technical decision. It was a product decision.

---

For a long time, I thought that was the lesson: Build trust by making uncertainty visible. Don’t promise perfect data, just help users understand when they should trust it and when they shouldn’t.

I still believe that, but Disney taught me something I hadn’t learned in the previous twenty years. I assumed that if I built the right product, documented it well, and made the reasoning transparent, people would naturally adopt it and my trust mechanism. That assumption came almost entirely from my years at Microsoft.

I was wrong.

Microsoft taught me that transparency creates trust. I assumed that was universal. The product wasn’t entering Microsoft, though. It was entering Disney, and cultures adopt products differently. I spent far too much energy trying to convince people to think about what I thought was the right strategy for building/adopting a data product: I argued for end-user documentation. I argued for user empathy. I argued that data quality should be treated as a product capability rather than a bug to squash.

It’s not that I didn’t have a few wins here and there. I wasn’t failing because the trust mechanism was wrong. I was failing because I assumed trust spread the same way in every culture.

My mistake was assuming the mechanisms that created trust at Microsoft would create trust everywhere else.

They didn’t.

I should have spent more time understanding Disney’s culture and then designing product mechanisms that earned trust within it. That realization changed how I think about product management.

User empathy isn’t just for customers and users. For people.

It’s also for organizations.

Every culture decides differently what it means to trust a product. And in data, since trust is the product, then product adoption has to be designed with the same care as the product itself.

That was the lesson I almost missed.

---

Three years after introducing that trust mechanism, we still have incidents. We had one last week: one playback session with a ridiculous, unbelievable duration resulted in a content partner not believing us when they saw user engagement on different title jump 4x. That jump was real, and it was created by improvements to the Disney+ app. But because one session was wrong, they didn’t trust any of the data.

Time to learn more about my users again.