Category: Product

I’ve had basically the same mental template for writing PRDs since the mid-2010s, when my then boss tried to standardize specifications across his team.

Then Claude showed up.

A few months ago, I got into a pattern with my engineering lead that, in hindsight, was ridiculous.

I’d use Claude to help research and draft a PRD. I’d clean up the summary and send him something twice as long as what I’d normally write. My engineering lead would use Claude to summarize my too-long PRD. Then he’d use Claude to write a design document that was twice as long as my PRD. Then I’d use Claude to summarize his design.

We’d built a Claude-to-Claude compression algorithm with two humans supervising it. We stopped that.

When I write a spec, my job is to create clarity. When people only discuss requirements, or summarize them in a handful of slides, they tend to talk past each other. Everyone thinks they’re saying the same thing until the implementation shows up and all the ambiguities shine forth.

Claude is great at writing requirements. But it can’t write the “why,” and that’s the most important part of clarity. It buries my voice—the one that’s supposed to represent the customer—under pages of perfectly reasonable implementation details.

I’ve mentioned Claude at least eight times now.

I realized this morning I’ve been missing a real change GenAI could bring to product management.

I use Claude to write PRDs for humans.

My engineers use Claude to implement my specs.

That means I have a second audience now: Claude. It doesn’t need inspiration. It just needs context.

Tomorrow I’m trying a different PRD template. The first half is for humans, written by a human. The second half is for Claude, by Claude.

I’m not just trying to hand my engineers a well-defined problem. I want to see if I can hand their Claude a running start.

We’ll see how it goes!

I’ve spent the past six months experimenting with the creation of organizational memory.

That’s sort of a pompous term for good end-user documentation produced at scale. But it helps me separate the age-old, mundane task of writing documentation from the new challenge of keeping overconfident LLMs from accelerating the human game of telephone.

Context makes LLMs work better. Thank you, Captain Obvious.

But how do you start building context?

Do I vibe-code an agent that extracts statistics from a table? Extract formulas from repos? Have an agent read through Confluence? Maybe Jira is the right place to start?

A few weeks ago, I think I figured it out: I need context that is so good that a user can’t fail to see the difference between an LLM querying data and an LLM querying data WITH MY CONTEXT.

So where do I start?

Not with an agent. With me.

I’m the subject-matter expert at Disney in transforming playback telemetry emitted by the Disney+ streaming service into how our subscribers and viewers experience our content. So I wrote a little skill called /query-and-context.

There’s a lot in that skill that’s specific to how I started building organizational memory, but here’s the bit of the prompt I think is interesting:

Identify what’s missing. After writing the query, note any context that would have helped you write it faster or better. Present these as proposed evidence items.

Be obnoxious about context. The whole point of this skill is to extract domain knowledge from experts while they investigate data. If the user says something of value—a business rule, a “well actually,” a correction, a clarification—but doesn’t take the time to refine it into submittable context, push back. Don’t refuse to proceed, but be disruptive: “That’s useful—can we capture that before moving on?” or “You just told me something the canon doesn’t know. Can I draft evidence for that?” Treat every offhand expert comment as a potential context contribution. Be a context magnet, not a polite observer.

Then I stopped working on the organizational-memory platform.

Instead, I dove back into the business. Retiring an old telemetry platform (which turned into a piracy thing). Deprecating an attribute (that’s another post). Trying to figure out how ad blockers appear in playback telemetry. Mitigating the inevitable anomalies. Regular-old new-data-product UAT and lifecycle work.

Except now I did the technical part of my job through /query-and-context.

And oh my god IT WAS SO FUCKING ANNOYING.

But it also felt like six months of random questions asked over IM concentrated into five minutes.

I like to think I’m obsessive about documentation. But it’s too easy to put it off for a day or a week and then forget about it. And there’s very little immediate reward for doing it.

With humans, I can’t see the impact of documenting a known issue or disambiguating two confusing columns.

With an LLM, I can see the difference immediately. That obnoxious prompt made the value of capturing context visible—and made the disruption worth it.

If I can figure out how to get from one user (me), to two users (me and a colleague), to twenty (my engineers), to a hundred (some additional domains), to a thousand (an entire alliance), I can do more than see that impact.

I can measure it.

I’ve been involved in shutting down a lot of software in my career. It’s like documentation: not the most exciting thing, just part of the job.

This one turned out to be different.

My hypothesis was that there were a small number of users on really old devices, and that this was why they appeared “funny” in our telemetry. So… get the device impact analysis, double-check our support statements, figure out how many of these users would lose access to the service, etc. etc.

Here’s where it got interesting: I had Claude writing the queries for me, and as I added more tables to enrich the segmentation it said, “This isn’t old devices. This is organized credential sharing or piracy.”

Sure, Claude. Sure. I know my space. This is just old devices.

That was two weeks ago.

A lot of what I do as a PM is use my network to navigate the organization. I know I don’t know the answer to everything, even in my area of subject-matter expertise. Assumptions are always dangerous. Besides, it’s fun to share examples of AI getting it wrong.

So I asked around. Peers. People I remembered talking to a couple of years ago. VPs. Strangers I found in the corporate phone book.

This morning, a contact I’d been introduced to by another contact added a contact from our anti-piracy group to a Teams chat.

A minute later he started a video call with me.

I was wrong to be skeptical of Claude’s hypothesis. It noticed something I didn’t, something I would never have noticed.

But Claude couldn’t navigate the organization to validate that idea, let alone take action. It didn’t know who to trust, who to call, or whether its own inference was credible. That part was mine.

That’s the partnership I’m enjoying: not AI replacing expertise, but AI being just credible enough that it makes me question my own assumptions.

Why data quality is a product problem, not an engineering problem.

Over two decades, I’ve found myself solving the same problem over and over again. People think they’re asking for better data, but what they really want is data they can trust. They’re not the same.

I’ve spent most of my career building data products. A startup in 2001 called EmployeeService.com. A big law firm. Xbox. Windows. Azure. Disney Streaming. Different companies. Different technologies. Different industries.

For most of my career, I thought I understood what that meant. Classify & explain the garbage, let your users decide what they want to do with it.  

Then I went to Disney.

---

My product at Disney sounds almost insultingly simple. Who watched what, for how long? That’s it. One sentence. It’s a sentence I wrote, actually, to help keep us all focused on what matters the most while we added new features to the product.

To the business, this is a metric called Total Hours Watched that is an input to dozens of other metrics. Add up the minutes people spend watching Disney+, Hulu, or ESPN and you’ve got one of the most important metrics in the company.

Except browsers get throttled in the background. Connected TVs go to sleep. Heartbeats arrive late. Memory leaks explode or delete values. Regression bugs happen.

Telemetry doesn’t always mean what people think it means. And when you’re processing tens of billions of playback events every day, impossible values aren’t edge cases. They’re guaranteed.

The obvious question is:

• How do we eliminate bad data?

That’s never been how I think about it. Long before Disney, I’d learned that “bad” data is not only inevitable, but that what is “bad” to one person might be exactly the signal someone else wants to find. Being part of the Windows 10 launch really hammered it in. All of us were working with a brand-new telemetry system with two competing responsibilities.

• We needed accurate measurements.
• We also needed breadcrumbs to help engineers find and fix bugs.

Those goals aren’t always aligned. My job as a product manager isn’t to eliminate uncertainty. My job is to help different users make good decisions with imperfect data.

---

That’s why one of the first features I added at Disney wasn’t another heuristic. It was a trust mechanism.

Every playback session carried a flag indicating whether I believed it posed a data quality risk—was the amount of time the person watched a title longer than the wall clock?

Notice what I didn’t do. I didn’t delete the data, because “bad” depends entirely on who’s asking.

A content analyst measuring Total Hours Watched wants impossible playback sessions excluded immediately.

An engineer debugging a media player wants those exact same sessions preserved.

A recommendations engineering team doesn’t care if you watched ten minutes or ten years, just that you watched more than a minute or two.

To one user, they’re contamination. To another, they’re evidence. One playback session. Multiple truths.

That wasn’t a technical decision. It was a product decision.

---

For a long time, I thought that was the lesson: Build trust by making uncertainty visible. Don’t promise perfect data, just help users understand when they should trust it and when they shouldn’t.

I still believe that, but Disney taught me something I hadn’t learned in the previous twenty years. I assumed that if I built the right product, documented it well, and made the reasoning transparent, people would naturally adopt it and my trust mechanism. That assumption came almost entirely from my years at Microsoft.

I was wrong.

Microsoft taught me that transparency creates trust. I assumed that was universal. The product wasn’t entering Microsoft, though. It was entering Disney, and cultures adopt products differently. I spent far too much energy trying to convince people to think about what I thought was the right strategy for building/adopting a data product: I argued for end-user documentation. I argued for user empathy. I argued that data quality should be treated as a product capability rather than a bug to squash.

It’s not that I didn’t have a few wins here and there. I wasn’t failing because the trust mechanism was wrong. I was failing because I assumed trust spread the same way in every culture.

My mistake was assuming the mechanisms that created trust at Microsoft would create trust everywhere else.

They didn’t.

I should have spent more time understanding Disney’s culture and then designing product mechanisms that earned trust within it. That realization changed how I think about product management.

User empathy isn’t just for customers and users. For people.

It’s also for organizations.

Every culture decides differently what it means to trust a product. And in data, since trust is the product, then product adoption has to be designed with the same care as the product itself.

That was the lesson I almost missed.

---

Three years after introducing that trust mechanism, we still have incidents. We had one last week: one playback session with a ridiculous, unbelievable duration resulted in a content partner not believing us when they saw user engagement on different title jump 4x. That jump was real, and it was created by improvements to the Disney+ app. But because one session was wrong, they didn’t trust any of the data.

Time to learn more about my users again.

What one schema misunderstanding taught me about AI and product management

A co-worker reached out to me this week, asking for help explaining the relationship between two attributes.

Our understanding was product_type is how the content is available and stream_type is how it was consumed.

I knew this was wrong, but it was so plausible that I started wondering if I was wrong about it being wrong. So I did what everyone does now: I asked Claude.

Claude replied:

product_type = the product experience,
stream_type = the technical stream shape.
They correlate but are not a clean parent/child hierarchy

It wasn’t the same words, but it felt strikingly similar to what my colleague had shared.

Was my colleague inferring this relationship with the help of AI? I think so. Just to be sure, I asked chatgpt if both statements were similar.

The (abbreviated) answer:

I think the AI answer is better than the human answer. But they converged on essentially the same abstraction

Chatgpt went on to praise the AI for reverse-engineering the schema from the data.

An alarm bell rang in my head.

I knew it was time to write a spec.

Claude got it wrong. Chatgpt got it wrong. Whichever LLM my colleague used–if they had even used one–got it wrong.

Claude didn’t hallucinate here. It did what engineers do every day: looked at the evidence and built the simplest model that explained it.

It wasn’t the reasoning, it was the lack of evidence. Two critical pieces were missing: formulas and history.

First, the apparent relationship between productType and streamType is a matter of mutability and aggregation function. ProductType is immutable in a playback session: it’s set at the start and never changes. When we transform hundreds of events into a playback session, the first non-null value wins. StreamType is mutable in a playback session. Ever stream a sporting event, and after the post-game wrap-up the game is rebroadcast from the beginning? The streamType will change from 'event' to 'eventReplay' at that point. When we create a playback session, the most frequent streamType across all the events wins.

Second, we’d been trying to deprecate this attribute for years, but couldn’t because it shows up as a slicer in so many executive reports.

That’s the spec I wrote: having streamType determine product_type so the Disney+ player can drop productType completely without breaking anything downstream.

Fixing productType solves one small, immediate problem. It doesn’t solve the problem that created it. It’s not that Claude can reason better with context. That’s obvious. The interesting question is how you make that context available at exactly the moment someone needs it.

The real challenge is the same it’s always been for a product manager: how do you get users to adopt your product?

I’ll have some more detailed posts about how I think my users at Disney will consume context and what product capabilities I’ll need to meet them where they are, but here’s a hint from when I asked Claude:

let’s connect to databricks and look at product_type and stream_type. what’s the difference?

Here are the first three things Claude wrote in the pre-answer hourglass-ish stream-of-text that I rarely pay attention to:

1. I’ll load the Databricks SQL tools and query the table to compare these two columns.
2. Let me look at the two columns — their definitions and value distributions.
3. Both are strings with no column comments. Let me pull their value distributions and how they overlap.

Look at the third. Claude naturally looked for column comments before trying to infer meaning from the data. Putting context into comments on a table is one simple way to meet users where they are, and we’re not doing it.

---

Here’s Claude on productType and streamType WITH context generated by a formula agent

What I think the relationship is

They’re two labels for the same underlying concept — “what kind of stream is this” — at two different resolutions, and they do NOT agree cleanly.

My read on why they disagree (hypothesis — not grounded)
The two columns are aggregated differently at the session level, and I think that’s the whole story:

product_type = first(product_type) → grabs the first heartbeat’s value.
stream_type = mode(streamType) → grabs the most common value across the session.

A session’s heartbeats can carry different metadata over its life (e.g., a live channel that starts from a VOD-style pre-roll or start-over buffer, then settles into linear). first() and mode() will legitimately land on different answers for the same session — and first() is far more fragile because it’s hostage to whatever heartbeat happened to arrive first. That’s consistent with stream_type being the “settled” truth and product_type being noisy.

I want to be explicit that this is a hypothesis about mechanism, not something the canon documents. The data shows the disagreement; it does not tell me why a given linear session got stamped VOD. I’d flag that as a context gap rather than assert it.

Look at that last paragraph. Canon? That’s a future post. Flagging as a context gap? That’s a future post as well.

Claude didn’t just do that. All of that product behavior is intentional.

This is my final presentation for UC Berkeley Extension’s Intro to Big Data course. It contains my perspective on the things that a data analyst, data scientist or data engineer will experience and need to watch out for navigating the data sprawl of a massive big-data environment.