Time to Write a Spec

What one schema misunderstanding taught me about AI and product management

A co-worker reached out to me this week, asking for help explaining the relationship between two attributes.

Our understanding was product_type is how the content is available and stream_type is how it was consumed.

I knew this was wrong, but it was so plausible that I started wondering if I was wrong about it being wrong. So I did what everyone does now: I asked Claude.

Claude replied:

product_type = the product experience,
stream_type = the technical stream shape.
They correlate but are not a clean parent/child hierarchy

It wasn’t the same words, but it felt strikingly similar to what my colleague had shared.

Was my colleague inferring this relationship with the help of AI? I think so. Just to be sure, I asked chatgpt if both statements were similar.

The (abbreviated) answer:

I think the AI answer is better than the human answer. But they converged on essentially the same abstraction

Chatgpt went on to praise the AI for reverse-engineering the schema from the data.

An alarm bell rang in my head.

I knew it was time to write a spec.

Claude got it wrong. Chatgpt got it wrong. Whichever LLM my colleague used–if they had even used one–got it wrong.

Claude didn’t hallucinate here. It did what engineers do every day: looked at the evidence and built the simplest model that explained it.

It wasn’t the reasoning, it was the lack of evidence. Two critical pieces were missing: formulas and history.

First, the apparent relationship between productType and streamType is a matter of mutability and aggregation function. ProductType is immutable in a playback session: it’s set at the start and never changes. When we transform hundreds of events into a playback session, the first non-null value wins. StreamType is mutable in a playback session. Ever stream a sporting event, and after the post-game wrap-up the game is rebroadcast from the beginning? The streamType will change from 'event' to 'eventReplay' at that point. When we create a playback session, the most frequent streamType across all the events wins.

Second, we’d been trying to deprecate this attribute for years, but couldn’t because it shows up as a slicer in so many executive reports.

That’s the spec I wrote: having streamType determine product_type so the Disney+ player can drop productType completely without breaking anything downstream.

Fixing productType solves one small, immediate problem. It doesn’t solve the problem that created it. It’s not that Claude can reason better with context. That’s obvious. The interesting question is how you make that context available at exactly the moment someone needs it.

The real challenge is the same it’s always been for a product manager: how do you get users to adopt your product?

I’ll have some more detailed posts about how I think my users at Disney will consume context and what product capabilities I’ll need to meet them where they are, but here’s a hint from when I asked Claude:

let’s connect to databricks and look at product_type and stream_type. what’s the difference?

Here are the first three things Claude wrote in the pre-answer hourglass-ish stream-of-text that I rarely pay attention to:

1. I’ll load the Databricks SQL tools and query the table to compare these two columns.
2. Let me look at the two columns — their definitions and value distributions.
3. Both are strings with no column comments. Let me pull their value distributions and how they overlap.

Look at the third. Claude naturally looked for column comments before trying to infer meaning from the data. Putting context into comments on a table is one simple way to meet users where they are, and we’re not doing it.

---

Here’s Claude on productType and streamType WITH context generated by a formula agent

What I think the relationship is

They’re two labels for the same underlying concept — “what kind of stream is this” — at two different resolutions, and they do NOT agree cleanly.

My read on why they disagree (hypothesis — not grounded)
The two columns are aggregated differently at the session level, and I think that’s the whole story:

product_type = first(product_type) → grabs the first heartbeat’s value.
stream_type = mode(streamType) → grabs the most common value across the session.

A session’s heartbeats can carry different metadata over its life (e.g., a live channel that starts from a VOD-style pre-roll or start-over buffer, then settles into linear). first() and mode() will legitimately land on different answers for the same session — and first() is far more fragile because it’s hostage to whatever heartbeat happened to arrive first. That’s consistent with stream_type being the “settled” truth and product_type being noisy.

I want to be explicit that this is a hypothesis about mechanism, not something the canon documents. The data shows the disagreement; it does not tell me why a given linear session got stamped VOD. I’d flag that as a context gap rather than assert it.

Look at that last paragraph. Canon? That’s a future post. Flagging as a context gap? That’s a future post as well.

Claude didn’t just do that. All of that product behavior is intentional.