VP of Growth says the biggest risk with AI is believing what the data appears to prove

Jon Chang is the VP of Growth Marketing at MyFitnessPal. Before this, his career spanned IBM Watson, Kickstarter, and Klarna. And he's been working with AI since long before it became mainstream.

We talked to Jon to understand how AI is changing advertising. Along the way, he told us which revenue functions are in desperate need of an AI overhaul.

Working on AI before it touched anyone's P&L

I'm Jon Chang, VP of Growth Marketing at MyFitnessPal, and I focus on the demand side of the business for revenue. This includes paid acquisition, app store conversion, and the gap between an ad's promise and a new user's actual experience upon arrival.

My path here includes IBM Watson, Kickstarter, and Klarna, and I've taught at NYU alongside my operating work. That first stop matters more than it might appear because I worked on commercial AI applications back when discussions focused on what these systems might eventually do, not what they were doing to anyone's P&L. Watching the field move from that period to this one instilled in me a durable skepticism about timelines and a durable respect for what actually ships.

The rest of that path led me to machine-driven channels long before anyone called it AI. Bidding came first, then audience selection, then creative rotation; each moved out of human hands one auction at a time. So, what's happening now doesn't feel like a break to me, but rather the same long trend arriving all at once — a genuinely different experience from seeing an entirely new trend emerge.

That history taught me that automation excels at optimizing toward correlation but remains completely indifferent to causation. As these systems accelerated, it became more obvious that incrementality — the question of whether your actions actually caused the outcome you claimed — was the discipline nobody invested in. I believe it remains the most undervalued skill in growth, and it only becomes more valuable as tools produce confident-looking numbers faster.

I write about this at Growth Marketing Musings on Substack.

A structure that forces better questions

MyFitnessPal is a consumer subscription business, mobile-first, competing in a category where acquisition relies almost entirely on auction-based channels and app store discovery.

That structure shapes everything because we don't manage a long sales cycle or work through a buying committee. The distance from impression to paid conversion is short, which sounds like an advantage and functions more like a constraint, meaning you get one attempt at each step and very little room to recover from a weak one.

It also means we operate in one of the harder measurement environments in marketing, which largely explains why incrementality has become central to my thinking. Multi-touch attribution genuinely breaks down in mobile, privacy-restricted settings where consent rates run low, which is why teams in this space lean on incrementality testing instead. Many organizations treat that as a limitation to work around, and I'd argue it's closer to a forcing function that leads to better questions.

How AI changed advertising

For years, building a tailored destination for every ad theme was so expensive that most traffic went to a single general page. Someone clicking an ad about one thing and someone clicking an ad about another thing landed in the same place. This meant the ad performed careful targeting, but the destination immediately negated that effort.

Across the industry, the cost of producing a variant collapsed once tools like ChatGPT and Claude became standard tools. When a variant is cheap, matching the destination to the promise stops being a luxury to justify and becomes the obvious default.

The effect is better documented than most in our field. One published analysis covering over a million Apple Ads ad groups found that ad groups pointing to a keyword-matched custom product page produced 23% more downloads than those pointing to the default page, with the same spend and keywords.

The key detail is what kind of matching works. A separate portfolio analysis found that apps seeing the biggest gains segment by user intent rather than by product feature. Pages built as visually refreshed versions of the default, with the same screenshot order and first hook, barely move at all. This is the key lesson: a language model makes it trivial to produce forty versions of a single idea, but that's worth almost nothing unless the intent work comes first.

I would add — and I think this part often gets skipped — that none of this is worth much unless you can determine if the change caused the lift. Platform-reported numbers on a matched page will almost always look good because you point better traffic at a better experience, and the reporting cannot separate those effects. The only way to know is to compare it against a control, which is slower and considerably less satisfying, but it's the only answer I would actually defend.

Where AI excels, and where humans are required

AI powers a fair amount at this point, not entirely by choice. AI now handles bid and budget optimization within platforms because manual levers are largely removed. Beyond that, AI's useful applications for us are in synthesis, not production.

Agents built on ChatGPT and Claude excel at taking large, messy market information and returning something a person can reason about. Category demand shifts, competitive movement, and the insights from platforms like Sensor Tower, AppTweak, and Appfigures become far more usable when AI can read across them and summarize changes. This shortens the distance between a signal appearing and a decision being made, and in an auction-based business, that distance carries a direct cost.

Three things stay firmly on the human side of the line.

First, we choose what to test and why, because the machine optimizes within a question but has no opinion on which question is worth asking, and that choice determines if anything else matters.

Second, we define what counts as a win. Nearly every disappointing AI result I've seen traces back to a proxy metric that remained healthy while the underlying issue didn't move. McKinsey found that 88% of organizations regularly use AI in at least one function, but only 39% report measurable EBIT impact. I'd argue that that gap is a definition problem, not an adoption problem.

Third, and the one I'd hold onto hardest, is the causal read. We do not automate our incrementality work, nor do I want to. Brand and power lift studies run natively inside ad platforms; geo holdouts are read against platform data alongside the mobile measurement partner; and the analysis on top of all that is deliberately human. Deciding whether a result is genuinely causal or merely correlated requires judgment with real money attached. No model will tell you that the number in front of you is confounded, because a confounded number looks exactly like a clean one.

A real-world revenue workflow

Here's a workflow from intent to destination and then back again.

We start by clustering demand around what someone is trying to solve, rather than by demographic or by whichever feature we want to discuss. That clustering draws on category-level visibility from app measurement platforms, specifically tools like Sensor Tower, AppTweak, and Appfigures, since those show search behavior and competitive movement that first-party data can't. Each cluster becomes a theme, and each theme receives both an ad concept and a matched destination designed to continue the same argument rather than restart it.

ChatGPT and Claude handle much of the analytical core, where I believe the real value lies. I'd stress to anyone building this that the model on its own is the least interesting part. An agent that can only reason is an expensive brainstorming partner, whereas an agent connected to the market data and the performance reporting you already pay for becomes more like an analyst with unlimited patience for repetitive work. The integration is the product, and what you achieve depends far more on what you connect it to than on which model you choose.

The integration also makes the spend defensible. We run paid acquisition in an auction, so every hour between a signal appearing and making a decision carries a cost. Surfacing a shift in category demand the week it happens lets you move money while it still matters, and that's a straightforwardly cost-effective use of the technology, unlike producing another two hundred asset variations.

We deliberately keep the read manual, which brings us back to incrementality. Every test carries a minimum volume floor set before launch, and we don't act on results below that floor regardless of how good they look. If you run 100 experiments where half have a real effect, and you run them at 10% power, you get roughly ten significant results, about half of which are false. Simply raising the power drops that false discovery rate from 50% to 11% without other changes.

Beyond the floor, anything we intend to scale meaningfully is validated causally rather than against platform-reported performance, because the platform has every incentive and structural reason to provide a flattering figure. That validation comes from lift studies conducted natively in the channels and from geo holdouts analyzed against platform and mobile measurement data. The measurement is native, the analysis is not automated, and I believe that division will matter more over time. Everything upstream of the read becomes faster every quarter, which means the read must remain consistent.

How AI can increase the rate of learning, but make those learnings unreliable

The biggest benefit of AI is pace. Ideas that used to sit in a backlog for a quarter now get built and read inside a single cycle. This compounds over time, since how fast you learn largely depends on how fast you can test.

The bad results come in two forms, with the second being more interesting.

The first is straightforward, and I already touched on it. We ran a campaign where top-of-funnel numbers looked strong, but downstream conversion moved in the opposite direction. The systems had done precisely what we asked. We had asked for the wrong thing, which is a target-definition failure rather than an AI failure, though it was executed with unusual efficiency.

The second is that speed multiplies your existing epistemic habits. If your default read on performance is correlational — and for most teams, it is — then faster testing doesn't produce faster learning. It produces faster accumulation of things you believe for insufficient reason. Adobe's published testing guidance makes a related point, noting marketers underpower a remarkable number of tests. If you keep doing it, false positives can end up rivaling or outnumbering true positives. Consider what happens when variant count rises and traffic-per-variant falls. You don't get more winners; you get more things that look convincingly like winners.

So essentially, AI raised our rate of learning and simultaneously increased our burden to prove what we'd learned was real.

Why AI struggles with customer context and trust

These systems struggle because they lack access to 'why,' which provides most of the information in a consumer subscription business.

Two people can search the same term, click the same ad, and arrive with completely different motivations, yet the data distinguishes nothing between them. A model reading behavioral signals confidently clusters them together because they are identical on every measurable dimension. The clustering is correct, but the conclusion drawn from it can still be wrong. This is a particularly uncomfortable failure mode because there's no error to catch.

For example, in market analysis, agents have produced clean, plausible, well-argued narratives about a shift in the category — summaries that read like a smart analyst wrote them. However, the underlying movement turned out to be an artifact: a reporting change, a seasonal pattern, or a definitional shift in how something was counted. The narrative was coherent because the model excels at building coherent narratives, but coherence is not accuracy. What made it dangerous, rather than merely wrong, was its finished sound.

That, in a sentence, is the trust problem. These systems do not register uncertainty. A conclusion drawn from thin evidence returns in the same tone as one drawn from solid evidence. A human analyst would have hedged, noted strange data, or checked something before writing it up. The model simply writes it up.

The subscription-specific version presents an even harder challenge. When someone cancels, the event looks identical whether they failed to get what they came for or succeeded completely and moved on. These are opposite outcomes with the same data signature, and no amount of modeling separates them, because the difference lies entirely in context the system never had.

Why forecasting needs an AI redesign

The instinct is to make forecasting more accurate by throwing better models at it, and I think that's the wrong AI redesign. Gartner puts the share of sales organizations achieving 90% or better forecast accuracy at 7%, and that number has remained stubborn for many years despite an enormous amount of tooling investment. When a problem resists that much money and technology, it's usually a sign that the problem was framed incorrectly, rather than that the solutions were too weak.

The framing error is treating forecasting as a prediction problem when it's really a decision problem. What a revenue leader needs is not a more precise number for next quarter; it's a defensible answer to what they should do differently this week. Those are related, but they are not the same question, and the second one is far more tractable. You can run an experiment that tells you whether a change caused an outcome. You cannot run an experiment that tells you what the world looks like in ninety days.

So, I'd argue for a redesign that shifts the center of gravity from prediction toward causal evidence. Instead of asking a model to interpolate the future from degraded signal, ask what you could hold out, what you could test, and what result would change your allocation. That produces less impressive artifacts and considerably better decisions. It also fails honestly, because an inconclusive test tells you it was inconclusive, whereas a bad forecast tells you nothing until the quarter closes.

The second half of the redesign is making uncertainty a first-class output rather than stripping it out before the number reaches leadership. Most forecasting processes collapse a wide range into a single figure because a single figure is easier to present, and everything downstream then treats that figure as though it were known. A forecast that carried its own confidence and that plainly stated which parts rested on solid measurement and which on modeled or inferred signal would be less comfortable and far more useful.

AI can help with all of this, incidentally. It's good at maintaining the evidence base, at flagging when an assumption underlying a projection has moved, and at doing the tedious work of keeping a picture current. What it should not be doing is manufacturing false precision at the end of a pipeline of weak inputs, which is unfortunately most of what people sell it to do in this part of the lifecycle.

Jon's Thoughts

Efficiency and differentiation currently work against each other, so every leader must deliberately choose a ratio, because the default ratio is not neutral.

Why generative models force a choice between differentiation and efficiency

Here's another problem. Generative models don't invent anything; they produce the most statistically probable output based on their training data, which means they reproduce dominant conventions and pull everything toward a common mean.

Efficiency and differentiation currently work against each other, so every leader must deliberately choose a ratio, because the default ratio is not neutral.

How revenue and marketing tasks must be unified and measured

In most companies, one team owns the ad while a different team owns the page, the demo, or the first conversation. Both teams are measured and tend to do fine. Nobody owns the seam between them, which is precisely where intent leaks out. Consequently, every function reports a solid quarter while the connective tissue loses the customer.

Roughly half of organizations report handoff misalignment between marketing and sales, with campaigns built in isolation and sellers learning about messaging changes from prospects rather than internally. That reads like a communication problem, but it's an ownership problem.

We changed how we treated the ad and its destination, viewing them as a single unit built, shipped, and measured together rather than as two assets belonging to two teams. The benefit was not only conversion but also interpretability, because when the promise and the payoff are one object, you can finally run a clean read on the pair. Two teams optimizing two halves separately will each produce numbers that look fine, but neither set will tell you what the combination did.

The equivalent question for a revenue leader is whether the story a prospect hears from marketing survives contact with the first sales call, and whether anyone can measure that transition causally rather than by counting activity on both sides of it. If those things are owned and measured separately, AI will make each half more efficient while widening the gap between them. Having leaner teams move faster across an unowned seam is a predictable way to lose money while every dashboard continues to look healthy.

Why CROs must own the objective function rather than delegating it

Overall, here's my advice: Own the objective function yourself rather than delegating it or inheriting whatever the platform defaults to.

These systems are excellent at maximizing whatever you specify, which means the specification becomes the strategy. If the number you hand the machine is a proxy for the outcome instead of the outcome itself, you'll end up with a beautiful chart and a worse business, and you'll get there faster than before AI existed.

The companion discipline, and the one I'd push hardest, is knowing in advance what evidence would convince you something worked. Most revenue organizations have a very sophisticated answer to what happened and a surprisingly weak answer to what caused it, and AI widens that gap because it makes the first kind of answer cheaper. If you can't articulate what a clean read would look like before you launch, you'll end up ratifying whatever the reporting hands back.

PwC's 2026 CEO Survey found that only 12% of CEOs saw AI deliver both revenue growth and cost reduction, while 56% hadn't yet seen meaningful financial benefit. I don't read that as the technology underdelivering; rather, it's evidence that most organizations pointed it at something they hadn't defined carefully enough and then measured it in a way that couldn't establish whether AI had delivered meaningful financial value.

Follow along

You can follow Jon Chang's work on LinkedIn and his newsletter, Growth Marketing Musings.

More expert interviews to come on The CRO Club!