You've hooked up all your data to AI. Now what?

If you use Google Analytics 4, you've already hooked up your data to AI, whether you meant to or not.

In June, Google embedded Ask Advisor directly into GA4 properties. It's a Gemini-powered chat that lets anyone ask "why did revenue drop?" and get an answer in plain English. Add the Copilot, ChatGPT and Claude connectors that plug into almost everything else, and for most organisations the question is no longer whether to connect AI to their data.

The question is: now what?

In his last post, my colleague Zac explained why AI didn't replace our analysts, but made them harder to replace. He covered what we built with Inspectre, our own AI analytics capability, and how it works under the bonnet.

This post is for everyone on the other side of that conversation. The CEO wondering whether they still need to hire analysts. The CMO who's been handed a chat window and a promise. The Head of Data trying to work out what their team looks like in two years. And, frankly, my peers in analytics who want to know what we think our jobs are turning into.

I'll tell you what's changed for my team. What we've learned the hard way. And the part almost nobody is talking about yet.

It usually goes something like this.

The first fortnight feels like magic. You ask a question, you get an answer, with a chart and a confident summary. Questions that used to sit in an analyst's queue for days now take seconds. Everyone's impressed.

Then someone takes one of those answers into a meeting. The finance director has a different number. Nobody can explain the gap. Before long, people stop trusting the tool with anything that matters.

That's the lucky version. The unlucky version is when nobody notices the number was wrong at all.

Experienced analysts protect themselves from this with something we call the sniff test. It's the quick gut check you do before you trust a number: does this look right against everything else I know? If sessions are up 40% but revenue is flat and nothing launched, something smells off. You go and look before you repeat it to anyone.

The sniff test relies on context. A dashboard gives you plenty of it: the trend line, the months either side, the related metrics sitting next to it. A chat answer strips most of that away. You get a single number in a sentence, delivered with total confidence, and very little to judge it against.

If anything, AI makes the sniff test more important, and harder to do.

I run the data and analytics team at Fresh Egg. Inspectre has changed our day-to-day more dramatically than any tool I can remember.

The biggest shift is that insight has moved from pull to push. We used to go looking for problems. Now they come to us. Every morning, before anyone opens a report, the system has already checked the data for every client it's connected to and flagged what looks unusual. For the first time, my team can keep an eye on all those clients at once, every day.

The second shift is depth. Following a question three or four layers down used to cost hours, so most of the time we simply didn't. That depth is now available on demand.

We measure something we call time-to-insight: the time between a business question being asked and the answer being found. It has shortened dramatically. A reasonably simple question could easily take a day. Now, in that same time, we'd have asked three follow-up questions and got to the bottom of what was actually going on.

That has changed where my analysts spend their attention. Less time working through rigid checklists and monitoring routines; that work now runs continuously and automatically, so our clients actually get more of it than before. More time in the data itself, verifying trends and finding the root causes of variance.

Inspectre has made us all proper analysts again.

That's the part I didn't expect. The work is faster, but it's also more interesting, because the tedious bits have been handed to something that doesn't get bored.

Now for the part I promised at the start.

For years, our industry has been obsessed with data quality, and rightly so. Is the tracking accurate? Are conversions firing? Is the consent setup working? We built whole services around getting that right.

That hasn't changed. If anything, AI raises the stakes. It will analyse badly collected data just as confidently as clean data, and far faster. Context can't recover what was never collected.

AI adds a second thing that needs exactly the same care: context.

Context is everything the AI needs to know that isn't in the numbers: what your business counts as a lead, and how the tracking changed in March. That the August dip happens every year because your customers are on holiday. That one campaign uses a naming convention nobody else understands. A good analyst carries all of this around in their head. An AI only knows what it's been told.

Get the context right, and the answers improve dramatically. Anthropic's own data team, which runs internal analytics on its own AI, published its figures this summer. Without the curated knowledge that tells the AI how their business and data work, it answered their evaluation questions correctly 21% of the time. With it, consistently above 95%.

Analytics industry legend Gunnar Griese makes the same point for GA4 data: writing that context is the step where you "onboard" or "train" your agentic analyst. And, as he says, it's not something you set up once and forget.

This is the bit that caught us out. We'd assumed context was something you write once, at the start of a project. In reality it changes all the time, because your business changes all the time. A new product launches. A definition gets refined. A campaign ends. A tracking fix changes how a metric is counted from one day to the next.

When context goes stale, nothing breaks loudly. The answers simply start going wrong, in ways that still look plausible. Anthropic saw their accuracy drift from around 95% to around 65% over a single month before they started treating the upkeep of that knowledge as seriously as the data itself.

In Inspectre, the context lives inside the interface. Anyone on the team can add what they've learned, and it immediately becomes available to everyone else. The knowledge compounds, as Zac put it last week. What one analyst discovers on Monday, the whole team benefits from on Tuesday.

It's one of the things that makes Inspectre so useful. It's also where I lose a bit of sleep.

And the risk I worry about most comes from inside the team: one well-meaning colleague who was wrong.

Somebody adds a note that a campaign has ended when it's only paused. Or that a conversion is double-counting when it was fixed last week. The AI now treats that as fact, in every conversation, for every user. Most people never see the raw context, so nobody notices. The answers just start drifting.

The security world has a name for the deliberate version of this: context poisoning. Honest mistakes follow exactly the same mechanics. One wrong entry, shared by everyone, trusted by the machine, invisible to the people relying on it.

We've found a good way to add to the context. The harder question, and I'll be honest that we're still working through it, is this: who checks that nothing in there has stopped being true?

The ideas we're exploring are all fairly unglamorous:

• Every entry carries who added it, when, and why.

• Entries expire unless someone re-confirms them.

• When the website, the tracking or the measurement plan changes, the related context gets reviewed at the same time.

• If the data starts contradicting something in the context, the system flags it rather than picking a side on its own.

None of this is clever, and it doesn't need to be. Context deserves the same discipline we've always applied to data quality, because it now shapes every answer just as much as the data does.

Zac explained last week that Inspectre never does the maths itself. BigQuery does the calculating, and the AI works with the results. That's the engineering.

On top of it sits a set of working habits my team applies to every answer. If you had a maths teacher who wouldn't accept an answer without the working, you'll recognise them.

• Show your sources. Every answer has to say where it came from: which data, which dates, which definitions.

• Show your working. Every answer comes with the query it actually ran, so someone can check it did what was asked.

• Recalculate at source. A number in a sentence isn't a number until it's been recalculated. Summaries are where figures drift, so we re-run them in BigQuery rather than accepting them as written.

There's a catch worth spelling out. Showing the working only protects you if someone can read it. To most people, the query is gibberish. To an analyst, it's exactly where you spot the wrong date range, the missing filter or the definition that doesn't match the question. That's one of the less obvious reasons to keep an analyst on your team.

The good news is that none of these habits are unique to Inspectre. If you're using Ask Advisor, Copilot or ChatGPT on your own data, you can start asking for sources and working tomorrow. The first time your tool can't show them, you'll understand why we insist.

This is a question I keep coming back to, and I don't think anyone has fully answered it yet. So treat this as thinking out loud rather than a rulebook.

There's a strong case for the client owning it. Only you know what your business actually means by a lead, when the pricing changed, or why that one region behaves differently. You're the authority on your own business, and you're the one accountable when something changes.

But I've seen what happens to documents that clients are asked to maintain on their own. Most measurement plans are out of date within a year. Nobody is careless. Everyone is busy, and nobody knows which details the machine actually needs.

There's an equally strong case for the analyst owning it. We know what the AI needs to hear and how to phrase it. We see the data every day, so we notice when something stops being true. But an agency can't be the authority on your business facts. We can only write down what we've been told.

So my current thinking is a hybrid: the client owns the truth, the analyst owns the curation.

The client is the source and the sign-off for anything that's a fact about the business. The analyst is the editor. They turn what they hear into context the AI can use, keep it tidy, spot contradictions and chase anything that looks out of date. Anthropic's data team landed somewhere similar: use AI to draft the documentation, but have a human own the definition.

The benefits go beyond accuracy:

• Clear accountability. Everyone knows who confirms what.

• Fewer conflicting definitions. Finance, marketing and sales have to agree what a word means before the AI can use it. That conversation is valuable on its own.

• Knowledge that survives people leaving. On either side of the relationship.

• Better questions from both sides, because the context becomes a shared record of how the business actually works.

We'll be testing this properly soon. Inspectre is currently in alpha, and we'll shortly be opening a closed beta with some of our clients. The ambition is to make it part of every ongoing client relationship we have. How we share ownership of the context with those clients is one of the things I'm most interested to learn. If you've thought about this problem too, I'd genuinely love to compare notes.

If you're a business leader hoping AI means fewer analysts, here's what actually happened to us: we ask more questions than ever.

That day-long investigation that's now three follow-ups deep? Those follow-ups are questions we'd never have had time to ask before. When the cost of an answer drops, you end up asking far more questions.

Which means the bottleneck moves. It used to be "we don't have enough people to run the queries". Now it's "we don't have enough people who can tell us which questions are worth asking, and whether the answers mean anything".

That second problem needs people. Adding more AI won't fix it.

So what does the analyst of the next few years actually do? From what I see in my own team, three roles sit on top of the analysis itself.

Guardian of the context

Everything in the context section above needs an owner on the analytics side. Someone who notices when a piece of knowledge stops being true, and has the authority to fix it. We're not alone in seeing this. In their State of Martech 2026 report, Scott Brinker and Frans Riemersma describe marketing operations roles moving from system admin to stack wrangler to context engineer. The same shift is happening in analytics.

Educator

Analysts have always had to explain what numbers mean. Now they also have to teach people how to ask.

This matters more than it sounds, because AI tools have a strong tendency to agree with you. Researchers at the UK AI Security Institute found that phrasing matters enormously: put something to a model as a neutral question and it stays balanced. State the same thing as a belief and it becomes far more likely to simply agree.

In analytics, that's the difference between:

• "Why did the redesign hurt our conversion rate?" This assumes the answer. The AI will happily find you reasons.

• "What changed in our conversion rate after 3 March, and what else changed at the same time?" This lets the data speak.

Teaching a leadership team to ask the second kind of question is one of the most valuable things an analyst can do right now.

Spotting the liar

To be fair to the machine, there's no intent to deceive. It simply doesn't know when it's wrong, and it sounds exactly as confident either way. So part of the analyst's job is teaching people to spot the tells:

• The answer fits your assumption a little too neatly.

• There's no date range, no definition, no mention of where the number came from.

• There are no caveats at all. Real data almost always comes with some.

• The number doesn't match a figure you already trust.

• You're twenty messages into a conversation and it's started agreeing with everything you say.

None of these proves an answer is wrong. They're reasons to apply the sniff test before you forward it to the board.

If you're building or growing an analytics team, the skills rising in value are:

• Measurement design: knowing what's worth tracking in the first place.

• Data collection design: tracking built around business questions, with a structured dataLayer and consistent events, because AI amplifies whatever you collect, flaws included.

• Data modelling and enough technical depth to check the working, even if they rarely write the queries from scratch.

• Critical evaluation of AI output: the sniff test, done well and done every time.

• Commercial understanding: knowing how the business makes money, so they know which answers matter.

• Communication: turning findings into decisions, and teaching others to ask better questions.

The work falling away is the repetitive, manual kind: building the same recurring reports, working through checklists line by line, and clicking through tags one at a time to check they still fire. The discipline behind data collection is going the other way.

There's an uncomfortable question here. If AI now does much of the work juniors used to learn on, how does anyone become a senior analyst?

Stopping junior hiring strikes me as the wrong answer. The better move is to change what the junior role is for. Less time producing reports, more time checking the AI's work, learning why an answer is wrong and building the judgement that used to take years of manual graft to develop. The organisations that get this right will have the best analysts in five years' time. Those that stop hiring juniors altogether may find they have nobody left who can check the machine.

Yes. But probably not the analyst you'd have hired five years ago.

Pulling reports is now the machine's job, and it does it well. What you need is someone who can look after the context your AI depends on, check its working, teach your team to ask better questions and tell you honestly when an answer doesn't add up.

If you don't have that person in-house, that's exactly the gap a good analytics partner should fill. It's also, not coincidentally, how we've set up our own team.

Last week, Zac advised turning on the GA4 BigQuery export today because you can't go back and collect the history later.

Mine is just as cheap and just as urgent. Write down your definitions, and decide who owns them.

What counts as a lead? A customer? An active user? Which conversions actually matter? How do those conversions rank in value, from least to most? When did any of those definitions last change?

It's the single most useful piece of context you can give any AI, whichever tool you use. You'll probably discover that your teams don't all agree. Good. That disagreement is the first, and most valuable, conversation to have.

AI won't replace your analysts. Used well, it amplifies them. It gives them more reach, more depth and far more time for the work that actually needs a human.

But that only holds with the right foundations underneath: data you can trust, context that's kept current, and a skilled human in the loop who knows the difference between an answer and the truth.

Next week, I'll show you what this looks like day to day, with real (anonymised) examples of what Inspectre has found, and how it's changed the way my team works.

If you've hooked up your data to AI and you're wondering what to do next, talk to us.

Full disclaimer, I have used Claude to help me organise my thoughts and make this article easy to digest but the story, the thoughts, opinions and recommendations are mine, based on my personal experience running a data team in a fast pace agency environment.