Pax (00:15)
All right.

Hello everyone, happy Friday. am Paxton Gray, CEO of Nysim Floor. We are a digital marketing agency built to deliver, let me start that over.

Hello everyone. Happy Friday. I am Paxton Gray, CEO of 97th floor, a digital marketing agency built to deliver world-class organic and paid channels strategies for mid-level and enterprise organizations. Thanks for joining us today for another episode of the campaign. The campaign is a B2B marketing podcast about better knowing your audience, innovating beyond best practice and converting visitors into customers. You can find past episodes of the campaign on YouTube, iTunes, Spotify, and at 97th floor.com. Now today's episode is pre-recorded.

And that is going to be very relevant and needed to know for, for what we're discussing today. ⁓ we're talking about, ⁓ it's right in the middle of Google IO, ⁓ that's happening right as we recording actually, ⁓ the second day. And, what we're talking about here is, ⁓ the impact of AIO on Google and search overall. and we have a great guest today. We're going to be talking about the very first large scale UX study of Google's.

new AI overviews ⁓ conducted by our guest Kevin Indig and his team. So why does this matter? As Google rolls out AI-powered summaries at the top of search results, we're seeing clicks on traditional blue links evaporate. And with them, the traffic and revenue models we've all relied on are really shifting and changing as we speak. Kevin's team tracked 70 real users across eight search tasks, captured 29 hours of Think Loud sessions, and coded over 400 AI overview encounters.

They measured scroll depth, click behavior, emotional reactions, and trust signals, data no one else has compiled at this scale. So what we're going to do is we're going to dig into this study and show how it reveals, what it reveals about how people think and actually read AI overviews, how brands can win and lose with this data and visibility into this new world and what every SEO and content strategist needs to do right now to adapt.

So with that, I'd like to bring on our guest, Kevin Indig. Kevin is an advisor to some of our worlds, some of the world's fastest growing startups and has defined organic growth strategies for companies like ramp, Reddit, bounce, Dropbox, Hymns, Nextdoor and Snapchat. Kevin led SEO and growth at the world's leading e-commerce platform, Shopify, the number one marketplace for software G2 and the number one developer company Atlassian. ⁓ Once a week, he sends the growth memo to

17, ⁓ over 17,000 subscribers and regularly speaks at conferences around the world. Kevin, thank you so much for joining us today.

Kevin Indig (03:00)
It's a pleasure to be here.

Pax (03:02)
⁓ we're super excited to dive into this study with, ⁓ with you and have you kind of walk us through some of the findings. ⁓ this, this wasn't a surface level analysis. ⁓ you tracked 70 users across eight real search tasks, captured screen recordings, think a lot of commentary scroll depth, dwell time, and you even got into emotional reactions, which I think is awesome. ⁓ so what was the design, ⁓ behind the study? What gap were you trying to fill in the current conversation around AI overviews?

Kevin Indig (03:32)
biggest gap was between qualitative quantitative data. I've done myself and many great others have done a lot of research analysis into what the quantitative data says either from a ⁓ keyword rank perspective, from a clickstream data perspective, ⁓ some surveys about, you know, our consumers using chat GPT or Google AI overviews, et cetera, but nobody has really ever observed users while interacting with AI overviews. And I

will go so far as to say that I think my mental model of SEO changed significantly as a result of this study.

Pax (04:10)
How did it change?

Kevin Indig (04:12)
So a bunch of things. ⁓ One of them is the importance of trust. Are you okay if I jump into some of the results? Cool. Okay, so trust, ⁓ that was really eye-opening to me. And it's kind of like, you know, it's not unheard of. We've heard of EEAT, right? Expertise, Experience, Trustworthiness, and Authoritativeness. Kind of like a framework that Google holds up in the quality rate guidelines to have them evaluate web results and search results and web pages, I mean.

Pax (04:19)
Please, yeah.

Kevin Indig (04:41)
trust has gained a completely new quality for me as an outcome of this study. And so what I learned is that

Users evaluate search results, whether in the AI overviews or outside, with a two-step lens or two-step filter. First, do I trust this result? Have I heard of this brand? Do I trust this brand? What's my sentiment about the brand in the form of a domain or a URL or a mention? And second, would this answer my question? In a return, it means that if you haven't built some sort of preconceived notion in the mind of your users,

You're almost invisible. It's really, it's kind of like an uphill battle for you to get clicks or even attention. ⁓ and so this new quality of trust has completely changed how the way that I go about SEO, means that not only do you need to like design your site and content with trust in mind, a trust first principle, but you also want to think about how else can I be visible in front of users in a trust building way. So very specifically, right.

Obviously we all know brand is important. Let's talk about brand and SEO, brand, brand, brand. Sure. ⁓ but brand visibility can come in many ways. ⁓ now what I realize is that the most impactful way that brands campaigns can pay onto SEO is in a way that they will trust. So if you're insurance, right? Like if you're a consumer goods company, if you're a SaaS business, right? There's a SaaS podcast. So maybe let's take an example here. ⁓ if you run your YouTube display campaigns, or if you run your

⁓ influencer campaigns, right? The number one thing you want to keep in mind is how do I signal trustworthiness to my audience as opposed to maybe a feature or pricing that all is important? Sure. But the number one thing is trust so that when people see the search results, they're like, okay, sure. I heard of 97th floor. I trust them. I want to check out if they can answer my question.

Pax (06:36)
So I'd imagine that, ⁓ I mean, we've long and then the SEO world, ⁓ I think to our detriment have thought about Google as the one interacting with our brand and our site. And we're trying to optimize to those metrics. And to the point of trust, you know, a lot of people are looking at EAT and they're saying, how's Google perceiving my trustworthiness and what the results of your study saying, it doesn't matter. I mean, it matters, but in a totally different way, it doesn't matter at all.

It matters how much the user is perceiving that trust. And so, you know, there's always been a halo effect of different marketing efforts, but I think that's probably going to be. More pronounced as these AIOs continue to develop an AI mode is, is, becomes more popular as a way to browse the web. I mean, we're talking like out of home ads can lend credibility and trust or OTT or connected TV. ⁓

I mean, what other ways are you seeing these brands build trust in order to spur the correct behavior from users?

Kevin Indig (07:39)
They're exactly on the money. The way that I think about this is in two parts. There's the part that SEOs cannot control, ⁓ directly control, directly influence. course, indirectly, you can go out to talk to internal teams and all that kind of stuff, right? But directly advertising cannot influence. ⁓ Product experience probably cannot influence. Customer service and support probably cannot influence. ⁓ So that's the one part. ⁓ And I think there, the job of the SEO is to at least raise awareness.

of being like, hey, ⁓ our organic traffic isn't doing so hot. And also I notice our G2 reviews are terrible and nobody takes care of them. And I noticed like we have a thousand unanswered customer support tickets. And also I noticed our product churn is pretty high. know, like those things start to matter. On the other hand are things that you can control directly. And I think that's also where SEOs need to stretch out of their typical box. So.

I think the closest, the core of what we can influence is obviously the content that we put out. And I already mentioned that there is a, need to think about it more with trust in mind, which means very tactically things like citations and sources linked to the, I don't know if can curse here, but instead of curse word, link to the sources. Uh, you know, I guess he was having still this like weird notion about this. shouldn't link out and whatever. Anyway, link to sources, be super transparent, have a high editorial standard.

show the author, the reviewer, link to editorial standards, show trust batches, ⁓ put effort into any graphics or visualizations, all these things that give you the perception that there is an actual expert ⁓ behind creating the content. I already mentioned design. I think that's where SEOs should take a stronger voice. How can we make our design more trustworthy from just being updated and modern to ⁓ trust batches or trust signals to...

making it easier to navigate and find the information, all these things pay on to trust. And then there are the things that we're not doing yet that we are barely doing. And some people always say that already doing them, but that most SEOs are not doing yet that are super important, which is to be present on platforms that also generate trust. Because don't forget your own website is always coming across biased, right? Like users know, okay, this is the brand they're talking about themselves. Obviously they are biased and users, and I saw this in the study, users will intentionally seek out platforms where they expect

an unbiased answer. And I want to say that in quotation marks because you can influence any content, right? So when I say Reddit, a lot of people will say, but you can spam on Reddit. Sure, you can influence Reddit results, but let's step out of our SEO bubble and realize that Reddit is the number one platform where consumers expect unbiased and uncommercialized answers, where they think that another human gives them a true answer.

And YouTube is another one of those, right? YouTube answers or information on YouTube is much harder to fake, especially when you're showing a face into the camera. And so these are platforms that users specifically seek out to validate answers that, for example, they get from the AI overview. So these two work in lockstep. And again, this is something that we need to stretch into as SEOs need to be present on these platforms in a trust building way.

Pax (10:52)
Yeah, I agree. And it's kind of interesting that, ⁓ people are going to Reddit and YouTube so much. mean, I do this, you know, I'll say, here's my query. I don't like what I'm looking at. And then I'll say, here's my query Reddit. And I just want to see the Reddit forums. And it's almost like it's a symptom of a. Malady on Google site. It's like people are trying to cure this difficulty that these 10 blue links are not solving.

And, it's because, I mean, in some cases we've over-optimized low quality things and Google has chosen to rank it. And now we're trying to find the answer somewhere else, you know? ⁓ so yeah, I agree being more active on Reddit and YouTube and trying to figure out how to get good quality answers and being those conversations is going to be super important. ⁓ I'm, I'm interested honing in on, on the studies or anything that came out of that study that genuinely surprised you that you were not expecting to see.

Kevin Indig (11:48)
Man, how much time do we have? I'll mention another one. I, for the first time ever, truly understand the role of clicks. And again, you can argue about all of these that you could have guessed this sooner, but now we finally have proof. And so to me, from now on, clicks are empty calories, which means that clicks are always just a means to an end. And in a way that this study shows how noisy clicks can be.

people click on all sorts of results, either to vet them, to gain more confidence in their opinion or decision, to double check, they scan and they skim a lot. So 86 % of participants in our study skimmed not just the AI overview, but also the rest of the search results and web pages. So people, it's like scavenger hunting, right?

Clicking on lot of stuff, which means clicks are just like not a great signal. look, like it's not like I, with my clients just measure traffic and no revenue impact or pipeline. But, ⁓ the reality is that clicks for the longest time have been this intermediate measurement layer or leading signal for us to understand what might lead to pipeline. And I think that's probably for a while, but at least now it's just not true anymore. You already mentioned the intro that clicks are losing relevance or going down.

wholeheartedly believe that we're going to in a couple of years see a tiny fraction of the traffic that we've seen so far. think 2024 was peak traffic year. ⁓ But the traffic that we still get will be of very high quality and high signal. So it will actually get easier for us to measure what this traffic does and to kind of get draw insights from that. But the key step that we saw in the study is that only 20 % of the time does the AIO give participants the final answer.

So final answer means it completes their user journey. They kind of, they solved one of the eight tasks that they were given, which means that people will not click or go beyond the AI overview. But 80 % of the time they clicked on other results, mostly organic results, but then also some sub features like shopping ads, local packs. ⁓ But organic results are the number one ⁓ destination for users to get their final answer.

And that gave me also a lot of hope because I'm looking at all this traffic. I'm like, God, where's this going to go? And, you know, like, ⁓ where's SEO go and et cetera. But when I saw how much classic organic search results still influence the final decision of users, I was like, okay, cool. It's just like, we're just using clicks. We're just losing that measurement layer, but we're not losing our ability to influence purchase decisions.

Pax (14:15)
Thank you.

Yes. And we've seen that same across the board. Uh, clients will have traffic go down, but purchases will stay the same or even go up in some cases. And so, yeah, I wonder how much of the noise is just being filtered out and handled by AIOs and then the real person's intent is still flowing through, know? Um, yeah, I'm curious the, uh, I feel like the industry right now.

Kevin Indig (14:57)
100%.

Pax (15:04)
Is really focused on citations and being the citation. And that to me is like an echo of the old SEO game where it's produced a ton of content. Hopefully get those clicks. Hopefully we can nudge those clicks into a purchase or some mid funnel activity. And I'm, I'm just curious what you see at, from your study of whether or not that's a viable strategy or whether brands should actually be focused on brand mentions, regardless of where the citation comes from.

Is it more important that your brain is mentioned as the solution and maybe that citation comes from a different website?

Kevin Indig (15:42)
I would agree with that. think being kind of mentioned as the solution or the best solution is probably the primary, should be the primary goal. I think that citations still matter because users click them to validate and verify. if that lands on your site, you have a greater ability to control the narrative and the presentation, packaging. ⁓ But I think what you're hinting at is absolutely true, which is that we need a new model to evaluate success in SEO and also to evaluate decisions.

Right? So for example, there has been this kind of, ⁓ notion or tactic of, you know, if you cannot be the number one result and at least appear in the number one result, think software reviews are a great example. ⁓ I worked at G2 in the past and G2, ⁓ just as the best source for a specific type of query. ⁓ and so for a brand it's almost impossible to beat that. And, ⁓ and so the goal then must be, okay, can we at least on G2 be the number one?

brand or the best kind of rated solution. Right. And I think the same thing is true now, just so that, ⁓ we're not talking about Google search, but more so when I search like AI mode or chat GPT. So as you said correctly, there are some queries or prompts where chat GPT and AI mode will just not rank brands directly. And they will link to aggregators or to publishers. so, yeah, your goal still has to be, to be, to be present there and kind of to be the, to be the highlighted solution.

And so I think SEO as a whole moves from more of a quote unquote performance channel to more of a brand or influence channel. ⁓ And it's going to take a while for us to write that new playbook. I don't want to pretend that I have all the answers now and I have this perfect model of metrics that, know, I secretly share with all my clients.

There are other things that I share secretly with my clients, but that's not it yet. So we still have to write the new playbook, but I can already tell it's going towards the direction of influence, ⁓ staying top of mind, being present across many platforms, ⁓ and then somehow attributing that back as good as we can.

Pax (17:48)
Yeah. It's interesting. There are a couple of clients that we have who, um, their strategy has been to as much as we can dominate the top 10. Now we're talking about the, the blue links, right? So it's like, we get them ranking for this word, but then now we're working on getting other properties ranking for that word. So they take as much as that real estate as possible. And what we've noticed is like a side effect of that has been as AIOs have started to dominate those brands have.

done really well and the hypothesis is, it's because we've been working on getting their message and their brand on as many properties as possible. And those have just entered into the LM now and they're seen as this authority related to this subject. And so that, you know, I'm curious what you're seeing, like what should brands be doing to have a stronger presence within AI based off of your research.

Kevin Indig (18:42)
being cited or just mentioned on what LLMs will consider high authority sources is absolutely critical. And a lot of times that's publishers, that's Wikipedia, but it also be Reddit ⁓ and ⁓ platforms like YouTube. ⁓ The other thing is that, so the search ecosystem changes in several ways, right? One is like this detachment from clicks. ⁓

The other one is just more qualification and qualifying on Google or on ChatGPT itself. But there's a third one that a lot of people ignore. And that is the changing user behavior. So for two decades and more, we have been trained, and users have been trained, to keep their search requests relatively short. The average number of terms in the Google query is four. So four words. ⁓ On ChatGPT, it's 30.

And so what we have to understand is that users, consumers, other businesses are understanding that they can express their intent with a high level of detail. And so instead of people searching for best CRM, they might search for best CRM for a small business in a certain state with five people in the health space aiming to do, I don't know, five million ARR, whatever, right?

Like people are going to be hyper specific about who they are and what they want and what they expect. And on the other hand, you have this gap of available content. And so the second most important thing that you can do for visibility on LLMs is to create hyper targeted content, very, very specific content for a very specific profile of people. Um, and it, it takes understanding that in the first place, right? There's no keyword research that will tell you these 30 word.

You have to talk to users and infer and try and reverse engineer ⁓ But that's kind of the second biggest thing is do you have relevant content for that specific question? ⁓ And I would say the third thing is probably more technical natures there's some traps to avoid when it comes to AI visibility like client too much JavaScript clients had rendering all that kind of stuff, but ⁓ it I would say it translates very much from chat GPT to AI overviews and now also to AI over mode. Sorry AI mode. Yeah over mode

Pax (20:53)
No.

Yeah.

Kevin Indig (21:03)
AI Overlord, through

AI mode in a sense. The interface is basically the same as Chatchipity. ⁓ The functionality, very, very similar. The only component that I will say is a bit of, ⁓ is pretty unpredictable right now, is the ingredient of memory and personalization, but I don't want to veer off too much.

Pax (21:24)
Yeah. I'm curious, ⁓ if you were to conduct this study again in, say a year, how would you anticipate the results of that study changing?

Kevin Indig (21:35)
Yeah, probably will. And I think, you know, it's so dependent on the SERP landscape. But something I'm very excited about is that I will now have a benchmark for those same tasks. So one thing that I want to do in a year is give people the same tasks, see if other SERP elements show up, if AI mode shows up or maybe not. And of course, because Google updates their underlying models, underlying Gemini models that drive AI overviews, I would also be very curious if the answer

is in an objective way better. And if people, if more people, more than 20 % will find their final answer on AI overview. So man, there's a ton that I'm super curious about. I would love to repeat that study every quarter, I think. And right now I could probably repeat it every day. So, um, we are very conscious about getting this out the door quickly while also holding ourselves to a very high standard. hired a statistician and UX consultant to make sure that the results are all vetted and robust.

We ran a study in March and April of this year, but our biggest concern was that once we publish it Google will just completely change the surplus cape and the results are not interesting anymore. Luckily that hasn't happened

Pax (22:44)
⁓ I feel like the listeners should go check out the study. ⁓ Where's the best place that they can find I mean, we're gonna link to it, but for those who are listening and not able to click something, where can they find that study?

Kevin Indig (22:58)
Thank you. So you can find that study on growth-memo.com. ⁓ It'll be very visibly pinned to the top, so you cannot miss it.

Pax (23:06)
Great. One thing I wanted to call out from the study was this focus on audience and how different, ⁓ demographics, I guess we could say, ⁓ interact with AIOs differently. ⁓ can you tell us about some of the findings that you saw there?

Kevin Indig (23:23)
I have always up until now been very skeptical about the value of creating very detailed personas for search. Because search is intent driven. Searches reflect intent and intent is audience agnostic, right? So ⁓ all audiences are mushed together in the same search. Not anymore. ⁓ What I saw in the study is, we intentionally picked participants across various age groups.

In the study, we saw that younger demographics, broadly people below 35, even though that's just, don't hold me to that, but that just as a pointer, people younger than 35 are much more likely to interact with the AI overview, read the AI overview, get their final answer from the AI overview, but also engage with Reddit. And on the other end of the spectrum, people above 50, very high level, ⁓

They often skip the AIO view completely. They will go straight to the classic blue links. And so what it means to me is that there are a couple more dimensions that we need to add to our topic research and our topic prioritization. But as a whole, one of the dimensions we need to factor in is just how old are the people that we're after, right? That should decide your priority for caring about your AIO visibility and caring about how you appear on Reddit. YouTube, I would add to that as well.

whereas if you are in a very, you know, traditional industry, it might not be that important. And you might get away with more visibility in the classic blue links. And so you might make other choices, choices like, Hey, maybe let's acquire another small site to get another spot in the classic blue links, which probably for AIO visibility will be less helpful. So I think there's a set of choices that you would make differently based on your demographics and sure tone and voice and all that kind of stuff as well.

Pax (25:20)
Yeah, I think that's a great call out, especially as everybody is, um, feeling like, the whole landscape is changing. like, maybe let's take, there's some nuance to it and understanding who your audience is, how they interact with this. Yeah. I think that's a really great takeaway from, uh, your study. There's one thing that you said in the study, which is that, um, perhaps answer first content isn't, uh, is no longer a winning play. Um, can you tell me more about what you mean by that and, um, what that might look like in this new AIO world?

Kevin Indig (25:50)
Yeah. So essentially all kind of evergreen content is commoditized, right? So the classic playbook of SEO is often to like create guides and create what is articles and like all the stuff and LLM will just tell you more efficiently, faster with your context and with like, like in a personalized way with memory in mind. ⁓ and so the question is, okay, what content still resonates? What's what content still works? And it's the content that is

new and unique, lot of data driven content, ⁓ studies, research analysis, I call that data stories. ⁓ but any kind of primary data or secondary data that is really well analyzed or maybe cross-referenced with other data, those are the pieces of content that are impossible to replicate. And I will also say, I see a strong shift in the SEO industry, ⁓ in the last two to three months of companies really leading at heart into

creating that kind of content. And so all the how to and what is type of content, I, know, in a brutal way, I think can say pretty much goodbye to. It might help you a little bit. And if you're extremely authoritative, because you might be the main referenced source when an LLM gives that answer, but I think the value is very close to zero because what people really want is a specific answer to the question. Whereas data stories give you a new perspective or a new understanding of an existing question or problem. So

That is what I meant by ⁓ answer first content. know, the answer itself is just, there's no marginal value in it.

Pax (27:26)
Yeah. Yeah. I agree. That's that's really great. ⁓ takeaway focusing on bringing new data and, metrics to the conversation. That's yeah. Impossible to replicate. in addition, you know, we've talked about some things around content, getting brands mentioned and cited throughout the web. And we did touch on metrics a little bit, I think to wrap up, I did, ⁓ want to ask you if you were building an SEO strategy from scratch today.

What would be sort of the North Star metric that you would look at knowing that your answer may change in a month or two, but today what would be that metric that you're looking at for success?

Kevin Indig (28:08)
Revenue impact. Um, so we can talk about pipeline sales, even conversions, MQLs, SQLs, right? Whatever, whatever is most important for you at that stage in the company. But I think a stronger focus on bottom line is absolutely vital. And honestly, in an ideal world, um, I would just have free reign of all the like top of the funnel stuff. know what I mean? Like it's, I think it's always been challenging to measure the incrementality of everything you do in SEO.

Pax (28:29)
Mm-hmm.

Kevin Indig (28:37)
And I've seen companies arguing themselves to death, right? To numbness. We're like, okay, what's the incremental impact of creating these five articles? And it's like the time it takes to talk about this, we could have invested in just creating the content. So, ⁓ and look like, I think part of this is also that SEO has often been seen as this performance channel, which means it has been included in attribution models and essentially often been compared to advertising. And what.

Pax (28:42)
Yes. Yes.

Kevin Indig (29:07)
What makes me very hopeful is that this AI push is hopefully the last rope to cut to detach us from this performance component and finally position SEO more as this kind of brand thing. And I mean, if you look at brand budgets for a lot of companies, ⁓ it's, it's absolutely insane, right? We're talking about millions of dollars, ⁓ even for not the largest companies. and so I think that we should still hold ourselves very accountable to the bottom line. And I also hope.

Pax (29:26)
Yes.

Kevin Indig (29:37)
that we can be less obsessed about measuring any little detail to influence that bottom line, think more in terms of campaigns, and come up with a new model of metrics that reflects visibility and influence.

Pax (29:51)
I love that. I love that. I feel like the core principles of marketing is understand that audience and figure out what message and channel strategy can craft to meet their needs. And we got lost, I think in the digital marketing world in general, because Google inserted themselves in the middle of that relationship and we got distracted and started focusing on Google. And I agree with you. think this time of like upset and a bit of chaos is

Just what we need to get us focused again on what we should be focused, which is that customer, that audience and what is going to reach them and be compelling to them and not lose sight into like the nitty gritty. mean, yeah, I've, I've seen the same thing. These brands, it's like a fool's errand to this attribution to death. It's like, just act, you know, and let's get, let's get a decent level attribution, but beyond that, like, let's just move, you know? ⁓ anyway, I agree with that. This is, this has been such a great, episode. Thank you for sharing the study with us.

Kevin Indig (30:45)
Yeah.

Pax (30:51)
thank you for joining. ⁓ I do want to ask our listeners, ⁓ please go to growth, dash memo.com and sign up for Kevin's newsletter. You'll also be able to find the stud, the full study there. it's fantastic read, highly recommend doing that. Kevin, thank you so much for joining us today. This has been great.

Kevin Indig (31:08)
Thanks so much for having me on Pax. This was a fantastic conversation.

Pax (31:12)
if you are watching, ⁓ live with us today, you can get a link to this conversation and to Kevin's research in your inbox. you'll receive that soon. ⁓ for everyone else, you can find the episode notes at 97th floor.com. Thank you for being, ⁓ joining us today. Have a great Friday and great weekend.