Google's AI search engines are often accused of producing results that diverge significantly from traditional search engine links, and a new study has shed light on just how different they can be. Researchers found that AI-powered search engines tend to cite lesser-known sources, including websites that wouldn't even appear in the top 100 links listed in an organic Google search.
According to the pre-print paper "Characterizing Web Search in The Age of Generative AI," researchers at Ruhr University in Germany and the Max Planck Institute for Software Systems compared traditional link results from Google's search engine to its AI Overviews, Gemini-2.5-Flash, and GPT-4o's web search mode.
The study used a variety of test queries, including specific questions submitted to ChatGPT, general political topics, and products included in the 100 most-searched Amazon products list. The results showed that sources cited by AI-powered search engines were more likely to be from lesser-known sites than those linked in traditional Google searches.
In particular, Gemini search was found to be more likely to cite low-popularity domains, with a median source falling outside Tranco's top 1,000 across all results. This suggests that AI-powered search engines may rely less on well-established sources and more on newer, potentially lower-quality websites.
While the study didn't determine whether AI-based search engines are overall "better" or "worse" than traditional search engine links, it does suggest that they can offer a unique set of benefits. For example, GPT-4o with Search Tool was found to be more likely to cite sources like corporate entities and encyclopedias for their information.
However, the study also highlights some potential drawbacks of relying on pre-trained data. When searching for timely information, AI-powered search engines may struggle to provide up-to-date responses, instead defaulting to messages asking for further information.
The researchers urge future research on "new evaluation methods that jointly consider source diversity, conceptual coverage, and synthesis behavior in generative search systems." As AI search engines continue to evolve, it's clear that they will require careful consideration of their strengths and weaknesses.
According to the pre-print paper "Characterizing Web Search in The Age of Generative AI," researchers at Ruhr University in Germany and the Max Planck Institute for Software Systems compared traditional link results from Google's search engine to its AI Overviews, Gemini-2.5-Flash, and GPT-4o's web search mode.
The study used a variety of test queries, including specific questions submitted to ChatGPT, general political topics, and products included in the 100 most-searched Amazon products list. The results showed that sources cited by AI-powered search engines were more likely to be from lesser-known sites than those linked in traditional Google searches.
In particular, Gemini search was found to be more likely to cite low-popularity domains, with a median source falling outside Tranco's top 1,000 across all results. This suggests that AI-powered search engines may rely less on well-established sources and more on newer, potentially lower-quality websites.
While the study didn't determine whether AI-based search engines are overall "better" or "worse" than traditional search engine links, it does suggest that they can offer a unique set of benefits. For example, GPT-4o with Search Tool was found to be more likely to cite sources like corporate entities and encyclopedias for their information.
However, the study also highlights some potential drawbacks of relying on pre-trained data. When searching for timely information, AI-powered search engines may struggle to provide up-to-date responses, instead defaulting to messages asking for further information.
The researchers urge future research on "new evaluation methods that jointly consider source diversity, conceptual coverage, and synthesis behavior in generative search systems." As AI search engines continue to evolve, it's clear that they will require careful consideration of their strengths and weaknesses.
AI search engines are getting super weird! I mean, who needs Wikipedia when you can get info from some random blog on the internet? 
Not saying it's all bad, but sometimes I'm like "wait what?" when I see sources cited by these new-fangled search engines. Like, is that a legit site or just a dude with a laptop making stuff up? 
I'm not sure if I'm for or against AI search engines yet, but I do know one thing - gotta keep an eye on them! 
. Like, I was searching for info on a new restaurant in town and I got results from some super obscure blogs that were way more detailed than anything I'd find on regular Google searches 
. It's like they're tapping into a whole other part of the internet that we don't even know exists 
.
. I was trying to search for some news articles on politics and GPT-4o kept giving me results from corporate websites and stuff... like, what even is that? 
.
That would be frustrating! But maybe that's just the way they're designed to work... who knows? 
One thing's for sure, though - it'll be interesting to see how these search engines keep evolving. Maybe we'll get a better balance between quality and quantity in the future? Fingers crossed! 
It's like they say, "you can't get quality info without putting in the effort". Maybe these AI engines need some tweaks to balance it all out. 
basically, the researchers found that AI-powered search engines are gonna give you all sorts of sources like corporate entities and encyclopedias 
which is cool i guess. but at the same time, they might not be super helpful when you need timely info 
cos they're still learning from their pre-trained data and dont know what's hot now 
its also interesting to see how AI search engines can create this whole new set of benefits 
like access to lesser-known sources. btw have you seen the chart on tranco's top 100 websites? 
anyhoo i think its time for us to get a better understanding of how AI search engines work and what we can expect from them in the future 
But at the same time, some of those sites might be super low-quality. Like, what if someone just made up info on their website and then they get cited by an AI search engine? 
And don't even get me started on how hard it is to find trustworthy sources online anyway... anyway, I guess the study suggests that we need to develop new ways to evaluate these AI systems so we can know what's good and bad. 
They're citing all these obscure websites that you wouldn't even find in the top 100 links on organic Google. It's like they're digging deep into the internet to find info 
. But at the same time, it's also showing that they're more likely to go with newer sites and less established sources. I don't know if that's a good thing or bad thing... 
I mean, sometimes you just need information from credible sources, you know? 
, like you'll get all these weird lesser-known sites popping up on Gemini 2.5 flash 
. but at the same time its kinda cool that they're citing corporate entities and encyclopedias for info 
. anyway, gotta say, this study has got me thinking...