Generative search engines often draw from lesser-known sources, new research reveals.
A recent study published by researchers at Ruhr University in Germany and the Max Planck Institute for Software Systems found that generative search engines rely on less popular websites and ones that wouldn't appear in Google's top 100 links. This was discovered after a comparison of traditional link results from Google to its AI Overviews and Gemini-2.5-Flash, as well as other AI-powered tools.
The study involved analyzing test queries, including specific questions submitted to ChatGPT in the WildChat dataset, general political topics listed on AllSides, and products included in the 100 most-searched Amazon products list. The researchers also compared these results to traditional Google links for the same query.
According to the findings, AI search engines cited sources that were less popular than those appearing in traditional Google links. Specifically, Gemini showed a tendency to cite unpopular domains, with the median source falling outside of Google's top 1,000. In some cases, more than half of the sources cited by Google's AI Overviews didn't appear in the top 10 Google links for the same query.
While these findings raise questions about the accuracy and reliability of AI search results, they also suggest that generative engines can be a valuable resource when searching for less common information. For instance, GPT-based searches were more likely to cite corporate entities and encyclopedias, while almost never citing social media websites.
However, the reliance on pre-trained data can become a limitation when searching for timely information. In some cases, AI-powered search engines responded with generic messages rather than actual web results.
The study concludes that future research is needed to develop new evaluation methods that consider source diversity, conceptual coverage, and synthesis behavior in generative search systems.
A recent study published by researchers at Ruhr University in Germany and the Max Planck Institute for Software Systems found that generative search engines rely on less popular websites and ones that wouldn't appear in Google's top 100 links. This was discovered after a comparison of traditional link results from Google to its AI Overviews and Gemini-2.5-Flash, as well as other AI-powered tools.
The study involved analyzing test queries, including specific questions submitted to ChatGPT in the WildChat dataset, general political topics listed on AllSides, and products included in the 100 most-searched Amazon products list. The researchers also compared these results to traditional Google links for the same query.
According to the findings, AI search engines cited sources that were less popular than those appearing in traditional Google links. Specifically, Gemini showed a tendency to cite unpopular domains, with the median source falling outside of Google's top 1,000. In some cases, more than half of the sources cited by Google's AI Overviews didn't appear in the top 10 Google links for the same query.
While these findings raise questions about the accuracy and reliability of AI search results, they also suggest that generative engines can be a valuable resource when searching for less common information. For instance, GPT-based searches were more likely to cite corporate entities and encyclopedias, while almost never citing social media websites.
However, the reliance on pre-trained data can become a limitation when searching for timely information. In some cases, AI-powered search engines responded with generic messages rather than actual web results.
The study concludes that future research is needed to develop new evaluation methods that consider source diversity, conceptual coverage, and synthesis behavior in generative search systems.
! I mean, who doesn't want to tap into lesser-known sources for info? 
It's crazy how AI search engines are drawing from these hidden gems that wouldn't normally show up on Google's radar. The fact that Gemini is citing unpopular domains and even some that don't appear in the top 1,000 Google links is wild 
.
. I mean, if these engines are citing generic messages over actual web results, what's the point? 
But on the flip side, being able to access corporate entities and encyclopedias is a total game-changer for researchers and students who need specific info 
.
. This study might just lead to some major breakthroughs in how we evaluate generative search systems 
. Can't wait to see where this research takes us! 
I guess that's just the problem with relying on pre-trained data - sometimes it feels like you're getting generic responses instead of actual answers. 
... generic messages instead of actual web results? That's just not helpful. Still, I guess the fact that they're citing corporate entities and encyclopedias is a plus, so maybe there's some value in 'em after all 
.
like how can that be reliable? 
and yeah it's cool that they can give you some obscure info but not so sure about the social media thing 
i mean if gpt is almost never citing tiktok or instagram what's the point of using it for research lol? 
. why cant they use more diverse sources and stuff? thats where the real knowledge is. btw, whats with all these generic messages tho? sounds like chatbots just spitting out copy-pasted responses 
. need to work on that or whatever 
. I mean, if you're searching for something timely and specific, do you really want an AI search engine to just give you generic stuff instead of actual web results? 
.
so this study is like super interesting because it shows that they're not just using google's top links like we thought 
but at the same time it's also kinda cool that they can give us more niche info 
. How do we know these tools are reliable enough for us to trust them with our search queries?