Over the past few years, Reddit has experienced surges of traffic from search engines. Why? Many internet users have adopted the practice of adding “Reddit” to the end of whatever they are searching for so they can find conversations and opinions from real people. Whether it be product reviews or travel recommendations, many people are looking to search within Reddit’s archives of discussions to find what they want.
Now, however, there seems to be only one search engine where users can actually do this.
As first discovered by the tech outlet 404 Media, Reddit has started blocking the vast majority of the internet’s search engines from crawling and indexing their website. This means that Reddit posts are no longer coming up in search results for users on these platforms.
There is one exception though where users can still search Reddit – and that’s Google.
Reddit is cracking down on AI bots
Which search engines are affected and why?
Searching Reddit now on popular search engines like Microsoft’s Bing and DuckDuckGo turn up scant recent results from Reddit.com. As 404 Media reports, more recent Reddit links aren’t turning up because they weren’t crawled and indexed before Reddit made the change.
A Mashable attempt to search Bing for any result from Reddit.com using “site:reddit.com” over the past week turns up zero results.
Mashable Light Speed
Reddit announced it was making a change to its Robots Exclusion Protocol (robots.txt) on its website on June 25. At the time, the company noted that it was making the change due to a surge of bots scraping its website. It did not seem from the statement at the time that this meant that organizations and platforms that legitimately help the broader internet and its users would be affected.
“Reddit believes in an open Internet, but not the misuse of public content,” Reddit’s protocol reads.
Blocking search engines entirely from Reddit would certainly be a controversial move on its own. However, there is one search engine that is pulling up recent search results from Reddit: Google, the search engine that has a financial relationship with the Reddit.
Google and Reddit’s relationship
Earlier this year, Reddit and Google entered into a $60 million deal that allows the search giant to use Reddit’s content to train its AI models.
In statements provided to multiple outlets, Reddit states that its recent changes aren’t related to its partnership with Google
“We block all crawlers that are unwilling to commit to not using crawl data for AI training, which is in line with enforcing our Public Content Policy and updated robots.txt file,” Reddit said in a statement provided to Engadget.
Engadget also claimed a source indicated that Reddit was excluded from search because Microsoft would not agree to the platform’s terms regarding AI.
Reddit’s move — if the speculation is true — seems fairly unprecedented, and highlights the hazards of a potential future where search engines become pay-to-play for even organic search results. Mashable has asked Reddit for comment, and will update this story if we find out more.
