Category: SEO & Web Design

OpenAI starts testing SearchGPT prototype, here’s what it looks like

OpenAI today announced SearchGPT, its long-awaited search product.

SearchGPT is a “temporary prototype” and limited to just 10,000 users and publishers at launch.

OpenAI plans to integrate the “best” parts of the new search features directly within ChatGPT in the future.

How SearchGPT works. SearchGPT responds to queries using its AI models plus information pulled from the web. It will include links to relevant sources, OpenAI explained in a blog post.

OpenAI is teasing SearchGPT like this:

“A prototype of new search features, using the strength of our AI models to give you fast answers with clear and relevant sources.”

What SearchGPT looks like. It looks fairly similar to what it has been doing since the arrival of GPT-4o – what is known as “browse mode.” OpenAI shared some videos of the “faster and easier” SearchGPT experience:

It starts with a search box asking you, “What are you searching for.”

After entering your query, SearchGPT will provide an answer that includes links to sources within the text answers and shows sources beneath the answer.

Additional results will be shown in a sidebar.

Here’s a search for [music festivals in Boone, NC in August 2024]:

Here’s a search for [best tomatoes to grow in Minnesota]:

For publishers. OpenAI is “launching a way for publishers to manage how they appear in SearchGPT, so publishers have more choices. Importantly, SearchGPT is about search and is separate from training OpenAI’s generative AI foundation models. Sites can be surfaced in search results even if they opt out of generative AI training.”

Coming soon. OpenAI said it plans to improve searches related to local information and commerce.

Waitlist. You can join the SearchGPT waitlist here.

The OpenAI search story so far. We first heard about OpenAI’s plan for a search product in February. Rumors heated up in May – but OpenAI didn’t launch its search product (GPT-4o launched instead). Then, earlier this month, The Atlantic CEO confirmed that OpenAI was “going to build a search product.”

Why we care. The next Google won’t do what Google does, as ex-Google CEO Eric Schmidt once said. OpenAI CEO Sam Altman has made it clear he has no interest in copying Google Search – but creating a new user-friendly search that combines LLMs and search without drowning you in advertising. While it’s too early to know whether ChatGPT will become a Google Search killer, this is clearly a story we’ll be watching closely over the coming months and years.

SearchGPT is a temporary prototype. ChatGPT plans to integrate search features from SearchGPT into ChatGPT in the future.

Read More

How to manage SEO campaigns during economic downturns

Economic downturns create uncertainty and changes in user behavior that impact markets and forecasts. They also affect the approach and resources allocated to marketing activities.

Because no two recessions or economic downsturns are the same, marketers operate in uncharted waters every time one occurs.

This article can help marketers manage SEO campaigns and show the value of their efforts despite a recession, during periods of market uncertainty, or during periods when external factors greatly change your target consumer’s behaviors.

Doing business during a recession

Consumers will naturally reduce their spending during economic downtimes and establish more stringent priorities. 

When sales decline, businesses begin to reduce expenses, lower prices and delay making new investments. Unfortunately, marketing expenditures are often the first to get cut. This approach to cost reduction is ineffective and should be avoided.

Clients often judge SEO and paid search as independent channels. SEO can be seen as “OK to turn off for a bit,” while a brand rests on its laurels, assuming that the current performance won’t worsen during the downturn.

But SEO isn’t the only channel that may see issues here. In 2009, the entire U.S. ad market saw a decline of 13%. This was predominantly driven by radio and magazines, which saw declines of 22% and 18%, while “online” only declined by 2%.

The argument for not reducing spending

It might be a good idea to keep costs down when entering a recession. But if you fail to support brands or examine how your core customers’ needs evolve, you will likely jeopardize your medium to longer-term performance.

Organizations and trade bodies such as the Institute of Practitioners in Advertising (IPA) refer to this as losing the “share of mind.” In the United Kingdom, the IPA has publicly advertised, warning brands not to reduce marketing spend in the months ahead.

It’s also worth highlighting that sometimes a plateau or deceleration in marketing spend can feel regressive.

In 2024, it is estimated that overall advertising spending growth in the U.S. will increase by approximately 10%. Compare this to an average growth of 23.3% prior to 2020.

These year-on-year spending patterns can be down to various reasons, including:

Market saturation: The level of investment has reached a point in which further investment would yield diminishing returns against the current consumer market.

Pricing pressures: As consumers go through times of economic uncertainty, consumer confidence and priorities in spending disposable income change. As a result, organizations look to maintain their advertising/marketing effectiveness without increasing spend.

Market consolidation: As consumers reduce spend, smaller organizations or those unable to compete effectively may exit certain markets. This often leads to fewer but stronger organizations still in the market, making it a more competitive environment with less competition for mental availability.

Reading through research papers and thought leadership content from the International Journal of Business and Social Science, Harvard Business Review and specialists from the Economist and Financial Times, five key marketing objectives for a recession begin to emerge.

Smarter spending and investment.

Retention of the existing customer base should be prioritized.

Leverage your competitor’s weaknesses.

Monitor the market and adjust spend targeting segments based on their behaviors.

Maintain your current spending at a minimum, just spend smarter.

Dig deeper: SEO for user activation, retention and community

How to keep your SEO efforts going during a downturn

The general message is “do not reduce spending.” That’s great. 

But when defending our retainer or contract renewal, businesses want to know how their SEO spend will tangibly impact the bottom line.

To address this, we can look to the 2008 recession and the recent pandemic – when other businesses pivoted and changed messaging across their digital marketing. 

Let’s apply the learnings to help our clients or employers power through an economic downturn without pausing their SEO efforts. 

Review your TAM and messaging

A total addressable market, or TAM, can be defined in several ways. 

The most common definition is the total number of people who could possibly use a product or service. For example, the TAM for a new smartphone might be the total number of people who own a cell phone.

Despite its limitations, TAM can be a helpful metric for investors to assess a company’s growth potential. Companies with large TAMs can be desirable to investors because they have the potential to generate a lot of revenue.

During a recession, businesses (in B2B) and consumers will react differently depending on their economic stability.

Depending on your TAM, you may need to pivot your messaging and value propositions. This then ties into your SEO strategy. Align activities to these messaging goals, depending on whether your product is deemed essential, luxury, postponable or expendable.

Essential products are often price-sensitive during a downturn. You may want to highlight the value proposition further for less economically stable consumers.

For those in your TAM who are better off, you should continue awareness campaigns (i.e., top and middle-of-the-funnel activities).

By comparison, luxury products can be communicated as precisely that – a luxury/treat to be consumed as a reward for austerity in other areas. They can also induce dopamine reactions and raise morale.

The other two categories, postponables and expendables, are the most difficult to pivot for.

An example of a postponable is a TV streaming service or magazine subscription. Users ahead of postponing may research cheaper alternatives to avoid missing out. During this research phase, you must be visible and fight to retain your existing users. 

For other goods that can be postponed (such as servicing a vehicle, replacing a tire, or updating home security systems), messaging should focus on the long-term financial and opportunity costs of not performing these actions now and providing support messaging.

Expendable products and services will likely impact local SEO more than other sectors. Rather than hire a gardener or decorator, consumers will choose to perform the maintenance and upgrades themselves.

This is both an opportunity and a threat to sell to consumers by enabling them or working to remain visible as a company and build trust.

To do this, you need to understand the overall confidence of your target consumers.

Get the newsletter search marketers rely on.

Business email address

Sign me up!
 Processing…

See terms.

Understanding your consumer confidence

In addition to reviewing your TAM, you need to review the confidence of consumers within your SOM (serviceable obtainable marketing) and SAM (serviceable addressable market).

Consumer confidence is a measure that gauges the optimism of households and how they feel about their financial stability at present and in the coming months.

Two common sources of this information at a macro level are the Michigan Consumer Sentiment Index (MCSI) and the Consumer Confidence Index (CCI).

When consumers are confident, disposable income is more likely to be spent on luxury, postponable and expendable products/services. When it’s low, there is a stronger focus on essential products/services.

You can collect your own data for this through surveys and engagement with your own community.

Talk to your sales team regularly. They speak with customers and potential buyers often. This can give you valuable insights into:

What frustrates your target audience.

What objections they have.

How they view their own finances.

Their perspective on your market.

These insights can help you understand your customers better, even if you can’t create your own economic index.

Smarter opportunity analysis and competitor targeting

Most businesses are focused on maintaining and retaining market position during a recession.

It’s an excellent time to identify consumers they currently hold and work to leverage them toward your products and services. Competitive targeting should be a staple of an SEO campaign anyway.

However, during a downturn, when sensitivity to price and value is heightened, your messaging and content can focus on pain points that consumers may have with competitor products and services.

Turn these into a competitive advantage to create a conversation with new prospects. You can produce competitor comparison content and highlight the competitor’s weak spots as non-issues or strengths with your product.

For example, if you’re providing a rotating proxy service and you know that your competitor, Bob’s Proxies, has issues with uptime, then make sure your content highlights that your service has no such problems.

Dig deeper: SEO SWOT analysis: How to optimize where it counts

Positioning for post-recession

SEO is a long-term strategy, but during an economic downturn, you must balance the longer term and the short term.

Typically, consumer trust and spending recover within one to two years of a recession. When consumers return to post-downturn spending levels (or establish new market norms), you want to ensure you’re prominent and visible in the vertical.

You can do so by maintaining a certain level of activity toward establishing and maintaining top-of-vertical awareness and remaining competitive for bottom-of-funnel, conversion-focused queries.

Maintaining SEO momentum during uncertain economic climates

Marketing may seem more challenging during an economic downturn. Customers’ spending habits often change, and you may have to go against your instincts.

Optimizing your budget and being strategic about your priorities is essential. This will allow you to continue marketing your products or services while meeting your customers’ needs.

A recession can be an opportunity to build customer loyalty and mental availability with your overall SAM and TAM. During a downturn, SEO can alleviate direct cost channels (such as paid) and offer long-term benefits and short-term stability.

Google and the other search engines will continuously update during this period. Competitors who remain stagnant and withdraw resources will suffer in the medium to long term, costing more in the future to regain lost performance and the opportunity cost of lost visibility.

Keep your marketing strong in tough economic times with insights on consumer behavior, market analysis and long-term positioning.

Read More

How to START building a modern digital marketing plan

It doesn’t take a lot of sentences to cover the number of disruptions in search marketing in the season (or era?) that we’re in.

Right now, Google is rapidly changing organic and paid search, causing excitement and confusion. New AI Overviews, leaked information and potential OpenAI search engines are creating distractions in our careers. Additionally, changing privacy laws and attribution challenges add to the complexity.

With so much happening, it can be overwhelming to keep up. It’s crucial to have a clear plan linked to your strategy so you don’t get distracted by new trends like AI or ignore changes and fall behind. While “plan” may sound boring, it’s essential for a solid digital strategy.

To help with the topic, I recently published my first book, “The Digital Marketing Success Plan.” In it, I go into granular detail with personal stories, processes and “how to” content centered on what I call the “START” planning process, and I will unpack those five steps to follow.

1. Strategy

The first phase of START planning is strategy. This is where you understand the business’s current state and what goals to reasonably set and achieve with digital marketing. 

Here, you’ll tackle everything, including setting goals, mapping marketing metrics to business outcomes and reviewing all past marketing efforts and our expectations before proceeding to the subsequent phases of the planning process.

In some companies, this step is completed with exhaustive research, business intelligence data and financial information backed up end-to-end through CRM or other platforms.

In other cases, businesses started adding digital marketing channels to the mix over the years (with good intentions) but don’t have all the dots connected for attribution and ROI.

Regardless of where your company is on the scale, there’s never a better time to ensure that all levels, teams and stakeholders are aligned on the end goals for digital marketing (and maybe marketing more broadly).

Then, you can work backward from there when you explore all possible tactics and ways to achieve your defined goals and strategy overall.

2. Tactics

The next phase is tactics. This is where you explore all potential avenues to reach goals, map out business outcomes and be open-minded about how to achieve them. 

You’ll evaluate all potential channels and networks (e.g., SEO, Google Ads, LinkedIn sponsored content) you can leverage using leading research and estimation tools to uncover opportunities to reach our target audiences.

This is how the strategy comes to life. You never want tactics to “be” the strategy or be more important than the strategy. These are the actions that drive us toward meeting and exceeding our goals.

You also have to consider your own biases toward certain channels based on your experiences, expertise or personal feelings about them. 

In some cases, LinkedIn may beat Google Ads. Or, programmatic display is required to support the customer journey in a hidden way you don’t realize.

Be open to exploring all channels and opportunities, and do the necessary research and projections to ensure they align with your strategy and goals.

3. Application

The third phase of the START planning approach is application. This is the time to map out the assets, large and small, that you need to get our house in order – or to build it.

Develop the inventory list you’ll need for all desired or defined tactics. This includes creating ads, graphics, webpages, copy and other necessary assets for a successful campaign.

At this point, it is tempting to start creating new ads or landing pages or examining your creative with a defined strategy and set of tactics. 

I will warn you to stick with identifying things at this stage and not start creating new things. This is where many great strategies get stalled out, and plans don’t come to fruition.

Getting the assets you need to launch or amplify your plan requires time and money. Suppose you dive right into designing or creating new things without properly identifying what you’re doing and connecting back to the “why” (Strategy and Tactics phases). In that case, you can do some expensive things quickly or push out the launch of your plan indefinitely.

Get the newsletter search marketers rely on.

Business email address

Sign me up!
 Processing…

See terms.

4. Review

The fourth phase is review. Here, you’ll define the aspects needed to manage and measure the successes and provide accountability for the plan. 

That means creating performance measurement dashboards and systems tied to the ultimate ROI equation, roll-up performance, channel performance and resource measurement. 

At this phase, you’re also setting up our project management system or aligning resource planning platforms to manage the work. Ultimately, you’re arriving at the benchmarks, budgets, reports and roles overall.

This phase may seem obvious, but I have seen a number of systems over the years, including dashboards and resource managers. Some are totally custom, and others are right out of the box. 

No matter what, they often miss out on tracking a full picture of digital marketing ROI and leave gaps that have to be filled with assumptions or cross-checking investment numbers by a CFO.

You don’t want gaps and want to make sure that our systems account for the full cost of digital marketing. That includes internal people, contractors, agencies, software, advertising spends and possibly more. 

If you aren’t accounting for those, then you could have rough meetings with the C-suite or investors down the road when they don’t see the full ROI that you might be calculating.

5. Transformation

The last phase of START planning is transformation. In this final phase, you schedule the tactics so they stay subordinate to the overall strategy.

You’ll craft a comprehensive calendar of tactics, optimization processes, agile strategy and reporting cycles. You will also map out the milestones, flights and content pushes, experiments, tasks and assignments overall.

Again, you might have a long history of SEO, Google Ads, social media marketing, email marketing, programmatic and more. You might have some great success stories in them or even with integrated, cohesive results. I love hearing those stories!

However, I would challenge you to consider that if you’re seeing positive results, they have likely come from focused efforts and not sporadic drop-ins to manage and see it through.

Regardless of your position, make sure you have resources planned, tactics and action plans scheduled and control of the plan.

Things will happen in your company, like new products/services, new audience focuses and new competitors. Things will also happen, such as new industry tech, changes in search engines, AI emergence and more.

Have a documented plan that also allows for agility and enough flexibility to be revisited and adapted at multiple levels so it doesn’t get obsolete or abandoned just a few months into the effort.

START your way to digital marketing success

Whether you follow the steps in START planning or adapt them to your own strategy and planning process, I strongly encourage you to revisit your current plan. Make sure that it is objective, specific, actionable and accountable.

A plan isn’t a plan if it is in one person’s head or if it is understood differently by multiple people. Things are changing really fast. Have a plan that can keep you on track, adaptable and up to date with changes.

My key goal is to ensure that you make wise, ROI-driven and accountable investments, whether you’re a business doing digital marketing or an agency doing it on behalf of your clients. 

Google, software, agencies and employees will get paid or take your money whether your digital marketing is working or not, and I hope that you see success in your digital marketing driven by a solid plan.

Get the newsletter search marketers rely on.

Business email address

Sign me up!
 Processing…

See terms.

Learn how to craft a robust digital marketing plan using the START framework to stay focused and adaptable.

Read More

Brand Protection: The complete guide by Adthena

The strongest paid search strategy builds on your brand’s unique value. Here, you control the customer experience and messaging, driving traffic that converts from search, via Google Ads. 

Your brand terms, message and unique selling points are yours to leverage – and in paid search, they should be your safe haven.

But with fierce competition, protecting your brand in paid search is crucial. Let’s unlock the secrets of brand protection, safeguard your reputation and optimize your campaigns for success.

What is brand protection?

Your brand deserves to be in control. But competitors, resellers and affiliates can exploit your brand for their own gain in paid search. How?

Keyword bidding: They might bid on your brand terms, causing your ads to drop in search results.

Trademark misuse: They could use your brand name or trademarks in their ads, misleading customers and diverting traffic.

Brand protection is your shield. It safeguards your online presence and reputation by:

Keeping your brand front and center in search engines.

Protecting hard-earned brand equity.

Ensuring your customers find you first.

Why brand protection is important

Brand protection is an essential part of paid search. it helps safeguard your brand from competitors, affiliates and resellers capitalizing on your brand’s equity, potentially damaging its reputation and driving up costs.

Trademark infringements divert as much as 21.5% of a typical brand’s clickshare to competitors. They’re stealing clicks that are rightfully yours.

Search is a critical part of the funnel and can be the single point of failure for your entire investment if you let rivals steal demand.

One lost customer (and associated data) can snowball into tens of thousands of dollars over that customer’s lifetime.

*Adthena industry average data taken from 450+ businesses’ text ads across the EU, U.S. and Australia. Includes brand terms and brand generic terms.

Protect your brand from affiliate ad-hijacking

Affiliate ad-hijacking is the process of affiliates creating fake ads that look like yours, using your brand name and targeting relevant search queries. Traffic is funneled to your website using UTM parameters, giving them credit for the sale, even though the customer landed on your site. This way, they essentially steal commissions unrightfully.

What are the risks of affiliative ad-hijacking?

Ad hijacking in paid search can be a real headache for businesses, leading to several risks and consequences:

Wasted ad spend: Hijackers might outbid you on your own brand keywords, forcing you to pay more just to appear in search results for your own brand name. This eats away at your budget and reduces the effectiveness of your campaigns.

Lost traffic and sales: If hijackers’ ads are displayed instead of yours, potential customers could be directed to the wrong website. This means lost traffic, leads and sales.

Brand damage and confusion: Hijackers might use misleading ad copy or even impersonate your brand entirely. This can confuse customers and damage your brand reputation.

Channel conflict: If unauthorized affiliates are hijacking your brand terms, it can create conflict with your other marketing channels.  Customers might get mixed messages or have a negative experience, hindering your overall marketing efforts.

Reduced click-through rates (CTR): Even if your ads appear, hijackers might use tactics that make their ads more appealing, leading to a lower click-through rate for your legitimate ads.

Erosion of brand trust: Repeated exposure to hijacked ads can make customers wary of clicking on your ads in the future, impacting brand trust.

Strategies to protect your brand from Ad-hijacking

Set boundaries: Agree to codes of conduct that specify what terms your partners can use and how you want your brand to be represented. Include search terms you don’t want to be associated with and consider alternative terms of value to whitelist so you’re not competing for the same clicks.

Prove it and don’t over pay: Make sure you’re only paying for traffic that you wouldn’t win otherwise. Capture evidence to enforce your contractual terms.

Use Adthena’s Ad Hijacking Detection solution: Monitor and report thousands of instances of affiliate ad hijacks across hundreds of affiliate networks with Adthena’s Ad Hijacking Detection. Automatically detect when an affiliate impersonates your ad and get evidence of fraud including ad screenshot and affiliate ID.

Take an interactive platform tour of Adthena Ad Hi-Jacker Detection

Protect your brand from trademark infringements in paid search

Trademark infringement in paid search happens when a competitor or another advertiser uses your legally protected trademarks in a way that’s likely to confuse customers about the source of a product or service. Here’s a breakdown of the key points:

Trademarks: These are unique words, symbols or phrases that identify the source of your goods or services. They help distinguish your brand from competitors.

Using your trademark: In paid search, infringement can occur when someone uses your trademark in several ways:

Bidding on your brand terms: This involves using your brand name or related keywords in their ad campaigns. While some platforms allow bidding on competitor terms, the focus shouldn’t be on misleading customers.

Trademark in ad copy: They might directly include your trademark within the ad text itself, creating the impression that their product or service is affiliated with yours.

It’s important to note that the laws around trademark infringement can vary depending on your location. However, most search engine platforms have policies in place to prevent advertisers from misleading users through trademark misuse.

What are the risks and consequences of trademark infringement in paid search?

Infringement carries significant risks for both the infringer and the brand whose trademark is being misused:

Legal trouble: Trademark infringement is a serious offense. If another advertiser misuses your trademark and you take legal action, they could face significant financial penalties and be forced to stop using your trademark.

Brand damage: Confusion caused by trademark infringement can erode your brand reputation. Customers who click on a misleading ad might feel frustrated or deceived, potentially damaging your brand’s trust.

Wasted resources: If customers are directed to a competitor’s site due to trademark infringement, you lose valuable leads and potential sales. This can negatively impact your marketing ROI.

Account suspension: Most search engine platforms have strict policies against trademark infringement. If you’re found to be infringing on another brand’s trademark, your ad account could be suspended, hindering your online marketing efforts.

Negative customer perception: Even if you aren’t the one infringing, being associated with such tactics can paint a negative picture of your brand to potential customers.

Reputational hit: Engaging in trademark infringement yourself can seriously damage your brand’s reputation. It can portray you as unethical and untrustworthy.

Strategies to protect your brand from trademark infringement

Don’t get sucked into bidding wars on your own search terms. The most effective long-term strategy against competitors infringing on your brand is to harness the power of search intelligence to understand their Google Ads strategies on a granular and continual basis. Then you can take action with Adthena’s Infringement Tracker.

Monitor bids and infringements on your branded search terms:

Make informed budget decisions on your brand terms.

Proactively monitor your entire digital brand landscape.

Prevent partners and affiliates from cannibalizing clicks.

Spot infringements and protect your brand value.

Take an interactive platform tour of Adthena’s Infringement Tracker 

Map and monitor competitive and partner activity around your brand terms

Search marketers experience many challenges when it comes to controlling the use of their trademarks. The two themes we hear most often:

“Trying to control the use of trademarks is tedious and time-consuming.” 

“Finding real-life trademark infringement examples is too difficult to do manually.” 

Manually collecting and processing the trademark violation data required to submit Google complaints properly can take up to eight hours each month. 

By automating aggregation and submission of infringements across devices at scale, you can confidently protect your brand and save time and effort. 

Take control with Adthena’s Auto Takedown

Adthena’s Auto Takedown allows you to map and monitor all competitive and partner activity around your brand terms. This lets you efficiently respond to trademark infringements and report examples directly to Google for removal. 

With the simple push of a button, you can:

Automatically monitor your brand trademark search terms at scale, 24/7.

Automatically submit evidence of trademark violations directly to Google. Adthena will also keep track of the responses from Google.

Save up to eight hours per month in manual monitoring

Measure the success of your brand protection strategy 

Now that you’ve implemented your brand protection strategy, it’s crucial to track its effectiveness. By tracking these metrics and focusing on both preventative and reactive measures, you can effectively gauge the success of your brand protection strategy in paid search and ensure your brand maintains control over its online presence.

Reduced infringement:

Manually track brand mentions in search results for potential trademark misuse. A decrease shows success.

Use automated tools like Adthena’s Infringement Tracker to detect infringement alerts. Fewer alerts indicate a winning strategy.

Metrics showing control:

Look for increasing brand search share, pushing out hijackers.

Maintain a healthy click-through rate for your brand terms.

Ideally, your cost-per-click for brand terms should decrease as competition lessens.

Indirect benefits:

A strong brand protection strategy can enhance brand image and awareness.

Prevent hijacking to improve conversion rates for your paid search campaigns.

Key considerations:

Integrate search engine and website data to identify user behavior discrepancies.

Brand protection is continuous, so regularly monitor metrics and adapt your strategy.

Focus on preventing infringements, not just reacting to them.

Protect your PPC brand terms with Adthena’s Brand Protection solution

Take a quick platform tour of Adthena’s award-winning Brand Protection solution and see how you could safeguard your brand.

Get started now!

Discover essential tactics to catch click theft, protect brand integrity and maximize ROI.

Read More

Microsoft confirms Reddit blocked Bing Search

Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site. “Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site,” a Microsoft representative told Search Engine Land.

What happened. On July 1, 2024, Reddit updated its robots.txt file preventing many search engines and AI tools from crawling the site. Reddit did not prevent Google from crawling the site, despite what some people may have thought earlier this month. But Reddit did prevent most other crawlers from crawling the site.

Earlier this morning, Mark Williams-Cook notified me that Reddit results were dropping out of the Bing Search index. Then several media outlets began covering the news. I wanted to confirm that Bing’s crawlers were indeed blocked, because Reddit was using IP detection to show search engines one version of its robots.txt file and humans another version of the robots.txt file – as I explained earlier this month.

Bing has thus stopped crawling new content on Reddit, which is why when you filter Reddit results in Bing Search for the last week, you see nothing:

Microsoft confirmed. A Microsoft spokesperson told Search Engine Land:

“Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”

Reddit statement. Reddit spokesperson Tim Rathschmidt says in a statement to The Verge:

“This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”

Why we care. With Reddit securing a licensing deal with Google, Reddit is able to play hardball with other search engines and AI tools. So Reddit has blocked most other search engines from crawling its content. Meanwhile, Google is driving insane traffic to Reddit these days, including testing special treatment in its search results for them.

It makes you wonder if other large websites can try to go down this route and where that might leave smaller publishers and content producers.

Meanwhile, don’t expect to see much new Reddit content being surfaced on Bing in the near future.

Reddit updated its robots.txt file prohibiting Bing and some other search engines from crawling its site.

Read More