The Pulse: Navigating the Shifting Landscape of Search, AI Crawling, and Technical SEO

In an era where the boundary between traditional search indexing and generative AI training continues to blur, staying ahead of technical requirements is more critical than ever. This week’s developments from Google and OpenAI underscore a period of intense evolution in how machines interact with the web. From server maintenance best practices to the emerging role of AI agents, site owners are being forced to refine their understanding of robots.txt, URL structure, and image rendering.

This edition of The Pulse breaks down the most impactful updates, providing the technical context necessary to protect your site’s indexability and content sovereignty.

Google’s Mandate: Mastering the 503 Status Code

Protecting Your Index During PHP Patching

On October 21, Google Search Central issued a timely reminder to site administrators: when performing server maintenance—particularly the urgent patching of PHP vulnerabilities—the only acceptable response to search engines is the HTTP 503 (Service Unavailable) status code.

Key Facts and Technical Context

The urgency of this directive coincided with the release of critical security patches for the PHP core, which affected a vast ecosystem of websites, including those running on WordPress. Google’s guidance is explicit: a 503 status code informs search crawlers that the site’s absence is temporary. This ensures that Google preserves your existing indexed URLs rather than misinterpreting a server error as a permanent site removal.

Google specifically warns against the use of "soft" 404s or 200 OK responses during downtime. According to Google’s crawl-rate documentation, the 503 status should ideally be served for no longer than 1-2 days. If a site remains in a 503 state for an extended period, Google may interpret the signal as a permanent departure and subsequently purge the URLs from its index.

Official Guidance

John Mueller, Google’s Search Advocate, clarified on Bluesky that the 503 response does not necessarily need to be triggered by the PHP environment itself. In fact, if your PHP is down, it cannot be triggered by PHP. Administrators should instead configure these responses at the server level (e.g., Nginx or Apache), through a Content Delivery Network (CDN) routing rule, or via a temporary maintenance server. Importantly, Google’s documentation on "pausing" a site warns that if you trigger a 503 in response to robots.txt requests, all crawling will cease immediately, potentially hindering the re-indexing process once the site returns.

The Robots.txt Frontier: Crawling vs. Content Usage

Clarifying the Scope of Access Preferences

One of the most persistent misconceptions in SEO is the idea that robots.txt acts as a catch-all guard for how content is utilized after it has been retrieved. Gary Illyes of Google addressed this in a recent Search Central video, emphasizing that, per RFC 9309, robots.txt is strictly an access preference protocol.

The Limitation of Standard Directives

Illyes clarified that robots.txt directives only govern what a compliant crawler is permitted to access. It does not dictate how a company, such as Google or OpenAI, uses that data post-crawl—for instance, in training a Large Language Model (LLM).

To restrict specific AI training, site owners must target the specific user-agent tokens defined by each platform. For Google, the Google-Extended token is the primary mechanism to opt out of content usage for Gemini model training and grounding in Vertex AI. It is vital to note that Google-Extended is not a separate crawler; it is a policy token. Blocking it does not stop Google from indexing your site for traditional Search results.

The Path Forward

While the industry currently relies on these fragmented, company-specific tokens, there is movement toward a more standardized solution. A draft proposal currently before the Internet Engineering Task Force (IETF), co-authored by Illyes, seeks to introduce a "Content-Usage" rule to the robots.txt standard. Until this becomes an industry-wide protocol, site owners must maintain vigilance over the evolving list of crawler tokens for every AI entity they wish to regulate.

OpenAI and Cloudflare: A Pilot Program for Discovery

Implications for ChatGPT Search

In mid-October, a joint announcement from Cloudflare and OpenAI revealed that a research pilot had successfully helped the OpenAI crawler discover and index hundreds of millions of new web pages. This initiative leveraged signals from approximately 1% of Cloudflare’s network to help the OAI-SearchBot identify active, updated content while bypassing dead links.

Implications for Site Owners

While this discovery process significantly expands the reach of ChatGPT Search, the companies emphasized a critical distinction: discovery does not equal permission. The blog post accompanying the pilot explicitly stated that a signal about a page’s existence does not grant the crawler access to the page itself.

Site owners must remain cognizant of the distinction between the two primary OpenAI crawlers:

1. OAI-SearchBot: Governs inclusion in ChatGPT’s search features.
2. GPTBot: Governs content usage for training purposes.

If you wish to allow your site to appear in ChatGPT’s search results but prevent it from training their models, your robots.txt should reflect an Allow for OAI-SearchBot and a Disallow for GPTBot.

URL Integrity: Raw vs. Encoded Brackets

John Mueller’s Correction on URL Structure

In a candid admission on Bluesky, John Mueller corrected a long-standing point of confusion regarding URL structure: raw brackets and their percent-encoded counterparts (%5B and %5D) are, in the eyes of Google, distinct URLs.

The Technical Reality

Mueller noted that he had previously provided incorrect guidance on this topic. Citing RFC 3986, he explained that because these characters are "reserved," they must be treated as unique identifiers. If a website returns identical content for both the raw and encoded versions of a URL, Google will attempt to crawl both, identify them as duplicates, and canonicalize them to a single version.

This "cleanup" process is inefficient and relies on Google’s ability to detect the duplicate content correctly. Mueller’s advice to webmasters is simple: "Pick one." Consistency across all internal links, sitemap entries, structured data, and canonical tags is the only way to ensure your site architecture is optimized for search efficiency.

The JPEG XL Dilemma

Image Optimization and Search Compatibility

Despite the growing adoption of the JPEG XL format in browsers like Chrome and Firefox, Google Images and Discover previews have yet to integrate support for the format.

Why This Matters for Image SEO

Google’s official image SEO documentation maintains a strict list of supported formats: BMP, GIF, JPEG, PNG, WebP, SVG, and AVIF. If your site serves images in JPEG XL, those images will not appear in Google Image search results or as rich media previews in the Discover feed.

For developers looking to leverage the superior compression of JPEG XL, the recommended approach is the use of the HTML <picture> element. By placing a JPEG or WebP file within the <img> fallback, you ensure that search engines can still ingest the image while human users with modern browsers receive the more efficient JPEG XL version.

The Future: Sundar Pichai on AI Agents

Browsing vs. Protocols

Looking toward the horizon, Google CEO Sundar Pichai recently discussed the evolution of AI agents in an interview on the Sources podcast. Pichai predicts a two-pronged future for how AI interacts with the web:

1. Human-like Browsing: Agents will continue to "browse" the web, interacting with pages much like a human user.
2. Agentic Protocols: The emergence of specialized protocols—such as the Universal Commerce Protocol—which will allow agents to interact with site backends directly.

Pichai views protocols as the more scalable, long-term solution, though he admits they are still in their infancy. Currently, Google’s "user-triggered fetchers" (agents that act on behalf of a specific user request) often operate outside the constraints of traditional robots.txt rules. As these agents become more prevalent, the industry will likely see a push for new standards that allow site owners to safely facilitate agentic commerce and data retrieval without compromising security or user experience.

Summary: A Theme of Signal Management

The overarching theme of this week’s developments is the importance of signal management. Every technical decision—from the status code you return during a server outage to the specific user-agent you block in your robots.txt—is a signal to the machine-driven web.

• 503s preserve your place in the index during downtime.
• Robot tokens protect your content from unintended AI training.
• URL consistency ensures your site architecture isn’t being "deduplicated" by search algorithms.
• Image fallbacks ensure your content remains visible in a fragmented browser ecosystem.

As the web moves from a human-browsed experience to an agent-intermediated one, the accuracy and clarity of these technical signals will define which websites remain authoritative and which fade into the background of a training dataset. Stay vigilant, audit your configurations, and ensure your site speaks the language of the modern crawler.