Technical SEO for AI: Six Checks That Actually Matter

Direct Answer

Technical AEO is mostly disciplined web delivery: let the right crawler reach the right public content, return useful HTML without depending on client-side JavaScript, keep responses fast and stable, preserve clear page structure, make structured data truthful, and repair valuable broken destinations.

Do not begin by allowing every AI bot. Search retrieval, model training, and user-triggered browsing use different crawlers. Decide what you want each company to do with your content, then encode and verify that policy across robots.txt, the CDN, firewall, application, and server logs.

Watch: Technical SEO for AI

Credit: This guide is based on Ahrefs AEO Course lesson 3.4 with Sam Oh, the supplied transcript, and the current official crawler documentation linked below. The purpose-based crawler matrix, verification procedure, redirect safeguards, and audit worksheet are editorial additions.

1. Separate Search, Training, and User Fetches

The crawler name matters because the same provider may operate several bots for different purposes. A publisher can often permit search visibility while declining model training.

This distinction corrects a common simplification: blocking GPTBot, ClaudeBot, or Google-Extended is not automatically the same as hiding from each provider's live search product. Policies and bot names change, so link the decision record to the official documentation and assign an owner to review it.

2. Check One: Audit Every Access Layer

Review robots.txt, but do not stop there. A permissive robots file cannot override a CDN challenge, WAF block, rate limit, authentication wall, geofence, DNS failure, or application error.

Ahrefs' study of roughly 140 million websites found 5.89% explicitly blocking GPTBot. That describes a training-crawler preference; it does not tell us how many sites accidentally excluded ChatGPT search or whether every blocked site wanted visibility.

A Purpose-Based Example

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

This example expresses “search allowed, training declined” for two providers. It is not a universal recommendation. Check legal, licensing, privacy, contractual, and commercial requirements before publishing a policy.

Verify the Real Result

1. Open https://yourdomain.com/robots.txt and save the effective response, not only the repository file.
2. Check specific paths against each relevant user-agent group, including the catch-all group.
3. Review CDN, Cloudflare, WAF, bot-management, authentication, and rate-limit rules.
4. Confirm the crawler's published IP ranges where the provider supplies them; a user-agent string alone can be spoofed.
5. Inspect access logs for requests, status codes, response times, and repeated blocks.
6. Re-test a representative article, product page, documentation page, and media asset.

Cloudflare's current controls also separate policy from enforcement. Its managed robots.txt can express search=yes, ai-train=no, and a content-use preference, while AI Crawl Control can enforce blocks. Cloudflare notes that robots compliance itself is voluntary.

Where llms.txt Fits Today

llms.txt is optional reference material, not a replacement for crawl controls, sitemaps, internal links, or useful HTML. The community proposal places a Markdown file at the domain root to summarize the site and point to important resources.

Current evidence does not support prioritizing it for search visibility. Google states that Search, including AI Overviews and AI Mode, does not use llms.txt. Ahrefs studied 137,210 domains in May 2026: about 38,000 served a valid file, and 97% of those files received no requests. Even a request only proves that the file was fetched, not that it influenced an answer.

3. Check Two: Inspect the Raw HTML

Important content should exist in the initial HTML response. JavaScript-rendering support differs across crawlers and can change. CDN rules may also serve a bot a challenge or empty shell even when a browser works normally.

1. Fetch the page without executing JavaScript or disable JavaScript in a test browser.
2. Confirm the title, canonical, H1, primary copy, product facts, links, and image alternatives remain present.
3. Check that navigation uses crawlable links rather than click handlers alone.
4. Compare the raw response with the rendered DOM and identify business-critical differences.
5. Use server-side rendering, static generation, or robust progressive enhancement for essential content.
6. Repeat the test through the real CDN and production hostname.

A no-JavaScript view is a useful warning test, not proof that a particular AI crawler sees exactly the same page. Confirm with server logs and provider-specific access tests where available.

4. Check Three: Make Delivery Fast and Predictable

Optimize reliability before chasing an unverified “AI speed factor.” The lesson argues that slow real-time retrieval may be abandoned. That is plausible, but there is no universal published timeout or confirmed citation boost for a particular Core Web Vitals score.

The engineering target is still worthwhile: return a successful response quickly, keep HTML compact, avoid redirect chains, reduce intermittent 403, 429, and 5xx errors, and cache public content responsibly. These improvements help users, search crawlers, accessibility tools, and retrieval systems.

5. Check Four: Use Clear HTML and Internal Links

Semantic structure helps navigation, accessibility, maintenance, and passage extraction, but it is not an AEO trick. Use one descriptive H1, ordered H2 and H3 sections, focused paragraphs, lists and tables where they clarify the task, descriptive link text, and stable internal links.

Google explicitly says sites do not need to split content into tiny “AI chunks.” Write sections that are complete enough to understand independently because that also helps people scan and link to the right answer. Preserve nuance when the subject requires a longer explanation.

• Keep essential facts in visible text rather than images alone.
• Link important pages from ordinary HTML navigation and contextual copy.
• Use canonical URLs consistently and avoid duplicate parameter variants.
• Keep headings descriptive instead of stuffing exact-match phrases into every label.
• Publish an XML sitemap for discovery; keep its URLs canonical and successful.

6. Check Five: Make Structured Data Match the Page

Use schema for established search features and entity clarity, not as a guaranteed AI-citation lever. Google says no special schema is required for AI Overviews or AI Mode and warns against overfocusing on structured data.

1. Select a valid type that matches the visible page, such as Article, Product, Organization, or LocalBusiness.
2. Keep names, dates, authors, prices, availability, images, and URLs consistent with the visible content.
3. Validate the generated production page, not only the template.
4. Remove stale or unsupported properties after redesigns and product changes.
5. Do not add FAQ or HowTo markup merely because the page contains headings; eligibility and supported features change.

Structured data can make a page eligible for supported rich results. It does not repair weak content, blocked crawling, misleading claims, or inaccessible rendering.

7. Check Six: Recover Useful AI-Referred 404s

AI assistants send a small but measurable share of visitors to nonexistent URLs. Ahrefs analyzed 16 million cited URLs and found AI assistants referred users to 404 pages at 2.87 times Google's rate: approximately 0.43% versus 0.15%. ChatGPT's measured rate was 1.01%.

Some URLs may be truly invented; others were real pages that moved without a redirect. Diagnose before acting.

1. Filter analytics and server logs for AI referrers landing on 404 responses.
2. Group variants and measure visits, engagement attempts, and business relevance.
3. Check whether the URL previously existed and whether backlinks or citations still point to it.
4. Restore the page when the resource should exist, or issue a one-to-one 301 to the closest true replacement.
5. Keep a real 404 or 410 when no relevant replacement exists.
6. Update internal links, sitemaps, canonical tags, and source pages that still expose the old address.

Do not redirect every missing URL to the homepage. Irrelevant redirects confuse users, hide information architecture problems, and may be treated as soft 404s.

The 60-Minute Technical AEO Audit

Run Ahrefs Site Audit or another crawler to scale the checks, then verify the important findings directly. A tool's AI-bot warning is a prompt for a policy decision, not an instruction to allow every crawler.

Continue the Ahrefs AI SEO Course

These requested companion videos cover the wider search and AI strategy.

1. The New SEO Playbook for AI Search

2. The Google Update That's Changing SEO

3. Give Me 8 Minutes and You'll Win at SEO

Video Chapters

Sources and Further Reading

• Ahrefs: Technical SEO for AI (primary video and supplied transcript)
• Ahrefs Academy: AEO Course Lesson 3.4
• OpenAI: Overview of OpenAI Crawlers
• Anthropic: Web Crawler Controls
• Google: Googlebot and Google-Extended
• Google: AI Features and Your Website
• Cloudflare: Managed robots.txt and Content Signals
• Ahrefs: AI Bot Block Rates Across 140 Million Websites
• Ahrefs: 137K-Site llms.txt Study
• Ahrefs: 16 Million AI-Cited URLs and 404s

YouTube lists the primary video's publication date as 13 May 2026. This article and its crawler notes were reviewed on 27 September 2026. Bot names, provider policies, firewall products, AI retrieval behavior, and search features change. Verify the current official documentation before editing production access controls.