Free audit
Menu

16 of 67 well-known websites block AI crawlers

Free SEO Audit & AI Visibility Checker

ChatGPT, Perplexity and Google decide in seconds whether to quote your site. Run 49 free checks on technical SEO, content, AI search visibility (GEO), answer engines (AEO) and Core Web Vitals, and get a fix list sorted by impact.

AI visibility here means AI search crawlers can reach and read your pages. We do not ask ChatGPT or Perplexity whether they mention your brand.

Free. No account. Results in under a minute.

A real audit of instagram.com, from .

Overall score: grade F, 54 out of 100

instagram.com scores 54. What about yours?

  • Technical SEO: grade C, 72 out of 100
  • Content and E-E-A-T: grade D, 69 out of 100
  • Answer engines (AEO): grade F, 17 out of 100
  • AI readiness (GEO): grade D, 60 out of 100
  • Speed: grade F, 50 out of 100

What we found on well-known websites

  • 16 of 67

    well-known websites we could audit block AI crawlers in robots.txt

  • 23 of 51

    of the sites whose real page we could read have no working llms.txt

  • 75/100

    is their average ratemysite.ai score

Scores count only sites whose real page we could read. Updated . See the ranking

Instant SEO audit

See what holds your site back

The audit runs 49 checks on technical SEO, content and speed. Every problem comes with a priority and a step by step fix guide.

Example recommendations

  • Title tag lengthHigh
  • Meta descriptionMedium
  • Image alt textLow

Example passes

  • Indexable (no noindex)
  • robots.txt
  • XML sitemap
Example Technical SEO score: grade B+, 89 out of 100

Technical SEO

Example score

AI visibility

Make your site easy for AI to cite

AI assistants must reach your site before they can cite it. GEO is a set of practices that make a site easy for AI assistants to crawl, understand and cite.

robots.txt: what we look for

  • User-agent: OAI-SearchBot allowed
  • User-agent: Claude-SearchBot allowed
  • User-agent: PerplexityBot allowed

llms.txt

A plain Markdown map of your site for AI assistants.

Speed

Core Web Vitals, measured

Speed data comes from Google PageSpeed Insights. Core Web Vitals are a set of Google metrics for loading, responsiveness and stability.

Core Web Vitals: pass thresholds

  • Largest Contentful Paintgood under 2.5 s
  • Interaction to Next Paintgood under 200 ms
  • Cumulative Layout Shiftgood under 0.1

Source

Google PageSpeed Insights, field data when available.

Check your site

Your report, step by step

  • Step 1

    One score, five colors

    Your grade, phone and desktop screenshots and a radar of the five categories.

  • Step 2

    What to fix first

    Every failed or warned check, sorted by impact, each linked to its fix.

  • Step 3

    Save it, share it

    Save as PDF, share the link, compare with a competitor, get score alerts and a site badge.

Open the full report
Check your site

What does the SEO audit check?

Made for the people who run websites

  • Business owners

    A plain grade and a short list of what to fix first, in everyday words.

  • Marketers

    See if Google and AI assistants can find, read and quote your pages, then compare with a competitor.

  • Developers

    The evidence behind every check, PageSpeed data for phone and desktop, and a Markdown version of each report.

More free tools

See all 9 free tools

How the audit works

  • 1

    Enter your URL

    Any public page. No account, nothing to install.

  • 2

    We run 49 checks

    SEO, content, answer engines, AI readiness and speed, in under a minute.

  • 3

    Fix what matters

    A list sorted by impact, with a guide for every problem.

Frequently asked questions

What does ratemysite.ai check?

It runs 49 checks across technical SEO, content quality, answer engine optimization (AEO), AI search readiness (GEO) and Core Web Vitals, then scores the page from 0 to 100 and lists the fixes in priority order.

Is it really free?

Yes. Audits are free and need no account. You can run 10 audits per hour.

What is GEO?

GEO is generative engine optimization: making a site easy for AI assistants such as ChatGPT, Claude and Perplexity to crawl, understand and cite. It covers crawler access, llms.txt, server-rendered content and clear brand data.

Does it check whether ChatGPT mentions my brand?

No. It checks whether AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot can reach your pages, and whether the content is easy to read, quote and cite. It does not ask ChatGPT or other assistants what they say about your brand.

What is AEO?

AEO is answer engine optimization: writing pages so that search engines and assistants can lift a direct answer from them, with FAQ and HowTo structured data, question headings and a short definition near the top.

Does the audit run JavaScript?

No. It reads the HTML your server sends, which is also what most AI crawlers see. If your content only appears after JavaScript runs, the audit will tell you.

How often can I re-run an audit?

A report is reused for 24 hours. After that you can re-analyze from the report page. Rate limit is 10 audits per IP per hour.

See your score now

Updated . Sources: Google Search Central, llmstxt.org.