16 of 67 well-known websites block AI crawlers
Free SEO Audit & AI Visibility Checker
ChatGPT, Perplexity and Google decide in seconds whether to quote your site. Run 49 free checks on technical SEO, content, AI search visibility (GEO), answer engines (AEO) and Core Web Vitals, and get a fix list sorted by impact.
AI visibility here means AI search crawlers can reach and read your pages. We do not ask ChatGPT or Perplexity whether they mention your brand.
Free. No account. Results in under a minute.
A real audit of instagram.com, from .
instagram.com scores 54. What about yours?
- Technical SEO: grade C, 72 out of 100
- Content and E-E-A-T: grade D, 69 out of 100
- Answer engines (AEO): grade F, 17 out of 100
- AI readiness (GEO): grade D, 60 out of 100
- Speed: grade F, 50 out of 100
- AI crawler accessHow to fix: AI crawler access
- Direct answer after the H1How to fix: Direct answer after the H1
- Largest Contentful PaintHow to fix: Largest Contentful Paint
What we found on well-known websites
16 of 67
well-known websites we could audit block AI crawlers in robots.txt
23 of 51
of the sites whose real page we could read have no working llms.txt
75/100
is their average ratemysite.ai score
Scores count only sites whose real page we could read. Updated . See the ranking
See what holds your site back
The audit runs 49 checks on technical SEO, content and speed. Every problem comes with a priority and a step by step fix guide.
Example recommendations
- Title tag lengthHigh
- Meta descriptionMedium
- Image alt textLow
Example passes
- Indexable (no noindex)
- robots.txt
- XML sitemap
Technical SEO
Example score
Make your site easy for AI to cite
AI assistants must reach your site before they can cite it. GEO is a set of practices that make a site easy for AI assistants to crawl, understand and cite.
robots.txt: what we look for
- User-agent: OAI-SearchBot allowed
- User-agent: Claude-SearchBot allowed
- User-agent: PerplexityBot allowed
llms.txt
A plain Markdown map of your site for AI assistants.
Core Web Vitals, measured
Speed data comes from Google PageSpeed Insights. Core Web Vitals are a set of Google metrics for loading, responsiveness and stability.
Core Web Vitals: pass thresholds
- Largest Contentful Paintgood under 2.5 s
- Interaction to Next Paintgood under 200 ms
- Cumulative Layout Shiftgood under 0.1
Source
Google PageSpeed Insights, field data when available.
Your report, step by step
- Step 1
One score, five colors
Your grade, phone and desktop screenshots and a radar of the five categories.
- Step 2
What to fix first
Every failed or warned check, sorted by impact, each linked to its fix.
- Step 3
Save it, share it
Save as PDF, share the link, compare with a competitor, get score alerts and a site badge.
What does the SEO audit check?
Technical SEO
14 checks, 25% of the score
Examples
- Title tag length
- Canonical URL
- XML sitemap
Content and E-E-A-T
8 checks, 15% of the score
Examples
- Author identified
- Visible date
- Readability
Answer engines (AEO)
9 checks, 20% of the score
Examples
- FAQ schema
- Direct answer after the H1
- Question-style headings
AI readiness (GEO)
12 checks, 25% of the score
Examples
- llms.txt
- AI crawler access
- Citable entity links
Speed
6 checks, 15% of the score
Examples
- Largest Contentful Paint
- Interaction to Next Paint
- Cumulative Layout Shift
Made for the people who run websites
Business owners
A plain grade and a short list of what to fix first, in everyday words.
Marketers
See if Google and AI assistants can find, read and quote your pages, then compare with a competitor.
Developers
The evidence behind every check, PageSpeed data for phone and desktop, and a Markdown version of each report.
More free tools
- Weekly leaderboardThe best scoring sites this week.
- Fix guidesStep by step help for every check.
- BlogPlain guides on SEO and AI search.
- llms.txt generatorBuild an llms.txt for any site in seconds.
- Schema generatorBuild JSON-LD for your site and see what it lacks.
- On every reportCompare with a competitor, get a weekly score email, add a score badge, save as PDF.
How the audit works
- 1
Enter your URL
Any public page. No account, nothing to install.
- 2
We run 49 checks
SEO, content, answer engines, AI readiness and speed, in under a minute.
- 3
Fix what matters
A list sorted by impact, with a guide for every problem.
Frequently asked questions
What does ratemysite.ai check?
It runs 49 checks across technical SEO, content quality, answer engine optimization (AEO), AI search readiness (GEO) and Core Web Vitals, then scores the page from 0 to 100 and lists the fixes in priority order.
Is it really free?
Yes. Audits are free and need no account. You can run 10 audits per hour.
What is GEO?
GEO is generative engine optimization: making a site easy for AI assistants such as ChatGPT, Claude and Perplexity to crawl, understand and cite. It covers crawler access, llms.txt, server-rendered content and clear brand data.
Does it check whether ChatGPT mentions my brand?
No. It checks whether AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot can reach your pages, and whether the content is easy to read, quote and cite. It does not ask ChatGPT or other assistants what they say about your brand.
What is AEO?
AEO is answer engine optimization: writing pages so that search engines and assistants can lift a direct answer from them, with FAQ and HowTo structured data, question headings and a short definition near the top.
Does the audit run JavaScript?
No. It reads the HTML your server sends, which is also what most AI crawlers see. If your content only appears after JavaScript runs, the audit will tell you.
How often can I re-run an audit?
A report is reused for 24 hours. After that you can re-analyze from the report page. Rate limit is 10 audits per IP per hour.
See your score now
Updated . Sources: Google Search Central, llmstxt.org.