MEIKAIAI crawlability audits: from SEO to machine readability
An AI crawlability audit checks whether automated systems can reach a site's important pages, render their content and extract the facts a human visitor can see. It should produce test evidence, not a general score for “AI readiness.”
The distinction matters because several systems may touch the same page: search indexes, model-training crawlers, user-triggered fetchers and browsing tools. Access for one does not establish access for the others, and crawlability does not guarantee that any assistant will cite the page.
Start with a page and a request
The audit unit should be a real page under a real access path. For each priority template, record the URL, requesting user agent, HTTP response, rendered text, canonical URL and the fields that were successfully extracted.
| Test | Pass condition | Evidence to retain |
|---|---|---|
| Access | The intended requester receives the expected status and content. | Request headers, response status, redirects and relevant server-log entry. |
| Rendering | Essential information is present without an unsupported interaction. | Initial HTML, rendered DOM and screenshot where layout affects meaning. |
| Extraction | Named fields can be recovered with their labels and units intact. | Extracted values compared with the visible page. |
| Identity | The page identifies the product, organisation or policy unambiguously. | Canonical URL, entity identifiers and structured data. |
| Freshness | Dates, availability and version information agree across surfaces. | Conflicts between page copy, metadata, feeds and structured data. |
Access is more than robots.txt
Robots rules are one checkpoint. The audit should also follow redirects, authentication, bot-management challenges, locale routing and CDN behaviour. A public URL can still return a challenge page or an empty application shell to an automated requester.
Server logs are the strongest evidence that a requester reached the site. A successful local browser test does not prove that the production edge served the same response to a named crawler or user-triggered fetcher.
Test the information that drives a decision
A page can render successfully while hiding the facts that matter. Product names, identifiers, price, availability, dimensions, ingredients, compatibility, policy dates and geographic restrictions should be present in visible text and associated with clear labels.
Common failure cases include:
- content loaded only after a click, scroll or client-side request;
- tabs whose inactive panels do not exist in the initial document;
- values shown visually without a nearby label or unit;
- structured data that disagrees with the visible page;
- regional pages sharing a canonical URL despite different products or terms; and
- error pages that return HTTP 200.
The last case is easy to miss. In our Alibaba citation-decay study, removed articles returned a small generic error page with HTTP 200. A status-only monitor would have classified those URLs as live.
Choose pages by risk, not volume
A useful sample covers templates and business consequences rather than selecting the most visited URLs alone. An ecommerce audit might include an in-stock product, an out-of-stock variant, a discontinued item, returns information and a regional restriction. A B2B audit might include pricing, technical documentation, a security page, a case study and a versioned policy.
For each template, define the fields that must survive extraction. “Page rendered” is too weak a pass condition if the model number or policy date is missing.
Turn the audit into a regression test
The first run establishes a baseline. The durable value comes from rerunning the same requests after CMS releases, navigation changes, CDN rule updates and structured-data changes.
Keep failures specific: which requester, which URL, which field and which layer changed. That gives engineering teams a reproducible defect instead of a screenshot labelled “AI cannot read the site.”
What the audit does not prove
A passing page is available for retrieval under the tested conditions. It does not prove that a platform indexed the page, selected it for a particular prompt, treated it as authoritative or will continue to behave the same way.
Crawlability, answer visibility and citation accuracy should therefore be reported separately. The audit establishes whether the source is technically usable; response-level measurement shows whether and how a platform used it.