Every AI visibility discussion eventually reaches content, and most start there. That is the wrong order.
A page that cannot be fetched, rendered and parsed is invisible to retrieval regardless of how well it answers the question. The foundation is not exciting, and it is where a meaningful share of programmes silently fail.
Crawler access, first and always
AI crawlers are distinct user agents controlled separately from traditional search crawlers. A robots.txt inherited from a template, or written before these agents existed, may be blocking exactly the systems you are trying to appear in.
This check takes five minutes and occasionally invalidates an entire proposed programme. Do it before anything else.
We have audited sites that spent two quarters producing AI-optimised content while blocking the crawlers that would have retrieved it. Nothing in the content layer compensates for that.
Rendering is the second failure point
Traditional search engines render JavaScript reasonably well. Retrieval systems vary considerably, and some fetch raw HTML without executing scripts at all.
If your commercial content only exists after hydration, you are betting on the rendering capability of every system you want to appear in. That is an unnecessary bet.
Server-render or statically generate anything that matters commercially. The test is simple: fetch the page with scripts disabled and read what comes back. If the answer is not there, no retrieval system is reading it either.
The technical foundation, in order of consequence
Each layer is worthless if the one above it fails.
Access
Crawlers permitted in robots.txt; no unintended blocking at CDN or firewall.
Delivery
Content present in server-rendered HTML, not only after hydration.
Indexation
Canonical URLs resolved, no duplicate paths, no accidental noindex.
Structure
Semantic HTML and structured data matching the visible content.
Performance
Fast enough that crawl budget is spent on discovery, not waiting.
Semantic HTML is not a formality
Headings in correct order give a parser the document outline. Real lists communicate enumeration. Real tables communicate relationships between values.
Div elements styled to look like headings communicate nothing. Neither do lists built from line breaks, or tables built from positioned divs.
This is one of the few areas where doing the accessible thing and doing the retrieval-friendly thing are exactly the same work. A screen reader and a retrieval system want the same structure for the same reason.
Duplicate paths and crawl waste
When the same content is reachable at several URLs, incoming signals split and crawl budget is spent re-reading known content rather than discovering new content.
This is endemic in mature ecommerce catalogues and large CMS estates, where categories accumulate multiple parent paths over years of restructuring.
The fix is decisive canonical selection, consistent internal linking to the chosen version, and redirects where appropriate. It is unglamorous and frequently the highest-value technical work available on a large site.
Structured data must match the page
Markup describing something the visible page does not contain is a liability, not an asset. It undermines trust in the rest of your markup and can attract manual action.
The rule is simple: mark up what is there, exactly as it is there. Aspirational markup is a short-term tactic with a long-term cost.
Common mistakes
- Commissioning content work without auditing crawler access first.
- Assuming all retrieval systems render JavaScript as well as Google does.
- Using styled divs instead of semantic headings, lists and tables.
- Leaving multiple URL paths resolving to the same content.
- Marking up FAQs the page does not actually answer.
- Testing in a browser rather than testing what a crawler receives.