Free Watch for AEO agencies
llms.txt Watch
Which sites publish llms.txt, and when its title or summary line changed.
Free Watch for AEO and SEO agencies
AI Crawler Watch
robots.txt rules that name AI crawlers, site by site, and when they changed.
AI Crawler Watch reads its source, robots.txt files published by each site, daily, and has confirmed 0 changes since Nov 6, 2026.
Source: robots.txt files published by each site (RFC 9309). Checked daily.
Sites watched50,000The Tranco top 50,000
History startsNov 6, 2026Nothing older
On timeCoverage numbers arrive after 14 days.
Latest changes
JSONNo changes yet. The robot checks daily.
Alerts by email come with paid Watches. This Watch's changes are free to read here and as JSON.
Find a site
More Watches
Free Watch for supply-path analysts
Seller Directory Watch
Which sellers an ad system adds to or removes from its sellers.json.
Free Watch for MSPs
MTA-STS Watch
Which domains publish an MTA-STS policy, and when it moves between enforce, testing, and none.
robots.txt rules that name AI crawlers, site by site, and when they changed. It does not determine compliance, approval, or intent.