Why does Astro NoCrawl exist?
To prevent accidental indexing of staging, preview, and internal sites, which can cause long-term SEO damage.
Astro & SEO Safety
Astro NoCrawl Plugin FAQs
Clear answers explaining how Astro NoCrawl safely prevents accidental indexing.
Questions covered
Answers
What does the generated robots.txt contain?
A simple rule that blocks all crawlers: User-agent: * followed by Disallow: /
How does allow-listing work?
Only exact hostnames listed in the allow option are permitted to be crawled.
Are subdomains implicitly allowed?
No. Subdomains must be explicitly listed to be allowed.
Does Astro NoCrawl execute at runtime?
No. It never runs at runtime and introduces zero overhead.
Is the output CDN-safe?
Yes. The robots.txt file is static and can be cached indefinitely.
What happens if robots.txt cannot be written?
A warning is logged, the build continues, and the site is not broken.
Is Astro NoCrawl open source?
Yes. Astro NoCrawl is released under the MIT license.
Want to try it yourself? Astro NoCrawl plugin or Astro NoIndex plugin or Astro Canonical plugin
Want to prevent accidental indexing of staging sites?