Astro Integration
Astro NoCrawl
Automatically block search engine crawling on non-production Astro sites using a static robots.txt file.
Last Updated: 01 April 2026
Usage Snapshot
23
Downloads in Last 30 Days
v1.1.0
Latest npm Version
Source: npm registry
Why This Plugin Exists
Staging and preview environments are frequently crawled by search engines by mistake, leading to long-term SEO damage.
astro-nocrawl prevents this by generating a blocking robots.txt file at build time for any site not explicitly allow-listed.
The result is static, cache-safe, and requires no runtime logic, middleware, or environment guessing.
Design Principles
- Build-time only, no runtime execution
- Explicit allow-listing, no guessing
- Static, cache-safe output
- Single responsibility
- Never blocks a deployment
What This Plugin Delivers
- Generates a restrictive robots.txt file for non-production sites
- Exact hostname allow-listing
- No HTML mutation or meta tag injection
- Works behind any CDN
- Adapter-agnostic and deterministic
Installation
npm install astro-nocrawl
The plugin runs during astro build and writes a robots.txt file only when crawling should be blocked.
FAQs
What does Astro NoCrawl do?
Astro NoCrawl generates a restrictive robots.txt file at build time to block crawling on non-production sites.
Does Astro NoCrawl run at runtime?
No. The plugin runs only at build time and produces static output.
How does Astro NoCrawl decide when to block crawling?
It compares the configured site hostname against an explicit allow-list and blocks crawling if the hostname is not allowed.
Implementation FAQs
Why does Astro NoCrawl exist?
To prevent accidental indexing of staging, preview, and internal sites, which can cause long-term SEO damage.
What does the generated robots.txt contain?
A simple rule that blocks all crawlers: User-agent: * followed by Disallow: /
How does allow-listing work?
Only exact hostnames listed in the allow option are permitted to be crawled.
Are subdomains implicitly allowed?
No. Subdomains must be explicitly listed to be allowed.
Does Astro NoCrawl execute at runtime?
No. It never runs at runtime and introduces zero overhead.
Is the output CDN-safe?
Yes. The robots.txt file is static and can be cached indefinitely.
Project Links
Source code, package distribution, releases, and documentation.
Need Implementation Details?
Read the FAQs for usage patterns, integration caveats, and rollout guidance.