Skip to content
Severity: Info

Finding: robots.txt Not Found

This finding means Nivaronix requested /robots.txt on your domain and did not get back a usable robots.txt file. It is an SEO/crawl-visibility signal, not a security finding.

Last updated August 12, 2026.

What it means

robots.txt is a plain-text file at the root of your domain that tells search-engine crawlers which parts of the site they may request, and can point them at your sitemap. Nivaronix requested /robots.txtand either got a non-200 response or a 200 response whose body doesn't look like a robots.txt file at all (empty, or an HTML page served at that path instead of a text file).

Why it matters

Without robots.txt, crawlers fall back to their own default behavior — most will still crawl your site, but you lose the ability to steer them: keeping staging paths, admin routes, or duplicate-content URLs out of the crawl, or pointing crawlers at your sitemap for faster discovery. This affects how efficiently and predictably your site gets indexed, not its security posture.

Does this affect your Nivaronix security score?

This is an SEO/visibility signal, not a security finding. Nivaronix reports it at Info severity, and Info-severity findings carry a 0-point deduction — this result does not change your Nivaronix security score.

What Nivaronix checks

Nivaronix requests /robots.txt over HTTP(S) and requires an HTTP 200 response whose body is non-empty and does not look like an HTML document (a common sign of a catch-all page routed to that path instead of a real robots.txt). If found and well-formed, it also counts the recognised directive lines present (user-agent, disallow, allow, sitemap, crawl-delay) — this is a presence-and-shape check, not a full robots.txt parser or a validation of crawl rules against your actual site structure.

Example evidence (illustrative — not live scan data)

GET https://example.com/robots.txt: HTTP 404

How to fix it

  1. Create a plain-text file named robots.txt at your site's root.
  2. Add at minimum a User-agent: * block and an Allow: / (or specific Disallow: rules for paths you want excluded).
  3. Add a Sitemap: line pointing at your sitemap.xml, if you have one.
  4. Deploy it so it serves at /robots.txt with a text/plain content type.
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

How to verify the fix

Load https://yourdomain.com/robots.txt directly in a browser and confirm it returns your text file, then re-run the Nivaronix scanner and confirm it now reports robots.txt as present.

Related

Check your robots.txt

Scan your domain free