📖 How to use
- Enter a public website URL or paste HTML, depending on the tool.
- Choose the available output or rendering options.
- Start processing and keep this tab open while the server works.
- Download or review the result when processing completes.
Home › Developer Tools › robots.txt Checker
robots.txt Checker
Fetch a website robots.txt file and inspect user-agent, allow, disallow and sitemap directives.
◉ Free🌐 Secure Web Processing🛡️ Private Networks Blocked
🌐 Enter Website & Process
HTTP/HTTPSPublic sites onlySafe redirect checks
Preparing…
🛡️ Public HTTP/HTTPS websites only. Private-network destinations are blocked by the processing server.
🛡️ Secure web engine
- Only public HTTP/HTTPS destinations are accepted.
- Private, loopback, link-local and reserved network addresses are blocked.
- Redirect destinations are validated again before fetching.
- Browser rendering jobs run one at a time on the current free server.
💡 Pro Tips
- The first request can take longer if the free Render instance was sleeping.
- Some sites block automated browsers or require login; those pages may not render.
- Very long screenshots are capped to protect server memory.
- Only process pages and HTML you are authorized to use.
❓ Frequently Asked Questions
What URL should I enter?
Enter the website homepage or any URL on that domain. The tool checks the origin /robots.txt file.