SolvedBot
If this bot is in your access log, this page explains what it is and how to stop it. Blocking takes one line and we honour it.
How to block it
Add this to your robots.txt:
User-agent: SolvedBot
Disallow: /We re-read robots.txt at least once a day, so that takes
effect within 24 hours and needs no confirmation from us. To slow us down
rather than stop us:
User-agent: SolvedBot
Crawl-delay: 10A Crawl-delay can only ever make us slower. We will not go
faster than our own default no matter what the file says.
If you would also like your threads removed from the index rather than just stopping future crawls, ask us here.
What it does
- User-Agent
SolvedBot/1.0 (+https://solved.3d3d.ca/bot)- Rate
- At most 1.0 request per second per host, with at most
2 connections open, jittered. Slower if your
robots.txtasks. - What it fetches
- Thread pages and the listing pages needed to find them. It does not log in, does not submit forms, and does not touch anything behind a paywall or a login.
- What it stores
- Thread text, so results can be ranked and a short excerpt shown. Never republished — visitors get a few lines and a link to your site.
- If
robots.txtcan't be read - We stop. An unreadable file is treated as a refusal, not as permission.
Why you might not want to block it
We send traffic to you. Every result links to the thread on your site; there is no cached copy and no reader mode. The premise of this project is that forums host the knowledge and should get the visit — which is also why the crawler says who it is on every single request instead of pretending to be a browser.
Something behaving badly? It is a bug and we want to know — email info@3d3d.ca.