Docs / Shared / Our crawler
Our crawler
What SoftonBot is, which sites it visits, and how to tell it not to.
What it is
Our crawler is called SoftonBot. It reads public job and event listings from Azerbaijani sites and publishes them through this API. It reads pages; it does not submit forms, create accounts, or follow links behind a login.
Block it in robots.txt, not by user-agent string. We send SoftonBot when we fetch your robots.txt and when we read event listings — but job listings are fetched with an ordinary browser user-agent, because several of the sites we read sit behind bot protection that fingerprints the TLS handshake, and a mismatched user-agent string makes that worse rather than better. So a rule at your web server matching on the name would not stop us, and we would rather say that than let you think it had.
It also answers to BirJobBot, and will keep doing so. That is the name this crawler used to run under, and site owners wrote rules against it. A rename that stopped honouring those rules would quietly start crawling everyone who had already said no, so both names are honoured and the old one is not going away.
How to tell it not to
Add a group naming it to your robots.txt. Either name works, and you do not need both:
User-Agent: SoftonBot Disallow: /
Narrow it if you would rather: Disallow: /jobs/drafts blocks that path and nothing else. The longest matching rule wins, and an Allow beats a Disallow of the same length.
It takes effect on our next run — within a few hours for jobs, which run three times a day, and within a day for events. We read robots.txt once per site per run, so there is nothing to notify us about.
Why User-agent: * is not enough
We honour groups that name us and deliberately ignore wildcard groups. That sounds backwards, so here is the reasoning: a wildcard rule is usually aimed at search engines and scrapers in general, and reading it as an instruction to us specifically would silently drop most of the sources on this platform on the strength of a rule that was not about us. A named group is unambiguous, and it is the one we act on.
If you want us gone, name us. It is one line and it is the version we cannot misread.
Removing something already published
A robots.txt rule stops future crawling; it does not remove what is already in the API. For that, get in touch and say which listings or which site — we suppress them at the view, so they leave every response including the archive, not just new ones.
If a posting contains someone's personal data and they want it gone, say that explicitly and we will treat it as an erasure request rather than a takedown.