The jobs fleet runs 97 configured sources three times a day, at 06:00, 11:00 and 15:00 UTC, four at a time. On a good day roughly 81 or 82 of them post a batch. The rest fail, or return zero rows, and the difference between those two outcomes is the whole subject of this post.
A source that returns zero rows is saying there are no vacancies right now. A source that fails is saying nothing at all. Both arrive as an empty result, and if you treat them the same you will eventually publish one of two lies: that a live job is closed, or that a closed job is live.
What made the question urgent
Until migration 0034 there was no column that could hold the answer. No migration contained a DELETE against current_jobs, no service expired a row, and the only thing that ever removed a posting from the API was a withdrawal on request. So a posting whose employer filled the role in March was still served in August, indistinguishable from one scraped an hour earlier except by reading ingested_at and guessing.
The obvious fix is the one the upstream we vendored already had: after each source finishes a run, mark every row of that source you did not see this time as inactive. It is four lines of SQL and it is correct on every day the scrape works.
It is also how you delete a source's entire corpus on the day it breaks.
A truncated run looks exactly like a small day
This is the part that is easy to miss until it happens. A failed run is harmless — it posts nothing, so the retirement pass never fires. The dangerous shape is a run that succeeds with part of the data.
75 of the vendored source modules paginate by breaking on the first empty page. So a fetch refused halfway through a walk returns pages 1 and 2 of 10 as though that were the whole source, and exits zero. Two instances are recorded from a single day, 2026-08-16: djinni was served an 88-byte "your IP has been blocked" page, and ejob_az a 404 anti-bot page. Both were logged as ordinary runs. Between them those two sources hold 9,198 and 2,744 rows.
A truncation that survives three runs outlives any grace period you would reasonably set. An unguarded retirement pass would then have retired nearly everything both sources had.
So the pass carries two guards, and they are the whole safety story:
- Grace. A row must be absent from this run and unseen for 24 hours. With runs at 06:00, 11:00 and 15:00 UTC that is roughly three consecutive misses.
- Coverage. If a run carried fewer distinct ids than a fraction of the source's live rows, the pass touches nothing and returns why.
The coverage guard refuses a real retirement whenever a source legitimately halves in a day. That trade is the right way round: the first case corrects itself on the next run, and the second — a source quietly blocked — does not correct itself at all.
Then the failure the guards do not cover
On 2026-08-29, measured on the server rather than from a laptop, seven enabled sources had stopped landing batches:
| Source | Status | Fronted by |
|---|---|---|
| jobu_az | 403 | Hostinger CDN |
| ejob_az | 404 | the site's own block signature |
| edumap_az | 403 | Cloudflare |
| royalesb | 403 | Cloudflare |
| un_jobs | 403 | Cloudflare |
| rigzone | 403 | — |
| premiumbank | 403 | Cloudflare |
Five of them went dark inside a 61-second window on 2026-08-15 at 16:30 UTC. unjobs.org is an international aggregator, royalesb.com is a Baku hotel, edumap.az is an education site. Three unrelated operators do not change bot policy in the same minute. Our egress address had been flagged.
The retirement pass could not help with any of it, because the pass only runs when a batch arrives and no batch was ever going to arrive again. Left alone, 6,194 rows would have sat at is_active = true forever — and true meaning "nobody has said otherwise" reads to every consumer as "verified live". That is the stronger false claim of the two available, so the rows were retired by hand in migration 0051.
The one that was not the same problem
jobsite_az was filed with those seven. It is not in the table above, because on the same day it ran from the server and landed a batch: 201 rows stored out of about 900 jobs seen. Its two-week stall was never an egress problem, so retiring it would have been wrong.
What that check protected is worth stating precisely, because the imprecise version is a mistake we have already made once in public. jobsite_az holds 12,667 rows. Only 1,199 of them are active — the other 11,468 had been retired weeks earlier, on 2026-08-12. A retirement pass only ever acts on live rows, so the population at risk was 1,199, not 12,667. The bigger number is a corpus total, and calling it an active count is exactly the error an earlier version of our own active field description made; it is corrected there now, and repeating it here while linking to the correction would be worse than useless.
The same discipline applies to the other figure. The issue behind that migration estimated around 19,000 rows would need retiring. Measurement said 6,194, and the migration header records the estimate alongside the real number rather than quietly adopting it. That gap — a three-fold overestimate, from a guess written by the person who also wrote the retirement pass — is the actual argument for the manual run. It cost one scrape.
What we publish, and what we do not pretend
In the database the column is is_active. The API publishes it as active, and its documentation is careful about which half of it is a finding.
false is a real verdict: the source delivered a complete scrape, this posting was not in it, and nothing had seen it for 24 hours. true is weaker than it looks. It is the default state, and the only thing that ever changes it is the retirement pass — which fires only when the source itself completes a run. A source that has stopped being scraped produces no verdict at all, so every one of its postings stays true indefinitely.
For the rows retired by hand in 0051, even false is not quite the documented meaning. Those postings may well still be open; the boards are up and publishing, we simply cannot reach them any more. The retirement reason recorded against them is deliberately its own value rather than being folded in with the ordinary absent-from-source one, so the two causes never become indistinguishable in the record.
It is an interim state and it is written down as one. The honest answer for those rows is a third value meaning "no recent run can say", and until that exists the API says the less damaging of the two things it can currently say.
None of this is reversible-by-accident either. Re-observation reactivates a posting — active back to true and deactivated_at cleared. The day the egress address changes, the next successful run undoes the whole migration on its own.
Derive it yourself, and you probably should
Because true carries no verdict on its own, GET /v1/sources publishes live_records beside records for every source. The gap between the two is exactly the population whose true means nothing, and a source whose last_ingested_at is old is one where that gap is unmeasured rather than small. Two sources currently report thousands of records and zero live_records.
The derivation is four lines. Compare a posting's ingested_at against its source's last_ingested_at: the source ran recently and carried this row means live; ran recently and did not means gone; has not run means nobody knows, including us. Nothing stops you being stricter than we are, and for anything jobseeker-facing you should be.
That is the whole reason the field is published with its weaknesses attached rather than as a clean boolean. A clean boolean would have been easier to sell and would have been wrong for every source in the table above.
The freshness page has the run times, the worked derivation, and what changes on a row when it is seen again. The jobs reference has the field list.