Docs / Shared / Data freshness

Data freshness

When each dataset is scraped, in UTC, and what changes on a row when it is seen again.

The schedule

All times are UTC, and the server keeps UTC year-round. Azerbaijan is UTC+4 with no daylight saving, so add four hours for Baku local time.

Job postings are the only dataset softon serves since 2026-10-16. The seventeen others were retired that day: their scrapers stopped and their routes answer 410. The changelog names them.

The three job runs are clustered inside the working day, so the gaps between them are four and five hours and the gap across the night is fifteen. Fifteen is the number to plan against — an average of eight would be true and useless, because nothing arrives overnight. A full run of the job fleet takes ten to sixteen minutes, so a row that lands in a run is queryable within the hour it started.

What changes on a row

A scrape does not replace a record, it re-observes one. Three fields say what that did, and they are what a polling client should read rather than re-downloading a collection.

A posting carries all three. ingested_at, first_seen_at and times_seen are on every row of /v1/jobs and /v1/jobs/{id}.

The difference between the two watermarks is large enough to be a budgeting decision as well as a technical one. A single refresh re-observes thousands of rows; across a whole day, roughly 1,400 postings are genuinely new — measured by walking first_seen_after=now-24h to exhaustion, 1,421 rows over 15 pages. Polling updated_after gets you every re-observation; polling first_seen_after gets you those ~1,400, which is about 15 requests a day.

If you are unsure what an endpoint accepts, ask it — a 400 for an unknown parameter lists every parameter that endpoint does take, and that list is generated from the router rather than written by hand, so it cannot be out of date.

What happens when a source takes a record down

On /v1/jobs, it is measured. Each time a source delivers a complete scrape, any posting of that source which the run did not carry — and which nothing has seen for 24 hours — gets active: false and a deactivated_at stamp. On the schedule above that is roughly three consecutive misses, which is deliberate: a single blocked run must not retire a thousand live postings. Nothing is deleted. Every field keeps the last value that was scraped, the id keeps resolving on /v1/jobs/{id}, and a posting that comes back is active again on the next scrape that carries it.

Ask for the half you want with active: true for postings whose source was still publishing them, false for the retired ones, and any, the default, for both. If you are showing postings to jobseekers, pass true.

Retirement deliberately does not move ingested_at, so that a retired row keeps looking unseen. updated_after matches deactivated_at as well, which means a client mirroring the corpus is told a posting died without having to poll for its absence.

One honest limit. The verdict comes from a source running, so a source that stops delivering altogether never produces one, and its postings stay marked active until a daily pass retires them: once a source has stored nothing for 14 days, every posting it still holds gets active: false and a deactivated_at stamp — at most about fifteen days after its last delivery. That false is not a finding about the posting, and the row does not say which kind it carries. The pass also declines when the silence looks like ours rather than the source's — while fewer than half of the sources that delivered in the last 44 days have delivered in the last 36 hours, or when what it would retire is more than a tenth of every live posting — and those postings then stay true until someone has looked. For those fifteen days, and for a refusal, the signal is the source's own freshness rather than the row's: its last_observed_at, below.

Deciding liveness yourself

active is our verdict, and a real one needs the source to have run. false is usually a finding — a complete scrape that did not carry the posting. true is the default state and no retirement pass ever writes it, so a source that stops delivering produces no verdict, and every one of its postings stays true until a daily pass retires it, at most about fifteen days after the source's last delivery. That false is the other kind: it says we stopped vouching for the posting, not that it closed, and the response does not tell you which kind you are holding.

That is not a corner case. On 2026-09-29, the last time that retirement was run by hand, it took 313 postings across seven sources off active: true — one of the seven an organisation that no longer exists. For the fortnight before the pass reaches a source, and for as long as it declines, true is still no verdict at all. We would rather tell you that than have you find it.

So if liveness matters to your product — and on anything a jobseeker reads, it does — derive it yourself. You have everything you need, and you do not have to accept our threshold.

The two facts, and the three answers

Both facts are published. ingested_at is on every row and moves every time a scrape re-sees the posting. The source's last delivery is last_observed_at on GET /v1/sources, which needs no key — fetch it once per sync, not once per row. Use that one and not last_ingested_at: a batch can land and store nothing, and three job sources did exactly that three times a day for five weeks while last_ingested_at said they had just run.

// once per sync — one row per job source, no key
const sources = await fetch(`${API}/v1/sources`).then(r => r.json());
// jobs rows only: a stem can feed two datasets, and last_observed_at is null off jobs
const lastRun = Object.fromEntries(sources.data
  .filter(s => s.dataset === 'jobs')
  .map(s => [s.source, Date.parse(s.last_observed_at)]));

// your rule, not ours
const STALE_SOURCE = 36 * 3600e3;   // a source silent this long tells you nothing
const MISSED_RUN   = 20 * 3600e3;   // carried this recently, relative to its source

function liveness(job) {
  const ran  = lastRun[job.source];
  const seen = Date.parse(job.ingested_at);
  if (!ran || Date.now() - ran > STALE_SOURCE) return 'unknown';
  if (job.active === false)                     return 'gone';
  if (ran - seen > MISSED_RUN)                  return 'gone';
  return 'live';
}

Both numbers are yours to choose. Ours are 24 hours of grace and one complete run; you may want a jobseeker-facing board to be stricter, and an analytics pipeline to be looser. Treating unknown as its own state rather than folding it into either is the part that matters — it is the difference between showing a stale vacancy and knowing you might be.

The order of the checks matters more than it used to. A posting the daily pass retired because its source went silent reads false, and it is the stale-source check, ahead of the active one, that files it as unknown rather than gone. Swap those two lines and every posting of a source that has been quiet for two weeks becomes a confirmed closure.

One thing this cannot tell you: a posting removed at someone's request leaves the API entirely rather than turning false. Re-fetch the id and check for 410.

A quiet source is not always a broken one

Records are scraped per source, and sources differ: some publish a handful of records a week, and a run that returns nothing new from one of them is an ordinary result rather than a failure. What the schedule guarantees is that the attempt was made, not that it found something.

What GET /v1/sources gives you

Every field the liveness check above needs comes from this one endpoint, and it needs no key. Fetch it once per sync, not once per row.

The gap between records and live_records is the useful signal. A source where they are far apart has retired most of what it holds; a source where live_records is 0 is one whose scraper has stopped and whose rows are all history. Two sources report thousands of records and zero live_records today. And a source whose last_observed_at is old is one where that gap is unmeasured rather than small — which is exactly the unknown state the check above keeps separate, and what stale_live_records counts for you.