Docs / Shared / Data freshness

Data freshness

When each dataset is scraped, in UTC, and what changes on a row when it is seen again.

The schedule

All times are UTC, and the server keeps UTC year-round. Azerbaijan is UTC+4 with no daylight saving, so add four hours for Baku local time.

The three job runs are clustered inside the working day, so the gaps between them are four and five hours and the gap across the night is fifteen. Fifteen is the number to plan against — an average of eight would be true and useless, because nothing arrives overnight. A full run of the job fleet takes ten to sixteen minutes, so a row that lands in a run is queryable within the hour it started.

Rates runs once a day, mid-afternoon in Baku — the table is UTC, so add four hours. That is deliberately after the bank's publishing window rather than inside it: CBAR issues one bulletin per business day, effective from that date, but does not commit to a time, and a run that fires too early stores yesterday's numbers carried forward under today's date. The single run asks for two dates, yesterday and today, so a bulletin that appeared after we looked is picked up the next day and corrected in place rather than being missed for good.

You do not have to care about any of that. Every row carries both dates and a carried_forward flag, so you can tell a fresh publication from a repeat without polling — and history never moves, because a bulletin for a past date says the same thing forever.

Genome is one probe a day of four upstream release markers, and it is the only line in this table that measures our reading rather than someone else's publishing. NCBI and EMBL-EBI cut releases months and years apart — IGSR's newest data freeze is from 2013 — so a daily check is not an attempt to keep up with them. It is there so that status and retrieved_at are about today: a release that has not moved and a probe that has stopped working are the same version string, and only the daily attempt tells them apart. Every probe writes all four rows whether the upstream answered or not, so a source that has gone dark appears as a row that says so rather than as a row that quietly stops changing.

Reachability is one probe a day of six watched URLs, and like genome it measures our reading rather than someone else's publishing — more so, because there is nothing upstream being published at all. A row is the answer a request got. Daily is what makes consecutive_failures a count of days, and that counter is the point: one refusal is noise, and sixty-one in a row is a fact about a relationship with a site. It is deliberately not more often than daily, for a reason peculiar to this dataset — two of the six targets sit behind bot protection whose verdict is partly a function of request rate, and probing them harder would make our own traffic a variable in the measurement.

Quotes has no schedule and is not going to get one. That pool was read once from an upstream database; quotations do not change, and re-reading them on a timer would restore the same rows with nothing but a new ingested_at to tell them apart.

What changes on a row

A scrape does not replace a record, it re-observes one. Three fields say what that did, and they are what a polling client should read rather than re-downloading a collection.

Only ingested_at is on every dataset. first_seen_at and times_seen are on /v1/jobs and nowhere else — so a sync loop written once against this page and pointed at events, quotes, rates, genome or reachability reads a key that is never there. On those five, ingested_at is the whole signal.

The difference between the two watermarks is large enough to be a budgeting decision as well as a technical one. A single refresh re-observes thousands of rows; across a whole day, roughly 1,400 postings are genuinely new — measured by walking first_seen_after=now-24h to exhaustion, 1,421 rows over 15 pages. Polling updated_after gets you every re-observation; polling first_seen_after gets you those ~1,400, which is about 15 requests a day.

/v1/events does take updated_after — use it, and do not walk all nine pages on every poll. It does not take first_seen_after. /v1/quotes takes neither, and does not need them: that pool was read once and quotations do not change. /v1/rates takes neither either — the watermark you want there is from, which filters on the bulletin's own calendar date rather than on when we stored it, and those dates are Asia/Baku. /v1/genome/releases takes neither and needs neither: it is four rows rewritten in place every probe, so there is no watermark to walk and nothing to page through — read the whole collection and compare retrieved_at, the last time we successfully read each upstream, against ingested_at, the last time we tried. /v1/reachability takes neither and needs neither for the same reason: six rows rewritten in place every probe, nothing to page through, and the history you want is already on the row as last_reachable_at, first_failed_at and consecutive_failures.

If you are unsure what an endpoint accepts, ask it — a 400 for an unknown parameter lists every parameter that endpoint does take, and that list is generated from the router rather than written by hand, so it cannot be out of date.

What happens when a source takes a record down

On /v1/jobs, it is measured. Each time a source delivers a complete scrape, any posting of that source which the run did not carry — and which nothing has seen for 24 hours — gets active: false and a deactivated_at stamp. On the schedule above that is roughly three consecutive misses, which is deliberate: a single blocked run must not retire a thousand live postings. Nothing is deleted. Every field keeps the last value that was scraped, the id keeps resolving on /v1/jobs/{id}, and a posting that comes back is active again on the next scrape that carries it.

Ask for the half you want with active: true for postings whose source was still publishing them, false for the retired ones, and any — the default today — for both. If you are showing postings to jobseekers, pass true.

Retirement deliberately does not move ingested_at, so that a retired row keeps looking unseen. updated_after matches deactivated_at as well, which means a client mirroring the corpus is told a posting died without having to poll for its absence.

One honest limit. The verdict comes from a source running, so a source that stops being scraped altogether never produces one and its postings stay marked active. The signal for that case is the source's own freshness, not the row's — a dataset whose newest batch is days behind the timetable above is the thing to look at.

Events and quotes have no such pass. There, ingested_at is still the freshness signal: a record several cycles behind the schedule has not been seen in the last several runs.

Rates has no such pass either, and on that one dataset ingested_at is not the freshness signal at all. A bulletin for a past date is immutable, so its rows are fetched once and keep that stamp for good while staying exactly as correct as the day they landed — judging them by age would retire the entire history of the feed. The freshness question here is “is there a row for today, and is it a repeat?”, and date and carried_forward answer it without reference to when we stored anything. Those dates are Asia/Baku, not UTC.

Deciding liveness yourself

active is our verdict, and we can only give one when the source has run. false is a real finding. true is the default state and the retirement pass never writes it — so a source that has stopped being scraped produces no verdict and every one of its postings stays true forever.

That is not a corner case. 15 of 96 enabled sources have not landed a batch in days, and one of them holds over 12,000 postings — all still active: true. We would rather tell you that than have you find it.

So if liveness matters to your product — and on anything a jobseeker reads, it does — derive it yourself. You have everything you need, and you do not have to accept our threshold.

The two facts, and the three answers

Both facts are published. ingested_at is on every row and moves every time a scrape re-sees the posting. The source's last run is on GET /v1/sources, which needs no key — fetch it once per sync, not once per row.

// once per sync — 121 rows, no key
const sources = await fetch(`${API}/v1/sources`).then(r => r.json());
const lastRun = Object.fromEntries(
  sources.data.map(s => [s.source, Date.parse(s.last_ingested_at)]));

// your rule, not ours
const STALE_SOURCE = 36 * 3600e3;   // a source silent this long tells you nothing
const MISSED_RUN   = 20 * 3600e3;   // carried this recently, relative to its source

function liveness(job) {
  const ran  = lastRun[job.source];
  const seen = Date.parse(job.ingested_at);
  if (!ran || Date.now() - ran > STALE_SOURCE) return 'unknown';
  if (job.active === false)                     return 'gone';
  if (ran - seen > MISSED_RUN)                  return 'gone';
  return 'live';
}

Both numbers are yours to choose. Ours are 24 hours of grace and one complete run; you may want a jobseeker-facing board to be stricter, and an analytics pipeline to be looser. Treating unknown as its own state rather than folding it into either is the part that matters — it is the difference between showing a stale vacancy and knowing you might be.

One thing this cannot tell you: a posting removed at someone's request leaves the API entirely rather than turning false. Re-fetch the id and check for 410.

A quiet source is not always a broken one

Records are scraped per source, and sources differ: some publish a handful of records a week, and a run that returns nothing new from one of them is an ordinary result rather than a failure. What the schedule guarantees is that the attempt was made, not that it found something.

What GET /v1/sources gives you

Every field the liveness check above needs comes from this one endpoint, and it needs no key. Fetch it once per sync, not once per row.

The gap between records and live_records is the useful signal. A source where they are far apart has retired most of what it holds; a source where live_records is 0 is one whose scraper has stopped and whose rows are all history. Two sources report thousands of records and zero live_records today. And a source whose last_ingested_at is old is one where that gap is unmeasured rather than small — which is exactly the unknown state the check above keeps separate.