# A wrong "closed" never comes back: measuring a job-posting liveness checker on real pages > Source: > Published: 2026-10-23 04:02:05+00:00 [career-ops](https://github.com/career-ops-hq/career-ops) is an open-source job-search pipeline: Node.js scripts driven by AI coding CLIs, with about 76k stars on GitHub. Before it adds a posting to your pipeline, `scan --verify` checks that the posting is still open. Where the job board has a public API, it asks the API. Otherwise it loads the page in a headless browser and reads it. A "closed" or "blocked" verdict goes into your scan history for good, and that posting is skipped on every later scan. So the two ways to be wrong cost very different amounts. A wrong "open" costs one more check later. A wrong "closed" makes a live job disappear from your search without a trace. Between 3 and 20 October 2026 I had 16 pull requests merged into career-ops, most of them in this liveness checker. This is what I found, how I measured it, and what the fixes look like. The board was busy: 135 open PRs and 208 open issues on 11 October. The project's CONTRIBUTING lets bug fixes go in without an issue, so instead of competing for claimed issues I read modules nobody was working on. I started with the ATS keyword and CV skill checks, which are close to my NLP background, then moved to the liveness checker. For the checker, I ran it on real postings and read every disagreement. The sources were public job-postings datasets (LinkedIn, USAJOBS) and live boards without an API. Several fixes also came out of the project's own conversations: a merge note from the maintainer, a follow-up the review bot offered, and three follow-ups from my reviews of other contributors' PRs. Many people run a DNS filter that blocks ads and trackers: Pi-hole, AdGuard DNS, Control D, a hosts-file blocklist. Such a filter answers `0.0.0.0` for a tracker it blocks. The browser check has a security guard that refuses requests to private network addresses, and `0.0.0.0` is on its list. Refusing the tracker is right. But the guard also marked the whole page "blocked", which `scan --verify` records permanently. I found it while reviewing another contributor's PR that touches how the guard handles DNS answers. To measure it, I took 35 postings on 19 boards with no API, all live the day before, and checked each one with `main`'s code and with the fix, behind two real filtering resolvers and with no filter as a control: `main`, 12 postings behind AdGuard DNS and 20 behind Control D read "blocked". The fix ([#5015](https://github.com/career-ops-hq/career-ops/pull/5015)) leaves the guard's decision alone. The request is still refused, with the same error. The guard only tags the refusal when the only private addresses in the answer are the filter's: ``` js if (blocked.length > 0) { const denied = new Error(`Access denied: Egress guard blocked private target IP ${blocked[0].ip}`); if (blocked.every(({ hit }) => SINKHOLE_ADDRESSES.has(hit))) denied.livenessCode = 'dns_sinkhole'; throw denied; } ``` A tagged refusal of a tracker, a script or an image no longer decides the page's verdict. Any other private address still counts, even next to `0.0.0.0`. I rejected the simpler option, ignoring every private answer for subresources, because it would drop the guard's signal on a host that really resolves into a private network. The follow-up, [#5075](https://github.com/career-ops-hq/career-ops/pull/5075), covers the posting's own host. When that host had no usable address, the guard reported "blocked". But no address is often about the run, not the posting: a DNS outage during the scan, or the user's own filter blocking the job board. Now it reads `uncertain`, and the posting is kept for another check. With the posting's own host failing DNS, 19 of 19 postings from 19 boards were dropped permanently on `main`, and 19 of 19 were kept with the fix. The PR states the trade-off: a domain that is really gone now reads `uncertain` too, because one failed lookup can't tell a dead domain from an outage. A kept dead posting costs one more check later. A pinned live one never comes back. This one turned up while I was measuring another fix: a posting whose host no longer resolved broke the checks after it in the same run. I reproduced it on the code from before my DNS change, to rule that out, and traced it to the browser library the next day. The checker runs every browser check in one tab. When a page fails to load (its host has no address, its certificate has expired, its connection is reset), the browser shows its error page a moment after the checker has moved on, and that interrupts the next page load. The interrupted page then finishes loading late and interrupts the one after it, often to the end of the run. Those checks fail as load errors, so `scan --verify` keeps the postings without having checked them, and the closed ones among them go into the user's pipeline. The root is a known Playwright issue ([microsoft/playwright#35944](https://github.com/microsoft/playwright/issues/35944), closed upstream without a fix). Through the real CLI, on one list of 45 URLs with 6 failing loads spread among 39 postings that were live the day before: Over 36 failed loads, the browser's error page landed 1 to 69 ms after the failure was reported, median 4 ms. So the fix ([#5299](https://github.com/career-ops-hq/career-ops/pull/5299)) waits for the browser's own signal instead of a fixed delay, for at most 2 s: ``` async function navigateSettled(page, navigate) { try { return await navigate(); } catch (err) { const canWait = typeof page.waitForEvent === 'function' && typeof page.mainFrame === 'function'; if (canWait && err?.name !== 'TimeoutError') { await page .waitForEvent('framenavigated', { predicate: (frame) => frame === page.mainFrame() && frame.url().startsWith('chrome-error:'), timeout: FAILED_NAVIGATION_COMMIT_MS, }) .catch(() => {}); } throw err; } } ``` A timeout is left alone: a timed-out load shows no error page, and the next load cancels it, so waiting would only add 2 s. I rejected parking the tab on a blank page after a failure, which another part of the code does after a search, because that load is interrupted the same way. The review bot pointed out that the old page changing its own address could end the wait too early. I couldn't make that happen in real Chromium in 26 tries, but fixed it anyway, with a test the first version fails. CI installs no browser, so the tests use a stand-in page that plays out the browser's order of events. The same sequences also run on real Chromium where it can launch. I filed [#4832](https://github.com/career-ops-hq/career-ops/issues/4832) for live postings on seven non-English boards that read as having no Apply button. getManfred, a Spanish tech-job board, was one. Its Apply button says "¡Apúntame!" ("sign me up"), which the checker didn't know, so every live posting read "no Apply control". getManfred's own feed marks every offer open or closed, so the A/B could measure against the board instead of against my reading of pages. The first run showed a case the issue had missed: one closed posting read open on both versions. Closed postings written in English often have an FAQ button such as "Can I apply if I live outside Spain?", and the English Apply pattern took it for an Apply button. The page does say the offer is closed, but that sentence renders near the bottom, after the checker's first look on some pages. Above the title, from the start, a closed posting says when it closed ("Oferta cerrada el 27/07/2023"). Closing patterns are read before the Apply button, so an FAQ button can't outvote the date. The fix ([#5350](https://github.com/career-ops-hq/career-ops/pull/5350)) is two patterns: ``` /\boferta cerrada el \d{1,2}\/\d{1,2}\/\d{4}/i, // a closing pattern, read before the Apply button /\bapuntame\b/i, // an Apply label, accent-free like the others ``` I left out the verb "aplicar": 73 of the 111 getManfred pages I measured have an FAQ button that uses it, closed ones included. On 136 real page loads: No Fluff Jobs, a Polish tech-job board, was the last of my board families from #4832, and the checker read it backwards. A live posting puts its Apply button ("Aplikuj") inside a page header, which the checker skips as page chrome, so every live posting read "no Apply control". A closed posting moves to a search page with a promo link that also says "Aplikuj", so most closed postings read as open. I'd planned label patterns. Then I found that the board's per-posting API says directly whether a posting is open. So the fix ([#5428](https://github.com/career-ops-hq/career-ops/pull/5428)) is an API rung, as several other boards already have. The API answers "published" (open), "expired" (closed) or "does not exist". A status the rung doesn't know reads `uncertain`, never closed, and an answer it can't read falls back to the browser. Through the real CLI on 102 postings (40 live from the board's search API, 59 closed from Wayback Machine archives, 1 gone and 2 made up): The review bot raised two valid points on it. I measured that they change no verdict on today's API answers and shipped them as a follow-up, [#5533](https://github.com/career-ops-hq/career-ops/pull/5533). VDAB, the job board of Flanders' public employment service, looks like the second case. A closed vacancy there keeps its page and its Apply button, and only a banner says it's closed. Reading the page while it renders, `main` read 13 of 84 loads of closed vacancies as open. My open PR, [#5703](https://github.com/career-ops-hq/career-ops/pull/5703), asks VDAB's per-vacancy API, which the page itself uses. One choice in it is deliberate: a "pending" vacancy reads `uncertain`, not closed, because one pending vacancy had never been published at all, and a closed verdict is recorded for good. Every PR adds tests that fail on `main` and pass with the fix. I checked each one with mutation variants: deliberately broken versions of the fix that the tests must catch. On [#4810](https://github.com/career-ops-hq/career-ops/pull/4810), 13 of 13 were caught; on [#4833](https://github.com/career-ops-hq/career-ops/pull/4833), 12 of 12. I also ran the full suite on each branch against `main`. Over three weeks it grew from about 11,600 to about 16,800 tests. Among the things the maintainer's merge notes thanked were the edge-case tests: in the keyword check, the ones that keep "IT", "HR" and "AWS" from matching ordinary words; in the checker, the one for a button that is hidden or sits in the navigation. ``` en ## Keywords extracted exit 0 coverage 80% de ## Extrahierte Keywords exit 1 No "## Keywords extracted" block found (or it is empty). es ## Palabras clave extraídas exit 1 same ko ## 추출한 키워드 exit 1 same zh ## ATS 关键词提取 exit 1 same ``` On the branch, all five exit 0 at 80%. Three more PRs are open: [#5535](https://github.com/career-ops-hq/career-ops/pull/5535) (HiringRoom's "Postularse" label, approved), [#5691](https://github.com/career-ops-hq/career-ops/pull/5691) (Tech in Asia's Apply link in the page header) and #5703 (VDAB's API). Since 9 October I've been on the project's contributors team. Thanks to the maintainer for quick merges and for merge notes that say exactly what they valued. All 16 merged PRs: [https://github.com/career-ops-hq/career-ops/pulls?q=is%3Apr+is%3Amerged+author%3Aroydonsequeira](https://github.com/career-ops-hq/career-ops/pulls?q=is%3Apr+is%3Amerged+author%3Aroydonsequeira)