A wrong "closed" never comes back: measuring a job-posting liveness checker on real pages

A developer contributed 16 merged pull requests to career-ops, the open-source job-search pipeline, most of them fixing its job-posting liveness checker. Testing 35 live postings across 19 boards behind real DNS filtering resolvers, the developer found that on main, 12 postings behind AdGuard DNS and 20 behind Control D were wrongly marked "blocked" because the egress guard treated sinkholed tracker addresses as page-level blocks; a fix tags only filter-sinkholed refusals as dns_sinkhole while still refusing the request, and a follow-up makes failed DNS lookups on the posting's own host read "uncertain" instead of permanently dropping the posting, keeping 19 of 19 postings that main discarded.

career-ops https://github.com/career-ops-hq/career-ops is an open-source job-search pipeline: Node.js scripts driven by AI coding CLIs, with about 76k stars on GitHub. Before it adds a posting to your pipeline, scan --verify checks that the posting is still open. Where the job board has a public API, it asks the API. Otherwise it loads the page in a headless browser and reads it. A "closed" or "blocked" verdict goes into your scan history for good, and that posting is skipped on every later scan. So the two ways to be wrong cost very different amounts. A wrong "open" costs one more check later. A wrong "closed" makes a live job disappear from your search without a trace. Between 3 and 20 October 2026 I had 16 pull requests merged into career-ops, most of them in this liveness checker. This is what I found, how I measured it, and what the fixes look like. The board was busy: 135 open PRs and 208 open issues on 11 October. The project's CONTRIBUTING lets bug fixes go in without an issue, so instead of competing for claimed issues I read modules nobody was working on. I started with the ATS keyword and CV skill checks, which are close to my NLP background, then moved to the liveness checker. For the checker, I ran it on real postings and read every disagreement. The sources were public job-postings datasets LinkedIn, USAJOBS and live boards without an API. Several fixes also came out of the project's own conversations: a merge note from the maintainer, a follow-up the review bot offered, and three follow-ups from my reviews of other contributors' PRs. Many people run a DNS filter that blocks ads and trackers: Pi-hole, AdGuard DNS, Control D, a hosts-file blocklist. Such a filter answers 0.0.0.0 for a tracker it blocks. The browser check has a security guard that refuses requests to private network addresses, and 0.0.0.0 is on its list. Refusing the tracker is right. But the guard also marked the whole page "blocked", which scan --verify records permanently. I found it while reviewing another contributor's PR that touches how the guard handles DNS answers. To measure it, I took 35 postings on 19 boards with no API, all live the day before, and checked each one with main 's code and with the fix, behind two real filtering resolvers and with no filter as a control: main , 12 postings behind AdGuard DNS and 20 behind Control D read "blocked". The fix 5015 https://github.com/career-ops-hq/career-ops/pull/5015 leaves the guard's decision alone. The request is still refused, with the same error. The guard only tags the refusal when the only private addresses in the answer are the filter's: js if blocked.length 0 { const denied = new Error Access denied: Egress guard blocked private target IP ${blocked 0 .ip} ; if blocked.every { hit } = SINKHOLE ADDRESSES.has hit denied.livenessCode = 'dns sinkhole'; throw denied; } A tagged refusal of a tracker, a script or an image no longer decides the page's verdict. Any other private address still counts, even next to 0.0.0.0 . I rejected the simpler option, ignoring every private answer for subresources, because it would drop the guard's signal on a host that really resolves into a private network. The follow-up, 5075 https://github.com/career-ops-hq/career-ops/pull/5075 , covers the posting's own host. When that host had no usable address, the guard reported "blocked". But no address is often about the run, not the posting: a DNS outage during the scan, or the user's own filter blocking the job board. Now it reads uncertain , and the posting is kept for another check. With the posting's own host failing DNS, 19 of 19 postings from 19 boards were dropped permanently on main , and 19 of 19 were kept with the fix. The PR states the trade-off: a domain that is really gone now reads uncertain too, because one failed lookup can't tell a dead domain from an outage. A kept dead posting costs one more check later. A pinned live one never comes back. This one turned up while I was measuring another fix: a posting whose host no longer resolved broke the checks after it in the same run. I reproduced it on the code from before my DNS change, to rule that out, and traced it to the browser library the next day. The checker runs every browser check in one tab. When a page fails to load its host has no address, its certificate has expired, its connection is reset , the browser shows its error page a moment after the checker has moved on, and that interrupts the next page load. The interrupted page then finishes loading late and interrupts the one after it, often to the end of the run. Those checks fail as load errors, so scan --verify keeps the postings without having checked them, and the closed ones among them go into the user's pipeline. The root is a known Playwright issue microsoft/playwright 35944 https://github.com/microsoft/playwright/issues/35944 , closed upstream without a fix . Through the real CLI, on one list of 45 URLs with 6 failing loads spread among 39 postings that were live the day before: Over 36 failed loads, the browser's error page landed 1 to 69 ms after the failure was reported, median 4 ms. So the fix 5299 https://github.com/career-ops-hq/career-ops/pull/5299 waits for the browser's own signal instead of a fixed delay, for at most 2 s: async function navigateSettled page, navigate { try { return await navigate ; } catch err { const canWait = typeof page.waitForEvent === 'function' && typeof page.mainFrame === 'function'; if canWait && err?.name == 'TimeoutError' { await page .waitForEvent 'framenavigated', { predicate: frame = frame === page.mainFrame && frame.url .startsWith 'chrome-error:' , timeout: FAILED NAVIGATION COMMIT MS, } .catch = {} ; } throw err; } } A timeout is left alone: a timed-out load shows no error page, and the next load cancels it, so waiting would only add 2 s. I rejected parking the tab on a blank page after a failure, which another part of the code does after a search, because that load is interrupted the same way. The review bot pointed out that the old page changing its own address could end the wait too early. I couldn't make that happen in real Chromium in 26 tries, but fixed it anyway, with a test the first version fails. CI installs no browser, so the tests use a stand-in page that plays out the browser's order of events. The same sequences also run on real Chromium where it can launch. I filed 4832 https://github.com/career-ops-hq/career-ops/issues/4832 for live postings on seven non-English boards that read as having no Apply button. getManfred, a Spanish tech-job board, was one. Its Apply button says "¡Apúntame " "sign me up" , which the checker didn't know, so every live posting read "no Apply control". getManfred's own feed marks every offer open or closed, so the A/B could measure against the board instead of against my reading of pages. The first run showed a case the issue had missed: one closed posting read open on both versions. Closed postings written in English often have an FAQ button such as "Can I apply if I live outside Spain?", and the English Apply pattern took it for an Apply button. The page does say the offer is closed, but that sentence renders near the bottom, after the checker's first look on some pages. Above the title, from the start, a closed posting says when it closed "Oferta cerrada el 27/07/2023" . Closing patterns are read before the Apply button, so an FAQ button can't outvote the date. The fix 5350 https://github.com/career-ops-hq/career-ops/pull/5350 is two patterns: /\boferta cerrada el \d{1,2}\/\d{1,2}\/\d{4}/i, // a closing pattern, read before the Apply button /\bapuntame\b/i, // an Apply label, accent-free like the others I left out the verb "aplicar": 73 of the 111 getManfred pages I measured have an FAQ button that uses it, closed ones included. On 136 real page loads: No Fluff Jobs, a Polish tech-job board, was the last of my board families from 4832, and the checker read it backwards. A live posting puts its Apply button "Aplikuj" inside a page header, which the checker skips as page chrome, so every live posting read "no Apply control". A closed posting moves to a search page with a promo link that also says "Aplikuj", so most closed postings read as open. I'd planned label patterns. Then I found that the board's per-posting API says directly whether a posting is open. So the fix 5428 https://github.com/career-ops-hq/career-ops/pull/5428 is an API rung, as several other boards already have. The API answers "published" open , "expired" closed or "does not exist". A status the rung doesn't know reads uncertain , never closed, and an answer it can't read falls back to the browser. Through the real CLI on 102 postings 40 live from the board's search API, 59 closed from Wayback Machine archives, 1 gone and 2 made up : The review bot raised two valid points on it. I measured that they change no verdict on today's API answers and shipped them as a follow-up, 5533 https://github.com/career-ops-hq/career-ops/pull/5533 . VDAB, the job board of Flanders' public employment service, looks like the second case. A closed vacancy there keeps its page and its Apply button, and only a banner says it's closed. Reading the page while it renders, main read 13 of 84 loads of closed vacancies as open. My open PR, 5703 https://github.com/career-ops-hq/career-ops/pull/5703 , asks VDAB's per-vacancy API, which the page itself uses. One choice in it is deliberate: a "pending" vacancy reads uncertain , not closed, because one pending vacancy had never been published at all, and a closed verdict is recorded for good. Every PR adds tests that fail on main and pass with the fix. I checked each one with mutation variants: deliberately broken versions of the fix that the tests must catch. On 4810 https://github.com/career-ops-hq/career-ops/pull/4810 , 13 of 13 were caught; on 4833 https://github.com/career-ops-hq/career-ops/pull/4833 , 12 of 12. I also ran the full suite on each branch against main . Over three weeks it grew from about 11,600 to about 16,800 tests. Among the things the maintainer's merge notes thanked were the edge-case tests: in the keyword check, the ones that keep "IT", "HR" and "AWS" from matching ordinary words; in the checker, the one for a button that is hidden or sits in the navigation. en Keywords extracted exit 0 coverage 80% de Extrahierte Keywords exit 1 No " Keywords extracted" block found or it is empty . es Palabras clave extraídas exit 1 same ko 추출한 키워드 exit 1 same zh ATS 关键词提取 exit 1 same On the branch, all five exit 0 at 80%. Three more PRs are open: 5535 https://github.com/career-ops-hq/career-ops/pull/5535 HiringRoom's "Postularse" label, approved , 5691 https://github.com/career-ops-hq/career-ops/pull/5691 Tech in Asia's Apply link in the page header and 5703 VDAB's API . Since 9 October I've been on the project's contributors team. Thanks to the maintainer for quick merges and for merge notes that say exactly what they valued. All 16 merged PRs: https://github.com/career-ops-hq/career-ops/pulls?q=is%3Apr+is%3Amerged+author%3Aroydonsequeira https://github.com/career-ops-hq/career-ops/pulls?q=is%3Apr+is%3Amerged+author%3Aroydonsequeira