Did That Job Close, or Did Your Salary Filter Hide It?

A developer documented a retrieval-pipeline pitfall in which a job listing's disappearance from URL lookup results is wrongly treated as proof the role closed, when the lookup's own salary and category filters may have suppressed it. The writeup proposes separating URL resolution from preference matching, preserving the last known record with a freshness warning on unresolved checks, and building resolution payloads from scratch rather than reusing search payloads. It also includes a Python URL canonicalization check that rejects malformed or tracking-laden links so they cannot trigger archival.

A saved job can disappear from a URL check because your own salary filter rejected it. You're about to connect requested - returned to your monitoring agent's archive action. The small dread: the next notification could confidently announce that an open role has closed. Lab note 028: can a URL lookup justify that transition? The catch arrives early: our job Actor applies salary and category filters during byJobUrls resolution. A successfully resolved listing can still be skipped. A missing record from filtered byJobUrls results does not establish that a specific job closed. The operation combines finding a listing with deciding whether to emit it. Those are different questions, even when they share an input payload. For an LLM pipeline, the dangerous step happens afterward: your application converts absence into a factual statement, saves that statement, and retrieves it as evidence later. Now the model has a citation for your inference. Splendid paperwork. Wrong conclusion. We'd give the pipeline separate lanes: php saved URL - resolution check - resolved / unresolved | v preference check | v matched / unresolved match independent closure evidence - archive permission An unresolved check should preserve the last known record with its freshness warning. It shouldn't manufacture a fresh “closed” chunk. Malformed links, nonexistent listings, and closed listings are skipped. That makes input hygiene part of lifecycle safety: a damaged saved URL must not become evidence about the employer's hiring decision. This filter accepts a conservative subset of canonical listing URLs. It deliberately sends unfamiliar forms to review rather than attempting clever repairs. Example saved input for the checks below, without any claim that this listing is currently open: python import re from urllib.parse import urlsplit saved urls = "https://himalayas.app/companies/sola-payments/jobs/senior-data-platform-engineer" def canonical saved url value : if not isinstance value, str : return None try: parts = urlsplit value except ValueError: return None path ok = re.fullmatch r"/companies/ a-z0-9 - +/jobs/ a-z0-9 - +", parts.path, canonical = f"https://himalayas.app{parts.path}" if not path ok or parts.query or parts.fragment or value = canonical : return None return canonical checked = value, canonical saved url value for value in saved urls accepted urls = list dict.fromkeys url for , url in checked if url rejected urls = value for value, url in checked if url is None Passing this check establishes structure, not availability. A rejected tracking link or unfamiliar slug belongs in rejected urls , where it can't trigger archival. Deduplicating accepted inputs also prevents repeated checks from producing competing lifecycle decisions for the same saved URL. A salary floor, ceiling, or currency filter excludes listings without published pay. Category filtering can also suppress a resolved listing that doesn't match. That creates a particularly unpleasant retrieval bug: changing a user's preferences can delete the underlying job evidence. Their preferences changed; the employer's listing didn't necessarily change with them. The resolution payload should therefore be constructed from scratch. Copying your search payload and changing mode carries too much baggage. This builder omits preferences and makes the emission allowance cover its batch: python EMIT LIMIT = 5000 def lookup input batch : if not 1 <= len batch <= EMIT LIMIT: raise ValueError "Batch must fit within the emitted-record limit." run input = { "mode": "byJobUrls", "jobUrls": batch, "maxItems": len batch , } return run input This removes a source of ambiguity. It doesn't turn a missing result into a closure certificate. We'd also keep preference decisions in application metadata rather than embedding them into the job description. Otherwise, retrieval can confuse “outside your salary preferences” with a property of the listing's availability. The default maxItems is 20. It limits emitted job records, so a larger saved set can outgrow the allowance before you know anything about its unmatched members. A batch difference then becomes a mixture of lifecycle uncertainty and your own output budget. Asking an LLM to explain that difference only gives the uncertainty better grammar. This collector splits the accepted URLs and uses the builder above. Preference keys are permitted only for a separate matching pass: python import os from apify client import ApifyClient client = ApifyClient os.environ "APIFY TOKEN" PREFERENCE KEYS = {"minSalary", "maxSalary", "salaryCurrency", "categories"} def fetch jobs urls, preferences=None : preferences = {} if preferences is None else preferences if set preferences - PREFERENCE KEYS: raise ValueError "Unexpected preference key." rows = for offset in range 0, len urls , EMIT LIMIT : batch = urls offset:offset + EMIT LIMIT run input = lookup input batch run input.update preferences run = client.actor "vexaloom-labs/himalayas-remote-job-finder-with-salary-and-tech" .call run input=run input if run "status" = "SUCCEEDED": continue rows.extend client.dataset run "defaultDatasetId" .iterate items return rows An unsuccessful batch contributes no positive resolution evidence. An exception stops this program before it constructs archive candidates. Streaming dataset retrieval also avoids making a dataset page boundary look like another missing-job event. The status message reports how many URLs were closed or invalid. It doesn't establish a closure reason for every missing URL. That distinction matters when an agent writes explanations. An aggregate diagnostic can't support “your saved role closed” unless you've evidence connecting that reason to that identity. | Evidence available | Unsafe conclusion | Safe application action | |---|---|---| | Locally rejected URL | The role closed | Review the saved address | | Missing from preference results | The role closed | Leave lifecycle unchanged | | Missing from a capped batch | The role closed | Correct the batch allowance | | Aggregate closed-or-invalid count | This particular role closed | Keep its reason unresolved | For reconciliation, use jobGuid , the canonical listing URL, rather than trying to identify jobs by title or slug. This filter accepts only an unambiguous requested identity whose jobUrl agrees with it: python def index rows rows, requested : buckets = {} for row in rows: guid = canonical saved url row.get "jobGuid" if guid in requested: buckets.setdefault guid, .append row return { guid: matches 0 for guid, matches in buckets.items if len matches == 1 and canonical saved url matches 0 .get "jobUrl" == guid } resolution rows = fetch jobs accepted urls resolved = index rows resolution rows, set accepted urls unresolved urls = set accepted urls - set resolved Duplicate identities stay unresolved here instead of silently choosing whichever record arrived last. Unexpected identities never enter the saved-job index. Neither condition deserves a closure story. Reimplementing salary and category matching locally has its own traps. Annualized pay still needs currency context, and category matching includes parent categories and tech tags. For exact Actor matching, we'd use a separate filtered pass and reconcile it locally. This costs another lookup pass, but avoids inventing an equivalent classifier from partial fields. The combined filter below creates application-owned state keys. Positive preference results establish a match; absence remains unresolved because a later lookup can fail independently. Example preferences for that separate pass: preference filters = { "minSalary": 120000, "salaryCurrency": "USD", "categories": "python" , } preference rows = fetch jobs list resolved , preference filters matched = index rows preference rows, set resolved monitor = {} for url in accepted urls: preference state = "not checked" if url in resolved: preference state = "matched" if url in matched else "unresolved" monitor url = { "lifecycle state": "resolved" if url in resolved else "unresolved", "preference state": preference state, } That distinction lets your agent say, “The listing resolved, but this check didn't establish a preference match.” It also gives retrieval a useful boundary: preference uncertainty can affect recommendations without erasing the source record. Keep check metadata separate from description chunks so it doesn't get copied across every chunk or outlive its own freshness. We'd use this operation to resolve saved listings. A missing row can't authorize closure. The worked example is our Himalayas Remote Job Finder with Salary & Tech Stack https://apify.com/vexaloom-labs/himalayas-remote-job-finder-with-salary-and-tech . Collection supplies evidence; your application owns the lifecycle transition. This final guard requires a separate source of confirmed closure evidence and blocks archival when the current resolution returned the job: python confirmed closed urls = frozenset def can archive url, closure evidence : return url in closure evidence and url not in resolved archive candidates = { url for url in accepted urls if can archive url, confirmed closed urls } Populate that evidence set through an independently verified closure process. Neither the model nor a batch-difference handler should fill it by inference. Repeated absence can justify retry or review. Repeating an ambiguous check doesn't make its explanation specific. minSalary , salaryCurrency , and categories . jobGuid values. Keep unmatched inputs unresolved in both checks. Example filtered input, with a placeholder listing whose current availability and preference match aren't asserted: { "mode": "byJobUrls", "jobUrls": "https://himalayas.app/companies/sola-payments/jobs/senior-data-platform-engineer" , "minSalary": 120000, "salaryCurrency": "USD", "categories": "python" , "maxItems": 1 }