{"slug": "open-world-perception-safe-handling-of-unknown-road-objects-via-a-taxonomy", "title": "Open-world perception: safe handling of unknown road objects via a taxonomy", "summary": "A new open-world perception system for driving scenes safely handles unknown road objects by using a hierarchical taxonomy, abstraction fallback, and segmentation cross-validation, as described in a reference implementation of F. Schaller's paper 'The Role of Semantic Models in Constraining Pattern Recognition in Modern AI Systems' (Intelligent Environments 2025, doi:10.3233/AISE250023). The system, built with YOLO, CLIP, and CLIPSeg, avoids misclassifying novel objects by abstracting up the taxonomy to a confident, safety-useful category, and validates detections against physical context and pixel-level segmentation.", "body_md": "Object detection for driving scenes (YOLO-style), but with a twist: instead of a\n**flat class list**, objects live in a **hierarchical taxonomy**. When a novel\nor ambiguous object appears — a horse-drawn carriage, an unknown work vehicle —\nthe system does **not** drop it or force a wrong leaf label. It **abstracts up\nthe taxonomy** until it reaches a category that is still confident *and*\nsafety-useful, then **validates** the detection against physical/semantic\ncontext.\n\nThis is a reference implementation of the runtime idea in:\n\nF. Schaller,\n\nThe Role of Semantic Models in Constraining Pattern Recognition in Modern AI Systems, Intelligent Environments 2025 (IOS Press), doi:10.3233/AISE250023.\n\n-\n**Abstraction fallback (don't fall into the void).** A detection descends the taxonomy top-down. At each level it only commits one step deeper if the evidence is decisive. If it becomes ambiguous, it*stops*— reporting the last confident, coarser category (e.g.`Truck`\n\n→`Transport Vehicle`\n\n) rather than a made-up leaf. -\n**Abstraction floor (don't become paranoid).** The more abstract a category, the more trivially true it is — labelling everything`Object`\n\nwould make a planner brake for the whole world. So every safety-actionable node carries a`floor`\n\nflag (🛡:`Vehicle`\n\n,`Living Being`\n\n,`Static Object`\n\n). Abstraction may never stop*above*the nearest floor. If even the floor is not confident, the detection becomes an explicit,**localized** handed to a conservative policy — not a useless generic bucket.`UNKNOWN OBSTACLE`\n\nPlus a **context-validation layer** that rejects physically implausible\ndetections (the paper's \"car flying above the clouds\" example): size and\nposition plausibility per category, inherited down the taxonomy.\n\n**Segmentation cross-validation (a second, independent perception path).** Alongside the box path, an**open-vocabulary semantic segmenter**(CLIPSeg) densely labels every pixel into its own stuff/things taxonomy (`segmentation.yaml`\n\n: road, sidewalk, vehicle, person, vegetation, sky, …). Each box is then cross-checked against it: a detection is**corroborated** when the pixels under it agree with its taxonomy branch, and**rejected** when they contradict it — the*data-driven*version of the position gate (a`Vehicle`\n\nbox sitting on`sky`\n\npixels is the classic flying-car false positive). Because the two paths come from different model families (YOLO+CLIP vs. CLIPSeg), their agreement is genuine evidence. Safety-first: an`UNKNOWN OBSTACLE`\n\nis never vetoed by segmentation.\n\n```\nimage\n  ├─ YOLO (ultralytics)            → boxes (\"there is an object here\")\n  │    └─ per box:\n  │         clip mode : CLIP zero-shot top-down descent over taxonomy nodes\n  │         yolo mode : COCO→taxonomy + confidence-based abstraction\n  │    └─ constraint validation    → reject implausible detections\n  └─ CLIPSeg (optional 2nd path)   → dense pixel labels (stuff/things)\n       └─ per box: cross-check region vs. taxonomy branch (confirm / conflict)\n  └─ annotated image + segmentation overlay + taxonomy tree\n```\n\n| File | Role |\n|---|---|\n`taxonomy.yaml` |\nThe semantic model: the tree + floors + constraints |\n`segmentation.yaml` |\nSegmentation stuff/things taxonomy (2nd path) |\n`hpercept/taxonomy.py` |\nTree loading, traversal, floor logic |\n`hpercept/detector.py` |\nYOLO wrapper (lazy model load) |\n`hpercept/classifier.py` |\nCLIP zero-shot over all taxonomy nodes |\n`hpercept/segmenter.py` |\nCLIPSeg open-vocab segmenter (lazy model load) |\n`hpercept/abstraction.py` |\nCore: hierarchical descent + floor fallback |\n`hpercept/constraints.py` |\nPhysical/semantic + segmentation validation |\n`hpercept/datasets.py` |\nLazy, streaming dataset registry |\n`hpercept/pipeline.py` |\nWires it all together |\n`hpercept/viz.py` |\nAnnotated image + overlay + HTML taxonomy tree |\n`app.py` |\nGradio UI |\n\nNothing is downloaded up front. Pick a set in the dropdown and fetch N samples — only those images are streamed (Hugging Face streaming mode).\n\n| Set | Purpose |\n|---|---|\nCODA |\nReal-world corner cases / unknowns — the primary novelty set |\nBDD100K |\nLarge known baseline for contrast |\nRoad Anomaly |\nUnknown objects on the road (animals, odd vehicles) |\nLocal folder |\nYour own images / extracted video frames (`data/samples/` ) |\n\nBecause the classification is\n\ntraining-free(pretrained YOLO + CLIP zero-shot), these sets are used forevaluation/demo, not training.\n\n```\npython3.12 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\nHeaviest dependency is PyTorch (~2 GB). Models download lazily on first use (YOLOv8n ~6 MB, CLIP ViT-B-32 ~350 MB, CLIPSeg ~150 MB). The\n\nYOLO-onlymode works without CLIP if disk is tight, and segmentation is off by default (its weights are only fetched when you enable the cross-check).\n\n```\npython app.py            # Gradio UI at http://127.0.0.1:7860\npython smoke_test.py     # offline logic check, no model download\n```\n\n**Descend threshold**— how decisive evidence must be to go one level deeper. ↑ = abstracts sooner (more cautious / \"paranoid\"), ↓ = dives to leaves eagerly.**Enforce abstraction floor 🛡**— toggle the anti-paranoia limit on/off to see the difference between a bounded`UNKNOWN OBSTACLE`\n\nand collapse to`Object`\n\n.**Min CLIP similarity**— refuse to commit on weak visual evidence.** Segmentation cross-validation 🖇**— turn on the second perception path; each detection gets a ✅ confirm / ➖ neutral / ❌ conflict verdict (conflicts are rejected), plus a segmentation overlay.\n\nExperimental (v3, merged to main). See\n\n[docs/spikes/open_world_feasibility.md].\n\nThe 2D pipeline cannot tell a real object from a flat 2D depiction (a car on a\nbillboard, a poster, painted livery). A **3D mode** adds a depth signal and, per\ndetection, a **flatness / foreground test**: a flat panel has near-planar depth\n(low internal relief once a plane is removed), a real object has relief.\n\n**Depth source.** Spike uses**monocular** depth (Depth-Anything), so it runs on the existing images with no LiDAR dataset. Production would feed the*same*test from**LiDAR** projected into the image (metric depth); see`hpercept/openworld/lidar.py`\n\n(stub).**Verdicts per detection:**`3d`\n\n(assessable, has relief) ·`flat`\n\n(assessable, near-planar → likely a 2D false positive) ·`n/a`\n\n(too small / distant to judge — the scale limit; metric LiDAR would extend the assessable range).\n\nRun:\n\n```\npython scripts/depth3d_spike.py 6     # writes figures/depth3d_*.png\n```\n\n**Findings (honest).** The depth map cleanly separates foreground from\nbackground, and large upright objects read as `3d`\n\nwhile small/distant ones are\ncorrectly held `n/a`\n\n(see `figures/depth3d_04.png`\n\n: a foreground truck+trailer is\n`3d`\n\n, distant cars are `n/a`\n\n). Limits: monocular depth is *relative*, and the\nrelief metric is shape-sensitive (a compact animal can read low-relief); the\ndataset has no billboard to demonstrate a true `flat`\n\nrejection. A reliable\nbillboard/flatness test needs **metric** depth (LiDAR) and better geometry. This\nis a promising capability, not a finished one.\n\nThe product idea is **HOWC**: extend a detector with the hierarchical taxonomic\nclassifier so untrained objects still get a safe coarse label or an explicit\nUNKNOWN. A class-agnostic proposer (MobileSAM via ultralytics) supplies boxes for\nthings the closed set misses:\n\n```\npython scripts/howc_spike.py 8    # YOLO vs MobileSAM + hierarchical labels\n```\n\n**Findings (honest).** MobileSAM hugely increases recall (19 YOLO detections vs\n148 extra regions on 8 images) but over-proposes background (sky, grass, walls).\nThe UNKNOWN gate filters ~76% as UNKNOWN; the rest is still noisy (vegetation\nlabelled `Living Being`\n\n). Class-agnostic proposals therefore need an objectness /\n**geometry filter** (the depth mode above) to be usable. Synthesis of the three\nopen-world spikes: recall (SAM) + precision (depth/geometry) + semantics\n(hierarchy) are complementary; the hierarchy is the differentiator and already\nworks. See [docs/spikes/open_world_feasibility.md](/freshNfunky/IE2025-Research-Paper/blob/main/docs/spikes/open_world_feasibility.md).\n\n**HuggingFace.** The stack is HF-publishable as-is (CLIP, Depth-Anything,\nMobileSAM/YOLO are all Hub/ultralytics-hosted, fetched lazily). Intended: a HF\nSpace demoing HOWC and a model card whose novelty is the *taxonomic abstraction\nlayer* over open-vocabulary features, not new detector weights.\n\nEarly rough rig — end-to-end pipeline, taxonomy, abstraction floor, constraints, lazy datasets and UI are in place. Next: real dataset-id verification for every source, video/webcam input, and quantitative evaluation on CODA.", "url": "https://wpnews.pro/news/open-world-perception-safe-handling-of-unknown-road-objects-via-a-taxonomy", "canonical_source": "https://github.com/freshNfunky/IE2025-Research-Paper", "published_at": "2026-08-14 17:00:32+00:00", "updated_at": "2026-08-14 17:11:44.653870+00:00", "lang": "en", "topics": ["computer-vision", "artificial-intelligence", "machine-learning", "autonomous-vehicles"], "entities": ["F. Schaller", "YOLO", "CLIP", "CLIPSeg", "Intelligent Environments 2025", "IOS Press"], "alternates": {"html": "https://wpnews.pro/news/open-world-perception-safe-handling-of-unknown-road-objects-via-a-taxonomy", "markdown": "https://wpnews.pro/news/open-world-perception-safe-handling-of-unknown-road-objects-via-a-taxonomy.md", "text": "https://wpnews.pro/news/open-world-perception-safe-handling-of-unknown-road-objects-via-a-taxonomy.txt", "jsonld": "https://wpnews.pro/news/open-world-perception-safe-handling-of-unknown-road-objects-via-a-taxonomy.jsonld"}}