Open-world perception: safe handling of unknown road objects via a taxonomy

A new open-world perception system for driving scenes safely handles unknown road objects by using a hierarchical taxonomy, abstraction fallback, and segmentation cross-validation, as described in a reference implementation of F. Schaller's paper 'The Role of Semantic Models in Constraining Pattern Recognition in Modern AI Systems' (Intelligent Environments 2025, doi:10.3233/AISE250023). The system, built with YOLO, CLIP, and CLIPSeg, avoids misclassifying novel objects by abstracting up the taxonomy to a confident, safety-useful category, and validates detections against physical context and pixel-level segmentation.

Object detection for driving scenes YOLO-style , but with a twist: instead of a flat class list , objects live in a hierarchical taxonomy . When a novel or ambiguous object appears — a horse-drawn carriage, an unknown work vehicle — the system does not drop it or force a wrong leaf label. It abstracts up the taxonomy until it reaches a category that is still confident and safety-useful, then validates the detection against physical/semantic context. This is a reference implementation of the runtime idea in: F. Schaller, The Role of Semantic Models in Constraining Pattern Recognition in Modern AI Systems, Intelligent Environments 2025 IOS Press , doi:10.3233/AISE250023. - Abstraction fallback don't fall into the void . A detection descends the taxonomy top-down. At each level it only commits one step deeper if the evidence is decisive. If it becomes ambiguous, it stops — reporting the last confident, coarser category e.g. Truck → Transport Vehicle rather than a made-up leaf. - Abstraction floor don't become paranoid . The more abstract a category, the more trivially true it is — labelling everything Object would make a planner brake for the whole world. So every safety-actionable node carries a floor flag 🛡: Vehicle , Living Being , Static Object . Abstraction may never stop above the nearest floor. If even the floor is not confident, the detection becomes an explicit, localized handed to a conservative policy — not a useless generic bucket. UNKNOWN OBSTACLE Plus a context-validation layer that rejects physically implausible detections the paper's "car flying above the clouds" example : size and position plausibility per category, inherited down the taxonomy. Segmentation cross-validation a second, independent perception path . Alongside the box path, an open-vocabulary semantic segmenter CLIPSeg densely labels every pixel into its own stuff/things taxonomy segmentation.yaml : road, sidewalk, vehicle, person, vegetation, sky, … . Each box is then cross-checked against it: a detection is corroborated when the pixels under it agree with its taxonomy branch, and rejected when they contradict it — the data-driven version of the position gate a Vehicle box sitting on sky pixels is the classic flying-car false positive . Because the two paths come from different model families YOLO+CLIP vs. CLIPSeg , their agreement is genuine evidence. Safety-first: an UNKNOWN OBSTACLE is never vetoed by segmentation. image ├─ YOLO ultralytics → boxes "there is an object here" │ └─ per box: │ clip mode : CLIP zero-shot top-down descent over taxonomy nodes │ yolo mode : COCO→taxonomy + confidence-based abstraction │ └─ constraint validation → reject implausible detections └─ CLIPSeg optional 2nd path → dense pixel labels stuff/things └─ per box: cross-check region vs. taxonomy branch confirm / conflict └─ annotated image + segmentation overlay + taxonomy tree | File | Role | |---|---| taxonomy.yaml | The semantic model: the tree + floors + constraints | segmentation.yaml | Segmentation stuff/things taxonomy 2nd path | hpercept/taxonomy.py | Tree loading, traversal, floor logic | hpercept/detector.py | YOLO wrapper lazy model load | hpercept/classifier.py | CLIP zero-shot over all taxonomy nodes | hpercept/segmenter.py | CLIPSeg open-vocab segmenter lazy model load | hpercept/abstraction.py | Core: hierarchical descent + floor fallback | hpercept/constraints.py | Physical/semantic + segmentation validation | hpercept/datasets.py | Lazy, streaming dataset registry | hpercept/pipeline.py | Wires it all together | hpercept/viz.py | Annotated image + overlay + HTML taxonomy tree | app.py | Gradio UI | Nothing is downloaded up front. Pick a set in the dropdown and fetch N samples — only those images are streamed Hugging Face streaming mode . | Set | Purpose | |---|---| CODA | Real-world corner cases / unknowns — the primary novelty set | BDD100K | Large known baseline for contrast | Road Anomaly | Unknown objects on the road animals, odd vehicles | Local folder | Your own images / extracted video frames data/samples/ | Because the classification is training-free pretrained YOLO + CLIP zero-shot , these sets are used forevaluation/demo, not training. python3.12 -m venv .venv source .venv/bin/activate pip install -r requirements.txt Heaviest dependency is PyTorch ~2 GB . Models download lazily on first use YOLOv8n ~6 MB, CLIP ViT-B-32 ~350 MB, CLIPSeg ~150 MB . The YOLO-onlymode works without CLIP if disk is tight, and segmentation is off by default its weights are only fetched when you enable the cross-check . python app.py Gradio UI at http://127.0.0.1:7860 python smoke test.py offline logic check, no model download Descend threshold — how decisive evidence must be to go one level deeper. ↑ = abstracts sooner more cautious / "paranoid" , ↓ = dives to leaves eagerly. Enforce abstraction floor 🛡 — toggle the anti-paranoia limit on/off to see the difference between a bounded UNKNOWN OBSTACLE and collapse to Object . Min CLIP similarity — refuse to commit on weak visual evidence. Segmentation cross-validation 🖇 — turn on the second perception path; each detection gets a ✅ confirm / ➖ neutral / ❌ conflict verdict conflicts are rejected , plus a segmentation overlay. Experimental v3, merged to main . See docs/spikes/open world feasibility.md . The 2D pipeline cannot tell a real object from a flat 2D depiction a car on a billboard, a poster, painted livery . A 3D mode adds a depth signal and, per detection, a flatness / foreground test : a flat panel has near-planar depth low internal relief once a plane is removed , a real object has relief. Depth source. Spike uses monocular depth Depth-Anything , so it runs on the existing images with no LiDAR dataset. Production would feed the same test from LiDAR projected into the image metric depth ; see hpercept/openworld/lidar.py stub . Verdicts per detection: 3d assessable, has relief · flat assessable, near-planar → likely a 2D false positive · n/a too small / distant to judge — the scale limit; metric LiDAR would extend the assessable range . Run: python scripts/depth3d spike.py 6 writes figures/depth3d .png Findings honest . The depth map cleanly separates foreground from background, and large upright objects read as 3d while small/distant ones are correctly held n/a see figures/depth3d 04.png : a foreground truck+trailer is 3d , distant cars are n/a . Limits: monocular depth is relative , and the relief metric is shape-sensitive a compact animal can read low-relief ; the dataset has no billboard to demonstrate a true flat rejection. A reliable billboard/flatness test needs metric depth LiDAR and better geometry. This is a promising capability, not a finished one. The product idea is HOWC : extend a detector with the hierarchical taxonomic classifier so untrained objects still get a safe coarse label or an explicit UNKNOWN. A class-agnostic proposer MobileSAM via ultralytics supplies boxes for things the closed set misses: python scripts/howc spike.py 8 YOLO vs MobileSAM + hierarchical labels Findings honest . MobileSAM hugely increases recall 19 YOLO detections vs 148 extra regions on 8 images but over-proposes background sky, grass, walls . The UNKNOWN gate filters ~76% as UNKNOWN; the rest is still noisy vegetation labelled Living Being . Class-agnostic proposals therefore need an objectness / geometry filter the depth mode above to be usable. Synthesis of the three open-world spikes: recall SAM + precision depth/geometry + semantics hierarchy are complementary; the hierarchy is the differentiator and already works. See docs/spikes/open world feasibility.md /freshNfunky/IE2025-Research-Paper/blob/main/docs/spikes/open world feasibility.md . HuggingFace. The stack is HF-publishable as-is CLIP, Depth-Anything, MobileSAM/YOLO are all Hub/ultralytics-hosted, fetched lazily . Intended: a HF Space demoing HOWC and a model card whose novelty is the taxonomic abstraction layer over open-vocabulary features, not new detector weights. Early rough rig — end-to-end pipeline, taxonomy, abstraction floor, constraints, lazy datasets and UI are in place. Next: real dataset-id verification for every source, video/webcam input, and quantitative evaluation on CODA.