# LAW-N Real-World Data Layer > Source: > Published: 2026-09-27 08:45:33+00:00 **Author:** Peace Thabiwa (PEACEBINFLOW) · SAGEWORKS AI **Status:** v1.0 — ready to post **Scope:** The full LAW-N Kaggle series built to date — 10 notebooks, from intro/setup through canonical builder, KPI collection, core laws, simulation, provenance, policy, risk scoring, and NSQL evaluation. LAW-N has moved past two notebooks. Ten now exist, forming an actual pipeline: raw telecom data goes in one end, and a queryable, policy-aware, risk-scored network state comes out the other. This whitepaper does three things: **Core finding, in one sentence:** the pipeline currently reports success while silently propagating empty or fallback-filled frames, because no schema or non-empty check exists anywhere between raw ingestion and the six notebooks that depend on it. The series is not ten independent notebooks — it's a single fan-out pipeline with one load-bearing joint. Everything from Notebook 5 onward consumes whatever the Canonical Builder (#4) hands it. `\`` mermaid flowchart TD subgraph L0["Layer 0 — Setup"] N1["#1 LAW-N Intro & Setup"] end ``` subgraph L1["Layer 1 — Ingestion & Normalization"] N2["#2 Real-World Dataset Builder (Cellular)"] N3["#3 Telecom KPI Collector (Multi-Source)"] end subgraph L2["Layer 2 — Canonical Fan-Out Point"] N4["#4 Telecom Canonical Builder"] GATE{{"⚠ NO VALIDATION GATE HERE\n(proposed in §6.3 / §6.4)"}} end subgraph L3["Layer 3 — Downstream Consumers"] N5["#5 Core Laws & Baseline Evaluation"] N6["#6 Signal Simulation & Time Windows"] N7["#7 Event Provenance & Causal Tracing"] N8["#8 Device Profiles & Policy Enforcement"] N9["#9 Risk Scoring & Severity"] end subgraph L4["Layer 4 — Query / Evaluation"] N10["#10 NSQL Core & Multi-LAW Evaluation"] end N1 --> N2 --> N3 --> N4 N4 --> GATE GATE --> N5 & N6 & N7 & N8 & N9 N5 & N6 & N7 & N8 & N9 --> N10 style GATE fill:#ffdddd,stroke:#cc0000,stroke-width:2px ``` ``\` **Why this matters:** notebooks #2–#4 are the load-bearing wall of the whole series. If the canonical layer is wrong, every downstream evaluation, score, and policy decision inherits that error silently — there is currently nothing at the `GATE` node above to stop it. The previous version of this review inferred the fallback problem by reading the code. A **captured execution log** from Notebook 2's actual Kaggle run confirms it happened, not just that it could: `\`` text 16.9s df["timestamp"] = pd.date_range(start="2025-01-01", periods=len(df), freq="S") FutureWarning: 'S' is deprecated and will be removed in a future version, please use 's' instead. 16.9s has_large_values = (abs_vals > 1e6).any() RuntimeWarning: invalid value encountered in greater 16.9s has_small_values = ((abs_vals < 10 ** (-self.digits)) & (abs_vals > 0)).any() RuntimeWarning: invalid value encountered in less ``\` | Log line | What triggered it | What it proves | |---|---|---| | `pd.date_range(..., freq="S")` | Only executes when `TIMESTAMP_COL` fails to match | The **fallback path ran in production** , against the real 16,829-row source — not just in theory | | `RuntimeWarning: invalid value encountered in greater` | `abs_vals > 1e6` compared against NaN | Pandas silently fails while formatting output — direct symptom of a frame that's **mostly empty defaults** | | `RuntimeWarning: invalid value encountered in less` | `abs_vals < 10**(-digits)` compared against NaN | Same failure mode, second comparison — confirms it's not a one-off | | `FutureWarning: 'S' is deprecated` | Deprecated pandas frequency alias | A **dated time bomb** : it works today, but the code path that's actually executing — the fallback — is built on an alias pandas will remove | A viewer skimming a `Normalized shape: (16829, 8)` summary line would have no reason to suspect any of this. The shape looks healthy. The values behind it are not. A "selected columns" export was pulled from the same source: `\` Columns: Timestamp, Locality, Latitude, Longitude, Signal Strength (dBm), Signal Quality (%), Data Throughput (Mbps), Latency (ms), Network Type, BB60C Measurement (dBm) \`\` All 10 columns are **correctly named**, matching the real schema exactly. Every one of the **4,468 data rows beneath it is empty.** This is a different failure mode from Section 3's column-mapping miss, and arguably more dangerous: `\`` mermaid flowchart LR A[Source CSV\n16,829 rows] --> B[Selection/Export step] B --> C["signal_metrics-selected-columns.csv\n✅ 10/10 correct column names\n❌ 4,468/4,468 rows empty"] C -->|"Schema-shape check"| PASS1[["✅ PASS — names match"]] C -->|"'Did it error?' check"| PASS2[["✅ PASS — ran clean"]] C -->|"Row-count / null-rate check"| FAIL[["❌ FAIL — 0% populated\n(does not currently exist)"]] ``` style PASS1 fill:#d4f7d4 style PASS2 fill:#d4f7d4 style FAIL fill:#ffdddd,stroke:#cc0000 ``` A pipeline stage watching only for "do the column names match" would pass this file. A pipeline stage watching only for "did normalization run without error" would also pass it. **Neither check catches an empty payload wearing a correct header** — which is why §6.2 proposes a row-count/null-rate assertion as a distinct, mandatory test. | # | Gap | Evidence | Why it matters | |---|---|---|---| | 1 | Column mapping never resolved | Notebook 2 execution log, `freq="S"` fallback firing live | Confirmed at **runtime** , not just in the source code | | 2 | Fallback data produces silent NaN formatting failures | Two `RuntimeWarning` s in the same log | The pipeline reports success while the underlying frame is mostly empty | | 3 | Deprecated pandas API in the fallback path | `FutureWarning: 'S' is deprecated` | The exact code path actually running will break on a future pandas upgrade | | 4 | Header-correct, data-empty exports | `signal_metrics-selected-columns.csv` — 10 correct columns, 4,468 blank rows | Schema-shape checks alone will not catch this; row-count/null-rate checks are required | | 5 | Canonical layer is a single point of failure | System map, §2 | Six downstream notebooks (#5–#10) inherit whatever #4 produces, with no validation gate in between | | 6 | No schema validation layer anywhere in the 10-notebook chain | All notebooks | Nothing in the current series stops a malformed or empty frame from propagating to Risk Scoring or NSQL | | 7 | No alignment to an external KPI standard | Dataset Builder, KPI Collector, Canonical Builder | `latency_ms` /`signal_strength` are self-defined, not checked against 3GPP TS 32.450 / TS 32.425 | **Severity read:** Gaps 1–4 are *observed defects* (proven by the log and the export file). Gaps 5–7 are *structural risks* — the reason gaps 1–4 were able to happen undetected, and the reason similar failures will recur without a fix at the architecture level, not just a patch to Notebook 2. Each test below is scoped to one gap from §5 and includes the actual check, not just a description of it. Treat `FutureWarning` / `RuntimeWarning` in the execution log as **build failures**, not noise — both are now confirmed symptoms of the fallback path firing. `\`` python import re, sys FAIL_PATTERNS = [ r"FutureWarning", r"RuntimeWarning: invalid value encountered", ] def check_papermill_log(log_path: str) -> None: with open(log_path) as f: text = f.read() hits = [p for p in FAIL_PATTERNS if re.search(p, text)] if hits: raise SystemExit(f"CI FAIL: warning patterns present in log: {hits}") ``\` Before any export or normalized file is accepted downstream, assert `row_count > 0` **and** `non_null_rate > threshold` per required column. `\`` python import pandas as pd REQUIRED_COLUMNS = [ "Timestamp", "Locality", "Latitude", "Longitude", "Signal Strength (dBm)", "Signal Quality (%)", "Data Throughput (Mbps)", "Latency (ms)", "Network Type", "BB60C Measurement (dBm)", ] def assert_non_empty_payload(df: pd.DataFrame, min_fill_rate: float = 0.95) -> None: if len(df) == 0: raise ValueError("Payload has zero rows.") for col in REQUIRED_COLUMNS: fill_rate = df[col].notna().mean() if fill_rate < min_fill_rate: raise ValueError( f"Column '{col}' is {fill_rate:.1%} filled — " f"below required {min_fill_rate:.0%} threshold" ) ``\` Declare the canonical schema once with `pandera` and validate every frame against it before it moves from Canonical Builder (#4) into Core Laws (#5): `\`` python import pandera.pandas as pa lawn_canonical_schema = pa.DataFrameSchema( { "timestamp": pa.Column("datetime64[ns]", nullable=False), "region": pa.Column(str, nullable=False), "provider": pa.Column(str, nullable=False), "latency_ms": pa.Column(float, pa.Check.ge(0), nullable=True), "signal_strength": pa.Column(float, pa.Check.in_range(-140, 0), nullable=True), "packet_loss": pa.Column(float, pa.Check.in_range(0, 1)), }, checks=pa.Check(lambda df: len(df) > 0, error="canonical frame is empty"), ) lawn_canonical_schema.validate(canonical_df, lazy=True) ``\` Because #5–#10 all branch from the canonical layer (§2), add **one** validation gate at that single point rather than six separate ones — cheaper to build, and it's the only place a fix covers every downstream notebook at once. `\`` mermaid` flowchart TD N4["#4 Canonical Builder"] --> GATE{"Validation Gate\n§6.1 + §6.2 + §6.3"} GATE -- pass --> FANOUT["#5–#9 (5 notebooks)"] GATE -- fail --> STOP(["Pipeline halts,\nalert raised"]) FANOUT --> N10["#10 NSQL"] style GATE fill:#fff3cd,stroke:#cc8400,stroke-width:2px style STOP fill:#ffdddd,stroke:#cc0000 \`\` Compare `latency_ms` / `signal_strength` against how those terms are formally defined in **3GPP TS 32.450** (KPI definitions) and **TS 32.425** (the underlying E-UTRAN performance measurements) — so "latency" means the same thing here that it means in an actual RAN performance report. Re-run any column-selection/export step against a frozen source and diff row counts against the last accepted export — this is what would have caught the empty `selected-columns.csv` before it left the pipeline. `\`` python` def assert_export_matches_baseline(new_export: pd.DataFrame, baseline_row_count: int, tolerance: float = 0.02) -> None: delta = abs(len(new_export) - baseline_row_count) / max(baseline_row_count, 1) if delta > tolerance: raise ValueError( f"Export row count drifted {delta:.1%} from baseline " f"({len(new_export)} vs {baseline_row_count})" ) \`\` `\`` mermaid flowchart LR subgraph BEFORE["Current state — no gate"] direction TB A1["Canonical Builder\noutputs empty/fallback frame"] --> A2["Risk Scoring"] A1 --> A3["Policy Enforcement"] A1 --> A4["NSQL"] A2 --> A5(["Risk score computed\non NaN latency —\nlooks valid, isn't"]) style A5 fill:#ffdddd,stroke:#cc0000 end ``` subgraph AFTER["With §6.4 gate in place"] direction TB B1["Canonical Builder\noutputs empty/fallback frame"] --> B2{"Gate"} B2 -- fail --> B3(["One loud failure,\nraised at the source"]) style B3 fill:#d4f7d4,stroke:#2b8a2b end ``` Right now, an empty or fallback-filled canonical frame can reach Risk Scoring & Severity or Policy Enforcement with **nothing to stop it** — a risk score computed on NaN latency is still a risk score, it's just meaningless. One validation point between Canonical Builder and everything downstream turns **six silent failure surfaces into one loud one.** | Priority | Item | Addresses | |---|---|---| | 1 | Fix the real column mapping in the Dataset Builder — resolve the placeholder columns against the actual `signal_metrics.csv` fields | Gap 1 | | 2 | Add the `pandera` gate directly after the Canonical Builder, before any of #5–#10 run | Gaps 5, 6 | | 3 | Add the runtime-warning and non-empty-payload checks to the CI/papermill run | Gaps 2, 3, 4 | | 4 | Trace and fix the selection/export path that produced a header-correct, data-empty file | Gap 4 | | 5 | Align `latency_ms` /`signal_strength` to 3GPP TS 32.450 / TS 32.425 | Gap 7 | | 6 | Add a second real source once the gate is in place, so multi-source merging (#3) is finally tested at N > 1 | Structural coverage | | Layer | # | Notebook | Role | |---|---|---|---| | **0 — Setup** | 1 | LAW-N Intro & Setup | Series entry point | | **1 — Ingestion & Normalization** | 2 | Real-World Dataset Builder (Cellular) | Raw → first normalization pass | | **1 — Ingestion & Normalization** | 3 | Telecom Real-World KPI Collector (Multi-Source) | Multi-source merge architecture | | **2 — Canonical / Fan-out** | 4 | Telecom Real-World Canonical Builder | Canonical schema — the fan-out point | | **3 — Downstream Consumers** | 5 | Real-World Core Laws & Baseline Evaluation | Baseline law evaluation | | **3 — Downstream Consumers** | 6 | Signal Simulation & Time Windows | Windowed simulation | | **3 — Downstream Consumers** | 7 | Event Provenance & Causal Tracing | Causal trace of events | | **3 — Downstream Consumers** | 8 | Device Profiles & Policy Enforcement | Policy layer | | **3 — Downstream Consumers** | 9 | LAW-N Risk Scoring & Severity | Risk/severity scoring | | **4 — Query / Evaluation** | 10 | NSQL Core & Multi-LAW Evaluation | Query layer across laws | `download.txt`, referenced in §3.