{"slug": "decomposed-relation-extraction-from-aviation-accident-narratives-evaluating", "title": "Decomposed Relation Extraction from Aviation Accident Narratives: Evaluating Transformer-Based Safety Knowledge Discovery under Relational Sparsity", "summary": "A decomposed relation-extraction framework separating relation classification from concept identification failed to hold up under exhaustive evaluation on 300 National Transportation Safety Board (NTSB) aviation accident narratives, according to a study posted to arXiv (2610.27013v1). The benchmark's 2,524 unique concept-text nodes expanded into 21,038 ordered candidate pairs, of which only 248 (1.18%) represented RESULTS_IN or CONTRIBUTES_TO relations; development Macro-F1 scores of 0.4738 for a TF-IDF Linear SVM and 0.4809 for a weighted RoBERTa classifier fell to 0.2741 and 0.2420 under sealed exhaustive evaluation, with Matthews correlation coefficients of 0.0128 and 0.0197. False positives dominated the results — the SVM recovered 96 reference relations but generated 7,018 false positives, while RoBERTa recovered 125 but generated 9,824 — leading the authors to conclude that known concept endpoints do not guarantee reliable relation extraction and that sampled-negative development performance may poorly reflect exhaustive evaluation.", "body_md": "arXiv:2610.27013v1 Announce Type: new \nAbstract: Relation extraction from aviation accident narratives is challenging because genuine safety relations are sparse among possible concept pairs. This study evaluates a decomposed framework separating relation classification from concept identification. A benchmark of 300 National Transportation Safety Board (NTSB) narratives containing 2,524 unique concept-text nodes was expanded into 21,038 ordered candidate pairs, with only 248 (1.18%) representing RESULTS_IN or CONTRIBUTES_TO relations. A TF-IDF Linear Support Vector Machine (SVM) and weighted RoBERTa classifier were developed using an independent graph-annotated resource with hard-negative sampling and frozen before sealed evaluation. Development Macro-F1 scores of 0.4738 (SVM) and 0.4809 (RoBERTa) declined to 0.2741 and 0.2420 under exhaustive evaluation, with Matthews correlation coefficients of 0.0128 and 0.0197. False positives dominated: the SVM recovered 96 reference relations but generated 7,018 false positives, while RoBERTa recovered 125 but generated 9,824. Post-test diagnostics revealed overlapping RoBERTa confidence distributions and false positives even within valid relation-category directions. Complementary end-to-end SmolLM2 evaluation identified missed and unsupported relations alongside malformed outputs. These findings demonstrate that known concept endpoints do not guarantee reliable relation extraction and that sampled-negative development performance may poorly reflect exhaustive evaluation. Controlling false positives and managing candidate-space sparsity remain essential for aviation safety knowledge extraction.", "url": "https://wpnews.pro/news/decomposed-relation-extraction-from-aviation-accident-narratives-evaluating", "canonical_source": "https://arxiv.org/abs/2610.27013", "published_at": "2026-10-21 04:00:00+00:00", "updated_at": "2026-10-21 04:18:10.643809+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "natural-language-processing", "ai-safety"], "entities": ["National Transportation Safety Board", "RoBERTa", "SmolLM2", "arXiv", "Support Vector Machine"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/decomposed-relation-extraction-from-aviation-accident-narratives-evaluating", "markdown": "https://wpnews.pro/news/decomposed-relation-extraction-from-aviation-accident-narratives-evaluating.md", "text": "https://wpnews.pro/news/decomposed-relation-extraction-from-aviation-accident-narratives-evaluating.txt", "jsonld": "https://wpnews.pro/news/decomposed-relation-extraction-from-aviation-accident-narratives-evaluating.jsonld"}}