{"slug": "show-hn-a-benchmark-for-flight-search-intent", "title": "Show HN: A benchmark for flight search intent", "summary": "A Hacker News user released a benchmark for flight search intent, testing AI models on parsing natural-language queries into structured search fields. GPT-5.6-sol led with an 87% pass rate (212/244), followed by GPT-5.6-terra at 86% (209/244), Claude Opus 5 at 85% (207/244), and Claude Sonnet 5 at 79% (192/244). The benchmark evaluates dimensions such as origin, destination, travel dates, and cabin class across German, English, and Spanish queries.", "body_md": "| 1 |\n**gpt-5.6-sol** medium |\n87%\n212/244\n|\n|\n**86%**\n*n=59*\n|\n**87%**\n*n=153*\n|\n**67%**\n*n=12*\n|\n**84%**\n*n=116*\n|\n**83%**\n*n=109*\n|\n**100%**\n*n=7*\n|\n**100%**\n*n=3*\n|\n— |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n240/244 | search type *oneway · roundtrip · multi* |\n223/233 | origin *departure locations* |\n225/233 | destination *arrival locations* |\n220/233 | travel dates *departure and return* |\n180/192 | stay length *nights* |\n72/82 | passengers *adults, children, infants* |\n15/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n28/32 | date validity *no past or out-of-order dates* |\n229/229 | trip shape *leg count, return xor duration* |\n228/229 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 742 / 242 / 120 |\n| total tokens | 240k |\n| total cost | no price set |\n| cost per case | — |\n| median latency | 3.9s |\n| errored calls | 0 |\n### By language| de |\n43/50 |\n|\n86% | | en |\n130/150 |\n|\n87% | | es |\n39/44 |\n|\n89% |\n|\n| 2 |\n**gpt-5.6-terra** medium |\n86%\n209/244\n|\n|\n**93%**\n*n=59*\n|\n**86%**\n*n=153*\n|\n**17%**\n*n=12*\n|\n**79%**\n*n=116*\n|\n**79%**\n*n=109*\n|\n**100%**\n*n=7*\n|\n**100%**\n*n=3*\n|\n— |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n237/244 | search type *oneway · roundtrip · multi* |\n222/233 | origin *departure locations* |\n222/233 | destination *arrival locations* |\n223/233 | travel dates *departure and return* |\n179/192 | stay length *nights* |\n66/82 | passengers *adults, children, infants* |\n16/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n31/32 | date validity *no past or out-of-order dates* |\n226/226 | trip shape *leg count, return xor duration* |\n225/226 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 742 / 205 / 86 |\n| total tokens | 231k |\n| total cost | no price set |\n| cost per case | — |\n| median latency | 3.3s |\n| errored calls | 0 |\n### By language| de |\n45/50 |\n|\n90% | | en |\n125/150 |\n|\n83% | | es |\n39/44 |\n|\n89% |\n|\n| 3 |\n**claude-opus-5** medium |\n85%\n207/244\n|\n|\n**92%**\n*n=59*\n|\n**86%**\n*n=153*\n|\n**75%**\n*n=12*\n|\n**86%**\n*n=116*\n|\n**80%**\n*n=109*\n|\n**0%**\n*n=7*\n|\n**67%**\n*n=3*\n|\n— |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n231/244 | search type *oneway · roundtrip · multi* |\n224/233 | origin *departure locations* |\n224/233 | destination *arrival locations* |\n213/233 | travel dates *departure and return* |\n182/192 | stay length *nights* |\n79/82 | passengers *adults, children, infants* |\n16/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n29/32 | date validity *no past or out-of-order dates* |\n230/236 | trip shape *leg count, return xor duration* |\n235/236 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 2725 / 272 / 0 |\n| total tokens | 731k |\n| total cost | $2.09 |\n| cost per case | $0.0086 |\n| median latency | 4.6s |\n| errored calls | 0 |\n### By language| de |\n37/50 |\n|\n74% | | en |\n137/150 |\n|\n91% | | es |\n33/44 |\n|\n75% |\n|\n| 4 |\n**claude-sonnet-5** medium |\n79%\n192/244\n|\n|\n**86%**\n*n=59*\n|\n**80%**\n*n=153*\n|\n**46%**\n*n=13*\n|\n**77%**\n*n=116*\n|\n**73%**\n*n=110*\n|\n**0%**\n*n=7*\n|\n**67%**\n*n=3*\n|\n2 |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n231/243 | search type *oneway · roundtrip · multi* |\n218/232 | origin *departure locations* |\n217/232 | destination *arrival locations* |\n214/232 | travel dates *departure and return* |\n179/190 | stay length *nights* |\n75/82 | passengers *adults, children, infants* |\n16/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n28/32 | date validity *no past or out-of-order dates* |\n229/236 | trip shape *leg count, return xor duration* |\n231/236 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 2725 / 302 / 0 |\n| total tokens | 742k |\n| total cost | $0.89 |\n| cost per case | $0.0036 |\n| median latency | 3.9s |\n| errored calls | 2 |\n### By language| de |\n36/50 |\n|\n72% | | en |\n125/151 |\n|\n83% | | es |\n31/44 |\n|\n70% |\n|\n| 5 |\n**gemini-3.5-flash-lite** medium |\n74%\n181/244\n|\n|\n**85%**\n*n=59*\n|\n**77%**\n*n=153*\n|\n**8%**\n*n=12*\n|\n**68%**\n*n=116*\n|\n**64%**\n*n=109*\n|\n**0%**\n*n=7*\n|\n**67%**\n*n=3*\n|\n25 |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n207/219 | search type *oneway · roundtrip · multi* |\n201/211 | origin *departure locations* |\n199/211 | destination *arrival locations* |\n194/211 | travel dates *departure and return* |\n156/171 | stay length *nights* |\n61/69 | passengers *adults, children, infants* |\n16/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n24/28 | date validity *no past or out-of-order dates* |\n204/209 | trip shape *leg count, return xor duration* |\n209/209 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 92 / 897 / 748 |\n| total tokens | 241k |\n| total cost | no price set |\n| cost per case | — |\n| median latency | 3.0s |\n| errored calls | 25 |\n### By language| de |\n35/50 |\n|\n70% | | en |\n120/150 |\n|\n80% | | es |\n26/44 |\n|\n59% |\n|\n| 6 |\n**gemini-3.5-flash** medium |\n74%\n181/244\n|\n|\n**80%**\n*n=59*\n|\n**78%**\n*n=153*\n|\n**25%**\n*n=12*\n|\n**72%**\n*n=116*\n|\n**72%**\n*n=109*\n|\n**0%**\n*n=7*\n|\n**67%**\n*n=3*\n|\n32 |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n207/212 | search type *oneway · roundtrip · multi* |\n201/204 | origin *departure locations* |\n196/204 | destination *arrival locations* |\n189/204 | travel dates *departure and return* |\n165/167 | stay length *nights* |\n65/65 | passengers *adults, children, infants* |\n15/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n24/26 | date validity *no past or out-of-order dates* |\n204/209 | trip shape *leg count, return xor duration* |\n209/209 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 92 / 986 / 830 |\n| total tokens | 263k |\n| total cost | no price set |\n| cost per case | — |\n| median latency | 3.8s |\n| errored calls | 32 |\n### By language| de |\n30/50 |\n|\n60% | | en |\n120/150 |\n|\n80% | | es |\n31/44 |\n|\n70% |\n|\n| 7 |\n**gemini-3.8-flash** medium |\n74%\n181/244\n|\n|\n**83%**\n*n=59*\n|\n**73%**\n*n=153*\n|\n**8%**\n*n=12*\n|\n**63%**\n*n=116*\n|\n**63%**\n*n=109*\n|\n**100%**\n*n=7*\n|\n**100%**\n*n=3*\n|\n44 |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n187/200 | search type *oneway · roundtrip · multi* |\n175/189 | origin *departure locations* |\n175/189 | destination *arrival locations* |\n174/189 | travel dates *departure and return* |\n141/150 | stay length *nights* |\n49/57 | passengers *adults, children, infants* |\n14/15 | cabin *cabin class* |\n5/5 | filters *stops, price, bags, connections* |\n20/21 | date validity *no past or out-of-order dates* |\n176/176 | trip shape *leg count, return xor duration* |\n176/176 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 92 / 834 / 704 |\n| total tokens | 226k |\n| total cost | no price set |\n| cost per case | — |\n| median latency | 2.8s |\n| errored calls | 44 |\n### By language| de |\n31/50 |\n|\n62% | | en |\n118/150 |\n|\n79% | | es |\n32/44 |\n|\n73% |\n|\n| 8 |\n**gpt-5.6-luna** medium |\n71%\n174/244\n|\n|\n**81%**\n*n=59*\n|\n**71%**\n*n=153*\n|\n**17%**\n*n=12*\n|\n**66%**\n*n=116*\n|\n**64%**\n*n=109*\n|\n**57%**\n*n=7*\n|\n**100%**\n*n=3*\n|\n— |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n234/244 | search type *oneway · roundtrip · multi* |\n223/233 | origin *departure locations* |\n204/233 | destination *arrival locations* |\n205/233 | travel dates *departure and return* |\n171/192 | stay length *nights* |\n61/82 | passengers *adults, children, infants* |\n15/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n28/32 | date validity *no past or out-of-order dates* |\n226/229 | trip shape *leg count, return xor duration* |\n228/229 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 742 / 257 / 135 |\n| total tokens | 244k |\n| total cost | no price set |\n| cost per case | — |\n| median latency | 3.4s |\n| errored calls | 0 |\n### By language| de |\n36/50 |\n|\n72% | | en |\n106/150 |\n|\n71% | | es |\n32/44 |\n|\n73% |\n|\n| 9 |\n**claude-haiku-4-5** |\n53%\n129/244\n|\n|\n**41%**\n*n=59*\n|\n**56%**\n*n=153*\n|\n**17%**\n*n=12*\n|\n**47%**\n*n=116*\n|\n**31%**\n*n=109*\n|\n**71%**\n*n=7*\n|\n**100%**\n*n=3*\n|\n— |\n*▸* |\n### Per-dimension *pass rate over the repeats that scored it*\naction *search the query or reject it* |\n204/244 | search type *oneway · roundtrip · multi* |\n171/233 | origin *departure locations* |\n180/233 | destination *arrival locations* |\n174/233 | travel dates *departure and return* |\n143/192 | stay length *nights* |\n61/82 | passengers *adults, children, infants* |\n14/16 | cabin *cabin class* |\n6/6 | filters *stops, price, bags, connections* |\n23/32 | date validity *no past or out-of-order dates* |\n184/197 | trip shape *leg count, return xor duration* |\n177/197 |\n### Cost & latency\ntokens in / out / reasoning\n*mean per call* | 2373 / 112 / 0 |\n| total tokens | 606k |\n| total cost | $0.72 |\n| cost per case | $0.0029 |\n| median latency | 1.9s |\n| errored calls | 0 |\n### By language| de |\n19/50 |\n|\n38% | | en |\n79/150 |\n|\n53% | | es |\n31/44 |\n|\n70% |\n|", "url": "https://wpnews.pro/news/show-hn-a-benchmark-for-flight-search-intent", "canonical_source": "https://fjmatrix.github.io/flight-search-intent-eval/", "published_at": "2026-09-03 13:59:00+00:00", "updated_at": "2026-09-03 14:23:12.075356+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["GPT-5.6-sol", "GPT-5.6-terra", "Claude Opus 5", "Claude Sonnet 5", "Hacker News"], "alternates": {"html": "https://wpnews.pro/news/show-hn-a-benchmark-for-flight-search-intent", "markdown": "https://wpnews.pro/news/show-hn-a-benchmark-for-flight-search-intent.md", "text": "https://wpnews.pro/news/show-hn-a-benchmark-for-flight-search-intent.txt", "jsonld": "https://wpnews.pro/news/show-hn-a-benchmark-for-flight-search-intent.jsonld"}}