Reconstructing the benchmark behind Luc Julia's 64% LLM reliability claim

A reconstruction of the reasoning benchmark behind Luc Julia's repeated claim that large language models are 'relevant 64% of the time' shows the figure comes from a single sentence in Bang et al. (2023), which reported 63.41% accuracy on 634 hand-picked questions, not the 'millions' of facts Julia has cited. The reconstruction, released as a runnable repository, scores GPT-3.5 at 63.41%, GPT-5.6 Sol at 96.50%, and Claude Opus 5 at 98.50% on a balanced 200-question subset. Julia, Chief Scientific Officer at Renault and often called a Siri co-creator, repeated the claim before the French Senate on 18 June 2025, but the original experiment was a reasoning evaluation, not a fact-verification task, and the rounding from 63.41% to 64% is mathematically incorrect.

A runnable version of the reasoning benchmark behind the "64%" accuracy figure that Luc Julia often introduced in French media as a co-creator of Siri has repeated for years as the general "reliability rate" of large language models. The number comes from a single sentence in the abstract of Bang et al. 2023 https://aclanthology.org/2023.ijcnlp-main.45/ whose reasoning evaluation graded a December 2022 ChatGPT snapshot on 634 hand-picked questions drawn from 22 series across ten categories. This repository rebuilds that evaluation from the paper and its released extraction code, reduces it to a balanced 200-question set, and documents an explicit grading rule so that any model can be scored on the same task today. Everything needed to reproduce a run is here: the questions and gold answers in data.csv , ten shuffled part .csv files to hand to a model, the merge and scoring scripts, and a per-benchmark description of where each item comes from. | Model | Score | Source | |---|---|---| GPT-3.5 | 63.41% | Original paper | GPT-5.6 Sol | 96.50% | This reconstruction | Claude Opus 5 | 98.50% | This reconstruction | The "64%" figure incorrectly rounded from 63.41% has had a public life well beyond the paper it comes from. In France it circulates as a general "reliability rate" of large language models. Its most visible proponent is Luc Julia, Chief Scientific Officer at Renault and routinely introduced in French media as a co-creator of Siri, who has repeated it for years in interviews, in his 2025 book, and on 18 June 2025 before the French Senate's Commission des affaires économiques: generative AIs are, in his words, "relevant 64% of the time ... that is, 36% of the time the AI says that something true is false." He has also stated on several occasions that the figure is lower today than when it was measured, because the web that models train on is now saturated with AI-generated text. Sources: Senate hearing https://www.senat.fr/actualite/ia-audition-de-luc-julia-concepteur-de-siri-5387.html video https://videos.senat.fr/video.5486945 685259f55eac4.ia--audition-de-luc-julia-concepteur-de-siri , Public Sénat coverage https://www.publicsenat.fr/actualites/societe/intelligence-artificielle-laudition-de-luc-julia-le-createur-de-siri-qui-a-enflamme-les-reseaux-sociaux , European Scientist interview https://www.europeanscientist.com/fr/opinion/chatgpt-cest-64-de-pertinence-il-faut-toujours-douter-de-lia-luc-julia-interview/ . Critical analyses of the claim: AFIS https://www.afis.org/La-controverse-autour-de-Luc-Julia-sur-l-intelligence-artificielle , nocodefunctions https://nocodefunctions.com/blog/luc-julia-ia-nuances-confusions/ . Three problems are worth separating: - The experiment described is not the experiment that was run. Julia has presented the number as the result of a study in which researchers submitted a very large set of verified facts "millions", in some tellings to ChatGPT and asked it to judge each one true or false. Bang et al. did nothing of the kind. The 63.41% is answer accuracy over 634 manually judged trials , hand-selected from 22 series across ten reasoning categories: deductive, inductive, abductive, mathematical, temporal, spatial, commonsense, causal, multi-hop, and analogical. There is no true/false fact-verification task anywhere in it, and no "millions" of anything: the entire reasoning evaluation is the 634 items reconstructed in this pack. The provenance is not even stable across retellings: the same interview points to arXiv:2302.12095 https://arxiv.org/abs/2302.12095 , a different paper on robustness to adversarial and misspelled prompts, and at least one version attributes the decline to an OpenAI report that nobody has been able to locate. The rounding does not survive contact either: 63.41 does not round to 64. It is possible the "64" comes from the February 2023 v1 abstract, which reported 64.33% over 600 samples before the final version revised it, but that is a reconstruction, not something the claim itself makes clear. - A single accuracy number is not a property of a model. It is a property of a model and a fixed question set. Bang et al. deliberately assembled hard reasoning items on which a late-2022 model was expected to struggle; move the difficulty and the same model returns any figure between 0% and 100%. Speaking of "the reliability rate of LLMs", without naming the questions, is a category error, and it is the reason a number measured once on one December 2022 snapshot cannot be carried forward as a standing property of a technology. - The claim that today's models would score lower is testable, which is the point of this pack. Benchmark after benchmark has gone from near-zero to saturated across successive model generations, so the prediction that a 2026 model underperforms GPT-3.5 on GPT-3.5's own evaluation set is a strong one. Rather than argue it, this repository reconstructs the evaluation and runs it: the results are in the table above. Note what this does and does not settle. It measures current models on the exact task behind the 63.41%, under a documented and auditable grading rule. It does not measure "reliability" in general, and no honest reading of a single benchmark could. This pack reconstructs only the reasoning evaluation behind the 63.41% sentence in the abstract of Bang et al. 2023 , not the paper's summarization, translation, sentiment, dialogue, multimodal, misinformation, or hallucination experiments. The final paper evaluated the 15 December 2022 ChatGPT snapshot in a zero-shot, manually judged setting. Primary references: ACL Anthology paper https://aclanthology.org/2023.ijcnlp-main.45/ , current arXiv record https://arxiv.org/abs/2302.04023 , arXiv v1 PDF https://arxiv.org/pdf/2302.04023v1 , and official extraction repository https://github.com/HLTCHKUST/chatgpt-evaluation . The paper and code do not define a single, fully reproducible formula that independently yields 63.41%. This reconstruction therefore preserves the released prompts, source ordering, and the final paper's trial allocation as closely as possible, while making the scoring rule explicit. - The February 2023 v1 abstract reported 64.33% and 600 reasoning samples; the final paper reports 63.41% and 634 samples. - For the Mathematics Dataset, Section 3 and Appendix F.2 say 7/30 , while Table 5 says 13/30 . - One sentence in the main text swaps bAbI task 15/16 labels, whereas Appendix Table 11 and the repository consistently treat task 15 as deductive and task 16 as inductive. - The final 634-trial accounting can be reconstructed as 17 series of 30, two SpaRTQA series of 32, and three StepGame diagnostics of 20: 17×30 + 2×32 + 3×20 = 634 . This count includes separate as-is and prompt-engineered runs over the same bAbI items, plus overlapping StepGame diagnostic items. - The released StepGame basic function removes ID 4 for a wrong gold label and thus returns 29 items even though the paper reports 30. The reduced set follows the code and excludes that item. - The released analogy extractor names a legacy NPZ that is absent upstream. The two selected problem-type indices 4 and 5 correspond to fix alphabet and sort in the upstream generator, so those types are reconstructed from the available all prob.npz . The reduced set contains exactly 200 rows . Counts were chosen by simple, interpretable sub-allocations rather than by forcing every benchmark to the same size: most 30-item series contribute 9 rows; MATH contributes 2 from each of 6 modules; each SpaRTQA series contributes 3 from each of 4 question types; the three StepGame diagnostics contribute 6 each; PEP-3k contributes 5 positive and 5 negative predicates; and letter analogies contribute 5 from each of 2 problem types. The 200 questions themselves live in data.csv and in the ten part .csv files. The per-benchmark sections that follow document each series and show one sample question from it, so that the prompt format and the expected-answer convention are visible without reproducing the whole set here. | Category | Benchmark / series | Original trials | Reduced trials | |---|---|---|---| | Deductive reasoning | EntailmentBank | 30 | 9 | | Deductive reasoning | bAbI task 15 as-is | 30 | 9 | | Deductive reasoning | bAbI task 15 prompt-engineered | 30 | 9 | | Inductive reasoning | CLUTRR | 30 | 9 | | Inductive reasoning | bAbI task 16 as-is | 30 | 9 | | Inductive reasoning | bAbI task 16 prompt-engineered | 30 | 9 | | Abductive reasoning | alphaNLI | 30 | 9 | | Mathematical reasoning | DeepMind Mathematics Dataset | 30 | 12 | | Temporal reasoning | TimeDial | 30 | 9 | | Spatial reasoning | SpaRTQA one reasoning type | 32 | 12 | | Spatial reasoning | SpaRTQA two reasoning types | 32 | 12 | | Spatial reasoning | StepGame hard, k=9 | 30 | 9 | | Spatial reasoning | StepGame basic, k=1 | 30 | 9 | | Spatial reasoning | StepGame basic cardinal diagnostic | 20 | 6 | | Spatial reasoning | StepGame diagonal diagnostic | 20 | 6 | | Spatial reasoning | StepGame clock-position diagnostic | 20 | 6 | | Commonsense reasoning | CommonsenseQA | 30 | 9 | | Commonsense reasoning | PIQA | 30 | 9 | | Commonsense reasoning | PEP-3k | 30 | 10 | | Causal reasoning | E-CARE | 30 | 9 | | Multi-hop reasoning | HotpotQA | 30 | 9 | | Analogical reasoning | Letter-string analogies | 30 | 10 | Total | 634 | 200 | First 9 examples of the Task 1 test split, matching the official extractor's first-N rule. The v3 Task 1 data are used. The prompt concatenates the provided entailment facts and the ARC-derived question. Dataset/source https://allenai.org/data/entailmentbank ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L422-L471 . | Sample question | Expected answer | |---|---| | if a place is in summer, then it will have the most sunlight. the northern hemisphere is a kind of hemisphere of earth. a hemisphere of earth is a kind of place. Melinda learned that days in some seasons have more daylight hours than in other seasons. Which season receives the most hours of sunlight in the Northern Hemisphere? | summer | First 9 story-question blocks from qa15 basic-deduction test.txt. The numbered bAbI facts and question are retained as in the official extractor. Dataset/source https://research.facebook.com/downloads/babi/ ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L319-L420 . | Sample question | Expected answer | |---|---| | 1 Wolves are afraid of mice. 2 Sheep are afraid of mice. 3 Winona is a sheep. 4 Mice are afraid of cats. 5 Cats are afraid of wolves. 6 Jessica is a mouse. 7 Emily is a cat. 8 Gertrude is a wolf. 9 What is emily afraid of? 10 What is winona afraid of? 11 What is gertrude afraid of? 12 What is jessica afraid of? | wolf / mouse / mouse / cat | The same first 9 task-15 blocks, transformed with the repository's deductive-reasoning prompt. These are deliberately separate trials: the underlying questions duplicate the as-is set, but the prompts differ. Dataset/source https://research.facebook.com/downloads/babi/ ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L319-L420 . | Sample question | Expected answer | |---|---| | Given facts: Wolves are afraid of mice. Sheep are afraid of mice. Winona is a sheep. Mice are afraid of cats. Cats are afraid of wolves. Jessica is a mouse. Emily is a cat. Gertrude is a wolf. The most recent fact is the correct fact. Based on the given facts above, do a reasonable inference on this question using deductive reasoning: What is emily afraid of? What is winona afraid of? What is gertrude afraid of? What is jessica afraid of? | wolf / mouse / mouse / cat | First 9 test examples from config gen train23 test2to10. The clean story is followed by the kinship query in the same orientation as the official extractor. Dataset/source https://huggingface.co/datasets/CLUTRR/v1 ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L522-L564 . | Sample question | Expected answer | |---|---| | Scott and Lewis are brothers. Jason is father of their father. Who is Lewis to Jason? | grandson | First 9 story-question blocks from qa16 basic-induction test.txt. The numbered bAbI facts and question are retained as in the official extractor. Dataset/source https://research.facebook.com/downloads/babi/ ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L319-L420 . | Sample question | Expected answer | |---|---| | 1 Lily is a swan. 2 Bernhard is a lion. 3 Greg is a swan. 4 Bernhard is white. 5 Brian is a lion. 6 Lily is gray. 7 Julius is a rhino. 8 Julius is gray. 9 Greg is gray. 10 What color is Brian? | white | The same first 9 task-16 blocks, transformed with the repository's inductive-reasoning prompt. These are deliberately separate trials: the underlying questions duplicate the as-is set, but the prompts differ. Dataset/source https://research.facebook.com/downloads/babi/ ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L319-L420 . | Sample question | Expected answer | |---|---| | Given facts: Lily is a swan. Bernhard is a lion. Greg is a swan. Bernhard is white. Brian is a lion. Lily is gray. Julius is a rhino. Julius is gray. Greg is gray. The most recent fact is the correct fact. Based on the given facts above, do a reasonable inference on this question using inductive reasoning: What color is Brian? | white | First 9 examples and labels from the alphaNLI test split. The expected answer gives both the A/B label and its hypothesis text; the original labels are 1/2. Dataset/source https://abductivecommonsense.xyz/ ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L473-L520 . | Sample question | Expected answer | |---|---| | Given: Jane was a professor teaching piano to students. Then: Jane spent the morning sipping coffee and reading a book. Select the most plausible explanation hypothesis : A. Two of Jane's students were early for their lessons. B. None of Jane's students had a lesson that day. | B. None of Jane's students had a lesson that day. | First 2 of the 5 paper-selected examples in each of 6 modules: linear algebra, add/subtract, differentiation, closest comparison, unit conversion, and base conversion. The official repository redistributes the exact 30 paper examples in math deepmind 30 samples.csv; this subset preserves all six modules. Dataset/source https://github.com/google-deepmind/mathematics dataset ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L115-L141 . | Sample question | Expected answer | |---|---| | Solve -282 d + 929 - 178 = -1223 for d. | 7 | First 9 test examples, matching the official extractor's first-N rule. TimeDial exposes two correct and two incorrect candidate completions; both correct candidates are accepted unless the second is 'none'. Dataset/source https://github.com/google-research-datasets/TimeDial ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L143-L178 . | Sample question | Expected answer | |---|---| | Given the conversation: A:We need to take the accounts system offline to carry out the upgrade . But don't worry , it won't cause too much inconvenience . We're going to do it over the weekend . B: How long will the system be down for ? A: We'll be taking everything offline in about two hours ' time . It'll be down for a minimum of twelve hours . If everything goes according to plan , it should be up again by 6 pm on Saturday . B: That's fine . We've allowed <MASK to be on the safe side . Candidate choices to fill in the <mask : 0. forty-eight hours, 1. 50 hours , 2. two hours , 3. 12 days , Note that there may not be enough information to certainly fill in the <mask , but from commonsense reasoning, you can surely narrow down what are the most probable choices to fill in the <mask . Please select the most propable choice from candidates and explain your choice. | 0. forty-eight hours OR 1. 50 hours | From the first 3 human-test contexts, take the first FR, YN, CO, and FB question whose reasoning type list has length 1 3 per question type . This mirrors the official selection loop and preserves the four SpaRTQA question types equally. Dataset/source https://github.com/HLR/SpartQA generation ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L44-L113 . | Sample question | Expected answer | |---|---| | Given the description: 'There is a block called A. In A there is a red triangle Which is above a black circle. The black circle is touching the bottom of A. A yellow square is touching the top edge of A and below that there is a red oval is touching the right edge of A. There is a second block call B Which is to the left of A. Inside B there are 3 squares, a brown one, a green one, and a red one. The green square is above the red square and touching the left edge of B. The brown square is touching both the bottom edge and the right edge of B. The red square is touching the left edge of the brown square.' . What is the relation between the yellow square and the black object? 0. left, 1. right, 2. above, 3. below, 4. near to, 5. far from, 6. touching, 7. don't know | 2. above; 5. far from | Scan the human-test set and take the first 3 FR, YN, CO, and FB questions whose reasoning type list has length 2. This mirrors the official balanced-by-question-type loop. Dataset/source https://github.com/HLR/SpartQA generation ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L44-L113 . | Sample question | Expected answer | |---|---| | Given the description: 'There are three blocks called A, B, and C. In A, there is a pink circle Which is near and to the left of a yellow square. B is to the left and far from A. In B, there is a red rectangle touching both the left edge of B and a green triangle. The green triangle is touching the bottom of B. There is also a white circle to the right of the green triangle. C is below both A and B. In C, there is a blue oval that is on top of a black rectangle. The black rectangle is touching the bottom of C.' . What block is far from the white thing? 0. A, 1. B, 2. C | A | First 9 entries of qa9 valid.json. The prompt is the repository's k=9 wording and asks for the spatial relation. Dataset/source https://github.com/ZhengxiangShi/StepGame ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L214-L276 . | Sample question | Expected answer | |---|---| | Given the description: S and A are horizontal and S is to the right of A. A is to the left of R with a small gap between them. V is positioned above T and to the right. P is placed on the top of Z. S is south east of Y. H is at the 6 o'clock position relative to P. R is to the top-left of Z. U is positioned in the lower left corner of T. H is to the bottom left of V.. What is the spatial relation, e.g left, right, above lower-left, .. of the agent Y to the agent P? | upper-left | First 9 valid entries from IDs 0-29 after excluding ID 4, which the official extractor flags as incorrectly labelled. The paper reports 30 items, while the released function actually returns 29 after the exclusion. Dataset/source https://github.com/ZhengxiangShi/StepGame ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L214-L276 . | Sample question | Expected answer | |---|---| | J is diagonally above B to the right at a 45 degree. What is the relation of the agent B to the agent J? Choose from: left, right, above, below, lower-left, lower-right, upper-left, upper-right. | lower-left | First 6 IDs from the paper repository's explicit basic-cardinal list. This is one of the three extra k=1 diagnostic samples used in the paper's 634-trial count. Dataset/source https://github.com/ZhengxiangShi/StepGame ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L214-L276 . | Sample question | Expected answer | |---|---| | C is sitting at the top position to Y. What is the relation of the agent Y to the agent C? Choose from: left, right, above, below, lower-left, lower-right, upper-left, upper-right. | below | First 6 IDs from the paper repository's explicit diagonal list. This is one of the three extra k=1 diagnostic samples used in the paper's 634-trial count. Dataset/source https://github.com/ZhengxiangShi/StepGame ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L214-L276 . | Sample question | Expected answer | |---|---| | If C is the center of a clock face, Z is located between 4 and 5. What is the relation of the agent Z to the agent C? Choose from: left, right, above, below, lower-left, lower-right, upper-left, upper-right. | lower-right | First 6 IDs from the paper repository's explicit clock-position list. This is one of the three extra k=1 diagnostic samples used in the paper's 634-trial count. Dataset/source https://github.com/ZhengxiangShi/StepGame ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L214-L276 . | Sample question | Expected answer | |---|---| | Y is sitting at the 12:00 position to U. What is the relation of the agent U to the agent Y? Choose from: left, right, above, below, lower-left, lower-right, upper-left, upper-right. | below | First 9 examples of the validation split, matching the official extractor. The expected answer gives both the choice letter and text. Dataset/source https://huggingface.co/datasets/tau/commonsense qa ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L566-L612 . | Sample question | Expected answer | |---|---| | A revolving door is convenient for two direction travel, but it also serves as a security measure at a what? A. bank, B. library, C. department store, D. mall, E. new york | A. bank | First 9 examples of the validation split, matching the official extractor. The expected answer gives both the 0/1 label and solution text. Dataset/source https://huggingface.co/datasets/ybisk/piqa ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L614-L665 . | Sample question | Expected answer | |---|---| | Pick from option 0 or 1 to achieve this goal:How do I ready a guinea pig cage for it's new occupants? 0: "Provide the guinea pig with a cage full of a few inches of bedding made of ripped paper strips, you will also need to supply it with a water bottle and a food dish." 1: "Provide the guinea pig with a cage full of a few inches of bedding made of ripped jeans material, you will also need to supply it with a water bottle and a food dish." | 0. Provide the guinea pig with a cage full of a few inches of bedding made of ripped paper strips, you will also need to supply it with a water bottle and a food dish. | First 5 plausible and first 5 implausible predicates, preserving exact class balance. The paper's Appendix F prefix is applied. The released extraction function returns bare predicates even though the paper says they were prefixed. Dataset/source https://github.com/suwangcompling/Modeling-Semantic-Plausibility-NAACL18 ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L180-L212 . | Sample question | Expected answer | |---|---| | Please judge if this predicate is likely plausible or implausible: "cloth absorb oil" | plausible | First 9 validation examples, matching the official extractor. The expected answer gives both the Choice1/Choice2 label and text; explanations are not scored, following the paper. Dataset/source https://huggingface.co/datasets/12ml/e-CARE ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L667-L727 . | Sample question | Expected answer | |---|---| | Choices: Choice1: The woman gave birth to a child. Choice2: The baby feels the awareness through physical sensations. Which one of the choices are causing the sentence: The child brought psycho-physical phenomena on a new life. | Choice1: The woman gave birth to a child. | First 9 examples from the exact 36-example excerpt redistributed in the paper repository. Each prompt includes all supplied context and asks for supporting facts plus reasoning steps; scoring is on answer correctness. Dataset/source https://hotpotqa.github.io/ ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L12-L42 . | Sample question | Expected answer | |---|---| | Given the following information: Adam Collis: Adam Collis is an American filmmaker and actor. He attended the Duke University from 1986 to 1990 and the University of California, Los Angeles from 2007 to 2010. He also studied cinema at the University of Southern California from 1991 to 1997. Collis first work was the assistant director for the Scott Derrickson's short "Love in the Ruins" 1995 . In 1998, he played "Crankshaft" in Eric Koyanagi's "Hundred Percent". Ed Wood film : Ed Wood is a 1994 American biographical period comedy-drama film directed and produced by Tim Burton, and starring Johnny Depp as cult filmmaker Ed Wood. The film concerns the period in Wood's life when he made his best-known films as well as his relationship with actor Bela Lugosi, played by Martin Landau. Sarah Jessica Parker, Patricia Arquette, Jeffrey Jones, Lisa Marie, and Bill Murray are among the supporting cast. Tyler Bates: Tyler Bates born June 5, 1965 is an American musician, music producer, and composer for films, television, and video games. Much of his work is in the action and horror film genres, with films like "Dawn of the Dead, 300, Sucker Punch," and "John Wick." He has collaborated with directors like Zack Snyder, Rob Zombie, Neil Marshall, William Friedkin, Scott Derrickson, and James Gunn. With Gunn, he has scored every one of the director's films; including "Guardians of the Galaxy", which became one of the highest grossing domestic movies of 2014, and its 2017 sequel. In addition, he is also the lead guitarist of the American rock band Marilyn Manson, and produced its albums "The Pale Emperor" and "Heaven Upside Down". Doctor Strange 2016 film : Doctor Strange is a 2016 American superhero film based on the Marvel Comics character of the same name, produced by Marvel Studios and distributed by Walt Disney Studios Motion Pictures. It is the fourteenth film of the Marvel Cinematic Universe MCU . The film was directed by Scott Derrickson, who wrote it with Jon Spaihts and C. Robert Cargill, and stars Benedict Cumberbatch as Stephen Strange, along with Chiwetel Ejiofor, Rachel McAdams, Benedict Wong, Michael Stuhlbarg, Benjamin Bratt, Scott Adkins, Mads Mikkelsen, and Tilda Swinton. In "Doctor Strange", surgeon Strange learns the mystic arts after a career-ending car accident. Hellraiser: Inferno: Hellraiser: Inferno also known as Hellraiser V: Inferno is a 2000 American horror film. It is the fifth installment in the "Hellraiser" series and the first "Hellraiser" film to go straight-to-DVD. It was directed by Scott Derrickson and released on October 3, 2000. The film concerns a corrupt detective who discovers Lemarchand's box at a crime scene. The film's reviews were mixed. Sinister film : Sinister is a 2012 supernatural horror film directed by Scott Derrickson and written by Derrickson and C. Robert Cargill. It stars Ethan Hawke as fictional true-crime writer Ellison Oswalt who discovers a box of home movies in his attic that puts his family in danger. Deliver Us from Evil 2014 film : Deliver Us from Evil is a 2014 American supernatural horror film directed by Scott Derrickson and produced by Jerry Bruckheimer. The film is officially based on a 2001 non-fiction book entitled "Beware the Night" by Ralph Sarchie and Lisa Collier Cool, and its marketing campaign highlighted that it was "inspired by actual accounts". The film stars Eric Bana, Édgar Ramírez, Sean Harris, Olivia Munn, and Joel McHale in the main roles and was released on July 2, 2014. Woodson, Arkansas: Woodson is a census-designated place CDP in Pulaski County, Arkansas, in the United States. Its population was 403 at the 2010 census. It is part of the Little Rock–North Little Rock–Conway Metropolitan Statistical Area. Woodson and its accompanying Woodson Lake and Wood Hollow are the namesake for Ed Wood Sr., a prominent plantation owner, trader, and businessman at the turn of the 20th century. Woodson is adjacent to the Wood Plantation, the largest of the plantations own by Ed Wood Sr. Conrad Brooks: Conrad Brooks born Conrad Biedrzycki on January 3, 1931 in Baltimore, Maryland is an American actor. He moved to Hollywood, California in 1948 to pursue a career in acting. He got his start in movies appearing in Ed Wood films such as "Plan 9 from Outer Space", "Glen or Glenda", and "Jail Bait." He took a break from acting during the 1960s and 1970s but due to the ongoing interest in the films of Ed Wood, he reemerged in the 1980s and has become a prolific actor. He also has since gone on to write, produce and direct several films. The Exorcism of Emily Rose: The Exorcism of Emily Rose is a 2005 American legal drama horror film directed by Scott Derrickson and starring Laura Linney and Tom Wilkinson. The film is loosely based on the story of Anneliese Michel and follows a self-proclaimed agnostic who acts as defense counsel Linney representing a parish priest Wilkinson , accused by the state of negligent homicide after he performed an exorcism. Question: Were Scott Derrickson and Ed Wood of the same nationality? Please find the supporting facts from the provided context, and use them to answer the question. Write all your reasoning steps. | yes | First 5 problems from upstream problem type 4 fix alphabet and first 5 from type 5 sort , matching the two type indices selected by the paper code. The released paper code names an unavailable legacy NPZ; the questions are reconstructed from the upstream all prob.npz and the upstream prompt formatter. Dataset/source https://github.com/taylorwwebb/emergent analogies LLM/tree/main/letter string ; paper extraction code https://github.com/HLTCHKUST/chatgpt-evaluation/blob/main/src/reasoning qa.py L278-L317 . | Sample question | Expected answer | |---|---| | Let's try to complete the pattern: a k c d e a b c d e m n o p c | m n o p q | - The authors used the public ChatGPT interface without an API, in a zero-shot setting, and identify the tested snapshot as 15 December 2022. - They sampled the first examples or explicit IDs produced by the released extraction functions. Prompt wording varies by dataset and is retained in data.csv , as the sample questions above illustrate. - Generated answers were manually judged against gold answers. The paper says rationales/explanations were also checked, although the reported metric is answer accuracy. For HotpotQA and analogies, a semantically matching or containing answer was accepted rather than requiring exact string equality. - The paper does not document temperature, decoding controls, a conversation-reset policy, grader agreement, or a complete category-aggregation formula. Those omissions prevent an exact independent regeneration of 63.41 from the publication alone. - Start ten fresh conversations with the model under test. Use one conversation for each part 1.csv through part 10.csv . - Attach exactly one part file to each conversation. Do not attach data.csv or this Markdown file, because data.csv holds every gold answer and this file exposes the sample ones. - Ask the model to fill only the Your answer column. - Save the ten completed CSVs with their original filenames. Run merge.py to merge them with the gold file, then manually fill Correct ✅/❌ . - Run score.py on the graded file. Report the final score over the 200 rows. Suggested instruction for each model conversation: Fill the attached CSV's Your answer column for every row. Answer each question independently. Preserve the Number and Question columns exactly, do not add, remove, or reorder rows, and return a completed CSV with the same filename. For rows containing several subquestions, give all answers in order, separated by / . Merge command: python merge.py --gold data.csv --parts . --output evaluation sheet.csv Score command: python score.py evaluation sheet.csv - Mark ✅ when the response clearly gives the expected semantic answer; labels and answer text are interchangeable when unambiguous. - Mark ❌ when the final answer is wrong, missing, noncommittal, or contradicted by the response's own explanation. - For bAbI rows with several subquestions, all listed subanswers must be correct and in order for the row to receive ✅. - For TimeDial, either listed correct option is accepted. For SpaRTQA multi-label questions, all required relations must be present and no incompatible relation may be asserted. - Explanations are not separately scored unless they contradict the final answer. This keeps the binary accuracy metric auditable while remaining close to the paper's manual answer judgment. - Grade blind to model identity when possible. If comparing multiple models, reuse the same 200 prompts and the same grading rules. The final score is the percentage of correct rows over the 200 rows of the benchmark. Let correct be the number of ✅ rows and n = 200 the total number of rows: Every row weighs the same, so categories with more rows contribute more to the final score. The ten part files use deterministic seed 20260805 . Every numbered row from 1 through 200 appears exactly once across the ten files, each file contains 20 rows, and row order is shuffled. Assignment minimizes repeated benchmark membership within a part; where a benchmark has more than ten rows, at most two of its rows occur in any one part. Identical or prompt-paired items are kept apart whenever possible. All CSV files are RFC 4180-style UTF-8 with a BOM for spreadsheet compatibility.