For now, here’s what I found from some light testing in Colab: I tried to reproduce the public numbers first, then deliberately looked for easy failure modes rather than assuming the headline accuracy was telling the whole story.
The short version is: the basic result held up better than I expected.
On the public 80/20-style split I reconstructed, I got:
That is broadly consistent with the results described in the repository and model card. I also tried a stronger grouped holdout where related lexical families were kept together rather than randomly split. The neural model was still essentially unchanged:
So I didn’t find evidence for the simple explanation that the reported neural accuracy is mostly coming from generic near-duplicate/template leakage.
The more interesting signal was narrower: L3 / Predictive seems to have much less diversity in how its core temporal relation is expressed than L2 or L4.
If I were extending the evaluation, my default route would therefore be quite small: That would probably tell more than simply adding another large random test split.
- What reproduced, including the negative resultOverall, the part I found most encouraging was actually the failed attempt to break the neural result with a generic lexical-family split. The obvious “99% only because the random split leaked the templates” explanation did not survive that check.
The next useful question therefore seems narrower: whether the four execution semantics remain stable when the same relation is expressed differently, and when multiple relations appear in one utterance.
If this were my evaluation budget, I would spend the next small increment on: Those are all relatively cheap, and each one answers a different deployment question without requiring a redesign of the core model.