Hi! I really like tomland, especially for its bidirectional codecs, and it made …me a bit sad to see it fall behind the spec.
Now that I have Claude at my side, I thought I'd put in the effort to push through some changes that were planned years ago (#248, #264, #373). There are a few of them and they are necessarily API-breaking, so I made sure to split them into individual commits that add up to full TOML 1.1.0 support.
The short version: against the official toml-test suite (the files-toml-1.1.0 list), the parser now accepts all 218 valid documents and rejects all 494 invalid ones.
Closes #373, closes #248, closes #264.
What's in here
The commits fall roughly into three groups.
Parser (dc09f7d.. dac66c7). Mostly about the TOML 1.0 rules that weren't enforced, plus the 1.1 additions:
- whitespace, newlines and comments are only consumed where the spec allows them.
scnow consumes only spaces and tabs and a newscnhandles newlines/comments. A key/value pair or header has to be followed by a newline; - whitespace around dots in keys, ASCII-only bare keys;
- string rules:
\eand\xHH(1.1), tabs and non-ASCII in basic strings, control characters rejected in literal strings,""before the closing""", etc.; - times without seconds (1.1), lowercase
t/z, range-checked offsets, no leading zeros in floats or0_1; - inline tables can span lines and have a trailing comma (1.1), and can be nested in inline tables and arrays.
There are two printer fixes in this group too, because the round-trip tests started failing on them.
MultiLinemode printedkey =followed by a newline, which isn't valid TOML. Control characters were printed withshow, so you'd get\ESC, which no TOML parser accepts. Mixed arrays and unquoted keys (c4d29d5,20fd317). Arrayholds[AnyValue]instead of[Value t]. There's a newarrayhelper for the homogeneous case and an_EitherBiMap to decode into a sum type.Pieceused to keep the quotes, sokey,"key"and'key'were three different keys. The spec says they're the same key, and a codec had to know how the file spelled it. Now the parser strips the quotes and the printer adds them back when a piece isn't a bare key. New AST and validator (a9b79c2). This one is the big one and it's basically #264. The oldTOMLhad two ways to encode a nested table, either as a prefix-tree sibling or as a nested sub-document, and which one you got depended on the order of headers in the file. As a result:- lookups and
Eqdepended on header order; - reopening a parent after a subtable duplicated the subtree;
int "a.b"andtable (int "b") "a"read different places;- the validator had no idea which tables could still be extended, so it accepted 43 invalid documents from toml-test.
Now
TOMLis a map from a singlePieceto anEntry:
data Entry
= EValue AnyValue
| ETable TableKind TOML
| ETableArray (NonEmpty TOML)
data TableKind = HeaderTable | ImplicitTable | DottedTable | InlineTable
Every level of nesting is exactly one piece, so a.b.c = 1, [a.b] + c = 1 and a = { b = { c = 1 } } all produce the same document. TableKind exists only so the validator can apply the 1.1 rules (inline tables are closed, dotted tables can't be reopened with a header, and so on). Eq ignores it. Value and TOML are now mutually recursive (Table :: TOML -> Value 'TTable for inline tables inside arrays), so they both live in Toml.Type.Value.
The validator is now a single sequential pass that tracks scopes, with a new ExtendClosedTable error.
Compatibility
This is a breaking change, so I've bumped the version to 1.4.0.0. I tried to keep the damage limited to people who work with the AST directly:
- the codec API is unchanged, and existing codecs should keep working.
list/nonEmptyread both[[key]]andkey = [ {..}, {..} ]; tomlPairs,tomlTablesandtomlTableArraysare still there as deprecated read-only functions with their old types, andToml.Type.PrefixTreeis kept as a deprecated module. My suggestion would be to drop them in 1.5;- two things can't be shimmed:
ArrayholdingAnyValue, andPiecebeing unquoted. The second one is the nasty one, becausePiece "\"a.b\""still compiles but now means something different. I wrote a "Migrating from 1.3" section in the changelog that covers both.
Testing
cabal test: 328 examples, 0 failures (GHC 9.12.4 locally; I haven't tried the rest of the CI matrix yet, so let's see what CI says).- toml-test, 1.1.0 list: 218/218 valid accepted, 494/494 invalid rejected. tomland doesn't ship a toml-test decoder, so this is an accept/reject sweep with a small
runghcscript that callsToml.Parser.parse. It doesn't check the decoded values against the expected JSON. Happy to add the script (or a proper decoder executable) to the repo if you think it's worth it. - For what it's worth, the 1.0.0 list gives 208/208 and 492/501. The 9 misses are exactly the things 1.1 relaxed: times without seconds, multi-line inline tables and trailing commas in them, and
\xescapes. There's no "1.0 mode", and I don't think one is needed.
Open questions
- Even split into commits, it's still a lot to review. The parser commits come before the AST change and stand on their own (the test suite passes at
dac66c7), so I can split them into a separate PR if that's easier for you. - I haven't measured performance. The new representation should be in the same ballpark (
HashMapper level instead of a prefix tree), but I haven't checked.