Figma to Code Before LLMs

Builder.io's first Figma-to-code system used a gradient-boosted decision tree model (XGBoost) that compared pairs of candidate node merges, feeding each candidate 21 geometry and structure features for 42 inputs total regardless of design size, to rebuild responsive layout hierarchies from flat Figma trees. The system trained on responsive websites rendered in Chromium, using getBoundingClientRect() bounding boxes and original HTML parent-child relationships as labels, after combining redundant nested rows and columns. The approach avoided copying every Figma node into an absolutely positioned element, which reproduces coordinates but leaves responsive structure unresolved.

Jev https://typesafe.ai/blog/introducing-system-one-models-and-jev launched last week, and people are already sorting inboxes https://x.com/ryanvogel/status/2100042788851101842 and even playing Doom https://typesafe.ai/blog/introducing-system-one-models-and-jev with it. I recognized the pattern: a model making structured decisions inside a larger algorithm . When I built the first version of Figma-to-code at Builder.io, I used a much narrower classifier. Our visual editor let marketers and designers place things where they looked right without learning the HTML mental model, but the result still needed a responsive layout . Figma and the web both represent documents as trees, but a Figma tree can be a much bigger mess . Nodes sit wherever the composition needs them, multiple frames overlap to create effects that could be a single CSS line, old iterations still exist hidden under some opaque white rectangle… I have seen it all. Frontend developers take that composition and decompose it into rows, columns, and constraints that can respond to different screen sizes. Copying every node into an absolutely positioned element reproduces the coordinates, but leaves the responsive structure for someone else to figure out. Data dominates As a big fan of Rob Pike, I started thinking in terms of data structures. A Figma design could be a single node or a 30 MB monstrosity, while the decision-tree models I wanted to try expected a fixed number of inputs . Models can work on variable-size inputs, including graphs, but a good-looking Figma design could be a flat frame with 400 sibling nodes. The hierarchy we needed was precisely the thing missing from the input Converting that frame one to one would make a terrible website. The breakthrough The breakthrough was to have the model compare two possible merges . We could use those comparisons to choose which nodes to group, then repeat with the new groups, instead of asking a model to turn the whole Figma tree into responsive HTML. Imagine rendering a responsive website, then throwing away its hierarchy. Keep each visible element’s bounding box, its position and dimensions , in a flat array: nodes = node1, node2, ..., nodeN A candidate merge proposes joining two nodes or groups under a new parent. One candidate might join a heading with its paragraph, another might join two neighboring cards. The model compares them and predicts whether both merges are valid, neither is valid, or only one is valid . compare candidateA, candidateB → probabilities for both valid, neither valid, only A valid, only B valid Each candidate became 21 numbers describing its geometry and structure: group dimensions, their enclosing box, alignment, distances, child counts, and information about alternative merges. Two candidates gave us 42 input features, regardless of the size of the design. The model did not read the text or look at colors. After experimenting with different models, we settled on gradient-boosted decision trees, using XGBoost https://xgboost.readthedocs.io/en/stable/ . A decision tree follows branches based on its inputs to make a prediction, and boosting combines multiple trees. Speed mattered because rebuilding a hierarchy meant repeatedly comparing candidates in a tournament, applying compatible winners, then building the next round. Training data The internet is our dataset Take a responsive website and the original HTML gives us a tree to learn from . Render it with Chromium, compute the relevant elements’ bounding boxes with getBoundingClientRect https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect , and keep their parent-child relationships as the known answer. After combining redundant nested rows and columns, we labeled a candidate merge valid when its two groups belonged to the same expected parent . Two independent branches could both have a valid merge, so we didn’t need to prescribe a single order for reconstructing the page. Comparing two candidates gave us one of the four labels: ...candidateA.features, ...candidateB.features, matchupLabel 21 values 21 values one of 4 classes A single layout could supply many rows for our training CSV as we reconstructed its hierarchy and encountered new candidates. We manually selected good websites, trained the model, and tried it against other websites and designs. The designs it could not reproduce guided what we added to the next training set. Data collection and training were heavily coupled and order-dependent, but we ended up with a small dataset covering a variety of designs and website structures. We could also inspect the trees for suspicious rules, but reasonable-looking conditions could still fit only the training examples. We still needed to test on designs outside the training data. A tournament between merges With 1,000 nodes, there are 499,500 possible pairs , before we even compare possible merges with each other. Our first algorithm filters out pairs that geometrically cannot occur. For separate boxes, a merge’s bounding box must not intersect a third node . A heading and its paragraph might be a candidate, while the heading and a card at the bottom of the page would enclose unrelated content. We handled containment before this search , recursed into children, and kept existing auto-layout structure. The demos use separate boxes to leave those cases out. We then compare every candidate against every other candidate . With candidates, that gives us matchups. The model’s most likely class determines how we update their scores: - Both valid: each candidate gets half the predicted probability. - Neither valid: each loses half that probability. - Only one valid: that candidate gets the full probability, the other gets nothing. If “both valid” wins with probability 0.8, each candidate gains 0.4. These contributions accumulate across matchups, so tournament scores can exceed 1 or fall below 0 . We sort by score and select the highest-scoring merges whose rectangles do not overlap . Two equally good candidates may share a node, in which case one has to wait. The September 2023 implementation used the maximum score as its cutoff , or 95% of that maximum for more than 20 candidates. Later, k-means https://scikit-learn.org/stable/modules/clustering.html k-means split the scores into two clusters, and we used how tightly they clustered to help adjust the cutoff. These demos use the earlier rule with recorded predictions from the currently checked-in model; scoring and merging run in the browser. Building the tree After each tournament, the merged groups become nodes for the next round . A heading and paragraph become a column, join a button, then join an image in a row. The model keeps answering the same question about larger groups, and we keep their children to build the tree. The main loop looks roughly like this: while more than one group remains: candidates = geometrically possible merges if there is only one candidate: merge it else: compare candidates in a tournament choose high-scoring, non-overlapping merges apply the chosen merges stop if no progress is possible When geometry blocked progress, one fallback temporarily removed a group, solved the rest, then joined it back. The algorithm could still fail , or finish with a hierarchy a developer would not have chosen. I used AIR to inspect each merge and its scores . Here is the original tool running again locally: From hierarchy to layout The model’s job ended at the hierarchy. We got colors, typography, borders, and other inline styles directly from the Figma nodes. Heuristics then decided alignment, margins, gaps, padding, and sizing. The recovered groups let us make layout decisions locally : spacing between cards in a row, or alignment within a column of text. Real Figma files Hierarchy extraction worked very well, but getting production-ready results was much harder when the team had not been careful with the Figma designs. For some customers it worked so well, and for others it was a disaster. Hidden iterations and overlapping shapes still needed handling, even with a plausible hierarchy. Passing one stage of the pipeline did not mean the output was usable. What I still like about this project is that hundreds of design nodes became comparisons with just 42 inputs .