{"slug": "scip-rocks", "title": "SCIP Rocks", "summary": "A software engineer built a tool that combines tree-sitter parsing with SCIP index data to produce a live node-and-edge graph of an entire codebase, aiming to restore hands-on understanding as AI-generated code grows. The author cites an estimate that 14 times more commits were generated this year than last year and JetBrains research finding almost half of all code is now generated by AI. The graph's Node and Edge structures record syntax type, file path, line and column, SCIP symbol and relationship kind, and can be rendered with libraries such as d3 or applications like Gephi.", "body_md": "# SCIP Rocks!\n\nA picture is worth a thousand words. Show, don't tell.\n\nWhen I first started out as a software engineer, I always found it easier to understand large codebases/systems when the information was visually represented (class diagrams, sequence diagrams, etc.). I also found that the more I worked in a system, broke things, fixed things, fiddled around, the better my mental model of it would reflect reality. This became especially useful when triaging issues, bridging the gap between business/eng, and generally helped when looking forward to what the future of an architecture should/could be.\n\nSome might suggest that today, you can \"just ask ai to make you a diagram\", or \"just ask it questions about the codebase\". Fair suggestion, but I cannot help but miss the feeling of *knowing* how something worked, and how *rewarding* it was to solve problems with that knowledge I had built myself. Trudging through walls of text in a chat interface is helpful, but not very stimulating.\n\nReality check. There's [an estimated 14x more commits generated this year than last year](https://x.com/kdaigle/status/2040164759836778878?ref=tanikella.ghost.io). According to [JetBrains research](https://blog.jetbrains.com/research/2026/08/how-much-code-do-developers-really-let-agents-write/?ref=tanikella.ghost.io), almost *half* of all code is now generated by AI. My personal crisis here is that a primary method for learning for me is in danger. How can I truly understand a system if I'm not messing around in it myself? I want a live picture that represents an entire codebase so that I can build a mental model of it and work in it more efficiently.\n\nThere are standards and practices for documenting how a system works in [UML](https://www.omg.org/uml/?ref=tanikella.ghost.io), [Mermaid](https://mermaid.js.org/intro/?ref=tanikella.ghost.io), [C4 Modeling](https://c4model.com/diagrams?ref=tanikella.ghost.io), and much more. These standards help engineers communicate with each other (and increasingly with agents). Some of these diagrams are drawn by hand, and some tools can auto-generate them (scip proto schema viz).\n\nOn the other side, there are many programmatic ways to read and navigate a codebase (`tree-sitter`, SCIP, LSP, LSIF). They're all based on similar principles. If you've ever read through [Crafting Interpreters](https://craftinginterpreters.com/?ref=tanikella.ghost.io), you've seen the world of scanning, parsing, grammar, etc. Tools like `tree-sitter` rely on these concepts. Combining `tree-sitter` with `SCIP`, you can do some powerful things!\n\n### What I Did\n\nIn my quest to build a \"live picture of a system\", I wanted to distill how a system worked in a graph structure. Most things executed in a codebase are nodes. Edges connect how those nodes communicate with each other (call, implement, extend, use type, etc.). This structure is really useful for visualization. You can use libraries like [d3](https://d3js.org/d3-force?ref=tanikella.ghost.io), or applications like [Gephi](https://gephi.org/?ref=tanikella.ghost.io) with this data to visualize and explore it! A simple version of `Node` and `Edge` are below so you can see what we're dealing with:\n\n```\ninterface Node {\n\tsyntaxType: SyntaxType; // (function, method, class, interface, etc.)\n\tname: string;\n\tfilePath: string;\n\tstartLine: number;\n\tstartCol: number;\n\tscipSymbol: string;\n\tinTestFile: boolean;\n}\n\ninterface Edge {\n\tkind: EdgeKind; // refers to type of relationship (calls, instantiates, extends, etc.)\n\tfromFile: string;\n\tfromName: string;\n\tfromSymbol: string;\n\ttoText: string;\n\ttoFile: string | null;\n\ttoName: string;\n\ttoSymbol: string;\n\tisExternal: boolean;\n}\n```\n\nTo get these two data structures, you can parse through SCIP index data, and use tree-sitter (needed to figure out `from` fields in an Edge). It's not the most efficient, as you're looping through large arrays, especially as the codebase gets larger, but we'll touch on opportunities to improve that later.\n\nHere's a series of visualizations for the [`excalidraw`](https://github.com/excalidraw/excalidraw?ref=tanikella.ghost.io) project using these data structures in a visualization project. Each node is a function/method/class/etc. in the project, organized relative to project root.\n\nThis is a radial graph that follows a project's directory structure. Folders are labeled, and the leafs are structured as `file -> symbol`. Every scip Symbol occurrence is represented as a Node all contained within a \"slice of the pie\". On hover, you can see the details of every node, file, folder, etc.\n\nAll the nodes here are on the outer rings. Their location is predictable (which comes in handy later), and at a glance, you can visually see where most of the logic lives in the project.\n\nCheck out the `AppSelectionTool` class from excalidraw:\n\nAt a glance, I can see what types of data this class is using, and all the operations it's doing.\n\nThis is especially useful when you introduce a `diff` capability to see how one graph differs from the other (*HUGE shoutout to* *difftastic* *for an excellent tool that allows you to parse through diffs semantically!*). Here's a PR I pushed to the project for the diff feature itself. The diff is highlighted, while the rest of the codebase fades. This was me trying to understand what kind of mess I'd made for myself in a React component while \"vibe coding\" a feature that required more thought.\n\nGraph-based traversal of codebases is not a novel concept. There are already projects out there that parse through tree-sitter/SCIP/LSP data to produce similar results (see: [Graphify](https://graphify.com/?ref=tanikella.ghost.io), [Ragex](https://ragex.hexdocs.pm/readme.html?ref=tanikella.ghost.io), [CodeGraphContext](https://codegraphcontext.vercel.app/?ref=tanikella.ghost.io), [codegraph](https://github.com/colbymchenry/codegraph?ref=tanikella.ghost.io)). I think there's power in visualizing such systems, and understanding them using a predictable shape/pattern. Now, at a glance, I'm able to understand what changes I made to a system, and their impact.\n\n### The Problems\n\nThere are a TON of problems with this approach.\n\n- The transformations on top of a normal SCIP index ([proto schema](https://github.com/sourcegraph/scip/blob/main/scip.proto?ref=tanikella.ghost.io) ) rely on you looping through nodes and edges to run calculations. We can make this more efficient.\n- The SCIP proto doesn't exactly match up with what we'd need for graphs, but it gets us most of the way there. We still need `tree-sitter` to figure out the enclosing symbols (\"where is this being called from?\")\n- SCIP indexers all act differently. There are some output inconsistencies between languages, and even indexer versions.\n- I breezed past the `git diff` . But that's a huge problem. Parsing through multiple graphs is doable, albeit costly.\n  - The other consideration here is a diff produces something between two commit SHAs. A large codebase in a large company will have MANY commits. [GitHub's uptime hasn't been great lately](https://github.blog/news-insights/company-news/an-update-on-github-availability/?ref=tanikella.ghost.io) . This complicates our path to understanding changes in a graph.\n- The other consideration here is a diff produces something between two commit SHAs. A large codebase in a large company will have MANY commits. \n- While this might be cool from a conceptual exploration / planning perspective, if you're in a text editor, you're much better off relying on an LSP server to give you similar information. This doesn't solve that problem. This gives you a zoomed-out view of that system.\n\n### The Future\n\nWe've reached a semi-predictable shape that represents a system. This shape also allows us to layer other data on top. Why stop at a git diff? A stack trace also gives us symbol and location information:\n\nA lot of people playing with this data are thinking similarly. \"How can a coding agent use this data to make decisions\". Sourcegraph (who created the SCIP protocol) has gone down the route of becoming the unified \"context layer\" for your AI Agents. Sourcegraph has also realized the utility of such layering with its \"Insights\" functionality in its platform.\n\nWhile this is undoubtedly useful, I want something different. I'd like to *communicate* with a coding agent using this type of view. I want to plan out a change, or review a coding agent's work *using* this visualization. I'm interested in making this a much more interactive experience.\n\nI especially like the direction that Meta's [Glean](https://glean.software/docs/introduction/?ref=tanikella.ghost.io) and Google's [Kythe](https://github.com/kythe/kythe?ref=tanikella.ghost.io) have taken. Glean in particular finds a clever way to abstract out the code, and the version control details, and exposes the same data (or in their terms, FACTS) in a queryable interface (Angle). Making this data queryable lets us separate the act of gathering this data (running indices, transformations, etc.) and actually using it.\n\n### Parting Thoughts\n\nMy next questions related to this are:\n\n- What are the limitations of the SCIP protocol? (I had some trouble with different indexers producing inconsistent results in some cases (if at all))\n- How can I tie memory/CPU usage data into these types of visualizations?\n- Should I build a personal server to store/query this graph data? (maybe?)\n\nIf you want to play around, I've cleaned up my mess and published [a slimmed-down version](https://github.com/dtanikella/atlas?ref=tanikella.ghost.io) here. In the coming days, I plan on expanding the tool and improving its performance so it works well on any project.\n\nI did this and a bunch of other things while learning with batch-mates at the [Recurse Center](https://www.recurse.com/?ref=tanikella.ghost.io). If you're a curious programmer, and you like learning in a setting with other curious programmers, you should apply!\n\n*I'm sorry for the bad writing. I wrote this myself. Used AI to proofread*\n\n### References and Shoutouts\n\n- AI-generated code\n- Documentation standards\n- Programmatic ways to navigate codebases\n  - tree-sitter, LSP, LSIF\n  - [SCIP proto schema](https://github.com/sourcegraph/scip/blob/main/scip.proto?ref=tanikella.ghost.io)\n  - [Glean](https://glean.software/docs/introduction/?ref=tanikella.ghost.io)\n  - [Kythe](https://github.com/kythe/kythe?ref=tanikella.ghost.io)\n  - [Crafting Interpreters](https://craftinginterpreters.com/?ref=tanikella.ghost.io) (background on scanning and parsing)\n- Other code graph projects\n- Visualization\n- GitHub reliability\n- Projects and community\n- Shoutouts\n  - SCIP / tree-sitter teams\n  - [Difftastic](https://difftastic.wilfred.me.uk/?ref=tanikella.ghost.io)", "url": "https://wpnews.pro/news/scip-rocks", "canonical_source": "https://tanikella.ghost.io/scip-rocks/", "published_at": "2026-11-02 00:58:27+00:00", "updated_at": "2026-11-02 01:21:10.898534+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["SCIP", "tree-sitter", "JetBrains", "d3", "Gephi", "UML", "Mermaid", "C4 Modeling"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/scip-rocks", "markdown": "https://wpnews.pro/news/scip-rocks.md", "text": "https://wpnews.pro/news/scip-rocks.txt", "jsonld": "https://wpnews.pro/news/scip-rocks.jsonld"}}