We recently designed a new feature in Plotly Studio called "Methodology". It surfaces the data source, columns, formulas, assumptions, and approach front and center in the interface. We designed this so that it'd be easier to verify and trust the results from the agentic loop and to prevent a feeling of mental atrophy that is becoming all to common in our AI experiences.
In many AI products and productivity tools, AI is generating the majority of the output and our role has transformed from creating things to reviewing things.
And in data products, reviewing and validating that the AI did the write thing is essential work.
In Plotly Studio, we've been thinking a lot about how we can make it easier to verify what the AI generated.
We've found that designing verification UX also solves another key problem in our use of AI - mental atrophy. AI can often generate too much and too fast for us to comprehend, and the default product experiences can often feel overwhelming. Verification ends up being a way to slow it all down.
In the most recent release of Plotly Studio, you'll find a new "Methodology" button below each step. The "Methodology" will display an easy to scan but detailed, thorough view of the most important aspects of the data analysis step.
Here's what it looks like:
In Plotly Studio, all of the analytics are backed by code. We've long believed in the power of code-based analytics. But many of the tools that we use for working with code in software engineering are ill-equiped for working with data.
Review is one of those. Each of these steps has 20-100 lines of code. While this is a managable set of code to review, it's not always easy to pull out the data-related details of the code. The Methodology section helps put it in context. It also provides an implicit review checklist, by surfacing the important things front and center like any assumptions that were made or if any data points were hard-coded.
Methodology is created by a second agent of the main agentic loop and context. This has a two benefits:
- It doesn't slow down the main agentic loop.
- It isn't "influenced" by the previous context.
AI sessions can sometimes get "stuck in their ways". They can fib or make up data in order to satisify user's request. By creating the Methodology as a separate AI call, we can avoid this contextual history and steering. The Methodology agent simply reviews the code and the result without any prior knowledge of the original user's request or the history of the session. This gives the results quite a bit more neutrality and consistency.
Methodology is now available in Plotly Studio. Give it a try and let me know what you think.