Yikes, it's already Monday night and I lost track of time because I've spent two whole evenings this long weekend trying to track down a stupid electrical circuit/breaker issue that should've been an easy fix but isn't.
Last week, (ref: I'm not learning coding from LLMs, but product design) I was saying how vibe coding up toy apps to myself was giving me a lot of practice in learning product design related skills. This week I wanted to elaborate a bit more on the point since I find the exercise surprisingly useful in my normal UX researcher work.
So, as a quick recap, I was hacking together a tool that takes text files on my local computer and uses a local LLM classifier to extract interesting tag information. Now, the back end of this is extremely simple, a totally naive solution in Python would be something only slightly more complex than [classify(i) for i in files]. There'd be a bunch of messy code up front to set up the classifiers, gather files, reading in data and saving the data to the correct output location, but the core job is just a single loop.
If this were to be a work project of some sort, I'd likely reinforce that basic pipeline code with very basic observability things, and maybe put in various safeguards for when things crash, get forcibly stopped partway, restarting job queues and so on. It all adds complexity but it is all extremely familiar backend glue code that the vast majority of us in data science are used to writing. The next leap in complexity is the step that I almost never take at work – making the web interface, or any visible interface, to the application. First, I write garbage html and have failed to learn any variety of JavaScript despite making multiple half-hearted attempts at picking up pieces over the years. I don't think I'm alone in this. I would venture to say that the majority of us in data science, with the lone exception of the people who love data visualization, are generally more comfortable in the back end shuffling numbers around than making interfaces.
So thanks to my inability to write an interface to save my life, I inevitably give the work of making front-end interfaces to things to people who know how to actually do it – actual frontend engineers and UX designers. These folks in tandem know how to arrange elements on a screen in a way that users can intuitively find what they want and accomplish their goals. I learn a bunch each time I watch them talk about their work.
So when I say that I'm "learning product design" making my dumb little personal apps, what exactly am I learning? Why is this relevant to my work?
First, it seems I have at least seen enough good and bad interfaces in my life to have developed some sense of "taste". It's not nearly as refined a taste as an actual frontend expert or designer would have, but at the very least I understand basic concepts like "related things should be together" and "more important/critical actions should be more prominent than surrounding things". I know to avoid giant walls of text. I know that users have certain interaction conventions they bring in from existing software use that we should work with instead of against – like for example how clicking on a top left logo of a page usually goes to the top homepage.
There's also other invisible conventions like latency standards, how errors should be handled and reported to users, asking myself what I expect to happen if I were to click a button that has a particular label. I've at least develop some sense of good or bad in relation to these things because I've had to measure all of these and more in my many years of analyzing how people use software.
So when I pull up my little app, and the main page takes 20 seconds to load because the LLM wrote some terrible SQL (as they usually do), I have a clear articulation of how the load time is terrible, how it should be within N seconds, how we should maybe design the page to not make such queries at all or to at least lazy-load in the data later. As a UX researcher, my primary job was to report something like "the load times on this page are terrible, >10s. We need to fix it because after 2s people start getting frustrated." After that report, a PM or engineer would take the feedback and go Do Something About It (tm), and I was almost never consulted on the actual solution. Now, it's on me to (make the LLM) Do Something About It. Hence what I mentioned last time about making lots of decisions that I wasn't used to making. Repeat for all the other bumps I ram into along the way.
So how does all this help my day job?
Well, I'm seeing a lot more stuff now. Usually if users hit upon a bug or obviously rough UX issue, no one ever tells me about it, they just fix the bug themselves. Most of us can name at least 5 different reasons why our data pipeline may fail because we've likely experienced most of them before. I don't think most of us can name 5 different ways a user can "do typical user things" to a UI to get it stuck into a broken state. We can know users are frustrated by a particular rough experience because a particular "intuitive feature" is missing, but then completely don't see the struggle to figure out how to make the feature actually intuitive.
I don't pretend that a bunch of vibe-coding will make me into a FE engineer or a designer. Instead, all these projects have given me space to fail at being those things. And failure is great, because that's how I learn about things.
Just this weekend, frustrated by how slow my 6yo desktop GPU is at tagging files, I worked on a "simple" feature to "spin up a cloud instance with a GPU, install Ollama on that and route the work to the faster machine there". At the start I poured in all the edge cases and failure cases I could imagine that would leave me with a mystery giant cloud bill. With that rough idea, those edge cases, I figured the LLMs could glue together the many complicated API calls needed to string together such a system... and it sorta did? At least, it successfully spent a couple of dollars of compute, did some benchmarks, came up with maybe a dozen start/stop/resume edge cases that we dutifully hammered away – and the feature is still quite janky. Even if I didn't have to write the job queue code lines myself, I got to directly experience, debug, and decide how to handle the many weird race conditions, blind starts, quota-based failures, and other nutty things that came up. I got to call bullshit on when the LLMs tried to do ridiculous things for the task. How was I supposed to know this stupid feature would find so many creative ways to fail?!? Now that I've slammed into these problems a few times, it's becoming less awkward to come up with potential solutions – despite most of us UX Researchers and former academics have some resistance to actually stating solutions out loud.
The point of all this work, aside from accomplishing my little weird tasks, is to experience the problems these other roles have. Now that I know this page is too slow, how do I fix it? Now that I know the navigation is a mess, what should it be? These are questions we don't need to ask ourselves if we're staying completely within our analyst roles. But it's what we need if we want to reach out beyond that box and it's never been easier to experience it.
Another fun thing I need to find time trying is modding video games. While I've loved games since I was a kid, making games is a whole separate thing. But one very interesting thing I've seen people do is they will make UI modifications with the help of LLMs. It is almost purely a design challenge of "how can I show information to myself that helps make the game fun to play". So maybe in a game of Civilization I make it remind me about the dozens of mechanics that I never learned to use (and thus keep losing because of that neglect). Or for some FPS I shuffle things around so I can get better awareness of the game state. Or I hate the control scheme to the point where I want to re-bind everything.
In a rapidly changing work environment where the lines between expertise is getting extremely fuzzy, having this broad experience and ability to switch between role archetypes is likely going to be useful. I have a better appreciation for what my colleagues need from me when I report results now. I can also (hopefully) hold a better conversation when discussing the same topics. Also hopefully when code breaks and telemetry goes wrong (as they do), I have more familiarity with hunting down the new sorts of stupid bugs that sneak out in the ridiculously high paced dev world we're finding in ourselves in.
Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.
Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.
"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.
Counting Stuff Official Forums: Discuss posts, or other data topics with the community.
About this newsletter #
I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise credited.
Supporting the newsletter #
All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:
- Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
- Send a one time tip (feel free to change the amount)
- Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
- Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!