In this episode, Scott Jenson, a veteran UX designer known for his work on the Macintosh, Google Maps, and Chrome examines the long-term stagnation of desktop operating systems and the limitations of current mobile and cloud-centric models. The conversation shifts toward specialised "Local-First" software, exploring how we can move beyond the "status quo" of interface design to create more intuitive, powerful, and privacy-conscious computing environments in the advent of the "LLM everywhere" era.
Key Takeaways
- Desktop operating systems have seen little fundamental innovation in two decades, focusing primarily on ecosystem lock-in rather than enhancing user productivity.
- While mobile dominates consumer attention, it lacks the depth required for complex productivity. Tablets are specialized for consumption, not creation, and cannot replace the desktop.
- Local-First architecture addresses the fragility and privacy risks of the cloud-centric era by ensuring data sovereignty and resilience against connectivity outages.
- Current interfaces rely on stateless direct manipulation. Future operating systems should implement ‘working memory’, context-aware scaffolding that tracks user activity and history.
- The future of LLM integration lies in embedding intelligence directly into application workflows, moving away from universal ‘chatbots’ toward functional, agentic interfaces.
Subscribe on:
Transcript #
Olimpiu Pop: Hello, everybody. I'm Olimpiu Pop, an InfoQ editor. And today I have in front of me Scott Jenson. For those of you, if the name doesn't mean anything, he's probably one of the UX designers that has a lot of impact on your life or the things that you're doing if you're using a MacBook or if you're using an Android phone because you worked on a lot of this stuff. Thank you, Scott, for talking to us.
Mobile vs. Desktop: Productivity vs. Consumption [[00:45](javascript:void(0);)] #
Scott Jenson: Great. I'm happy to be here. Yes, I basically started off at Apple. I worked on the Macintosh System 7. I then worked over and moved over to the Newton, which was a big handheld at the time, got a lot of hype, then moved over to do a lot of desktop work. I ended up working at Google and worked on the first version of Google Mobile Maps. Went over and worked on the Chrome team, had a project called the Physical Web, and then did quite a bit of consulting. I was at Frog Design for a number of years as creative director and worked on some big projects for say HP and Disney. And went back to Google at the end of my career. About two years ago, I left Google. I'm now working on my own.
Olimpiu Pop: I liked one of your talks and the way how you framed the current work that you're doing. You mentioned that you are repenting for your sins in the last period, and that's why a lot of your work is currently in open source. Thank you for taking the energy and time during this period and doing that much exploration. And that's why we are together because you did have, I think almost a year ago, it was a Ubuntu, I think, conference where you spoke about the past, present, and future maybe, but more on the present and future of what UX for desktop means. That got a lot of traction. I think it was last time when I checked, it was half a million views. And then I saw the presentation that you had at Local-First, not that much traction because it's more current than the other one. But what you had different there, you had a couple of ideas that you shared.
Actually, you put your fingers or your ideas, creative ideas into practice, and you provided a couple of ideas how UX can look for desktop and to better organize your work. But let's not get ahead of ourself and take a step back. Let's look a little bit, as you mentioned, you grew to the phases of what desktop means. Let's look at how the UX in the desktop space evolved and maybe touch a bit if you think it's useful on the differences between mobile and desktop. Because even though mobile is around with us for some time now, we are still using desktop and there is no sense of something that will come and replace them, that people try to replace them with the tablets, but that's nowhere near into that term.
Scott Jenson: Well, I think you covered a lot of the key hits I've been talking about over the last couple years. The first one is just that the desktop has basically stopped evolving for the last 20 years. Apple has introduced 12-ish features in the last, say, 15 years, but they're all about making the ecosystem better, universal clipboard or the phone being used in the desktop or AirDrop or things like that.
So the vast majority of the improvement with Macintosh has been to tighten its control around its ecosystem. And the desktop itself has not fundamentally changed. And that transitions to what you were just mentioning about mobile, that a lot of people feel like all the hotness, all the excitement is in mobile. But the point I made in my talk is that mobile won consumers. It didn't win productivity. So back around 2015, all these companies tried, multiple companies.
At Google, they tried this. They called it Tablet Tuesdays. And they was like, "Everybody bring a tablet to work and do all of your work on the tablet". And that was a disaster. It's not that tablets are bad, it's just that tablets are hyper-focused on things like content and gaming and social media. And so mobile and tablets are very good at a very narrow set of things. And so a lot of my work has been trying to say, "Let's not throw desktop out. What is it about desktop that is so powerful? And then how can we inject some of that into mobile?" It's not either or. It's about what's good.
And so I've wanted to both improve mobile, but in the talk I gave at Ubuntu was I also wanted to improve desktop. And my general thesis is that we don't seem to be that excited about improving things. We just accept the status quo. And I gave a couple of ideas on that.
Olimpiu Pop: This thing where you're saying, "Yes, we have tablet and it's useful for some stuff. We have mobile and it's useful for some stuff". But desktop is still king in a couple of categories and that's very useful. That sounds a lot like cloud where you have everything across the board and then regardless of what you're using, you'll have those things. But given that our discussion started around Local-First and what that tries, I had a similar conversation with Adam Wiggins, which is involved both with Local-First, but he also was one of the pioneers of the cloud with Heroku and bringing things between developers. And one of the conclusions of our conversation at that point was we have a tendency as technologies to go from one extreme to the other.
We moved from desktop, we moved all the way to the cloud. And now we are realizing that that abstraction is too much and we missed some of the points and now we have to find a sweet spot. And that doesn't have to be something set in stone, but it'll be probably something like gradient. You're moving left and right depending on what's the scenario or different types of approaches. What's your take on it? How do you feel about it? What will be the things that we can take from, I don't know, maybe mobile, maybe desktop, and what will be the points that we need?
Privacy and Practicality in the Cloud Era [[06:21](javascript:void(0);)] #
Scott Jenson: Again, you touched on many points there. I think the big issue with the cloud. Well, there's lots of things to talk about, but the two big ones is privacy. The idea is that with a cloud, all of your data is with one particular provider. And so this is a well-known issue people talked about. I'm not the first one to mention this. And that's why Local-First gets a little bit worried that you're giving all your data to Google or all your data to Microsoft. And so there's that aspect of it.
But the other one is just simply pragmatic, which is to say we live in a world where you don't always have connectivity. And so how do you continue to work on something if it's in the cloud because you're cut off? And whenever there's an AWS outage, the world seems to always rediscover this fragility of it.
And so there's both a practical and a philosophical reason, I think, for talking about it. But like you said, we don't want to throw the baby out with the bath water. It's not like cloud is inherently wrong. I think Local-First is just trying to build it so it's robust. So I can keep all of my data. You can't take it away from me. And if the cloud goes away, I can continue to work on it. And that feels like a very powerful thing that cuts through all of Local-First.
Olimpiu Pop: Yes. And I think, well, it's more heavyweight than what we're discussing, but distribution of load is something quite important. And that goes around in pretty much all aspects of technology in life. I'm just thinking now I'm based in Romania, so that means that we have a huge neighboring country called Ukraine that has a problem for the last couple of years. And that means that if the things are not distributed, they have a huge problem with bombardment just taking out electricity. And I think that's something important also for us. Just distributing stuff, as you mentioned across, we have different copies and there are other problems to that, but that's a whole different conversation, as you mentioned.
So now we have more or less two different trends, if you want. We have the Linux-based family of systems, and I'm thinking now about all the open source distributions or the ones that got forked from them more or less like macOS back in its infancy. And then we have Windows. What do you think? Where does the creativity coming from? Because they have a tendency of copying each other. It's just like a game of tag. Where is the creativity coming from at this point?
Scott Jenson: Yes, that's an interesting question because as a UX designer, what I would say was that Apple was always trying to initially just really solve deeper problems. I worked at Apple for eight years and it was just like, what did people need? And what do we want to do? And I think all tech companies have changed recently, so I don't want to single out any one particular one, but Apple and frankly Google have been, what do we need? It's why I felt like I had to leave Google. It was all about the money. And when I was at Google, we really were a bunch of Boy Scouts and we just wanted to do what was right and the money would follow. It's not like we didn't want money. It was just more like if we did the right thing, it would take care of itself. And it's become so clearly about lock-in.
I mean, what Google has done with Android and Chrome and what Apple has done with Safari and locking out browsers, this is not for users. This is not a user empowerment strategy. This is locking you into our ecosystem so we get our 15% of the revenue. And so yes, I think there's a natural frustration with us, for lack of a better word, late stage capitalism that is showing up in these companies. And I refuse this framing like, well, you got to eat somehow, which just gives up any attempt at ethics or longer-term thinking. And I believe that there is a win-win here. You can make money and you can think longer term.
And in fact, as a UX designer, you're frequently making this play to companies, which is to say, "Yes, if we do this UX work, it's going to cost a little bit more money, but you're going to get brand loyalty, you're going to get less errors, you're going to have less callbacks". So this idea about investing in the future to get bigger returns is at the heart of social democracy. It's a longer term vision that means that we can do the right thing and possibly make money.
Olimpiu Pop: Yes, I agree with you. And while listening to you, an idea was just spinning in my head. It's all nice and cozy thinking about having partially the data that you normally need in your pocket on your phone or tablet or whatever. But the tendency of the standards, and I know that that's a topic that you touched upon, is that the images, for instance, they're so huge. And once you put them on iCloud, they have a tendency of providing very small thumbnails on your phone, but you still need the cloud.
So even though I have what I consider something that is very generous, it's 250 gigabytes of memory, of space on my phone, it's always full. And whenever I want to check a photo, it's like, "Okay, your connectivity is not that great. So please rejoin the Wi-Fi". So I think that's one of the challenges into probably a degrading way of looking at it.
I remember back in the day, 10 years ago, we developed an application and we had a degrading level of quality. For instance, if I was on my mobile phone, the photo was up to a given resolution. And then if I was on a more powerful device like a tablet or even a MacBook, I was providing different degrees. And then that allowed us to zoom in or zoom out depending on the resources that we had. And as you mentioned, there is an ecosystem and everybody tries to lock you in. And I do have to admit that the Apple ecosystem is very well integrated. Everybody else is ears, light ears away. But that's my problem, as you mentioned. A very simple scenario. For me, it happens that I need to send emails from my mobile phone. I download something and I cannot find the file anymore.
And then it's a lot easier to just go to my MacBook. But I think these are the points that you had as well. And going back to desktop, what's the real problem or how you call it? The curse of direct manipulation, I think you referred to it. What are the building blocks that you were starting when we're looking at desktop?
The Curse of Direct Manipulation [[12:53](javascript:void(0);)] #
Scott Jenson: Yes. Well, the previous point you made about the down a file and then not being able to find it on your phone, that is a fundamental problem of mobile because what mobile did was it hid effectively files within each application. So if I want to edit a file in another application on mobile, it's nearly impossible. The only way I can do it is to share it to the other application, which often makes a copy. And there is a file manager, but most apps don't use it. In fact, I have a whole blog series about this and how we can have the best of both worlds. We can have the simplicity of hiding files within applications, but we can expose them as files. That's a separate conversation about just evolving the mobile system to be more file-friendly, but keeping this whole self-contained model that makes them so much simpler.
But now going back to your curse of direct manipulation, I used that kind of with my Alan Kay quote that, "Perspective is worth 80 IQ points". And so by understanding a little bit more about what the desktop has historically done, it gave us perspective to figure out what we wanted to do to improve it. And so the idea there was that direct manipulation is stateless. You cut, you paste, you type, you open, you close. And there are all these atomic elements that let you manipulate a document. It's all good. No one's complaining, but it's by design stateless. It doesn't remember what you did. It has no history of what you've done.
And so I used that as my perspective to say, "What would happen if we had an operating system that actually remembered a little bit more of what you did?" And then I had three different prototypes that kind of showed what that was like because that would allow effectively a little bit more stuff to happen and gives us support.
Because to me, the whole purpose of an operating system is to support the user doing work. And direct manipulation is one level of support, but then the next one is this ability to help me maintain my context so I can come back from lunch and I can keep working.
Olimpiu Pop: Yes. So taking a step back, it's something that just circled in different environments. So people are saying, "Okay, data is the new oil", and stuff like that. So that's kind of true also for us because in the end, if you get a laptop and doesn't have anything on it's just a machine that is very powerful, but doesn't have the ability. Once you have an operating system on it, you have a collection of applications that allow you to do basic stuff. But once you start using it for you, you have files and all those aggregated. But if you'll open the downloads folder, for instance, that's crazy. It'll be impossible.
And what you mentioned and what you underline now as well is it should be an extension of ourself, our brains, and have also a working memory, some kind of metadata that will allow you more context. And the index is the facility to which we are mentioning to do that. It'll allow you to make things a lot easier and then just resume at different points and then avoid the different types of systems that you need to just do by hand.
Scott Jenson: Yes, exactly.
Olimpiu Pop: You had also a couple of suggestions. You didn't just put the finger on the wound, but you had a couple of proposals, three of them, if I remember correctly. What do you think at this point would make our lives as users of desktop easier? What would be the options that we have on the table?
Scott Jenson: Well, yes, I have a whole talk I can give on that one if you want. But no, the-
Olimpiu Pop: Go ahead.
Three Prototypes for Future Desktops: Spatial, Episodic, and Associative memory model [[16:33](javascript:void(0);)] #
Scott Jenson: No, no, no. But the three prototypes that I showed at the Local-First Conference was about this idea of effectively working memory. The first one was about trying to not be blinkered by the small screens on mobile or frankly, even laptops. And so imagine we had a bigger screen and what could we do with that? And so I had effectively a window manager that allowed you to spread things out so that you had a wide range. And that gets into human perception and how it's very easy for the human eye to look left and right on the horizon and see things in the distance. And so the idea was to have all of the activity that you're doing spread out in a very helpful way that gives you the ability to find things quickly.
I have a lot of issues with people in the Linux community because they are very smart and they're very organized, and that creates a certain type of very rigid software. And the example I gave is virtual desktops. Virtual desktops are very powerful, but they only really work if you're really smart and really organized. And I think a lot of the world is very organic and messy. We all have desktops where things are piled on it. And so I kind of wanted to have a windowing system that was kind of the anti-Linux approach, which was just more organic.
So that was the first one was a window manager that spread everything out. But the other one was to say that we've all been in a situation where you cut and paste something and it's lost because you cut something else on top of it. And yes, all operating systems now have a clipboard history, but we don't really use it. They're kind of a pain. It's like, "Oh, it's the seventh thing down". Or you could, in theory, go into one field and cut every single field out and go to another field and paste them all back in again. But that's nearly impossible. It's tedious. It's just so hard to cut.
So the idea was to say, "How could we elevate this whole copy-paste history model?" So I created a system where a little bit like Obsidian, we had a little side drawer, and you could put all of your clipboard history in there or the files that you're working on or sub-things. It was meant to be effectively a little mini context window that allowed you to store all your data for that particular project. And you could open and close it like a file, but it acted like a folder. And so it was meant to be a way for you to have working memory of that stuff.
And then the final one was more of a history that was totally stolen from a classic paper in the '90s called Lifestreams. And the idea was to say, well, what have I been doing the whole day? Just give me a ribbon of everything I've been doing, but provide some clever visualizations. Because once you give someone some scaffolding, a memory scaffolding, it's like, "All right, that's what I did right before I talked to Olimpiu", or "This is what I did before lunch, this is what I did". And it gives you a little bit of a structure and you can find things effectively by adjacency. And so these aren't the only three things that we could be talking about, but it is an attempt at me to articulate and give examples of how we can have an operating system go well beyond what it's been doing today. And I hope it got people excited.
Olimpiu Pop: Okay. So we are looking about the working memory and how the things are being done and different perspectives on top of that. One of them, as you mentioned, was spatial and that allows you to organize it in a different way, but not in the way with the virtual desktops we have. Because my feeling is that always when I'm looking at virtual desktops, it feels that I have my ski suit on and I have so many pockets, but I'm not ever able to find the car keys or anything else, especially when I'm wearing the ski gloves. That's one. And then that allows you one important aspect that for me it was really impressive was the fact that even the smaller files are useful. So meaning that you can see the data there and you can just use them. Well, for instance, you had an example of a music player and then that's depending on the way how it's zooming in or zooming out, it just changes shape and is still useful.
The other one is in terms of a timeline of what we actually worked and then allows you to look at the things differently or the clipboard being organized in an associative manner where you have different things that are grouped in the way that it's better. What do you think other things that would be useful just as guidelines like best practices? You mentioned you have more, so if there is anything that needs to be brought to light.
Scott Jenson: Well, my overall thesis is the desktop and frankly mobile have gotten frozen in time. And all I want us to do is to try a little harder, to just explore a little bit. So I've written on my blog, I have the future of files and the presence of files that talks about how we can extend mobile. I've done work in the past with Google. I did a project called Eloquent, which is also on my blog that talks about how we could significantly improve text editing on mobile. And then of course, these three concepts I just did on the desktop.
So all I want us to do, my primary goal here is to say there's so much we could be exploring and the world is not set in stone. But so often people think that I want to blow everything up. And there's a crowd of people that says, "It's taken us 30 years to get to this point. Don't blow it up". And I'm like, "Oh, I don't want to blow it up". I mean, things are always changing and we're not going to use the same desktop 50 years from now. Something's going to change. And right now, the way things are looking is we're just going to let Apple hand us the keys to the future because they're going to define what we want to do. And they clearly aren't earning that right. And frankly, neither is Google.
Now I know I'm becoming a little bit more of a radical here, but I mean, my point is to say let's appreciate that there's more to do. And so I gave you three examples. I know there's a gajillion more. And so I don't have anything specific other than those three. But what I am doing is I'm actually trying to turn my prototypes into real code because that also helps.
So the last one I gave you, the live streams that shows me, I've now got it written as a Chrome extension and as a Swift extension. And it's capturing desktop events and it's capturing Chrome events and it's putting it together into a single thing. And I'm going to be giving another talk in September at KDE, and I'm hoping to have it done in time for that. And then I'll open source that, of course, so everybody can play with it. And it raises, by the way, all sorts of privacy issues. But the goal right now is to just try and to see if it's powerful and if it's helpful. And then if it is, then we have to then have further conversations because there is this tension between a system that remembers and does things for you is also a privacy risk. How do you navigate that? And how do you make sure that that's done in a way that's protective of the user? And so that there's a deeper conversation to have there.
Olimpiu Pop: So we are looking forward to your work. Good luck with getting it in time in September. There is not that much time until then, so best of luck with that. And I'll try to add another point based also on the conversations we had and also on your talks. And that's standards. Because now there are a couple of ecosystems, and I'm just thinking now about Apple. I would've stayed away from that. But you did mention earlier AirDrop. And my feeling is that AirDrop is such a great technology, especially when you have people in the proximity and you don't want to go all the way to the cloud and send them on different type of messaging apps. But that works only in the Apple ecosystem.
Scott Jenson: That recently changed. Now Android basically has figured that out and so now it works.
Olimpiu Pop: And that's where I wanted to get to, is the fact that there is a lot of effort on the European Union side to just open that up, to create the standard based on it. AirDrop started it. And it seems that the points regulation is useful as well. Do you think standards is also an important aspect to just have this bridge across different technology gaps? Is it something worth to keep in mind?
The Role of Regulation and Standards [[24:40](javascript:void(0);)] #
Scott Jenson: Oh, absolutely. But whenever you say the word standards, every tech bro in the room will stand up and yell, "You're killing creativity". And so I want to make sure we acknowledge that this is a challenging issue, but say for example, getting Apple to get rid of the lightning connector to go to USBC, that would've never happened if it hadn't been for the EU. And so that was, I think, very good pressure. And again, it shows the fact that from an e-waste point of view, it's the right thing to do to have one universal connector, but it was not in Apple's best interest. It was in the world's best interest. I wouldn't call that a standards conversation. I would just call that kind of a compliance conversation.
And so I do think that there is a bigger role for governments to force companies to think longer term. And I'm going to go back to my radical roots here about late-stage capitalism. It's not that these companies are necessarily evil as much as people like to say they are. I like to say that they are amoral. They're not immoral. They just don't care. They literally only care about the very, very short-term effects, which is why we have global warming and other things as well.
So to me, the way of framing it is what's the longer term goal? Because there's money involved. It costs money to think longer term. And so how do we do that in a way that is fair to these companies, but frankly, is also fair to us? And because I think the vast majority of the decision-making has been only what's fair to the companies, it's been extremely one-sided. So I do believe it's time for a little bit of pushback.
Olimpiu Pop: I would normally refrain from having these kind of comments, especially in the time-bonded context, but you mentioned radical twice already, so I have to mention my radical perspective as well. You're totally right that companies tend to look only at their interest, but that interest is fueled by our comfort because we have a tendency and you can see that all across as humans we'd have, okay, it's a lot easier to bookmark a page. But rather than bookmarking it and being organized, and that gets us to the initial point, we'll use Google search engine. And that's obviously something that goes all across the internet. It's burning fuel and so on and so forth. And now with LLMs, that's even more so.
Scott Jenson: Agreed.
Olimpiu Pop: Ethical aspects, environmental aspects, and so on and so forth. So if we would be more concerned about the future, because the general population will be the most affected by climate change and all the other problems, the companies will have to stand up to their standards. And you can see those points mainly across the last couple of months when there is this sovereignty movement in Europe and where we just realized that we do half things as well. And I'm happy to see that we still have people that are aligned across the pound in terms of what are the European, the Western values and what do we think to just make sure that greed is not the only thing that is fueling technology and everything else.
Scott Jenson: No, I completely agree. And there's multiple forces involved. So we talked just a few minutes ago, we talked about the role of governments to force these things, but also there's the role of technology. For example, what's going on right now with LLMs and the big data centers and so forth. There is likely going to be a technological force that's outside of the governments, which is to say we're going to, over the next five years, have machines that are significantly more powerful, and we're going to have open source local models that can run locally. And this mirrors what's happened throughout the history of technology, whether you're going server side. We went from mainframes to laptop, mini computers. And from mini computers, we went to cloud and we went back to phones.
Beyond Chatbots: Agentic UX [[28:28](javascript:void(0);)] #
And so this idea of going cloud and local and cloud and local has repeated itself throughout history. And I think we'll see the exact same cycle with LLMs. Just yesterday I was trying to do some coding using a local model and it was pretty terrible, but I had to try. But I did switch to an open coding agent and I used DeepSeek and it was shockingly good.
Now, DeepSeek has its own issues. I'm not trying to say it's perfect, but I mean I'm saying we can see the vector. And so as we get a more ethically trained model, which still I don't think that mark has been hit, they all have ethical issues right now. But the point is if we get an open source model that we can run locally, that provides, I think, a fantastically interesting opportunity to switch power away from these big guys too. So there's lots moving. That's interesting.
Olimpiu Pop: Thank you, Scott, for moving the direction towards LLMs because I know your opinion and I align with that because they are at points immoral and they are a terrible disaster for the green footprint, but they're very useful at this point and they'll be able to accelerate it. And now as you open Pandora's box, I have to ask, the guys from Google tried to make everything a webpage with ChromeOS. Well, that didn't take off. Now the guys from OpenAI want to make a small box with just one button and that's it. Is the discussion that we just had useful in the context that everything will be a chat window and everything else will be just typing in?
Scott Jenson: Okay. How much time do you have? The short answer is hell no. And I think as a UX designer, I mean, again, I try to write my thoughts down. So on my blog, I've got a post called Boring. Boring is good. And I think that there is a lot of ways in which we can use LLMs that are smaller and easier, but also we need to understand that intelligence is inherently social. It's a construct that you and I agree on as to what is intelligent. And so much of what we do that is work is actually part of our cultural construct. It's not a mathematical concept of intelligence. It's something that we share as a society. And the same thing applies to LLMs and work. So much of the work is more nuanced, it's got context. And so I do believe that we're going to get more LLM-based support, but it will move towards the problem.
And slapping a chatbot and everything is going to become the first resort of losers. I mean, yes, you can do it and there is some value to it, but I do believe that having the LM built into your problem is more work, but it will allow you to do far more interesting work. So I don't want to go into the details of the specifics, but I do believe that we will get past this chatbot phase as more and more apps are written. But to be clear, it's not as easy. It's cheap to throw a chatbot on.
Olimpiu Pop: Okay. In my evil moment when I was working for a company where dollar was king or where euro was king, I was trying to frame the offer of the company during the takeoff of the L&M. And what I was pushing for was augmented user experience, augmented interfaces. So at points, as you mentioned in our previous conversation where, I don't know, you want to find the right temperature balance with the voice interface, that can be quite problematic when you can be a knob, like just going left and right a bit more, a bit less. It could be frustrating. And that's the feeling also with LLMs at points. But what I think an LLM can do is provide the mechanism at a given point of time where you want to fast onboarding.
Let's say you have an accountant, 20 years of experience, he worked his whole life into a given tool, a given software. He's used only with that. He knows what to do. He moves next day to another one and he doesn't know what to do, but he knows exactly what he should ask. So that, from my perspective, is a good way of looking at it. And it can be voice, it can be a couple of buttons, it can be a chat window. And that's a good way of interacting with an LLM that will provide you the exact tables and charts that you would imagine. And that will be a lot faster than low-code from my perspective.
Scott Jenson: Oh, yes. I mean, just the other day I had a whole bunch of data and I just said, "I want to organize this, but not quite sure". And he goes, "Oh, you should probably use a pivot table". And I'm like, "Well, what's a pivot table?" Because I'm not a big spreadsheet guy, and it explained what it was. And now it still was chat-based. And so what I would challenge these systems to do is to then move to the next step, which is to say, "Okay, I'll actually input the pivot table for you". And in fact, actually, I take it back. I went to Gemini and I asked about it as a pivot table. And so I went into Google Sheets and I take it back. It built a pivot table for me. And so that's what I mean by going into the app and doing more of that.
So yes, I think that there are lots of ways to do that. And I find myself really leaning into the education side of things, which is to say once I learned that, I said, "Okay, now tell me about pivot tables. Why are they there and how do they use?" And it gave me a little mini-documentary. Now, to be fair, this is entirely a chat-based interaction. So I don't want to say chat-based is entirely useless, but I'm trying to draw the distinction between learning and doing. And I think we're going to be doing more doing as these things get more mature.
Olimpiu Pop: Yes. Well, it's being refined. I think last year I had a conversation with the guys from Ideal and they had a lot of research into the AI space and they did multiple things, bringing together QR codes, LLMs, dots on pictures and all that stuff. And they tried different mechanisms and different aspects. And I think it's a whole creative evolution, but probably we'll see what comes next as long as we remain creative, as you mentioned. We just try to push ourself into finding the right balance and the right tool.
Scott Jenson: Before we go, I want to make sure that we don't go down this naive path of just adding more tools to the AI because what I just said implied that I just wanted to have more skills and then I could talk to it and it'll just do things for me. Whereas the example I gave in my boring post was about a company that I don't want to encourage them. I'm not trying to recommend them, but it's called Napkin AI where you would talk to it, but it would produce effectively an intermediate artifact. And a lot of the options were built into the user interface.
So in other words, it gave you a whole bunch of choices and each one had alternatives that you could explore. And it was a classic UX. It's like, do you want this one or this one, this one? So you could have talked about it, but my point is a lot of the AI's options revealed itself into classic UX elements.
And so that's quite separate from throwing more skills at a chatbot approach, which is I think the only hammer that these systems can swing is to just throw more skills into the chatbot. So that's the distinction I want to make.
Olimpiu Pop: Yes, most probably just listening to you, what I think I will deem useful, especially in ideation phases as you mentioned, is the ability of switching dynamically between different environments. We started this conversation, but if I'll tell you about, "Scott, I want this kind of frame and with that, that and that", probably it'll be a lot easier to just take a white paper, just draw on it, and then just add colors and so on and so forth. So probably that would be a more morphic ability of doing it, but we'll have to see how things are evolving.
Scott Jenson: Well, yes. And I think it's one of the reasons why UX designers are getting so excited about using coding agents because it means that we can see the results of the coding immediately. And so when I'm working on a more creative idea like this timeline, I've done the research, I have the UX, I know what I want, but there's a lot of details as to how it should be presented to the user in a very nuanced way that just requires a lot of iteration. And what effectively agentic coding allows a UX designer to do is to explore their prototypes significantly faster.
How LLMs Enable Designers to Showcase Their Ideas [[36:56](javascript:void(0);)] #
So yes, I'm using a chatbot to write code, but then I'm playing with the code to say, what am I learning and what's different? And I know multiple designers who are extremely excited by this new switch. It doesn't replace programmers either, by the way. It just allows the UX designer to play at a much, much higher speed than they were before. And I think it actually produces significantly higher quality just because you can try so much and you can fail so much that it gives you that opportunity. So there's a lot of power there.
Olimpiu Pop: What I feel is that before I was limited by the knowledge of using particular tools. For instance, if I was deploying something to the cloud, I used, I don't know, GCP or Azure or whatever because I knew it and that was the ones. But there are other ones out there now. It's just an abstraction point. And even if it's not ideal, and that's the same thing with UI. I don't know, I used React. Now I have the ability of using other frameworks easily that I like how they look or even just go and say, "I like the way how those things are looking. Make me something similar, but tweak it in that way". And that is a lot easier to get to the point that you want.
Now about the quality, that's a whole different perspective and that's where things need to be improved. I'm just thinking about the issues, both on OpenAI or Anthropic where they had agents that just went through the guardrails and just created havoc on the internet. So let's hope for a beautiful future.
Scott Jenson: I guess the closing point I'd like to make is that my mindset is not that these are the answers. It's more that these are the problems and I just want us to see the problems. So often when I talk about the fact that the desktop is dead, I get people to go, "But it's a standard. Why do you want to blow things up, Scott?" And I'm like, "No, can we see that we have so much more that we have to do?" So that's all I want to focus on is to see the problems, get excited by them, and to explore new solutions.
Olimpiu Pop: Yes. So it's about the evolution and we should not comply with the status quo because it's something that we got used to. We have to get out on the comfort zone and just move out and make things better.
Scott Jenson: It's surprising how this feels so conservative or the world's gotten so conservative.
Olimpiu Pop: Yes, that's sad. I remember at some point I was reading an article about the movies and music becoming so boring. All of them are just copycats of the other ones because somebody just looking to the numerical part, just took out the soul, took the creative out, and everything became money. We should take a step back, look at the problem, improve it, and then hope for a better future.
Scott Jenson: Exactly.
Olimpiu Pop: Thank you, Scott. Have a great day.
Scott Jenson: Thank you. Enjoyed it. Bye-bye.
Mentioned:
Physical WebMacintosh System 7Lifestreams (Classic research paper)Napkin AILocal-First Conf Talk on Desktop UXUbuntu Conf Talk on Modern UX Lack of ImaginationHow LLMs need to evolve blog post by Scott JensonRich (Python terminal formatting library)Niri (Scrollable tiling Wayland compositor)
More about our podcasts
You can keep up-to-date with the podcasts via our [RSS Feed], and they are available via
[SoundCloud](https://soundcloud.com/infoq-channel),
[Apple Podcasts](https://itunes.apple.com/gb/podcast/the-infoq-podcast/id1106971805?mt=2),
[Spotify](https://open.spotify.com/show/4NhWaYYpPWgWRDAOqeRQbj),
[Overcast](https://overcast.fm/itunes1106971805/the-infoq-podcast)and
YouTube. From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.