 What happened when I was at Yahoo many years ago in the mobile iteration is that they recognized that this is not just a revamping of the existing technology, which they did to us at Yahoo at first. Oh, just use the web APIs. No, it's going to be a transition. It's going to be a shift. That's why you have material design now. That's why you have, you know, you know, these teams at Apple that want to make sure that the experience is meeting what the expectations are. So developers with designers can design these applications that are going to work really super well but we don't have that defined in the future yes we are going to be able to have something using ai in that development process so that all of my software is going to match well with whatever type of design language that the designers come up with and that's not just on the unit on the ui it's the ux so that i can define an experience from the product management's perspective and be able to have a way of implementing that quickly and deploying it i hope that's sort of my dream. But in the meantime, there, you know, this paper is, I've been waiting for something like this. This is a great paper to start with. Well, I'm going to get off my soapbox. No, I really enjoyed that, Mike. And you said a couple of really key things to me that sort of I've noticed as well. And the first thing is the chatbot era will be looked back on like the DOS era, you know, before we had GUIs. That's the first thing. But there was a phrase you used over and over again in that discourse then, and that was with me. I want this done with me. And that really comes the the crux of that paper in many respects because it was you know that that that bi-directional communication of watch and this is coming from that paper what should this agent achieve how should how should they achieve it what tools should they be calling and more more than that is the education part as well. What can this agent actually do? What is it currently doing? Is it doing it with me? And ultimately, did it achieve its goal at the end? And am I happy with that? And that all takes some kind of UX design. And as you rightly said there, this is so nascent. The paper is extremely readable, and I'm very glad that they've made this call for design patterns and principles, because nobody is an expert in this yet. So someone's got their hand up, it looks like. Simon. Yeah, just a few thoughts maybe to get people thinking. So when we talk about the first thing, I'm extremely, my perspective, I'm extremely skeptical about this whole thing. I think it's just a hype cycle. We've all been here before. But like, the integration piece, I think is pretty, it's a lot harder than we kind of make it out to be. But also, there's like this fundamental problem with interacting with an agent that can kind of do whatever, which is intent. Intent resolvers are extremely difficult. And if you get it kind of right, it's a terrible UX. It needs to be 100% accurate. And there is no way to know if you've gotten it right or wrong until you've made that guess. And so like, when we think about UX, we'll think about like, yeah, it's a lot harder. Getting this right is a lot harder than we think even in a very narrow scope context. Like, yeah. And it's, I think it's a human thing that unless you are a human talking to a human, that you understand the context and you understand the people, place, products, and processes, which is human, you are unable to understand the intent. The same way that everybody on this call would describe the same tasks differently. And I probably wouldn't be able to guess what you're talking about until you started talking about it. So that's just, I think, you can't just like, we'll figure it out eventually. It's a lot harder than we think it is, I think. No, I think you hit on some really important points there. And one is that the user experience for me is I'm going to expect to be much different from you. And I'm expecting that these systems are going to be able to learn how I interact. That, you know, at 8.25 in the morning, this is the way that you can expect me to be communicating. And what I would be expecting is like the rush to get to the kids to school, the time, like the way I interact right now is going to be very different from, let's say at five o'clock tonight. and that the agents should be able to customize my interaction and the dynamically work based on yes by the way you have a flight to catch by the way you know knowing what the like you said is is understanding what the intent and having some way of being able to collect the feedback from that i mean in mobile in mobile, we did that for many years. I mean, it was, you know, analytics. Every single phone, every single app is just spewing tons of analytic information that can be used to then be able to provide the feedback to the agency. Well, Mike, I'm going to push back on analytics about like this agent stuff because exactly because it's non-deterministic, the being like analytics and maybe I'm in the wrong, the wrong round table. And then I'll stop talking after this, but like fundamentally that, that you do not know what the person is going to say. You do not know how your user is going to interact with it. You also don't know what the person is going to say. You do not know how your user is going to interact with it. You also don't know how the processing is happening because it's like kind of black box and other ones. And so you effectively do not know what your product is outputting. And from a product management perspective, that puts you, like you don't know what the user experience is, which is just so wacky, like such a wacky premise to come up with. And then also, if you want to try to analyze that, you're unable, just because of the variabilities and the non-deterministic nature of these interactions, you can't have a really nice flow because like, maybe this person is using the same five words, but in a different way than the same person who used the same five words. So like, you need the intent. It's just so complicated. And these aren't sci-fi problems. I am having this problem right now at work. And it's not with an agent. It's just with an intent-based chatbot. And so it's just like it scales in just this crazy way that I think we're underestimating how hard it is. Absolutely complex. Yeah, I must admit, given this some thought as I have been, I'm not underestimating it, that's for sure. It's pretty scary stuff, to be honest. And I agree with you because I'm dealing with large language models and I'm trying to analyze intent. And the only way I can do it is kind of if people start repeating the same question. I know I haven't nailed it. I'm moving from the deterministic. Put in X, get out Y every time. Now it's like, I'm not sure. So much of it is an education piece as well. But the whole world's kind of trying to engage with this. You know, hey, you're dealing with probability now so yeah it may it may not work you know but if we're if we're gonna get i i mean as i say uh stripe released an sdk so that large language models can can do financial transactions on your behalf this is it's kind of like uh that like, I actually, sorry to toot my own home, but this is how I concluded the post on the Microsoft thing. Yeah, building powerful AI agents is one thing. Ensuring they're transparent, trustworthy, and aligned with the human need and expectations is another. And I think that everybody that's like AI agents, AI agents, is not considering this aspect probably most have gone off to do a technical talk they'd probably prefer to be there then consider hold on a second the thing that might really hold us back is the human aspect so i'm there with you is what i'm trying to say simon i'm dealing with the same same stuff and it's tough it's hard yeah yeah okay anybody else anybody got any comments anything they want to add to it anything anything. Hi, everyone. Hi. I come from a very different background. I just started diving into AI like a couple of years ago. I come from an operations background, finance, trading, and all that. So I built my first AI agent, which was very static, not built on any framework, not Autogen, not any other framework. That was like sometime. And then I was wondering now when I listen to what everyone was saying right now, what we expect from an AI agent differs from each. Like everyone has a different expectations. What's an ai agent differs from each like everyone has a different expectations what what's an ai agent what does it do what because it also depends on what features are we giving this ai agent yeah and once we give uh ai agent some features we program it to do something we our expectations of ai agents will differ let's say today I gave it some features. Okay, I see some potential. I expect something different from, let's say, someone was using the same AI agent yesterday, for example, and they were using a different, let's say, the older version of the same AI agent, for example, or compared to a different AI agent developed by someone else with different capabilities, for example. So we don't have a unified expectations. So our experience, there's no, let's say KPIs or something to measure the experience or what should be the standard or the benchmark for experiences we get from AI agents, because each AI agent is built differently and for a different purpose as well. So there are so many variables going on and there are no unified frameworks. Everyone, like every now and then there's a new framework coming up and some frameworks are getting updated or all of that, you know? And also AI agents are being implemented for different reasons. For example, some AI agents I've been working on, there's one project I was trying to prove solopreneur can work so one person one man show and surrounded by AI agents multi-agents even using your multi models you know so my expectations when I'm using those AI agents which I'm using them for internal purposes to handle work to do things autonomously and all that would be different from someone, let's say using a custom GPT, for example, with some features, for example, let's say, or an AI agent for the development as a front, let's say, a front office functionality to handle customer service, you know? So how do we exactly define what's a good experience from a bad experience related to ai agents so i think it's very difficult to narrow down or set a benchmark or something like that this is my input on this thank you oh yeah i totally agree with that uh but ultimately in the end you're either going to have a good experience, a mediocre experience, or a bad experience. Whether it's a program that is a mobile app, or it's on your computer, or it's an agent that has an interface that you're working with. And what your feedback is, I mean, one-star review when I had a mobile apps, oh my goodness, the VP would be at my desk. You are going to have, and that was my point back in terms of analytics, you are still going to have a way of providing feedback to whoever the provider of the agency is to say, this is not working. I didn't get my trip vacation to LA set up properly with this agency. I don't want to use this agency anymore. Perhaps I should use another travel agency. So yes, it is going to be, again, you're right, there is no clear definition, frameworks, KPIs, and ways of being able to manage this. And yes, because a lot of this is behind this is a non-deterministic LLM that is making decisions. It's not something of somebody can sit down and create a test suite with 1500 different interactions and knowing exactly what words would be said in every single time. We're not doing that. So, and again, the expectations also is, I want my experience to be different from stewards. And we have different ways that we'd like to see working with a travel agent. And that's good because now we can do that, right? If anyone's ever tried to create a customer support chatbot before from scratch or from any of the frameworks where they give you this, you know, tree structure that you have to put all the questions in and try and help your users, it's horrible. But if you have a way of leveraging LLMs in an agentic mechanism where you're using the reasoning and you're being able to get those types of responses where, again, you can by building in guardrails as well as being able to build in ways of modifying what the inference and the response that you're giving to the person so that you can have that experience. And in the end, I'm either going to give the guy a five-star review or a one-star review. And that's going to determine whether or not that agency is working. Thanks, Mike. Gil, did you want to say something? Yeah, thank you. I just want to add to what has been said so far. I think we do have KPIs and metrics I just want to add to what has been said so far. I think we do have KPIs and metrics to make sure the user experience. As Mike said, it's irrelevant for the end user how we create this magic behind the curtains, right? We can make sure maybe not the outcome, the output, maybe that is the tricky part, but the outcome, we do have metrics for that. We have a task success rate, time on task, error rate. Those are outcomes that we can measure and see if this system or this tool that we are creating is providing value to an end user right so just wanted to point out that and then and this is more kind of like a question sometimes i get confused when we talk about ai agents versus an ai assistant right In my understanding, AI agents are part of an ecosystem around the model that enables the model to do more things, right? It's sort of like plugins, right? It's not something that the end user needs to be aware of or that brings value directly to the end user. There is a middle tier in there. In my opinion, that's what I would call an assistant. And the assistant is the one that is going to interface with the user. It could be voice, it could be an actual chatbot UI, or it could be also just automating tactical tasks that are repetitive and boring so that the end user doesn't have to take care of them anymore. It doesn't have a UI, right? And the AI agents are helping with that, right? So from the assistant point of view, that's a tricky part in my opinion. That's where we need to create these experiences that connect to the end user, that relate to the end user. But again, I don't feel that that is something related to AI agents. AI agents is a technique, it's an architecture, it's something else that we use when we develop the solution. But that front end that presentation layer that interaction layer is what becomes more more relevant um taking all that work that has been done by the model and the agents and uh presenting it in a way that solves somebody's problem right and one last thought I also think and this is a really personal standing the u.s industry and especially its process its methodology is broken in my opinion my background is in architecture and when you are learning how to become an architect there is a lot of technical learning that comes with that you need to understand construction techniques you need to understand uh structural analysis right it takes years to to become an architect and the same goes to industrial designers, right? But when it comes to UX, people can do a boot camp in three months, right? And they are able to interview users, right? And create documentation based on that and maybe some models, right? But most of them don't understand the technical side of things. They don't know how to manipulate this medium software, right? But they, most of them don't understand the technical side of things. They don't know how to manipulate this medium software, right? For which they are designing. And I think that that's another issue. There was always a gap between design and development. And now with these new technologies, if designers are not more aware of how these things work for instance agents right and if they don't understand that interaction design is the u.s skill that overlaps a hundred percent with ai agent workflow design and that things like finite state machine theory is relevant because it's the glue between interaction design and the tech that supports it right then we have a problem right then this gap is just going to become bigger so we keep talking about the user-centric side of things and in my opinion most of the time is virtual virtue signaling why because if i spent hours interviewing people but i don't understand how to take advantage of this technology and the technical constraints too right in order to provide a solution that nails their problem then i am just making drawings up in the air and having a lot of back and forward with developers. So I wish we can go back to what we had before. You know, back in the 2000s, designers knew how to code. They would create amazing animations on Flash and things like that. They were aware of HTML and CSS when they were aware of html and css when they were designing um and we we came from something called human computer interaction right and i feel that that's the part that is missing so designers should get a bit more involved into understanding computer science understanding how a model works um why guardrails are important, because that's part of the user experience, how we can get that done, how agents work, and how they can manipulate these different workflows. You don't need, you don't have to design just user flows, you can design agent flows, right? And that's a responsibility for the designers. So anyway, just wanted to share those thoughts and see what you guys think about that. But I think that's what is making it hard to solve. As we've only got the 15 seconds left, two things that I really do agree on. The first is that where there's no definition of AI agents, it's difficult. And the second one is users won't care how we architect it, just like you said, they just want a result. So. Right. Thanks, everybody. Thank you.
Agent UX // Roundtable // Agent Hour
A bi-weekly "Agent Hour" event to continue the conversation about AI agents.
Sponsored by Arcade Ai (https://www.arcade-ai.com/)

This Roundtable is facilitated by Stuart Winter-Tear, Head of Product - Genaios