Ep. 5: Building Reliable AI: Testing, Governance, and Agents – with Andrew Freed

Welcome to Bespoken AI podcast. Press one to start the show.

Hey, welcome to the Bespoken AI show. Uh this is our fifth episode and we're really pleased to be joined by Andrew Freed, who is a distinguished engineer uh and inventor with IBM. Um so it's really uh a great pleasure to have him here. Andrew and I have had a chance to work on a few projects together. And so uh yeah, it's it's really nice to have him here as a guest. Uh so Andrew, if you want to just briefly introduce yourself.

Yeah, thanks for having me, John. Great to be here. Uh so my name's Andrew Freed and I'm a distinguished engineer at IBM. I've got 20 years in the industry, about 10 years or so in in AI. Uh I first joined uh AI projects when IBM uh IBM had a computer named Watson that played Jeopardy. Um and I joined shortly after that and I've been doing uh AI projects ever since.

Yeah. And uh I'm I'm glad you mentioned uh the extent of experience that you have in the space. Uh when I met you I think you'd already published your first book. Uh this was a few years ago on conversational AI. And I think I was introduced to you uh by somebody else at IBM. And I picked up a copy of the book, you know, I saw your name on the meeting invite, picked up a copy of the book. And I you know, I read read it anyway, but I read it and I just found myself in violent agreement with so much of what it said. And especially um something that I was really grateful for was you had sections in the book dedicated to testing.

Which, you know, is is the core of what we do and you know, I I think you and I found some common cause around the fact that uh we think that it's you know, something that people are underestimating and under investing in in their work. And you know, it's just basically overall look. Um And I thought that was fantastic and just overall around I thought the book was terrific.

Well, thanks so much uh John and it's in a sense it's kind of simple, right? Like chatbots are software and software needs testing. So of course we're going to need some sort of testing. Uh the nature may be a little bit different, but as I look at uh you know, testing AI it's it's not actually so different to me. Um there's still there's still corollaries to unit testing, there's still corollaries to integration testing, to system testing. And maybe the way you do them is a little bit different, right? But that concept of a testing triangle and even knowing, "Hey, what is this thing supposed to do and how do I make sure that it does it?" That's not a new problem, right? That's that's software.

Yeah, I I strongly agree with that. Um I guess do you think that the testing has it become more important with AI in general like when we're doing NLU based systems, do you think that that it take on a more importance then and then even more so now as we get into generative AI?

Yeah, I think it's it's it's always been important. It's probably getting more important as these systems can do more and more things. Yeah, like in in the beginning maybe it was {quote} easy that I'll I'll have to do an NLU to make sure it understands me and I'll do some basic integration testing to make sure I can hit all the right APIs, right? But now as I've got uh agents involved, it's not just those couple of things, but now I'm adding in even more things, right? Like, "Hey, is it going past its security window? Is it saying things on its own that I don't want it to say?" Right? Um Lots lots of more uh capabilities or even hey, it's allowed to come up with its own plan, but I'd really like it to go this this direction. I want it to call this thing and never that thing.

Does it do that? Right? So it's not going to go down the need for testing and the other thing that won't go away is knowing what good looks like or knowing what right looks like. You know, I I I I like to say it's fun to pretend. It's fun to pretend that I don't need to know what the right answer looks like. The AI will just figure it out. I know. The AI will go it'll certainly go somewhere, right? And it may be right almost all the time, right? But if you don't know what right looks like, you can't tune it and you really can't effectively build it.

Yeah, I mean we would strongly agree with that. Now somewhat you're mentioning that it's sounded to me that you're describing guardrails. Have you found as people started to adopt generative AI more that guardrails would be an essential part of implementing?

Yeah, they're certainly an essential part and I would even to to me there's there's a fundamental tension between the agent autonomy and and rules, right? And essentially guardrails is is one more way of getting at rules. And the tension is do I want to code more rules in ahead of time? Right? In which case it's not a very exciting system. We've had rules-based systems before, but you know for sure it'll follow your rules. Right? Do I want to instruct the agent and and pass it, you know, hey, please do this, never do that. Do I want to add an additional reflection?

Say hey, after you've processed these steps, you know, go essentially go double-check yourself. Or do you want to add on essentially a rules-based guardrail on top of it, right? And so all sorts of trade-offs in the How do I say it? How how um, sure fire it is, right? You know, is it is it 100% uh, going to follow this or is it almost certainly going to follow this? Uh, the the latency involved uh, in the rules checking or that um, you know, editing a editing a reflection loop on the LLM will take a little bit longer, right? So, these are all very interesting trade-offs that are happening right now.

Yeah, and I I like the fact that you're referring to them as rules uh, and also rules-based systems. So, one of the uh, first jobs I had was working with rules-based systems. And in the financial services sector and even essentially built my own rules-based system as part of the company that I was with. And it wasn't it wasn't cuz I was working in AI at all. Um, I sort of that was the first encounter I had with AI, I guess, professionally. Um, cuz it was considered a space adjacent and some people said it was a part of it. Um, but it was because we wanted delivered the ability for non-programmers to express logic. Um, and in the financial services industry, we were doing underwriting rules. And so, it was for like, okay, if somebody's credit score is better than this and their, you know, uh, loan to home value ratio is less than this, then it's okay, go ahead and approve uh, approve the loan. And it was because they were very complex and it was non-programmers who were doing it, it was seen as very advantageous to state these things as plain English.

Um, I wasn't thrilled with how it all came together. I mean, that company was was reasonably successful cuz, you know, the financial services is a good space, but I started to joke that what we really were building was a uh, programmer identification system, not so much a rules-based system. Cuz what you tend to find is there's one person in the organization that had the wherewithal to spend the time to kind of sort this stuff out. Um I guess now that I see more rules being adopted sort of again, I have this optimism that they could you know, I I always like the idea. I mean, stating your business policy in plain English is to a system is great.

And then hopefully the systems can you know, uh negotiate some of the contradictions or ambiguities of those business policies better than they did before. Cuz before it was basically like it would just kind of short circuit where there were contradictions or ambiguities. Now it can fill it in. And yeah, I mean, it it's I've been delighted to see how people are doing that. I guess from where we sit, I don't totally know if it's working completely. Um but it does seem like it's is a very um you know, potentially beneficial path to go down.

Yeah, and that that's that's one of the the the magic parts of this this technology, right? Is that I I can I can use plain English, right? Um people used to joke with me when I said I was an AI, they said, "What? Do you talk to the AI all day?" I'm like, "No, I write rules and you know, a little machine learning." Now we kind of do talk a little bit all day to it. Um

Yeah.

But but um that's that's much easier for folks, right? Like whether you're writing Java or Python or whatever, you know, there's a syntax, you got to be very specialized to get into it, it's very fragile, um very very brittle, like like you said. Um and the promise to write in your own language is very helpful, right? Um but that's that's also uh ambiguity ambiguity is a double-edged sword, right? The ability to be ambiguous helps, but also leaves some some undefined behavior. Um but I think that that actually the best part of it is uh writing it down in the first place because a lot of folks come in with the the thing, "Hey, I want this to be my second brain, right?" And that's fine, but like what's what's in your what's in your brain? And you'd be surprised how much of it you have to write out, right?

So, whether we encode it as a natural language prompt, whether we do a hybrid of prompt and rules, you know, you there's no getting around knowing what that policy looks like, and that is frankly, that's where a lot of these projects um spend their time. It's that these policies are not actually documented not to the level that um people may think they are.

Yeah, I mean, that's an excellent point. I So, do you have any particular tools you use to get that information out of the subject matter experts cuz that is a very tricky process?

Uh, I don't know if I've I've got a tool to get it out, um, but the I mean, I just start with well hey, pretend you just hired me. Like, how would you train me to become you, right? I don't I don't know anything of what you you do. You have 10 years of experience. Make me you, right? And and don't tell me I have to sit here for 10 years stumbling through all the things that that you had to stumble through, right? Um so so no no magic to it, but uh sometimes it's just it's just talking and and writing out that spec, right? I I really love the spec driven development process.

Um, and it can be hey, I just interview you. How do you do it? And maybe I run it through an NLM to say, "Hey, how do I convert this this transcript into a prompt, right?" My favorite GenAI trick right now is just trans convert this transcript into, right? Convert it into test cases. Convert it into an architectural decision document. Convert it into a spec for an agent, right? Because those are still tedious to write, right? It's it's easy to talk. It's fun talk.

Yeah. Yeah. Yeah, that's that's a great point. Um So, one area that you and I have talked about in the past, you know, some of the different channels uh for AI. Uh we both worked extensively, you know, within sort of chat interfaces, uh you know, web chat, I guess especially on our side, as well as voice. Do you you know, what do you see working the best out there? What do you think is you know, most valuable for customers? Uh what's sort of your view on the different channels and, you know, kind of handicapping them and looking at them?

Yeah. So, I mean, in in the end, I think we're we're we're going to go through all of them, and I I feel that the natural evolution is that the the web, the digital stuff gets there first, and then voice kind of catches up. And it's because in a digital channel, at least I know what they actually typed in, what they actually said. Um whereas in voice, I'm typically dealing with what the uh AI hear. Right? And I usually have a little bit of extra latency as I go that um that that second jump, right? From from uh from audio to text, then text to meeting. Uh so, usually that voice takes me a little extra time, but seeing a lot of interest in in both right now.

Yeah. I mean, do you Would you agree with the statement that voice is more difficult to do than chat?

Yeah, because again, you have that that second that second layer. And I know some of the models are are sort of skipping it by doing it at once, right? But in in the old world, we would do a speech to text, and then we would do a text to meeting, right? Or a text to next step. And that that second layer just makes it all the more interesting.

Yeah. How much do you think I mean, in terms of issues with you know, some of the sort of persistent challenge is with speed. Um you know, accents. background noise, um, you know, like I'm using a noise reducer now cuz I've got some construction going on. Um, you know, the variability of languages, uh, and performance of speech recognition engines between those languages. How much of a factor in the challenge do you see that being nowadays? And do you think it's improved in the last three or four years or, you know, is it about the same?

I

think it's getting better. The The challenge has always been the the context. The context is very short in in speech. Um, I'm working a use case right now where the the average user utterance is like three words. And I'm thinking, man, three three words, that's fantastically long. I've never had to deal with I've never gotten three words before, right? When, um, I I did a lot of work in medical systems where the a callers they say one word. Right? And you've got literally one one chance at it. There's no context to say what was the word before and after that kind of nudges me to the right place. Um, so that's what makes voice hard in general.

Um, I think I think it'll take some time for the user population to get used to speaking to a a robot with that additional uh additional text context. Um, we've all frankly been trained by IVRs to be kind of short, right? To say one or, you know, it's very menu driven very menu driven very very very keyword driven.

Yeah, I mean my impression, I I guess I thought you know, I was saying this to you just before we started recording. We've seen a lot of customers, seems like in the last 6 months, that have come on and said, you know, they're they've always been using generic for a while, but perhaps the focus has been more on chatbots. Now it seems like a lot of customers are focused on or enterprises are focused on getting voice bots out there. And the speech recognition has done better than I expected so far. And you know, and I it's not my feeling, you know, from the past thing that we've done and the sort of benchmarking that we do internally, I don't feel like the raw ASR has improved very much or changed underneath the covers a lot, but I do think you know, these systems do tend to encourage people to, you know, have longer utterances, to speak more conversationally, and so you do get that additional context that you know, it seems to kind of um outside of you know, sort of typically thorny things like member IDs and weird alphanumeric issues, you know, as long as there's some additional context, if they're a little more verbose, it seems to do a great job of figuring it out.

It it does, but that that thorny bit that you mentioned, like that doesn't go away, right? That's that's where a lot of these uh get get stuck, right? Cuz there's typically an intent gather, there's some sort of, you know, authentication or member ID, you know, tracking ID, order ID, blah blah blah, and then it's downstream, right? And um for for your your spoken intent, I can get a word wrong, I can get a syllable wrong, I can probably catch it. Um if I get a number wrong in a track uh tracking ID, an order ID, I'm out of luck. I'm not going to find it, right?

And so that's that's what it takes to get these voice agents really out there.

Yeah. Yeah, I mean, that the authentication steps, I mean, we have found have been a huge obstacle for customers that have pushed forward in more conversational things. I mean, that's I it And there is, I don't want to not exactly ironic, but you know, there's an aspect for people who are pushing towards very much sort of something state of the art, um really trying to bring in new technology, and then you get stuck on what's the very first step in the sequence, which is going to be you know, and and they really do great on everything else.

Yep.

But you know, if if your first step is like a complex alphanumeric uh that that can lead to some disappointment for everyone.

Yep. And and so one of the techniques that's that, you know, I try to do with these customers, is there any way you can skip it or is there a a preempting um can I know what number you called me with? Can I use that to to assist me? Sometimes I can, right? And sometimes I can't. From a a you know, compliance point of view, I really do need you to say that ID. So so I'm proving it's you, not just that someone took your phone. Right? Um but that's one of the ways that we try to to help with that. Any context I can bring in um I I had a time can can make it make it easier.

Yeah, one of our favorite tricks always to recommend to customers was like you know, if you have any other additional information then use that to down select and then

Mhm.

I I want to I don't know how compliance perceives this because I don't know if a lot of customers took us up on it, but then you know, past like you know, if the person said John Kelby and you don't even know, I mean, what is Kelby, right? I mean, there's plenty ways it might come up with to spell that. But if you do some fuzzy matching against the database with that name, you pull out 20 IDs

Mhm.

use those as a recognition and on your speech recognition I mean your success rate is going to go up. I mean, it does go up immensely. Um you know, it from a compliance perspective, I could see someone saying that I don't know if I would say that's cheating or not, but I mean, it's you know, that context is huge.

That That's the way I'd like to see it as can I bring up the hints as as you called it rightfully, right? So so we'll look up the hints based on my phone number. I The caller still has to say it, but I can nudge towards the hints, right? And I'm not really I'm not giving away your number, um but I'm I'm more likely to hear you say it properly if I already knew, "Hey, this kind of thing is probably coming." Right. That That's That's sort of the the sweet spot I feel for these.

Yeah. Yeah, I mean, I I certainly think that that's a great approach. Um So, when I first met you, at one book, and then you wrote a stick and puck that got into um you know, adopting uh you know, it's like chatbots and conversational AI, but really with gen AI covered additionally. And there's a lot of great stuff in there. I mean, I think like the the way you talk about rag is really great and a really great overview. Um but one thing you touched upon that I really liked was governance. And you know, I So, how you know, how do you see organizations approaching governance in this age? Cuz it it does seem like a huge question and challenge for companies. Like, who's Who does the compliance responsibility sit with? How do they actually get involved? I mean, are they supposed to look at the logs? I mean, how are organizations engaging on that question that seemingly is rather difficult?

It It is difficult, and I think this is probably the weak spot of most deployments in that Oh, we'll we'll we'll we'll It's almost like we'll get to the governance, right? And they're they're taking it as, "I'm going to test for the things that I'm worrying about ahead of time, and I'm going to, you know, sign off on a build that I tested against, you know, hallucinations and groundedness and all these different things, right?" Um which I think, you know, if you're you're light on light on time, that's probably the best that you can do.

But in terms of um afterwards, like, "Hey, you know, someone does have to be looking at these logs. Someone um you do have to define what what out of bounds looks like uh and then what you're going to do about it, right? Um so there there is there is a fuzziness in this space where there's a lot of you know doom and gloom like hey, what if it says what if it says something wrong? Right? Um and one one silver lining here is like well, how often are your humans saying something wrong, right? Your your goal your goal isn't necessarily perfection, right? Ideally, you're doing better than the system that that you replaced.

Right? Um and I think that's that's the the saving grace on this. Do I need to look at every single thing? Well, it's like well, do you look at every single do you listen to every single call that you're you know, that your humans say? Or is it more likely hey, this this agent has low this you know, human agent has low scores. I'm going to go look at a couple of their calls. Um I'm going to now do a deeper dive, right? Which I think is more typical on the other. So, I think from a you know, an AI system's point of view it it's probably going to be the same kind of way, right? I won't be literally reading every single one of these, but um I'll be looking for for indicators and when I see one then I'll then I'll dive a little deeper on what's happening.

Yeah, I I mean, I think that analogy is really apt because you know, one of the things that we've seen an explosion of uh you know, besides you know, the agents themselves is now okay, let's use AI to monitor real human agents. And so there's so many I mean, there was always QA tools and QA tools now talking about you know, we're doing quality assurance on the actual conversations humans are having with each other. But I I think that they were mostly like keyword detectors, you know, I mean, you're looking and they would there was always sentiment analysis, but you know, the sophistication of the previous generation of sentiment analysis is definitely you know, I mean, your mileage will vary uh or could vary greatly. But now, I mean, people are delivering tooling that summarizes every conversation, flags them effectively, um you know, I mean, it seems like there's a huge explosion of stuff like let's figure out what the act for humans are saying to these people. And it does, I think, um really, you know, people are raving, I think, into the forefront of people's minds of what the heck have these agents been doing? Cuz we really didn't know. I mean, we knew if people started cursing, right? And, you know, if they were hanging up all the time and there's lots of ways like that of people that were, you know, disconnecting and uh acting like it was, you know, I thought that it was with the network and it wasn't.

Yeah.

But now, they actually have much more data on what's going on in those conversations. And I do think yeah, I mean, what is how comfortable is the robot agent versus the human agent? I I haven't seen exact numbers on that. I mean, there's all sorts of stats thrown out there, but it does seem like um Well, well, I mean, I I mean, just on a personal level, like I tend to trust uh Gemini, ChatGPT more than I trust a doctor now, right?

Mhm. Well, it depends on the the case, but yes, sometimes sometimes for sure. Um and so, what you were talking there about, you know, the LLM as a judge, like that's a fantastic pattern, right? Whether it's dealing with, you know, humans or or another LLM itself. Um we we do know, of course, that they LLMs tend to judge themselves favorably, so you want a different judge um LLM than than the other. But, um I mean, it's it's a great pattern, and um again, it kind of comes to hey, what does good look like? If I'm asking it to judge, the judge is I'll say only as good as its criteria, right? It'll make up a criteria, but that's probably not what you want.

Um and it's worth taking the time to think, "Hey, what is a what is a good conversation look like? What are the things that I actually want flagged?" Um maybe you already have that from your reviews of your humans, but but maybe you don't, right? And this is a great time to define all that.

Yeah. Yeah, I uh I think that that is uh really critical and it I guess it dovetails with another aspect that I think you've been really uh reinforcing for quite some time now, which is uh how much you need to kind of work on these systems after they go live. So, I mean, in both your books you talk about how to improve systems over time. I think that was a particular focus for your second book of like, how do you get these systems once they're in production to um a level that's really satisfactory for a performance perspective. I mean, how do you Look, I mean, I'll make a statement about what we tend to see, which I think is an anti-pattern, is we do see and I I think it's improving a little bit, but for years what we saw with people deploying uh conversational AI systems was they saw it as a typical project where, you know, 80% of the work was going to be done before it went into production and then, yeah, there's maintenance, but the maintenance is sort of a 20% after.

You know, you know, for a smaller system. And for conversational AI, I would say that's completely flipped around. I mean, you really want to be doing um I don't know, it's like 80% of the work, but you you know, you would really have something that you have to take care of and nurture and grow it along. I always like that analogy of a baby. I think these systems are beyond babies now, but it is something that you've sort of adopted and now need to steward along through its life. You can't just sort of put it into production and throw it to a maintenance team. Um but that is still too frequently what we see folks doing.

Um where they just, you know, there's like a consulting crew that takes it on after, you know, the consultants disappear and and it can be unfortunate. Um how do you guide people on that? Do you think it still is pressing now? Uh what do you think about that subject?

Yeah, no, it's it's the ever-present challenge, right? Um maybe getting all a bit better here, right? But um no, this this hasn't gone away. Maybe it's a little bit easier with with the agents and the ability to, you know, tune a prompt versus, you know, write brand new rules in the chatbot. Um but all the things that we saw before, like they're still present. Like, hey, customers are going to be asking different questions. My own internal policies are going to be changing. Um you know, so on and so forth. And again, I go back to the old days. Like, how did you deal with like was software ever static? And like, I guess sometimes, right? But but hey, if I didn't have regression tests, could I ever change my old-school software with confidence? Like, not not really, right? And so, it's the same thing here. If I don't have a ground truth, if I don't have a set of um scenarios that I'm using to tell, hey, the bot still functions the way I thought it did. Right? Um that discipline still applies. Right?

Yeah. Yeah, I I mean, I guess something that uh I mean, the similar thing that I was saying that I was thinking, well, you know, a common thing for any type of application is, you know, you should expect to throw away the first version. So, it's not really true that, you know, uh you ever can just build something and be done with it. But I do think within organizations, they often are doing that. And even though it's not like it's not great if you do that with a web and mobile app, but you know, with an AI-based where maybe it's going to understand 20, 30% of the things people will say when it first goes into production, you know, there can be truly disastrous where at least the bugs feel um so much more glaring than they do with the web app where it's like, "Look, you click the button and it does what it's supposed to do." Well, nobody understands it. That's okay, that's user training or they'll they'll figure it out, it'll be fine.

versus it's like, well, 70% of the time people are just not being understood. That That sounds really terrible, right?

Absolutely. Absolutely. And I do think they're being understood much much better, but um the agentic fluency is is been a a boon at the beginning of of the calls. Um frankly, I see a lot of time spent on the the tuning and the performance especially. Right. So, one of the major things right now and we we we alluded to this earlier is do I want to be, you know, instruction driven um or do I want to code in some rules? And um one of the things I I see in a lot of agents is, "Hey, you know, no matter what the user says, do this thing first.

Call this tool first." Right? Uh and then go on and think about it and whatever. It's like, "Well, that's a rule." But, it's it's implemented in natural language that the agent has to think about. It has to read through it, has to reason about, and has to make the same decision every time. Right? And I kind of liken it to do I want an interpreted language or a compiled language? Right? Interpreted is great for for many things. Right? It's faster to write, um but it's not faster to run. Right? So, that's that's one of the the tradeoffs I'm seeing a lot right now in these builds.

Um cuz anytime you're invoking, you know, one, two, three, five LLM calls, there's going to be a latency cost, there's going to be a cost cost to to all this stuff. Right? And I can deal with that by moving it moving it out of the LLM, but is that agentic enough for me? Right? Sometimes that's even a discussion.

Yeah. I mean, those are those are excellent points. I mean, and that But that does get us something else. So, as we have pushed into doing more agentic testing, um or work with more customers and more you know, we've had a great deal demand or customers that want to you know, essentially let's use agents for everything. You know, you can use agents to come up with your data, to run your tests. Uh agents can then execute the tests. They do a nice job of talking to the other agents and navigating through test scenarios. Um and if they're both based on gen AI, they can be fun to watch, uh especially if they go a little haywire and start having conversations with each other.

Uh and then you can also use the LLM as a judge to you know, let's evaluate it uh across not even not even just correctness. We can look at many different dimensions. But then at least and I haven't you know, it's so far we've just been presenting all this data back to customers, right? Here's here's all the data and and they seem to like it. But I will say uh it's a lot of data and it's a lot of metrics. And like the in the discourse I see online, I see people, you know, let's optimize across or you know, here's 10 different metrics you might want to look at. Uh or seven different metrics you want might want to look at.

How do you I mean, how do you think customers can deal with it? I do for me at least when I look at that, I'm like, how much of this are they really onboarding? How do we make it more effective to them for them to the you know, it starts, you know, ideally to me, we like to roll up the data to a binary. You say like, this is good, this is bad. Uh because our experience is people don't want to look at 10 different metrics, much less try to optimize across all of them.

Yeah, that's that's too much for anybody. I think past like three three metrics is is even even a challenge. Right? Um So uh and that there's there's not an easy easy answer here, I suppose. It's um we try to help them test for each of these different things. Like, "Hey, you're worried about jailbreaks. Okay, well, let's let's use a standard set of jailbreak questions. That's probably not unique to you. Right?" Um groundedness, well, you know, hey, there's no getting around I need your data here to to look at what is what level of groundedness is is going to work for you.

Um and then there's some uh subjective things. Like, "Hey, I want it I I love those really wordy gen AI answers." Or, "No, no, I want a super duper short one. Right?" Like That's again, what why do you why do you want it to be how do you define what what good looks like to you? Um so I mean, you can try to bring in practices from from other other places, right? Um so you're at least not starting with you drop the book down and say, "Well, here's the 10 metrics. Which ones are we doing today?" That's probably not going to be super successful, right? Cuz the folks who just thought they could hand it to the AI AI and you know, be on their way.

Yeah. I will say though, as soon as you said that, I was like, "Oh, verbosity. That's one we don't have on our list of metrics." Well, you know, we should have that.

A nice easy one, too, right? You can count that with without a rule. Yeah.

Yeah. Well, I I'm interested to see how people navigate that. I you know, it is um I mean, there's so much to look at and the the systems are changing so quickly. Um Yeah. Uh in terms of new models coming out as well as new tooling around it. Um I guess you know, as we start to kind of wrap up, is there particular tools or you know, I mean what What things that are on the horizon that you've started to use or that you see coming? You know, what things are you most excited about? What do you think might be the most impactful?

Yeah, I mean for for me coding agents have been uh spectacular. Uh I use IBM Bob. That's the the tool our company produces. Um for for all sorts of questions around this. Like, "Hey, I want to design I want to design the test suite." Right? Um what are some things I I ought to think about or how could I present this gigantic spreadsheet in a way that's that's helpful, you know, to to my customer? Those have been fantastic. Um again, I don't get to get out of the thinking, right? Uh I I still have to describe, "Hey, this is what it does. This is what success looks like to me." Um and I get help in like writing the test cases themselves was kind of dull, right? Um or pick writing uh you know, member IDs that I was going to test on. Kind of dull, right? So, I'll let the I'll let the the gen AI take care of that. Um but what's what's going to be good for me? Um that I'm still very interested in controlling myself.

Um and I know it could generate all this stuff, right? It could generate like 900 test cases. Um and in the voice system, what if that takes me overnight, right? Is that what I want? Right? Cuz maybe that's fine. Every night I'll run you know, 900,000 test cases and come in the morning and see what went wrong. But um I tend to want a faster feedback cycle, right? And and if I'm going to say what's going to run, um what can I run in a couple minutes? Um sure, maybe I'll let the tool pick that for me, but I probably need to guide it as well.

But in in the end, where we have the true symbiosis, the coding agents are probably the most exciting part for me.

Yeah, I mean I guess I would totally agree with that. It's really amazing uh it's moved our internal development forward and it's also moved my external side project development forward. Not that I'm not able to uh it really finish anything, but I mean I now have, you know, two orders of magnitude more side projects, you know. Yeah, at the same time. Um and I will say also on the point of you just getting it around the size of the data sets. I mean, one thing I liked and I think it was in your second book, is you you know, you talked about uh having I think a set of golden utterances, I believe, or you know, golden data set. And I will say like one anecdote uh that we mentioned is like so I had you know, when when I was getting started in this, you know, we were working with kind of small organizations. Um and then I finally I met with somebody from Apple that worked on the QA team there. And this was on the, you know, before Siri goes out the door. Worked on that team.

And I was like, well so, you know, what's your data set like? Uh and she said, well you you know, we have a thousand golden utterances that we've run to. And I was like, no, that's it. She's like, that's it. Now, that's not to say I'm sure the internal Apple team then, you know, trains that model has a lot more than a thousand utterances.

Sure.

But for that core set, uh you know, for the team that's doing the end-to-end testing at the end, you know, I mean, that was enough for Apple. And I don't know, they sort of gave me the confidence to go to or I liked that anecdote with customers where, you know, cuz a lot of them are like, well we've got a hundred thousand test cases. And it's like, well

Yeah.

I mean, Apple got by with under a thousand.

So so their quality is is is on point, right? And a thousand good ones is better than a hundred thousand bad ones, right? So if I look at my my phone, what do I do with it? Like I I set an alarm, I create a reminder like a hundred times a day, right? So I'm sure you know, those sort of very simple task race task-based ones, they've got a set of those. And then you've got it folks like I don't know, you know, play me a new song or, you know, go go look up uh who who won the football game last night, right? So they probably got categorization. I want, you know, a hundred of these, a hundred of those, you know, representative. It's a very intentional set that they've that they've uh chosen. That's that's the best way to do it.

Yeah, and I think it also just it fit with our own experience of things which is like, yeah, in theory you've got I mean you know, I mean going back to some of these conversations like, yeah, I mean, five years ago you could have a hundred thousand test scenarios. I mean, now you could have if you did the math on it, it it really could be a billion. You know, I mean, I it's small in that practical purpose, but it's like

Mhm.

Are you going to run through a hundred thousand test cases? I don't think, you know, uh people don't have the appetite to review all that. So our guidance has been try to keep it simpler, try not to uh make yourself crazy with it. Make sure those core things are working well.

Yeah, absolutely.

Uh well, nice. I guess uh any other uh you know, final words of wisdom you'd like to share or anything else we can

I I I'll I'll I'll end like I started that's, you know, chatbots are software and software needs to be tested. Uh it doesn't go away because it's it's AI.

All right. Well, thank you all for joining us, Andrew. Uh I think this has been a terrific conversation.

All right, this is so much fun. Thanks for having me.