Ep. 8: The Hard Part of AI Isn’t the Model – with Cobus Greyling
Show Notes
AI models are advancing faster than ever — but is having a powerful frontier model really enough to succeed in production?
In Episode 8 of The Bespoken AI Show, host John Kelvie sits down with Cobus Greyling, AI Evangelist and prolific industry writer, for a deep conversation about what it actually takes to build, manage, and operate reliable AI systems in the real world.
From AI harnesses, context engineering, and low-latency model routing to open-weight fine-tuning, model distillation, enterprise governance, and voice AI word error rates, they unpack the connective engineering behind modern AI — and why the model itself is rented while the harness is owned.
Topics & Key Takeaways
- Why the AI harness matters and what happens beyond the model (Anthropic, OpenAI Deep Research, and coding agents)
- AI orchestration, low-latency model routing, and the rise of open-weight models
- Fine-tuning with LoRA, model distillation, and running AI in private enterprise environments
- Decision models, fleet engineering, and dialing AI autonomy up or down by use case
- How enterprises should approach AI safety, governance, and incremental adoption
- Continuous improvement, hill climbing, persona simulations, and AI evaluations
- The gap between AI industry hype and what is actually happening in production QA
- Voice AI accuracy, speech recognition (ASR) word error rates, and the hard engineering behind reliable systems
Key Moments & Chapters
- 00:00 Introduction & Welcome with Cobus Greyling
- 05:09 Why the AI Harness Matters & What Happens Beyond the Model
- 12:00 AI Orchestration, Model Routing & OpenRouter
- 18:24 Open-Weight Models, Fine-Tuning & Model Distillation
- 27:51 Decision Models, Fleet Engineering & Dialing AI Autonomy
- 34:46 Enterprise AI Governance, Continuous Improvement & Hill Climbing
- 43:01 AI Industry Hype vs. Production Testing & Evaluation Reality
- 50:57 Voice AI Accuracy, ASR Word Error Rates & Engineering Reliable Systems
Full Episode Transcript
Welcome to Bespoken AI podcast. Press one to start the show. All right, this is the Bespoken AI show. This is our eighth episode, I believe, and we're joined by Cobus Greyling, who is an AI evangelist. I would uh hopefully in a way that doesn't embarrass you. I would introduce you as one of the most curious, if not the most curious person I know that works in AI. and I'm just always amazed with how you manage to keep up with this industry and everything. So, it's really a pleasure to have you joining us.
Great. Thanks, John. Yeah, I think we met personally once in Washington once or twice. And yeah, I I I'm happy to make it into the top 10, this being the eighth episode. And yeah, I I think it it helps to be curious. I think someone I read in someone's bio these days online they said curiosity skilled the cat. So I think being naturally curious and trying to understand how things work it helps. Well can make a career out of it to be honest. [laughter]
I would agree. I I mean it seems to me it's the most valuable trait one can have now. So, I guess to get us started, and I've asked this question of everybody else, and I I'm interested in people's reaction to it, but you know, as someone who has worked in the space for some time, um, and I might have even first reached out to you, I think when you were with uh, Vodafone,
I think I was I was maybe trying to prospect or or uh, you know, get you as a customer. I don't I don't remember exactly, but so you've been in space for a while. You know, going back to like the 2018 or, you know, the the 2010s, let's say, did you conceive of that we would end up here or anything like this? I guess you know, good question, John. I think I like people often ask, was there AI before ChatGPT? You know for most people the advent of ChatGPT was the beginning of AI and I always say well before that we had NLU and NLP models and Mr. you know, which I guess you also know from our community. She once said with NLU and NLP, we used to have a problem of scarcity and then all of a sudden with large language models, we had a problem of abundance. You know, we had to build harnesses to control it.
But I would say it's it's very hard to to have imagined how far we've come and it's hard to say where things are going. I'll give you two examples, John. The one is if I just look at how the technology around large language models develop just to control it right so we you know you get first at prompting then you had system prompts then like we started to build like harnesses and then we realized there's a context problem so we need to add rag to inject the context so I find it interesting that as like the use of language models expanded we we build the infrastructure and the systems to support our systems and that's continuously evolving you know in terms of you think think about now we've got harness engineering and fleet engineering and loop engineering and context engineering so that's one part but secondly and and lastly on this question I think another aspect that really that's blowing me away and I couldn't envision is AI models that's able within a harness to to write and execute code. I remember a few months ago, Anthropic floated this idea of universal agents and the idea was that you won't have tools, right? So tools are really the the predefined connection points of agents. It's the hands and feet of agents. And they said, well, you're not going to have to have tools.
You're going to have agents that actually write the code. It needs to execute the instructions. And I mean I'm running croc build cli on my on my MacBook and the way I give it an instruction and it it writes the code to execute that instruction you know that's for me kind of like beyond comprehension. Um and and again it's not the model right? It's the model. It's the harness around the model. And I would love to maybe get into this whole harness idea later on, but the ability of of agents to write code to achieve its objectives that for me is is still kind of magical, you know, because always for us software was expensive and it needed a certain level of longevity or durability. And all of a sudden software is ephemeral, right? you you you create it, you use it, and you dispose of it, right? It it's it's it's become really disposable. So, I would say that part of agents writing code and and executing, I I find that really still quite astounding, just that whole idea.
I, you know, would strongly concur with that. I think you're more up to date on this than me, though. So I I was reading some of your articles about harnesses recently
and so I mean tell me about that. I'm I'm interested in that and why
I didn't totally get it like what makes it so significant.
Right. So, so, so maybe three things, John. I would say the first thing like just recently, I think yesterday, Coer had a article, in my opinion, it was overblown of a Qwen model that tried to phone China or tried to send data to China. And I did a blog on that yesterday. But, but the thing is a model, if you take a language model, it's tokens in and it's text out, right? So, a model on its own, it's got no capability. it's just text out or instructions out in the form of text, right? And I think one's got to keep that in mind. And so the first time I I got this notion of a harness, right, was I I went so often I go to Open AI's cookbooks, you know, they they Python notebooks and I just it's publicly available and I browse around to see what's in there. And then I found this notebook where OpenAI this is like just I think a year or more ago where OpenAI shared the architecture of their deep research API right and so I think in my mind and in the mind of the collective there was a a language model sitting somewhere exposed by API and you just that's what you used and when OpenAI shared the the architecture of a deep research API. There was actually a whole harness. So you had the API, but you had multiple models that were orchestrated that you had a model that disamiguated.
You had a small model, cheap model. You had a different model that uh established the intent and then formulated the query before it went to the more expensive deep deep research model to kick off the search and come back. And obviously that model's got internet access. And when I saw that I realized well this is not a model the deep research API it's not a model it's a it's a harness it's actually a whole system right the a few years back or maybe a year back Nvidia and you can google this Nvidia released artic a research paper where they said the future of agentic systems is multiple smaller models that is orchestrated at that stage it was revolutionary right Today we there maybe a third example also in their cookbooks OpenAI released the architecture of ChatGPT and somehow we had this notion of ChatGPT is one model with a web UI wrapper and but there's also there was a strong harness underneath research also showed that that Claude Claude Code also has a strong harness and many of the updates that came through on Claude was not necessarily model improvements but harness improvements. If you look on arXiv, you'll see that um many studies that show that they've got there's up to 50% increase improvement in performance when they tweak the harness instead of the model. So, and that's where we're currently at, right? So, it's not just models anymore, but it is the the harness that sits around the model. And so, the harness can manage one or more models. It manages the context. It manages the memory. So any conversation needs context and context in a conversation is in intelligence. Context is normally saved. Context is informed by memory. Memory is saved as data in a database. And so that's the se that sequence and all of that is managed by the harness. The harness also got the rules of what access the system has right to to to to to a CLI or does it have internet access?
Can it SFTP? What protocols does it have access to? So the harness is really this this control encapsulation within which the the one or more models run and that's the enablement you know for that you know for that model to run. And so I find it kind of I find it amusing if people look at the model and if you take an open weights model and you run it in a aircapped environment highly controlled there's just no way it's going to be nefarious if you have the safeguards in place it's just a model but I will be very uh concerned if I'm an enterprise or any company and I don't have my own harness my own own control environment uh you know with within which I um I host and I and I run my models. I don't know if that makes sense, John. That's why I got
No, it does. And it brings up so many different topics to kind of go from there. I guess so. All right. So, one question I had though around the harnesses is do you like Okay, and I think you know we're naturally I think going to gravitate towards talking about the state of the industry. [laughter] So, you know, hopefully people will bear with us as we all pine on what's happening. I mean to me the way you're describing a harness one it makes me want to look at all these things that you've mentioned and and to look at that OpenAI project that they open source the deep research that you mentioned but it does also sound like you're talking about orchestration and yes
I did read some of the summaries I'm not going to claim I read the whole S1 from Anthropic I'm not that guy but I did read summaries of the S1 and I was interested I kind took from it. Anthropic is saying it sounds like they're saying it's orchestration. Now, I've seen also many vendors who are that are not that miss the frontier model wave. Uh I won't name any names here, but I think people can probably think of them that that are not seen as being on that cut, you know, bleeding edge, cutting edge of OpenAI, Anthropic. And there's probably some people who are seen as that that are not actually there. And they tend to emphasize
orchestration and then its cousin governance. And I've started to think of orchestration, it's it's the last refu refuge of the scoundrel. It's like if all you got is orchestration, you really don't have much. But I I feel like you're saying, "No, no, there's a lot." I mean, I don't know if you're saying it's defensible or not. But to me, that doesn't seem very defensible. If that's, you know, like that everyone's going to have a good harness, I sort of assume, or where are you going with that? I mean, it's because that's kind of where my head goes. Is this is are we all just reduced to orchestrators and that's that's not so great uh if you're making software.
Yeah. I so maybe three angles to it. So I think the one angle John is like many technologies or terms are being reborn. I won't name the company but recently I was listening to a podcast of a CEO of AI company and he said he said guys like we did something revolutionary. We are sending AI over PSTN public switch telephone networks. We broadcasting and receiving AI over a phone call. Your phone is voicebot. And I I looked at it. I thought like this has been done for the last 10 years. NLU NLP this has been around you know. So but there was this opportunity by a a native company to say guys like AI over PSDN we're doing it. And the second example I want to use is one morning I woke up and there was a new thing graph engineering like graph engineering graph engineer just and then I looked at it and that's how we used to build chatpots with graphs nodes with edges that connects the nodes based on conditions you know and so graph engineering was born but it was it just it was just a it name change right call flows um graph data um it's always been around you nodes connection between nodes so I I think one's got to some somehow just look deeper and so loop engineering was also coined it's just a cron job that kicks off at certain times and runs a process that's kind of intelligent so I think there's a bit of confusion where like old terms are being re reborn That's that's the one thing but I I think sec secondly I think the orchestration I think that's where the business opportunity is because you've got different systems and you've got different models and you you need to orchestrate this whole environment into one environment. I'll give you an example. PayPal recently released a paper and they said that most organizations just have one model. They just perform token burn to token maxing just burn tokens for better results. There's like usage sprawl o all over and they can't manage the APIs. They can't manage the cost and PayPal released a a study and I I might have it wrong. It comes from memory but they had they had hundreds of models they orchestrated. They had a gateway with less than 4 milliseconds latency.
They've got 12,000 developers, but they mentioned how many tokens they run through each day, but it's this this whole environment where they orchestrate the different models, right? And you know, recently, you know, I think Stripe bought OpenRouter and so model routing is the next frontier. the the intelligence layer is moving away and pay the PayPal study showed that models be becomes utilities interchangeable. The the the intelligence layer is moving out of the model into the routing proxy like OpenRouter Hugging Face has got one there are a few others people make their models available at a at a cost.
So, and I think this is like seriously frightening for the frontier model providers because all of a sudden someone is leaprogging them and said, I'll be the gatekeeper. I'll run this proxy and I'll give you low latency, low cost and I'll orchestrate to to the best model. And I mean I saw some research raised recently. I think it was from from OpenRouter where for the first time more than half of their queries went to open weight Chinese models
and not American frontier models. Right. So you and and then obviously there's there's cost savings in that.
So so I think one's got to think of of of harness engineering and orchestration more widely, right? I'll give you another example John and then I'll stop talking but you know working with enterprises I I came up with this this mantra data demands diligence like it's all fun and games but the moment you work with an enterprise a multinational and you start touching the data then like the vibe coding ends data demands due diligence or extreme diligence and so then it then you move into that area where you start orchestrating the data like the the the like the the data lake you know the you know the lakehouse where you surface your data where you clean it where you prepare it so that's also a level of orchestration so so I think gone are the days where you've got the API and a frontier model and off you go it's it's becoming the environment is developing and becoming more complex and I think enterprises organizations they have to orchestrate the environment themselves.
Okay. So, I think that's a really interesting point. I, you know, you know, one, I mean, thank you for that answer because I think you covered on a lot of really interesting things and there's a few things I want to follow up with there. I mean, one of them and and the safety part that I think you touched on upon the end it, you know, that's interesting to me and interesting to us as a company at Bespoken. But on the subject of you know you brought up the proprietary and the open weight models and and also the you know the companies that are doing the sort of arbitrage between them which I just sort of assume we will all be using. I and I will say full disclosure I don't use them yet because our mentality is we're investing in this we like Anthropic or we like Gemini and you know we go all in on that for a particular use case but I also just think in my head once we get this thing let's prove out the use case get adoption for it of course we'll use something like OpenRouter I mean it's just a given in my mind
um that we will
but how do you see that playing out? I mean, what do you think about the open weight versus the proprietary? I mean, where do you think we will land medium longer term on those?
So, so maybe two sides to that, John. The one side is I think people will always use frontier models for research and coding. I I used the latest cursor model when when SpaceX bought Cursor. Kiss was available in their harness like I think two weeks before the announcement. I I saw a a mark how do you say in English a remarkable improvement in terms of performance using the cursor model. So when when I code or create utilities I want to use frontier models and I think that's going to always be the case for for coding for building for research. When it goes to operational side, I think there's it it will get to people already are using open weights models airgapped. Um I mean I won't use a DeepSeek model hosted by DeepSeek but you know perplexity that's one of the first things they did. They like download the model they they fine-tune the model to you know for their preferred political persuasion and off they went. You I think there are like three three sides to it. Like the one is that there's some very good research where the training data is available. So Nvidia is really good at at showing the training data. I I played with a project recently where you could actually click through and see the data the model was trained on. Right? So if I ask who's John Kelvie and it comes back and it says tech mogul, I can actually click on on that and I can find that.
Someone needs to check that model for pollution. [laughter] But so [clears throat] I can actually click through and find a document that model was trained on. Right? So that's really transparency when it comes to to data and a model being open with the data being available. But with open weights the the original training data might not be available but you can fine-tune that model. You can train that model to to react and to have the behavior you want it to have. So I would say people enhance the the the harness for performance and they fine-tune the model for behavior. Um so so so I think you know companies that's serious about you know AI I I see it companies running their own models in private data centers in a control environment they fine-tuning those those models uh it's their own environment. If you just look at the amount of models that's being deprecated or discontinued, right? So imagine you fine-tuned or you invested in that model. So so open weight models definitely orchestrated that's the that's the solution. I just want to mention one more thing John. I find it interesting that fine-tuning is kind of like still very much like a dark art to people. you know, it's it's this enigma and I I want to encourage people start experimenting with fine-tuning because if you think back with NLU and NLP, we used to fine-tune NLU models daily.
Oh, yeah.
I remember when I was at at at the telco provider, we fine-tune the NLU models virtually daily. We had acoustic model for for ASR for speech recognition, converting speech to text. We fine-tune that acoustic model every every few weeks, you know, measure the word error rate, roll back or or implement a new acoustic model. And we're going to get to a stage where we're going to, as a rule, fine-tune fine-tune models. And I've been fine-tuning Qwen models on my MacBook using LoRA. It's a fine-tuning method. And it's amazing the the behavior changes you can instill in a model by fine-tuning it. So, so I think we're like really well you asked earlier about what will the future hold. I think we're at the infancy really now. I I think models will become more common more local inference will become common models will become more powerful and and fine tuning will be become common place.
If I can mention this as well, John, I I I'm absolutely in love with model distillation and I think model distillation is really there's nothing wrong with it, right? You can have distillation attacks like Anthropic ad from they say from China and we can talk about that if you like. But what I like about model distillation is you can take you've got a a model a frontier model and you only using a very small piece of that model, right? And you can take that input and output from your usage and you can train a small you can distill that into a smaller language model, right?
And and emulate that same behavior in a small open-source model. Um there's there's been a recent paper called harness inclusive distillation where you take the behavior of the harness and you also distill that into a smaller model. And and I'm I'm kind of surprised that more people aren't performing model distillation um because that's like a just a a good way of extracting your behavior like your organization's behavior from the model fine-tuning a small language model on that and and running the model. Sorry.
I mean I think that's [laughter] a really
that's a really interesting approach. I was not familiar with that. I will say I have been writing about and then like Braden Ream who was the last podcast that we published uh talking about with people different things around how you sort of down select from a large language model to a smaller more performant you know more coste effective version and I I so I like the idea of resurrecting ing rules-based systems because I spent an early part of my career working in AI on rules-based systems and I and I will say I was very cynical about AI based on that experience because I was not in any way an innovator or researcher on rules-based systems. I was part of a company that I co-founded but uh we were trying to achieve plain English policy management within the financial services industry and rules-based systems were the sort of parliament for doing that and there was but there was people who had come out of MIT that had rules-based systems companies and I'm going to start digressing but you know I mean when you dug into it you're like there's not a lot of intelligence here at least certainly not in the way we conceive of it today. Um, but they were effective in terms of like how to make policy and logic transparent to people that were not programmers.
Uh, when I was talking to Braden, we were talking about this concept a bit and he was talking about
and now I'm forgetting the term that he used, but essentially, you know, encode certain pathways, you know, use embedding models to I think he referred to it as hardening, but essentially, oh, did I already get this request or something pretty darn similar to it? If so, give them the same response.
Um,
and I mean that sort of steers us into the safety concerns which is,
you know, sort of foremost in our minds, but I really like that.
Uh, and I will say it's it's somewhat, I suppose, with the, you know, doing a distilled model. But, uh, I think that has a lot that that sounds beneficial, too. What I like about these other approaches whether you call it rulesbased system there's a great paper from Amazon they wrote on a neuro-symbolic approach to validating systems means rules as far as I'm concerned or doing this sort of hardening which I see is a just in time creation of rules that you're making it more deterministic which and this gets into sort of our somewhat narrow perspective on things which is how do you make these things testable and trustworthy And we still cling to um and maybe it's um maybe we're we're clinging too hard to it, but we cling to the idea that it has to be repeatable. And so I like these approaches and not everything needs to be repeatable. I've been impressed by how people have been willing
to deal with the unpredictability of these systems that business people have embraced the unpredictability of it in a way that I never expected them to. But still you have mission minimally things like authentication. We see within customers the authentication workflow is quite rigid and it's quite deterministic or completely deterministic and and there's other steps in the process where they're trying to find like well we want it to feel conversational and we want to give the users you know we want to take advantage of this new technology but if they're inquiring about their bank balance you know I mean however we achieve it through guard rails or through you some, you know, upfront filtering and orchestration. It's a narrow set of responses they're going to get back.
And so I I really I I find that such an interesting topic and and I'm interested in these different approaches.
Yeah, I I fully I fully agree with you, John. You're actually ahead of your time, but if you think of of the the company that was founded now, TypeSafe, and they introduced this model called called Jev, it's a decision model, right? And I mean I I got early access and I was playing around with it and and all of a sudden it's like like exactly what you're saying. We didn't want to have this like wild answers like we just want a answer machine. This is the input. give us an answer and then subse so this the founder I forgot his name the CEO used to be with OpenAI and then OpenAI also released a decision model and all of a sudden decision models are all the rage and the other day just you know my my spare time I took a Qwen model on my MacBook and I fine tune it to be a decision model right so I took I got training data from I think I think Hugging Face and I said well you know I classified this training data according to a few dispositions. a call center agent will put a disposition to a call accounts retention whatever the case might be and I fine-tuned the Qwen model again using LoRA and I said I'm training you to be decision model these are the examples and when you get a input you've got to give me one word back accounts closure so I train this model to be a decision model right and come back with a probability and an answer so I agree with you and I think John that's back to this whole idea of orchestration and what you want to achieve like we recently I looked at example for a bank where they had multiple agents that were orchestrated.
Uh so the fancy word for that apparently is fleet engineering. But they had one supervisor agent and then they some agents were really deterministic, some were really flow-based, right? So applying for a credit card, applying for a loan, that's flowbased because you've got to you've got due diligence you want to take the user through. And so just to book in that idea, I mean that brings me to this whole thing. There's this great talk by Andrej Karpathy at Y Combinator and he spoke about when he was leading autonomous vehicle autonomy at at Tesla and he spoke about autonomy being a dial. So autonomy lives on a spectrum or agency I see agency as freedom. It lives on a spectrum and you want to dial it up and down depending on the use case, right? And and when I that really rung true for me when he said that because within an enterprise and and that's exactly what you're saying John there are certain policies and procedures there are certain due diligence steps sometimes you want to use a decision model that's intelligent but it gives you a decision and a confidence but sometimes you don't need that right if you've got an agent that monitors sentiment of a caller for escalation that can't be probabilistic it can't be determined istic you know you want to you know you want it to have that freedom to to reason so I think it's all about the use case and you want to dialogue agency up or down maybe like another example I was listen this these days oh quite a while back actually the CEO of a tech company spoke about agents and full autonomy and then we were in a meeting there was a CTO in the meeting and he said it scares me what these guys are saying because no one in my organization has full autonomy not even the CEO. So, how do you want to introduce a AI agent that's got full autonomy? Like, no one's got that, right? There's approvals, there are due diligence, there are steps. So, I I like this idea of having agency or autonomy live on a spectrum and you've got to dial it set it depending on the use case and and the scenario. If that makes sense.
I I love that. I I've got to make sure to go back and capture all these notes from this conversation because I mean you've got you give me so many things to read and look at. So that I mean that paper from Karpathy I'm so interested in that. So what what did he see as the primary levers for dialing up and down autonomy? So if I remember correctly, I think the thing that was interesting was that he said that like world changing technologies normally diffuse top down and normally from the milit military. So he took GPS as an example. You know GPS diffuse top down until I'm wearing a gin you know I'm using GPS now I'm on my wrist right on your phone. He also spoke about the internet how the internet diffused top down uh academic organizations military and and from their companies and then it most people use internet at first time at work and then in home internet and then he spoke about AI that inverted this diffusion right so it's not a top down it's a bottoms up diffusion hence you've got this whole problem of shadow shadow AI which the MIT study coined where people take their own AI to work and it's not, you know, officially sanctioned AI by the company. Most people don't want to use the company's AI. They want to use their own AI because they know what good AI looks like. So, so I think one of the challenges is that you've got this bottom up diffusion um you know by mere mortals and people are figuring out control as as things go along. But I think the biggest analogy he used was autonomous driving um and different scenarios where you would want more human oversight or less human o you less human oversight and I think that's really important within an enterprise and it back to your whole idea of of of having like decision engines and having more of a graph approach. There are instances where you you want to have more human control and more human oversight and other other instances where you don't necessarily need that.
You know, it just can go off and and and process the data. So I I think that's what makes enterprise enterprise AI a challenge. And I we spoke I think before recording we spoke about data that demands due diligence or extreme diligence. So it's also what data you're working with in terms of what level of agency or freedom you want to you want to make use of. Well, and so then that dovtales with another of my favorite topics, which is I think you've done such a I love all these concepts you're introducing around potential ways to manage the AI. It's just this is delightful to think about.
How do you see or how do you think organizations will then on the governance topic, how will they govern using these tools? How are they doing it so far? and how will they proceed do you think?
Right. So that's a good question, John. Short answer is I don't know. But just to just to throw out some ideas, right? So so so recently when I looked at a few studies just to try and figure out what's happening in the market and because I think the problem with with AI, there's so much sales talk. I mean the marketing is just like beyond comprehension. So I think people are looking for real conversations. So when you go to at trade shows and stuff, people are really just want to have a conversation. Uh but there there are a few studies that that shows practically what's happening on the on the ground. And so many companies there are companies that went big bang with with frameworks. And many of them went open source with with frameworks and they went with open source frameworks. It's accessible. it's cheap and they had this initial success right but then it comes into managing it and I think later we can touch on hill climbing and this continuous improvement and AI evvels but then also there was so two problems there was instability so now you've already got this you know a language model what's the right word non nondeterministic right you you've got like the the framework behavior and then the Second thing they struggled with was technology churn. So they continuously had to update and maintain their frameworks as technology changes and um so many companies reverted back according to research from Anthropic and also on arXiv on just a much more simple approach where it's like really a very rudimentary simple approach. So, two companies I I spoke to recently, fairly large companies, just started with getting their data center environment, installing models, starting to fine-tune those models, self-hosting those models and slowly like that ratcheting ratcheting it up. So, not going this big bang approach, but having like a slow incremental rolling out of functionality within the organization.
So I think that's one of the I think that's like a more responsible approach than spending millions on on systems and solutions that companies don't don't fully understand. Um so so I think that's the that's the important thing. I you know I was listening to to Satya Nadella a while back obviously the CEO and president of of Microsoft and he was like accused why did Microsoft abandon the the the frontier model race right why why aren't they part of this front and and I I thought his answer was very astute and he said like that was a conscious decision and if you look at their their latest like announcements it's all about local AI like AI on machines on laptops and it's all about orchestration and for me that's a a very astute approach you know they're focusing on running locally you basically running offline and having this approach of hill climbing where they continually improve the harness and the orchestration environment
so so I think that's where I think that's where the value is like a more incremental measured approach um than than buying a system or a service that's hugely opaque and you don't know how how it ties together. Can I quickly touch on hill climbing, John, or or
No, please do. Yeah.
Yeah. So, so I I've been recently that fit in with this whole thing of orchestration and and and the the harness because if you think of traditional software, someone developed that software, it was released, there were different versions and when someone found a bug, they would like log a Jira ticket and describe I click here, there and here and this happened and then some guy will will will open that ticket, replicate the error, fix it and it goes into the next release. So I almost see it. Jensen Huang calls traditional software pre-recorded. So it's pre-recorded software and it's almost like a film. I don't know if kids know what a film is and different frames in the film and you just take one frame out from the pre-recorded software. You fix it, you put it back and and off you go.
Because soft software was deterministic, right? It was it was pre-recorded. It was set. But now with generative AI software is non-deterministic, right? You get different answers all the time. And so how do you improve a system? So you've got this harness now with network access and different models. How do you improve it continuously? Because it's not pre-recorded. It's different all the time. And something we've done is we've looked at continuous improvement as a triangle, right? So, so we run evals at the one point of the triangle is personas. So, different user personas you can glean from your from your logs. Then you've got different use scenar use scenarios and different criteria in terms of response time um and and model behavior and then you can run simulations with with sessions, customer sessions or or or logs. run simulations and continuously improve your honors in terms of uh latency, cost, answer accuracy, customer satisfaction, you know, those can be your metrics and through that process of of running evals, you can continuously improve your working environment and so to say hill climb, right? So you might think like what if I swap this model out, model A with model B. Well, then you run evals and you check the performance of those metrics. If it's worse, then don't swap it out. If there's improvement, you can swap that model out. And so, it's almost like I always say an AI agent equals harness plus model or models, right? So, the agent equals the harness plus the models. The model is rented, the harness is owned, right? So any gains, any incremental gains I make on my environment, I don't want to bank that in the model. I want to bank that in my own harness, my own operating environment for AI. And I really see that's where Microsoft is heading, focusing like they always did on enterprises.
And uh so it's almost like this operating system for AI, right? And for me that that creates that stickiness. It's hard to swap out. it's hard to migrate and uh that's where the true value sits. You know, you really got to see the model is rented but your your harness your AI operating system is uh is owned. [clears throat] I think that that's that's a really interesting approach. I think for Microsoft that makes sense. I guess for individual organizations I do I you know one of mys that I have come up with over the years that's empirical you know which I do better with my empirical thesis than the ones that I deduce myself [laughter] deductively I end up in weird places but I will say I have seen over the years and Even going back to starting this company, I thought, you know, when you say hill climbing, I get excited about that because I think of those, you know, that's exactly the type of problem that brought me into the space and I thought, well, everyone's going to be doing this and everyone's going to be doing optimization and this is so exciting and I got into this around Alexa and Google Assistant. That's what brought me in. But I'd been doing speech recognition and I was [clears throat]
working on optimizing it [cough] sort of tirelessly and I thought, well, now everyone's going to have to do this. Well, everyone did not. And the basic I mean the wall that I found myself hitting uh and I'm not saying this is everyone, but I saw it as our customers almost universally, you know, the sort of salailable audience that's out there. They do not want to move towards optimization and probability. And any technology that forces people to sort of confront that and deal with it is not going to be embraced.
Now of course I say that and obviously people you know I mean everything we're dealing with now is based on you know probabilities and non-determinism. But I like to go back to the governance question I met with a QA person. This is one of the things that convinced me of my thesis was I met with a QA person, incredible lady that had been leading one of the QA teams at Apple and I was like, well, how you know what's that test bed look like for Siri? And I was expecting to say I'm gonna exaggerate 50 million utterances for all these scenarios because I was my test bed was more than 50,000 for what I thought was a relatively small model. [clears throat] And I really I and I often felt like I was not doing a good enough job. And she's like, "Oh, we've got about 500 golden utterances." Now, I'm not talking about the AI team at Apple. I'm sure it looks very different if you're in R&D, [clears throat]
but from the QA, this thing's going out the door 500. And I was like, gee, I don't I don't think that, you know, the sort of big data probabilities are going to happen. And what I've seen so far from a governance perspective and this is limited. I mean this is based on a handful of customers that we have seen and I'm talking about on the voice AI side. I could talk about chat bots as well and it's not dissimilar but it I would say with the chatbots it's just complete lack of concern. On the voice side, it's we've seen folks saying, "We're going to go into production this year, come hell or high water, and we're going to do 50 to 100 sort of golden utterances, maybe even a little beyond that, but not it's not going to be a loop.
It's not going to do hill climbing. It's just make sure the thing doesn't go off the rails." And one of the things actually I will say I take is vindication on some of my thesis which I I don't know if I should [clears throat] that makes me sound petty but you know we felt this demand from customers
to give us evals these multi-dimensional evals let's glory in the data
every every evaluation every simulation every you know persona and you know I mean you can get into these like nd dimensional analyses and and we presented some of that and the first or you know say the second version of our products really kind of went nuts with that. Now, as we've been out there and we've seen people and it's almost the same customers that were asking for that, they're saying, "Gee, could we maybe pull back on what we sort of maybe we could we still like all that data, don't get us wrong." You know, we're interested in all these different dimensions.
That's that is great. But uh one, you know, when I show this multi-dimensional thing to my manager, the one that's in red that says 25, they tend to kind of bug me about that. So, you know, we like let's uh you know, xnay on the on that part. And also like maybe we could just boil it down to sort of a thumbs up and thumbs down on the main screen, the main dashboard. [laughter] And and then we've got the supporting data. Now, I think it's early to say that that's the end place, but I will say I that fit with my perspective of the market that and I guess I don't think that this is a question that we can answer. Maybe it's a good one for us to leave this on, which is I deter nondeterministic non-deterministic systems are here, but they are operating within people with deterministic mentalities. And what I see so far is that the deterministic mentality is we see the system and it's an up or down. That's what we see as governance so far.
Yeah.
And I'm I'm impressed that a lot of them are getting the thumbs up quite frankly.
John, yeah. While while you're speaking many things came to mind, but let me just say this. I I think there's maybe two things. The one thing is I think there's a massive delta about what is hype and spoken about and what's actually happening on the ground and you just pointed to it in terms of like test beds and an amount of testing it goes in and that's what I like it's two nuggets for me if you see what PayPal is doing on the ground practically or or other companies if you find those those nuggets because there's a big delta I believe between marketing and hype and what's actually happening on the ground. That that's the one thing. The second thing is I saw research recently there's also a big delta between software that's AI generated you know so developers vibe coding so existing a AI software and what's actually going into production. I think that was a16z or it's very small there's a big delta a lot of software created only a small amount of that data goes into production. So I think those deltas are big and I agree with you if it comes to like testing. I mean guard rails became AI safety um you know like you know improvement became hill climbing I don't know how many companies are actually doing that right I one of my favorite pastimes is someone says okay we've got advanced to perform hill climbing I like to say what what exactly do you mean by hill climbing or I think yeah AI evals and I said but what do you understand stand on eval how do you actually do it? And then people floundered like it's hard for them to to explain so so I agree with you. I think that delta especially when it comes to testing and quality it's massive and what people speak about and advertise and what they actually do on the ground and I think many people don't want to hear the truth like you said about you people say that take away that route all that red give me less red in my report people don't want to to see the bad news you know I've seen lots of creative reporting when it comes to quality
well and I don't mean to sound cynical either I am a little cynical so it's it's it is uh unavoidable. But it's I guess I mean my point is a little more I mean it it borders on philosophical where I just think and I guess I see this inselves. I mean I I'm the CEO of my company in theory I should be spending less time on the technical parts and I find you know for myself I wanted let's get this two-way binary even if the system underneath it is using lots of probabilities. And so I I think that that impulse is natural. So yeah, I mean I don't mean to sound like, oh, I'm exposing these people.
They don't really know what they're doing.
So So I'll give you a practical example, John. So when I was with a Telkom, we telco, we we had a ASR. [clears throat] What is ASR? Is it advanced speech recognition?
Automatic speech recognition. Yeah,
automatic speech recognition. Sorry, English is my second sandwich. So it's basically it's basically speech to text converts speech into text and then you've got a word error rate right you calculate the word error rate you've got a a test set you know golden set and what the right translation is which is human translated you know what the machine gives you and you find a delta and we like we had hundreds and thousands of utterances we've transcribed like really work hard and we were about 16 to 18% on word error rate and people has built voice parts they I'm speaking their language but 16 to 18% error rate and we couldn't bring it down and I said to my manager I can I can over fit right there are some it can take out
but you know if if 16 to 18% that's where we were at
so a hyperscaler came to sell their system to us I won't mention the name and the person said like he laughed at us like amateurs like we've got sorry
I was just going to say he's going to say 2% or something like that
3%. [laughter]
So he said we got 3%
and we and this guy was like you know we are just you know Africans South Africans trying to figure things out. This is someone from the US from the hyperscaler and we just muted and we said like there's no way there's just no way you're going to get 3% word error rate like it's just not possible on a on a phone like on a you know but you know the the confident well we didn't go with them because the technical team just said that's not possible but then you get other companies say well it's tough 15% that's so so I think one's got to be realistic about the areas as well like you you've got to have the scars you know to prove it and like if you're down in the data you know what's realistic because these are just systems right uh it's not magic you know and and and I think that's your probably your mantra as well if you're down in the data running these benchmarks you know what's true and what's not true and those are the realities and sometimes you just can't get it better.
You You can't get it better on the ASR, right? That's what it is. You've got to work with that tolerance of of 16% error. Maybe you've got to reprompt. Maybe you've got to rephrase. There are ways of of trying to to mitigate. But I I think people's got to stare reality in the face and see exactly what's what and and what quality looks like. And sometimes just simple graph, that's the answer. One, I have to say I'm so grateful for you for bringing up that example just because I like to pull out those numbers like that and uh you know because I I and I don't use this as a talk track anymore, but I would occasionally sort of challenge customers back in the day being like, "What do you think is a good error rate for a speech recognition system?" And they said, "Well, we're using Google uh or you know, whatever." And my understanding is it's around 5%. And I'd be like, "Well," and you I would usually only say this to people after they contracted with us.
[clears throat] And I'd be like, "We're going to go ahead and test your system." And let me just tell you, if you come in at better than 25%. Be grateful, you know, and then and they'd be like, "What are you talking about, Kelvie?" You know, and come in with stuff. You know, this people don't believe me. And I do think it's better now because the LLMs with their context do fix a lot of these problems. Uh so I think that I can over talk it but I nonetheless I do think these examples are still very much out there.
Um
you know alpha numeric are a great example of it. Uh you still got to work around those which is you know the member ID step the first step in the process you know like immediately falls down. But we would see people like
they'd come in at the 60% error rate and they'd be like, "Well, we can't." And that'd be the end of it and you'd feel I'd feel terrible because I'd be like, initially, the first time we had that with a customer and now I don't know if we should even close. It'd be like I'd be like, "This is great. We're going to go on this incredible journey together and get this thing fixed. Believe me, we've got the tools to do it. It's it's context and training and everything." and they would be like a week later call us back and they'd be like that project is over [laughter] and the big vendor we brought in is fired and so are you. Thanks John for [laughter] saving us from disgracing ourselves and I be like do I get any bonus with that? No. Yeah, [laughter] but but but you know, but I think maybe if I think about my experience building a voice bot, you know, with a big company, I mean, maybe that's an answer to your previous question about like what's the harness like cuz we kind of like developed our system like step by step, right? So we focused on the speech synthesis, the the the text to speech. We we trained the acoustic model. We tested that with vernacular local languages. Okay, really good speech synthesis. This guy's mom thought he was speaking, the voice artist. And then we focus and then we had a team. I was part of the ASR team.
I love it. So we had all these cool recordings, anonymize, transcribe these hands sitting. We had a system where you transcribe at night if you have spare time. And then there was a guy that worked on the NLU that was plugged just before large language models. Chat there was GPT2 time guys that worked on the NLU the intent recognition and a team that worked on the the API latency. Um because obviously with voice you need like you know sub 500 millisecond latency. But I mean maybe that's the answer to your question previously like how you build a system and you bring it together and it's complex and different teams take different pride in take pride in their element and then to your point of just the honesty you know intent recognition is just not good enough. Our word error rate is not good enough and optimizing those things. So, so I think you need like a high level of honesty, you know, to to to know where you're at and and and how you can improve it. Sorry, I'm rambling on.
No, no, I agree that. And I think but I think that the uh the other side of it to sound a more optimistic note is I mean you and I are both talking about our enthusiasm for these problems. I mean I think that's why we're in this space. I mean, we'd love these problems. And I think again, I mean, it I I feel bad that I sometimes I maybe sound like I'm disparaging some of these teams that I've worked with. I mean, like on the contact center side, thinking of those teams, you're talking about teams where once the thing goes live, one dude, this was not uncommon, one dude who's like, "You're in charge of the code, the training, the monitoring, everything." and and it probably was more than one dude, but I mean it really was like I mean such a skeleton crew that they would be investing and now and I don't I won't pretend I'm not Bret Taylor. I'm not calling into the CEO of we have a great we do have Fortune 500 customers, a lot of them, but I'm not calling into the CEO by any means. The decisions come down to our the teams we work with from above. But my sense is those those folks that are higher up, they're supporting the budget. they're giving them more resources. And I think we're going to find more and more enthusiasts built into these teams that are excited to be like, I'm at 82% and next quarter I'm going to get to 85%.
And and that's I mean to me that's like a it's such an exciting thing to see people going on that journey and I think it is definitely early stage from my perspective. But I definitely see like
I don't want to give away too many secrets here, but I mean like the engagement we see and the sort of urgency the customers feel not just because they're worried about things going badly, but just because they want to understand these systems better. It's
it's night and day from five years ago. And I guess that's probably obvious, but
but but just on on that point, John, and I wanted to say something and now it slipped my mind while you were speaking. No, I can't remember. But I should have write I should have wrote it down. I slipped my mind now. Anyway, sorry.
We can we can wave and then edit it [snorts] in. [laughter]
Oh yes, yes, yes. Yeah, now I remember like I mean we just on what you said now about that hard work, right? we uh and we spoke about it you know off camerara just be prior is that I think people underestimate the complimentary hard connective architecture that goes into it and and what you are doing in terms of testing that's part of that hard work and I think people underestimate that it's it's not a magical technology you still want business outcomes you want still still want customer trust and that's been like my mantra when it comes to uh you know that flag some I think Lex Fridman tweeted about it. It said we did not do this. We're doing this not because it's hard but because we thought it would be easy. [laughter] That's that's like many projects I started when I when we worked in enterprises we didn't do it because it was hard it when typically in a call center often one in 10 to 30% of the calls or 40% of the calls especially with telos or banks but then you get to that smaller margins where it's really tough to to make the improvements but I think what what I'm trying to say is like it's hard work and you can't just deploy something especially something That's customer in the customer care environment that's really connected.
You're connected to the customer. You connect it to backend systems. You connected to the cloud. Whatever. It's not something magical. It it really takes a lot of hard work.
Well, I think that's a delightful message to wrap up on. [laughter] Thank you, Cobus, so much uh for joining us. I think this is a fantastic conversation and I won't try to get a commitment from you but you know you and I could turn this into the Joe Rogan show talking for three hours. [laughter] So
yeah I feel like I carry on back we could do the extended version that goes on until [laughter] [sighs and gasps] but anyways thank you so much for joining us. Thanks. Thanks, John Jetson.