Name: Build vs buy: the tradeoffs nobody lays out
Uploaded: 2026-10-20T17:46:35.076Z
Duration: 28 min 49 s
Description: Build vs buy: the tradeoffs nobody lays out

Transcript for "Build vs buy: the tradeoffs nobody lays out": Hi, everybody, and welcome to the second, webinar in Front series on AI agents in customer work. I'm really excited to be here today with you all. My name is Meredith Finn. I'm the COO and CFO at Front. In addition to my many responsibilities at Front, one of the things I do, one of the teams I lead is on... Is our corporate AI and systems team. So that team is responsible for building many of the agents and applications using AI internally that our customer teams run. So I'm gonna open with a quick stat based on Front commission research of leaders who already have an approved vendor list for AI agents. 78% say they wouldn't wait for that vendor to offer an agent that fits, and 44% would go around that list entirely. So leaders are just trying to really move forward with AI in any way that they can. They're experimenting. They're building regardless of internal policies and IT approval. So when the approved option really isn't enough for the AI agents out there, there's definitely a question that teams are facing today on do you build or buy. And I'm so excited to be joined by Nassim Benmalek from, AWS, who is a solutions architect who works through that decision with enterprise accounts. Nassim, welcome. Thank you so much for joining us. Thanks for the invitation. Well, we'll get started with, you know, where teams start and, you know, what surprises them. So first question, Nassim, would be when a customer is weighing whether to build their own agent or buy one, what's the first thing that you ask them? Yeah. So at Amazon, we we love to working backward. So we work backward from our customers. And, usually, my first question is, what is the experience you want to provide to our customers? And, also, what is your unfair advantage? For example, if I can provide an example. Let... Let's say you want to build a chatbot on your web page. If if it's only a chatbot using, your internal... Your documentation, There are plenty of offerings, out there, plug and play. If you, have built a strong experience, in customer support, you understand your users, you have built processes, you have tremendous amount of data, you know how to escalate. You might want to build it yourself. Make... That makes sense and, you know, very much reflects what we're seeing amongst our customers, as well. What surprises teams about the ongoing work of running an agent? So when you make the decision to build yourself, you know, or even if you buy it, what's the ongoing work that you're seeing teams need to do? It really isn't what we've seen internally at at Front is it's not a set it up once and forget it, situation. You really need to maintain both agents that you're building and agents that you're buying. So what what have you seen teams face that they... That was unexpected? I would say that, the deployment is the easy part. It's easy to prototype. It's easy to deploy. And then as you said, you need to maintain it. And compared to traditional softwares, where you have something deterministic, You you know the inputs you'll receive. You could build logic around this this inputs. You know the error codes, for example. If you receive this code, then you do that. Here with agents, you don't have that, and so you need to build evaluation. You need to evaluate. You need to know whether, your customer's experience is good or not, and this isn't something as easy as with traditional software. So you have to to build this evaluation. You have to to to build observability, and there are plenty of challenges with the agents, regarding guardrails, permissions. So, yeah, maintenance is a is a bunch of work. Do... Have you found that customers will need some human support, for those agents once they're For example, let's say for the evaluation, deployed. one strategy I've seen, is using another LLM to evaluate whether, your current LLM, behaves in the the right way. But someone has to evaluate the evaluator. So you need someone who knows the business, who knows your customers to say whether. this is a good answer or decent. And so, yeah, you have, obviously, human work, after, the deployment, just because of that. Yep. Yeah. We found the same internally, and I think our customers are experiencing it as well. And as a CFO, I can I can say that people don't always budget for that when they're thinking about the trade off between build versus buy? So I would say thinking about maintenance costs from a CFO perspective, thinking about any human costs or other AI costs that you're going to have long run to maintain should be included in that build versus buy decision, because as many of us who are running businesses as executives know, people costs still are the majority of our costs today. The technology underneath keeps moving as well. I think we've seen that the foundational model companies know, are releasing pretty significant updates, at a really, you know, strong pace, and quick pace of of innovation. How have you advised teams or helped teams, decide what to build now in a way that will really hold up through the transformation that's happening underneath with the foundational models? First of all, I would say that the the the field and the current pace is quite exciting. You you see a new model, being released one day, and the following day, you have a conversation about it with, with customers. You see AI company going from zero to 100,000,000 in revenue in in about three days. So, this is quite, quite exciting, and it moves a lot. And even the foundations, move a lot. My advice, and this is, basically our idea, at AWS, behind, some of our services. We want to provide the choice, and you should build, on top of, a foundation where you can easily move. Especially, for example, for for the LLM, you you want to be able to to to switch models because you can see that, models are evolving quite quite quite fast. You can see that they perform better at a lower cost, and you want to benefit from that from that innovation. So you you you need to be able be able switch, to, quite easily. So build on, foundations, that provide the the the choice. That would be my my advice. Yeah. No. That makes a ton of sense and also reflects our experience as well. And I'm fully willing to admit, you know, we we made some decisions early on in our internal tooling and how we were building things where we bet on one foundational model provider versus another, and then we had to do some work over a few months to really stand up the infrastructure to be able to switch between between models. So for anybody who's just starting, which don't feel that you are behind in any way if you are, one of the things to really look out for is to make sure that you are building for model flexibility. It's important not only to make sure that your model, that your agents have, I think, the latest capabilities and to see... I'm I'm sure you agree with that, but also from a cost optimization perspective, putting my CFO hat on again, you really want to be able to make sure that post launch of an agent, you're able to monitor the cost of that agent, the performance of that agent, and see if there's ways in which you can optimize, model select and the models that are used underneath for both cost and performance. Nassim, have you seen any tooling, or how are... How have you seen people are are maybe monitoring, that cost and performance, and when do people decide to potentially switch models? Yeah. Again, at AWS, we we think that there's no, no best model for everything. You need to to choose the best model for your use case. So, again, we come back to knowing your business and your customers very well. So at first, you need your product people stating what is good, what is a good answer. For example, let's say you you you provide a a summary to our customers. So this is an example an example. So what is a good summary? It should be concise. It has to include all the things in the original content. We want to provide this fast. What is fast? When once you have all those metrics, you want to test multiple models, and and you you choose the model, that meets all the criterias, while being the cheapest and with the best latency. So you need to evaluate, and, again, it requires understanding the business, the product, and your customers. Yeah. So a lot a lot of measurement is happening across, the agents as well, and that's another point I would just make to anybody who's building, really making sure you've got the instrumentation correct, not only from a techno... Technical perspective to, evaluate performance of the agents, but also to make sure that you're able to measure their costs so that you can properly budget for maintenance going forward. The reality is what we've also seen, in many instances is the cost today may not be the cost tomorrow. Sometimes it's going up, sometimes it's coming down, depending upon which models you're using, and maybe how much human support and human in the loop maintenance is required. We'll sort of, take a a bit of a turn. You know, with agents getting easier to build, what would you suggest leaders do to stay in control? So, obviously, everybody across... Is seeing across their companies that employees who have access, to AI tooling are are sometimes building their own agents. Have you given any advice to your clients in terms of how to control cost, how to control standards, how to control security? And is there anything that AWS provides, to be able to help support that in particular? Yeah. At the earliest, we think that, security is our top priority. It matters, a lot for our customers. But before anything else, before, mentioning guardrails, controls, and all of that, I really wanted to say that, the agency can generate era is quite exciting and an an amazing opportunity. So what I would say first to to, to to leaders, is that you should leverage, this technology, and, you should, again, with coding agents, you can build faster. You should build what your customers are are dreaming, and you have, now the tools to do it, in a very quick way. Now, for, for, for the controls and the guardrails, mentoring the guardrails. For example, you don't want your your agent to to talk about, I don't know, topic... Topics outside of your product scope. So you want to have guardrails able to to... Like, the agent is able to say, I'm not able to under... To to answer to that, because, this is not in my scope. Or to also filter all the things you don't want to receive, from your customers. It could be PII, personal information. And so we we have, these types of filters, at AWS. It's called, bedrock guardrails. So we offer that, out of the box. For, the permissions layer, the permission layer and the security shouldn't rely on the agent. They are not deterministic. So you want your security to rely on your services, the services behind the services that the agent will be using. You want, like, for example, an agent that will access, an internal service, your tool. You want that the permission happens at the service level, not at the agents. You don't. want to rely on a prompt of that. So, yeah, I don't know if I said everything. So guardrails, permission, authentication, you want your agents to reuse, your user authentication, but on... Scopes to only the the services he he he needs it needs you to access. And, again, this is a a common principle we have in security, this privilege. So you should, you should, apply the least privilege rule. Yep. Makes sense. Yeah. It's many of the same principles, I think, that we've thought about with application software, and security across, from an IT perspective today. But, yeah, you have to keep that, I think, in in frame when you're thinking about, building, or buying agents. And sometimes the advantage for buying is some of those decisions have been have been thought through as well. So, if you're a nontechnical user and some of the things that we're saying are scaring you, you know, there's always there's so many options on, opportunities to buy, as well, where you can still get many of the benefits of automation, but, without, you know, the... Necessarily the technical, overhang. And so at at front, you know, we have our autopilot product that, is an out of the box solution, that can be purchased, if you don't feel comfortable building your own agents. But we also deeply believe that there's an opportunity for every customer to build what works best for their business, and that's why we've opened up our platform to bring your own agent as well. So whatever is the best, modality for you, build, buy, we're really looking to optimize the front platform, for, both experiences. I forgot to mention at the beginning that we are gonna have a q and a session. I, we are broadcasting live right now. So if you have any questions, feel free to put them in the chat. A couple people must have already, had had a few. So we've got a couple that have have come in, that I'll start with, but feel free to send any additional questions, that you guys have that have come off of, the conversation thus far. But the first question... So we've already built something internally. Is it too late too late to switch to buying or the reverse? You know, Nassim, have you seen folks who maybe have built something and realize that the cost or complexity to maintain is too high and then they switch to buying something or or vice versa? Not in this, agent ticker... In the agent tick space, not on AI agents. But I would say, it's never too late. Even if you build on on... If you build something, and as we said, the pace of, the Geni and the agent, were all these, very fast. So you always have to adapt and and build on top of, flexible foundations. But, yeah, I've not seen, any example, of that, in the agenting space. Yeah. Yep. I... We've certainly seen this upfront internally, the other way around where, you know, a a year ago, we probably were buying more prebuilt agents, and then with, I think, many of the dramatic advancements over the last twelve months, in the ease to build both agents and applications, with the models that have have been released, we have moved much more to experimenting with buying... Sorry, with building ourselves. I would say, you know, where we maybe have run into, you know, in a building situation where we may have run into, hey. We may wanna look to go buy something is around that long term maintenance question. Do we have the right people internally to be able to continue to maintain this from a time perspective, from a technical perspective. And that's where we, you know, have made decisions as well to continue to move forward with tooling that is developed as well. And I... Just a a personal anecdote on that is really with data agents. We've definitely built some data agents internally, that are working great, but we've also, used third party data analytics agents as well. And just. just. to to... Sorry. But just to add something, I've not seen, any example of that, in product, like, agents in product, but at some point, I've seen many customers building their own, internal productivity tool and chatbots. And, they, a lot of them switched to existing offering, because, again, of this maintenance cost, and, because, there's no answer advantage. So, yeah, I've seen, this, four internal productivity tools. For For example, Yep. at Amazon, we have Amazon Quick, but, you have many offerings that Yes. exist. Agreed. Agreed. We've we've seen that too where even we've tried to build our own internal tooling to look at, how to measure spend, how to measure ROI. We're also seeing some tooling come out, that's really helping us from a third party perspective, and there's dedicated product teams that are thinking through how how to interpret that data. So we've, we've, done the same in terms of switching back and forth between build versus buy there. Let me get to the next question that's come in. How do you estimate the ongoing cost of maintaining an agent so it can go, in a business case? I I would say it depends. The the the thing with, agents and AI, it's not a deterministic science. It's so empirical. I I... Like, you you need to to run POC prototypes, and you need to start from, if we take this, agentic support, case, I don't have any other example in mind. So so, you should start from how much does it cost, for you to do the job. And then, you run prototypes, you build POCs, and you can compare. Does it make sense? Does it provide. value? Does it bring value to my customers? Is the experience better? Does it cost less for for us, to run this? And then decide. So I won't have a perfect answer with, estimations, formulas, you can apply. Yeah. This is empirical. Yep. Agreed. I think... I don't think anybody has really perfected a formula for how to cost this stuff out. We certainly, internally within our finance org, have been doing the best we can on estimates, but, you know, find that sometimes we have forecasted too too low. Sometimes we forecasted too much. You know, the cost to run these things is also changing pretty dramatically over time, and you can you can also influence your cost by changing models to, one of the points that we discussed earlier in terms of having the infrastructure be really flexible to change underlying models. We too start with, hey. As a baseline, what's it costing us to maybe do an activity with humans today? That's the cost that we need to try to determine. Do we think it's going to be more or less expensive to try to build something ourselves? Now the the thing you can do when you're looking to buy an agent is if you know the volume of conversations that you are likely to run through this agent, If you can get an estimate of resolution rates, we obviously are in the in the customer support space, so resolution rate is is a popular metric. You can sort of estimate the cost to to buy. Right? And so we look at both, what's it costing with a people first team today? What would it cost us to buy? And then do we have the ability to potentially build something ourselves, cheaper. So I I would also say, like, you can look at a benchmark too, of the cost to run, prebuilt agents out there. And from a budgeting perspective, when you make that build versus buy decision, really trying to understand, hey. Is the cost to run the prebuilt agent going to be less than trying to maintain this going forward? And I think we've mentioned a couple of things around maintenance that you just wanna make sure are included in any models, beyond just the token cost, which again, I think is really, really hard to forecast of you... As you've pointed out, Nassim. But the people cost, I think, is also hard in a place where people can, I think, both overestimate and underestimate, the time and and, people required in the loop in order to maintain the agent? Okay. Got another question here. For a team with a small technical stat... Staff, what's a realistic first step? So, Nassim, this is probably a great one for you. You know, if you're just trying to really start to experiment with building with building agents, but you've got a small team, where would you start? I, for Soul, I would say that I've seen small team, building, like, amazing amazing tools and products. I... I'd say, at Amazon, we usually say, think big, start small. So you don't have to build the perfect agent that handles all your all your use cases and, that plugs into all your data, with all the tools, the perfect agent. No. Start small, but what... By what you can do. If it brings value, then you can iterate on it. I usually say to my customers, an architecture. is not final. You can always iterate, on top of it. And, we can come back to to what we said. You need to build on, That's weird to say that, but flexible foundations. So be able to to to easily, switch between, solutions, frameworks, LLMs. But, yeah, you can always iterate. Start small. If it brings value, then you can add another tool, another use case, another another data, another knowledge base, and then you keep, iterating. Yeah. No. I couldn't agree more. I mean, we had a relatively small team that we started our internal experimentation with. It was really a team of three from a technical perspective, and just start building. And don't be discouraged that the first thing you build doesn't work. I think you just keep keep persisting, keep trying things. It is an amazing time. The tech is also changing so fast as we've said numerous times, throughout this event, that coming back to it too, even a couple weeks later, performance might be different. You know, your ability to build something might be different. So, yeah, I I couldn't agree more that small teams can do a lot today with, the power of the technology and only continue that only expect to have that continue to increase. And so the amazing thing will be small companies, small teams will be able to do so much more than ever before, and that's really the power of AI. Nassim, I wanna thank you so much for joining us you. today. Really appreciate your insights and sharing more about AWS's offerings that can support companies with building agents. Thanks everybody who joined us live or who's listening to this after the fact. You know, we look forward to continuing our customer conversations with you and assisting you in either building, or buying agents. Thanks so much for joining us. Have a great day, everybody.