Isn't malleable software dangerous? Won't users build things that hurt them?
This is exactly what vendors are afraid of. And I think rightfully so. We've seen a few too many headlines where AI has deleted production, or wiped user data, or modified something that we don't like. And I think a lot of us have used agents, have seen the agent barrel into a task that we really don't want it to do, and only because we've stopped it before it gets there are we able to prevent anything bad happening.
So it's a totally common fear, and one which I would say is present in almost every vendor that we speak to. Malleable software has a lot of different applications, and the problem is that sometimes when people implement malleable software, they do it in a way that can be dangerous. So they do it in a way where a user can accidentally remove all their data without understanding it properly.
And so that's a real fear, and something which vendors need to make sure they're using a provider that understands the domain and understands the category. And they also need to make sure that they themselves are doing a lot of good training for their staff. Otherwise they might end up with these issues.
What ways do vendors implement malleable software, and how they do it safely?
The first way, the most common method that people will use for malleable software using AI, is that they will give an AI access to their API. And they'll say, well, whatever users can do with our API, then the AI builder that we have inside our app can also do the same thing that anyone else can do with the API.
So that seems right. That kind of passes the sniff test. But actually that is a really dangerous way of thinking. The reason for that is that normally when somebody's actually implementing with your API, they're going through a process where previously they were talking to an engineer, or a product person, or a sales engineer. They were going through some sort of human process to validate that the thing that they were building was going to do what they were expecting it was going to do.
With malleable software, you don't really have that anymore. So the person who is thinking about the idea is also the one implementing it. And it's not going to go through many layers of validation or verification that this is a good or a bad idea. And so what you often get is people building things that are probably not. Something, for instance, if somebody wants to import a bunch of data, and they don't think about the fact that by importing this data it will wipe all the data in the entire system clean. Right? Like, that would be something that you can do with an API and would go through a lot of checks and balances as you implement that with the vendor. But if you're going to be building it through an AI builder, you can just think of that concept and then implement it within a few minutes. So we don't think that's a great model.
And the next model you have is your MCP model. So this is actually quite interesting, because we've seen people dial back what's possible with an API on their MCP servers, because they understand that with an MCP server, somebody might be just chatting to their Claude instance, and that Claude instance might call a function on the MCP server, and that's not something that they want to do. So we kind of see that people already, there's this concept emerging, that the type of behaviour you expose on your MCP server is not the type of behaviour that you expose necessarily on your full-fledged API.
Then you get into a third thing, which is when you are entering the software through the vendor's own UI. Oftentimes the vendor will have a different way of interacting with their own UI than what they will expose to you through their external APIs. So they'll have an internal API, which is much more feature rich and much more specific to their UI, and then you'll have an external API, which is doing your cloud-based functions.
And so where we sit in this whole equation is that we don't use the external API. We don't use the MCP server. We don't sidestep what the vendor has already decided is appropriate for its internal API. We don't even use the vendor's internal API directly. We use the client-side SDK, under a model of what we call UI sandboxing.
So we take the UI element, we wrap it inside a client-side SDK. And because we do that, if you have a delete all data function and you're exposing that inside something like Rough, what you can do is on the client side you can say, hey, if this function is called, show a confirm dialogue box. Right? So you're never going to get somebody accidentally deleting their data through some sort of AI-driven model, because you have controlled that part of the UI, and you put all of the checks and balances that you want before it hits the server.
Is there a tension between safety and how much genuine freedom the customer gets?
Yes, there is a tension between the freedom and the safety. And this is a tension that vendors have to be comfortable with first.
It doesn't matter if your customer really, really, really wants to build a delete all users button and you're not comfortable providing that capability to them. As the vendor, it's your decision to come up with what you are comfortable with people doing, just like you would with an external API. And then over time it'll be good to extend those capabilities until you're comfortable with more and more surface area of what to build.
But yes, it is entirely up to the vendor as to what they're comfortable with. If you have a financial data product, that data is going to be very, very important, and you can't provide things like edit data functionality, or you might not be able to provide specific data addresses that you might be able to do for some other application.
So it is entirely up to the vendor what power customers get, what power sales teams get, what power CS teams get. And this might be a function of training. You may extend the capabilities for specific team members. But you can't just go to a vendor and say, hey, we do everything your API can do. Or, hey, we do everything that your MCP server can do. Or even worse, hey, we do everything that your internal API can do. Because that is taking the choice away from the vendor and leaving them with something that could be damaging to their brand and their product.
What does the future look like with malleable software?
I think what we might see happen is that we have customer-specific capabilities. So certain customers are given the ability to do more complex or more dangerous things. Or certain team members, or certain departments. And that may be how we evolve this concept of malleable software.
Because when we think of malleable software, we don't just think of end users changing the software for themselves. A lot of the time, especially in a B2B capacity, the end user isn't really thinking from a product perspective. But a lot of your internal staff are. Right? So your sales teams, your CS teams, they are very close to your customers. And oftentimes they understand what would solve a customer's problem here and now quite intimately.
And so when we think about malleable software, we think about, well, how do we enable not just the end user, but all of those people that are surrounding the end user and supporting them? And so those people can be given much more advanced capabilities, because they can go through training, they can understand the limits of what they're able to build and what they're not able to build, and they can provide a bit of their experience for the end user at the end of the day.
So where I see this going is that we probably have some sort of stratification, where some users are given a pretty cut down read-only view, and some users are given the ability to build much more complex features.
Long, long term, I think software will become more and more malleable as people think about these products as being an interface to their own data, rather than data that lives on some other person's servers or some other person's machine. So I suspect what will happen is it won't just be UIs that are malleable. It'll be things like the data itself is malleable, and we'll see more single-tenancy data models and things like that, which might change the way that we approach malleable software.
What does something built dangerously look like, versus the same thing built safely?
A great version of this is bulk editing tables. Right? So with a bulk editing table, you're kind of running the gamut of almost everything you can do in a piece of software. You've got viewing the data, you've got modifying the data, you've got potentially deleting the data. And so within that model we can see what would happen if different types of users built a bulk editing table.
If the users had access to a delete all data function, then a bulk editing table when built by an AI may also have delete all data as a function that you call through a handy button. And clicking that button, if the vendor hasn't set it up correctly, may delete all the data instantly. Right? So that's a good example of when you have a malleable surface and you haven't protected it very well, then these things can happen.
And so what you want to be doing, in that same example, if you want to be providing it safely, you would have a confirm box when the user clicks delete all data, so that they understand they're about to delete all the data in their system. You may only provide your editing functions for specific types of data, maybe not all data in the entire system.
And then when somebody says I want a bulk editing table, they're going to get something that's pretty safe to use, and that's providing genuine value without the vendor getting cold sweats looking at it.
What does good training actually look like?
Malleable software has to come from the leadership team first and foremost. It is a fundamental shift of how we think about products, and it just doesn't work if it gets pushed up from engineering or pushed across from the sales team. It has to come from a cohesive leadership team that's aligned on the concept. And then that needs to be proliferated down into an organisation.
So you can't just take something like a malleable surface, put it into your product, and then after that do a couple of workshops with some key members of your sales team and then expect that to go really well. Once that surface is there, the cat is out of the bag. People can build whatever they want.
And so you need to first start with an announcement across your organisation, so that people are aligned with what is going on here, and that this is a shift in how we think about products in general. And then from there you need to be going through departmental training. So you need to be telling, hey sales team, the purpose of this is to close deals, but you also need to be closing deals within our strategy. Hey CS team members, the purpose of this is to bring churn down or drive upsells, but these are the things that we want to be seeing and these are the things that we don't want to be seeing in this.
There needs to be a decent amount of training across the org before you can roll something like this out. And that's why we provide that training to organisations that need it.