Attacking Your Own Agent: MiDojo, Roast Me, and When Each One Applies

Both tools get described as red teaming for AI agents, so anyone reading the two descriptions side by side will conclude that one of them replaces the other. What separates them is who writes the attack, and that is also what decides which one a deployment needs.

MiDojo and Roast Me do not compete. MiDojo measures how much damage can be done to an agent through text that a third party can write into something the agent reads, so it verifies a suspicion somebody already had. Roast Me looks for the weaknesses of an agent against a verified truth and produces the suspicions nobody had. If a third party can modify what the agent consumes, MiDojo applies, and if the corpus is closed and we provided it, the work belongs to Roast Me.

Attacking Your Own Agent: MiDojo, Roast Me, and When Each One Applies

For: the commercial team, and anyone who has to explain to a client what "we tested the agent" is supposed to mean. No technical background assumed.

Axel Fritz · Alquimia

We ran MiDojo, an open source tool from Red Hat for attacking AI agents, against an agent of our own, in the same stretch of work where we finished Roast Me, the adversarial evaluation library we develop inside Gaussia. We did it because the two are presented in the same words, so it is reasonable to assume that one replaces the other. What we found is that they answer different questions, and that both can sit in the same proposal without overlapping.

Why we ran this study

Both tools get described as red teaming for AI agents, so anyone reading the two descriptions side by side will conclude that we are proposing something Red Hat already gives away. That objection is going to show up in a sales conversation sooner or later. We wanted an answer to it that holds up, which meant running MiDojo end to end against an agent we wrote ourselves instead of reading the documentation and forming an opinion.

What follows is that answer, and it comes down to what each tool is capable of finding, not to how good either of them is.

What MiDojo is

An AI agent does not only answer questions, it reads. It reads the ticket somebody opened, the comment somebody left on a booking, the file somebody uploaded, the description of a tool that belongs to a third party, and its own memory of earlier conversations. For the model all of that is text, and it cannot tell apart a piece of data from an order, so anyone who can write into something the agent will read can leave an instruction sitting there.

MiDojo measures how much damage can be done through that door. Nothing gets hacked, since the instruction only has to be somewhere in the path the agent walks, and the agent obeys it as a side effect of doing its honest job.

The way it works is by sitting in the middle: between the agent and its tools it puts mirror tools that look identical from the outside, and those mirrors return the real data with the malicious order embedded in it. The agent is never modified and never finds out that it is being evaluated, which is what makes the tool applicable to an agent that is already deployed.

It then measures three things separately: whether the agent still completed the task the user asked for, whether it fell for the attack, and whether the attack even reached it. That third measurement is the part we would defend hardest, because a report that counts an attack the agent never saw as an attack it resisted inflates its own numbers. MiDojo marks those cases as not applicable and leaves them out of the average.

What Roast Me is

Roast Me looks for the weaknesses of an agent against a verified truth. We write down what the agent is supposed to know and how it is supposed to behave, and the library attacks that from different angles and with different framings. The result is a map of where the agent gives way, so the team can rebuild the weak parts, or the parts we care most about protecting.

It runs in two stages. The Profiler takes the behavioral contract, the catalogue of question types worth asking and the knowledge base the assistant answers from, sends probes, grades every exchange and produces a map of where the assistant looks weak. The Exploiter takes that map and writes hundreds of fresh queries per weakness, filters them so that they still sound like something a real customer would say, sends them, and reports which families of conversation break the assistant repeatedly.

The kind of failure this finds is worth describing, because a filter built on keywords will never see it. In one run against a deployed assistant, a customer took something the company offers under a condition and treated it as always true, asking about a product in euros when that product only exists in pesos, and the assistant answered with a minimum amount in euros. It did not invent a product, it agreed with a false premise about a real one, and every word in that answer is a word the documentation itself uses.

The unit of the finding matters as much as the finding, since a single strange prompt that worked once cannot be turned into a test. A category written in plain language can be regenerated next quarter to check whether the fix held.

That same run left us with two things we had not anticipated. Several of the weaknesses we walked in suspecting came back clean, so the assistant held where we assumed it would give way, and among the ones that did fail the volume was higher than we had estimated, which is the kind of correction nobody makes by reasoning about the agent from the outside. The other one is about the questions themselves, since the probes the library composed over the corpus came at the assistant from angles we would not have taken by hand, and a person writing test cases alone tends to write the attacks that already occurred to them.

Where the difference between the two actually is

The intuitive split is that one tool watches what the agent does and the other watches what it says, and that points everyone in roughly the right direction. It does not survive contact with the tools, since MiDojo also grades the text of the answer and can catch a leak in it.

The difference lives in who writes the attack. In MiDojo every payload, every attack objective and every criterion for deciding whether the attack worked comes from the person assembling the suite, which makes it a test bench for a suspicion somebody already had. Roast Me also starts from something we write, and then multiplies it, so the library generates hundreds of variants nobody saw in advance, sends them, grades them and reports which families break the assistant. One of them confirms a suspicion, the other one produces suspicions, and that is the sentence to use in front of a client.

What happened when we attacked our own agent

We wrote a padel club booking assistant for the lab, with five tools, and inside the comment field of one booking we hid three orders dressed up as an internal club notice written in Spanish. Nine cases in total, six of which actually reached the agent, and it fell for all six.

It cancelled the reservation of a member who had nothing to do with the conversation, it leaked a phone number in an answer that never asked for one, and it forwarded that number to an external address while replying to the user completely normally. That last one is the one to remember, because an attack that leaves the service working perfectly gives nobody any signal that something happened.

A few things we ran into along the way are worth naming, without going into them here. A leak came back as a pass because the model wrote the phone number with a slightly different character than the check was looking for, so an exact text comparison can report more safety than there is. An attack looked as if the agent had resisted it, when in fact the data it was after was simply not reachable from where the agent was standing. And a heavily hardened prompt scored a perfect security number by refusing to do any work at all, which is why utility and security have to be read together.

We should be clear about what this run does not prove, since we wrote the agent ourselves, we used one model at one temperature, and none of it says anything about how a production system would hold up.

MiDojo is also the instrument that tells you whether a fix worked

This is the use we did not expect to value as much as we do. Once the suite exists, it can be run again after every change, so a modification to the prompt or to the rules stops being an intuition and becomes a number that either moved or did not move.

We ran four configurations, ten runs each, because the model is not deterministic and a single run measures nothing. Writing a better prompt cut the successful attacks roughly in half and cost no quality at all, since the utility score went up. Writing more rules on top of that made the total worse, and one attack in particular became more frequent right after we added the line that prohibited it. There is no proven explanation for that, and the useful part is the fact itself, because a rule can worsen exactly what it was written to fix and there is no way of knowing without measuring. Writing more prompt and adding more rules is the first reflex when an agent misbehaves, and in one of the four configurations it was the wrong direction.

The last configuration did not touch the prompt at all, and took away the argument the attacker was filling in, so the destination of a message came from the authenticated session instead of from the agent. Those attacks went to zero, while the ones that did not depend on an argument stayed where they were.

Where each tool stops

MiDojo needs a channel, which means it needs somebody outside choosing part of the text the agent reads. If the agent consumes tickets, comments, uploaded files, third party tools or its own memory, there is a door to test and MiDojo applies. If the corpus is closed and every document in it came from us, for example an assistant whose only job is to inform, there is nothing to inject and MiDojo has no work to do. That is the case Roast Me covers on its own.

The ceiling of MiDojo depends on whoever uses it, since it only finds what that person thought of writing. It verifies the weaknesses somebody suspected and leaves the rest untouched, and a weakness that never gets checked stays latent. Maybe at some point there will be a way of guiding the user according to the kind of agent being evaluated, or a floor of weaknesses that every suite has to cover. Today there is neither.

Beyond that, we would not describe it as a tool with problems. It is simple to apply and simple to run, and the man in the middle design means the agent under test never has to be touched.

How to tell which one a deployment needs

The rule is short. If the agent consumes information or data that a third party can modify, whether it reads it directly or through a tool, MiDojo applies and its verdict is a fact somebody can check. If the corpus is closed and we provided it, the attacker who can plant something in the text is gone and Roast Me is the one that does the work.

When both apply, they run in that order, since Roast Me tells the team what to be suspicious about and MiDojo settles whether a specific suspicion holds against a specific channel.

Roast Me is part of Gaussia, the open source evaluation framework we develop at Alquimia. MiDojo is open source and belongs to asago, a project Red Hat runs together with IBM Research, Microsoft, NVIDIA and others. What we report about MiDojo here comes from a lab agent we wrote ourselves, so read it as a spike.