Insu Yun
Patch Team Lead
Assistant Professor at KAIST
Every Patch Agent has its Own Story (1) - Martian: Exploring the Unknown with Sophisticated Tools
As we mentioned in our previous blog post, we enhanced the patching capabilities of Atlantis by ensembling multiple patch agents. In this series of blog posts, we will introduce each of our patch agents in detail and explain the rationale behind their designs. Diversity for Good To maximize the effectiveness of ensembling, it is crucial to have diverse agents. If all agents are similar, the ensemble will not perform significantly better than any individual agent. Therefore, we intentionally designed our patch agents to be diverse in their approaches, methodologies, and also models used. We newly developed six patch agents, each with its own unique architecture and motivation, as summarized in the table below.
Vincent, One Puzzle for Our Ensemble Toward High-quality Patches
As mentioned in the previous post, our strategy for patching is to prepare multiple agents to ensure both the robustness and correctness of the system. To this end, we developed various patch agents, each specialized for different LLM models and tools. In this post, we would like to introduce Vincent agent, one of the patch agents running under our ensemble-based patching system. Right Root cause, Wrong Patches What surprised us during the competition was that LLMs alone are already quite doing well at generating proper patches. Given a sanitizer report, LLMs could freely explore the codebase by itself and reason correctly about the given bug—especially when the problematic code appeared near the call stacks in the report.
Ensembles of Agents for Robust and Effective Automated Patching
Why Ensemble for Patching? In the AIxCC competition, finding vulnerabilities is only half the battle. Once a vulnerability is discovered, it must be patched to prevent exploitation. This is where the Atlantis-Patching system comes into play. As the AIxCC’s ultimate mission is to make software secure, it awards more points for patching vulnerabilities than for finding them. In particular, the competition rewards 6 points for patching a vulnerability, compared to just 2 points for discovering it. As a result, to win the competition, it is crucial to have a robust and efficient patching system that can quickly generate effective patches for discovered vulnerabilities.