Recent advances in LLMs and harnesses have enabled AI coding agents to complete ambitious tasks that were previously thought impossible (or at least unfeasible) in mere days. This is most evident in large-scale refactors and rewrites such as Bun’s Rust rewrite and Cursor’s SQLite Rust implementation.
These projects are an amazing engineering achievement, and kudos to the teams behind them for pushing the envelope. So what made the Bun and SQLite migrations possible? an extremely detailed spec, either in the form of documentation or a test suite. Having such a spec helps guide agents in long-horizon rewrites and refactors, and even serves as a hard quality gate the agent must pass before completing its task. In fact, having a good spec is key when implementing agentic loops and graphs, which are the next big thing (at least according to my X/LinkedIn feed).
So for repos that do have good tests or documentation, the question becomes: how well can today’s AI agents actually handle a full repository translation? And how do we measure that rigorously?
Introducing RepoMod-Bench
This is where RepoMod-Bench comes in. Our latest research work, published at KDD 2026, provides a dataset and a framework for evaluating large-scale repository-level rewrites and migrations.
The core idea: take a source repo’s existing tests, convert them into an implementation-agnostic test suite that interacts only through high-level interfaces such as REST APIs or CLIs, and use that suite to score the translated repo. The test suite is derived from the original repo’s tests and determines the final score for the migration, while remaining hidden from the agent to prevent gaming the task.
The key insight is simple: if a Python web server exposes the same REST endpoints as its Rust translation, and both return the same responses for the same requests, then the translation is functionally correct, regardless of how either one is implemented internally.
How We Built It
We designed a four-stage pipeline to construct each benchmark:
Repository Selection: We filter for repos that expose a standardized interface (REST API or CLI), check test coverage, and verify licensing.
Test Suite Creation: We parse the source repo’s existing tests, convert them to pytest, and critically, filter out any implementation-specific tests. What remains are tests that only interact through the public interface.
Configuration: Each benchmark gets a benchmark.yml, a Dockerfile for reproducible builds, and a translation prompt.
Validation: We run the generated test suite against the original source repo. If it doesn’t hit 100%, the benchmark doesn’t ship. This guarantees the test suite is a faithful representation of the repo’s behavior.
The Dataset
To demonstrate the framework, we curated a benchmark dataset of 21 real-world repositories of varying sizes, rewritten across 8 programming languages. This totaled over 11,600 unique tests covering over 1.6 million lines of code.
Repository sizes range from 14 LOC to 211K LOC, spanning three complexity tiers: Small (<10K LOC), Medium (10K–50K LOC), and Large (>50K LOC).
What We Found
We used this dataset to evaluate four agent–model combinations on the task of full-repo translation:
Agent
Model
Build Success
Pass Rate
Avg Time
Claude Code
Opus 4.5
100%
48.2%
38.5 min
OpenCode
GPT-5.2
100%
43.0%
16.4 min
OpenCode
Opus 4.5
100%
42.0%
21.6 min
Codex CLI
GPT-5.2
95.2%
30.4%
20.3 min
A few things stood out:
The scaling collapse is dramatic. Average pass rates drop from 91.3% on small projects (<10K LOC) to just 15.3% on large projects (>50K LOC). The largest project in our benchmark (211K LOC) maxed out at 19.5% pass rate across all agents. Agents aren’t just getting a little worse at scale; they’re hitting a wall.
The harness matters as much as the model. Claude Code outperformed OpenCode by 6.2 percentage points using the exact same underlying model (Claude Opus 4.5). This tells us that agent architecture is a first-class variable, not just the model powering it.
Building compiles; passing tests is another story. Build success rates exceeded 95% across all agents. They can reliably produce code that compiles. But functional correctness, i.e. does the translated code actually do the same thing, is where they struggle, especially at scale.
Single bugs cause cascading failures. In several large repos, a single misunderstood abstraction or incorrectly translated core module would cascade through dozens of dependent tests. The bottleneck isn’t context management, it’s maintaining architectural coherence across thousands of interdependent files.
Why This Matters
At Modelcode we tackle large, messy, complex migrations and rewrites on a daily basis. For us, evaluation is the difference between a successful project and a never-ending bug hunt. That’s why we’re excited to share this work, and hope that it gets adopted and expanded upon by researchers and practitioners alike.
The beautiful thing about this evaluation framework is that it can be used as a feedback signal for the agent performing the migration/rewrite, so for example, an agent can stay in a loop until at least 95% functional equivalence is achieved. But that raises another question - what if my repo doesn’t have such an extensive test suite? What if my documentation is sparse and out-of-date? How do I still benefit from such mechanisms?
That’s a topic for another post, but if you made it this far - check out Morph and find out how our platform connects the dots to enable an end-to-end code modernization process at any scale.
📄 Paper: RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing
💻 Benchmark & Code: github.com/Modelcode-ai/mcode-benchmark
🔗 Modelcode: modelcode.ai