Successive requirements
Work unfolds across a sequence of development tasks, each introducing the next set of requirements.
A NEW CODING BENCHMARK
One codebase.
The next challenge.
Software keeps evolving.
Can coding agents keep up?
Multi-step-SWE evaluates coding agents across successive requirements in the same codebase.
REPO / EVOLUTION STATE CARRIES FORWARD
one codebase main
01Modernize the buildBUILD
02Reshape the projectSTRUCTURE
03Lock dependenciesDEPENDENCIES
04Expand compatibilityCOMPATIBILITY
05Make builds reproducibleREPRODUCIBILITY
06Close the standards gapSTANDARDS
NEW REQUIREMENT. EXISTING CODE.
Concept illustrated with a Python packaging task.
01 / THE BENCHMARKSOFTWARE IS A CONTINUING PROCESS
The next task starts
where the last one left off.
Real development moves forward through new requirements, changing dependencies, and evolving standards. Multi-step-SWE brings that continuity into coding evaluation.
A codebase that carries forward
Changes become the starting point for subsequent work. Earlier implementation choices shape what comes next.
A closer look at each step
Step-level evaluation makes it possible to inspect progress throughout the sequence. The full evaluation protocol is coming with the paper.
02 / INSIDE A TASKAN EXAMPLE FROM THE BENCHMARK
Six steps.
One evolving Python project.
Follow a packaging modernization task from its legacy build system to reproducible, standards-aligned releases.
STEP 01 / 06TASK OVERVIEW
Modernize the build system.
Move the existing Python project from a legacy setup.py build to a PEP 517 build system.
setup.pyPEP 517 build
WHAT CARRIES FORWARD
The existing project and its behavior are the starting point for the migration.
New requirement. Same repository.
Summarized task requirements. Evaluation results and complete task specifications will accompany the release.
03 / THE RELEASE IN PREPARATION
The full picture.
Coming soon.
We're preparing the paper, first evaluation results, and release materials. They'll be available right here.
Meet Klavis AIResearch paper
Task construction, evaluation protocol, and findings.
Leaderboard
Model comparisons and detailed evaluation results.
Code & tasks
Release artifacts and instructions for getting started.