AI Codes Great But Fails at Long-Term Maintenance

Think AI is ready to take over maintaining large codebases? Think again. A new benchmark called SWE-CI, released by Alibaba and collaborators, delivers a sharp warning about how today's AI models handle long-term code maintenance. Writing a short snippet is one thing, keeping a real codebase healthy over months is another, and the gap is bigger than most people assume.

What is the SWE-CI benchmark?

Unlike one-shot coding tests, SWE-CI does not just check whether a model can fix a single bug. It measures how well AI maintains a codebase over an extended period, simulating a realistic software development workflow built around a continuous integration (CI) loop.

To keep the evaluation grounded in reality, the researchers pulled 100 real repositories from GitHub, filtered with strict criteria: over 500 stars and at least 3 years of active maintenance. Each repository was tracked across an average of 233 days of development, with roughly 71 consecutive commits per task, a substantial workload by any measure.

The simulation uses a dual-agent setup: an "Architect" that analyzes bugs and plans the work, and a "Coder" that writes the actual code, mirroring how a real development team operates.

The shocking part: AI still struggles over the long haul

Today's models are genuinely good at writing short code snippets, but the study found they struggle badly at maintaining a system without breaking existing functionality. Most of the 18 models tested (from 8 major providers including OpenAI, Anthropic, Alibaba, and Zhipu AI) scored below 25% on the zero-regression rate.

In plain terms, that means in 75% of cases, when the AI updated the code, it accidentally broke tests for features that already worked. This is a strong reminder to review AI-generated code carefully, especially on large projects.

Only two models cleared the 50% mark: Anthropic's Claude Opus (version 4.6) and Zhipu AI's GLM-5 (China). Some reports even put Claude Opus 4.6 as high as 76% on certain specific criteria, a promising sign for the future. Still, a 25% baseline across the board remains a serious challenge for the industry.

A new problem: AI-generated legacy code

AI has learned how to create technical debt too. Developers are noting that AI now produces a new kind of "legacy code": code that runs fine in the moment but is difficult to maintain and can quietly break future updates.

This is a concept worth paying attention to for anyone using AI to help build software. Ignoring architecture and long-term scalability can lead to serious consequences down the road, making future maintenance harder and more expensive. It also underscores why reviewing AI-written code matters so much.

What this means for software engineers

This research reinforces the idea that AI cannot fully replace people anytime soon, especially in the role of maintaining large systems. Humans remain essential for code review, testing, and handling the edge cases that AI tends to miss as a system grows. Edge cases can be genuinely complex and often require real-world experience to spot and handle correctly.

Engineers are shifting from writing every line of code themselves to acting as "conductors" coordinating AI agents, focusing more on system architecture than line-by-line detail. That shift opens new opportunities for software engineers to move into architecture and management roles rather than pure hands-on coding.

Key metrics from the study, like EvoScore (accuracy on future modifications) and the regression rate, are also detailed in the paper, giving a fuller technical picture of how this benchmark actually works.

Do you think AI will close this gap soon? Let us know what you think.