Real-SWE: Benchmarking AI on Real-World Enterprise Codebases

A new benchmark called Real-SWE has been created to test how well today’s best AI models can handle the tangled codebases of actual, private companies. The project, led by Snagnik Das, Siddhant Paliwal, and Janak Sunil, asks a fundamental question: can coding agents do the real work of a software engineer in a production setting? Real-SWE uses problems taken directly from the licensed production code of real businesses, creating a much more authentic challenge than tests based on synthetic problems. We will examine the benchmark’s design, its initial results, and what it means for both AI development and the software industry.

Overview

What sets Real-SWE apart is its use of tasks copied verbatim from private production environments. This reflects the messy context and inherited product complexity that engineers grapple with every day. The benchmark avoids neatly designed or artificial tasks. Its challenges are found in both the code itself and the specificity of the instructions given. Real-SWE’s evaluation also uses native harnesses, which mimics how enterprise engineers actually work and allows for an assessment of the entire “model-and-harness combination” instead of just an isolated model. During testing, high reasoning was applied to all models.

One task, for instance, requires fixing an invoice billing system so it applies taxes correctly and avoids charging customers who are exempt. This single problem involves multiple tax settlement methods, destination-based pricing that relies on external tax authority providers, and the proper handling of VAT registrations for business in Europe, all while integrating with services like TAX_JAR_URL, PROD_TAX_JAR_URL, and INFLUX_URL. The codebases chosen for Real-SWE went through a rigorous selection process that prioritized real companies with significant user bases, strong engineering teams, and heavy production workloads. Early results show the difficulty. Initial data reveals that 71.4% of rollouts under 10 minutes failed, a figure comparable to the 73.4% failure rate for longer rollouts, pointing to the immense challenge of diagnosing multiple systems and grasping existing business logic. Every task is provided in Harbor format, with verifiers based on or taken directly from the codebase’s own test suites injected at the time of grading.

Key Takeaways

• Real-SWE is a new benchmark testing top AI models against the code of private, real-world enterprises.
• Its tasks come from licensed production codebases, mirroring the genuine problems engineers solve inside existing products.
• The evaluation process uses native harnesses, testing the model and its tools together rather than the model alone.
• A sample task demonstrates the complexity, involving an invoice system that requires the AI to manage diverse tax rules and integrate with external ledger and tax services.
• Initial findings reveal major difficulties for AI, with 71.4% of attempted rollouts under 10 minutes failing.

Industry Implications

The arrival of Real-SWE signals a major change in how the AI software engineering industry measures its own progress. By rooting its tests in genuine, private enterprise code, the benchmark shifts the focus from abstract problem-solving to the concrete realities of production engineering. Developers of AI models must now build agents that can understand an existing architecture, preserve behavior that users depend on, and implement changes within the operational constraints of a live system. This will likely drive the creation of AI capable of handling complex, cross-functional modifications that touch many parts of an application and its underlying business rules.

Real-SWE’s prompts are sometimes slightly underspecified, a characteristic noted as being similar to DeepSWE and Terminal Bench. This suggests that future AI models will need to get much better at discovering implementation details on their own by investigating the codebase, just as human engineers do. The high failure rates are stark. The fact that “71.4 % of rollouts under 10 minutes failed” highlights the massive performance gap between what current AI models can do and what enterprise software development actually requires. This points to a critical need for breakthroughs in AI reasoning, contextual understanding, and the ability to work with complicated legacy systems. The benchmark may serve as a catalyst for new training methods that better prepare AI for the messy, context-heavy world of production code.

Conclusion

The release of Real-SWE provides a crucial new standard for evaluating AI models within the software engineering field. The benchmark offers a true measure of an AI agent’s capacity to handle the deep-seated complexity of production systems by focusing on private, real-world enterprise codebases. This work throws the gap between current AI capability and the practical needs of enterprise development into sharp relief, especially when it comes to understanding existing architecture and business logic. Real-SWE is positioned to speed up the creation of AI agents that can finally integrate into and contribute to the difficult and intricate world of real enterprise software.