AIChilles: Automatically Uncovering Hidden Weaknesses in AI-Evolved Systems

Computer Science > Artificial Intelligence [Submitted on 14 Jun 2026] Title:AIChilles: Automatically Uncovering Hidden Weaknesses in AI-Evolved Systems View PDF HTML (experimental)Abstract:The computer systems community has recently seen growing interest in AI-driven system evolution, where AI agents iteratively rewrite systems. Frameworks such as AdaEvolve and Engram report 12-60% score improvements over human-designed algorithms. While these results are promising, there are practical concerns if these AI-evolved programs can perform worse on unseen workloads and exhibit scalability regressions. Given the speed and scale of AI-generated code, we need automated mechanisms to uncover such identify hidden weaknesses in AI-evolved systems programs. To this end, we develop AIChilles that takes as input a baseline program $P$ and an AI-evolved program $P'$, AIChilles searches for valid workloads where $P'$ regresses relative to $P$ in correctness, runtime, memory usage, or output quality. To tackle the diversity in system applications, weakness types and potential bugs, AIChilles combines deterministic workload-parameter extraction, agent-based constraint inference, differential oracles, and code-frequency coverage to discover diverse failures. Across five system applications and 30 AI-evolved programs, AIChilles finds 49 distinct hidden weaknesses. We also show that explicitly including AIChilles in the AI-driven development lifecycle can mitigate several of these weaknesses. Current browse context: cs.AI References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer (What is the Explorer?) Connected Papers (What is Connected Papers?) Litmaps (What is Litmaps?) scite Smart Citations (What are Smart Citations?) Code, Data and Media Associated with this Article alphaXiv (What is alphaXiv?) CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub (What is DagsHub?) Gotit.pub (What is GotitPub?) Hugging Face (What is Huggingface?) ScienceCast (What is ScienceCast?) Demos Recommenders and Search Tools Influence Flower (What are Influence Flowers?) CORE Recommender (What is CORE?) arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

이 뉴스, 독자들은 어떻게 느꼈나요?

공식 발표 ↔ 진영별 보도

'research' 카테고리 뉴스

A Definition of Good Explanations and the Challenges Explaining LLM Outputs

Dr-DCI: Scaling Direct Corpus Interaction via Dynamic Workspace Expansion

Relational Structural Causal Models

arXiv의 다른 기사

Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

OSGuard: A Benchmark for Safety in Computer-Use Agents

Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling