swe-bench

Google DeepMind researchers Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim released a preprint paper titled "XYEval: Agents say yes to bad advice," proposing a meta-evaluation framework XYEval to assess the performance of agents when faced with misleading suggestions from users. The paper extends the "XY problem" to the agent scenario, where the user inquires about their attempted solution X rather than the actual problem Y.XYEval retains the original task environment and verifiable answers, only adding seemingly reasonable but misleading user suggestions in the instructions. The misleading suggestions are generated by a generator after seeing the complete task and are appended in the form of confident suggestions rather than hard requirements, allowing the original task to remain solvable. The evaluation covers six benchmarks: τ2-bench, SWE-bench Verified, SWE-bench Pro, Terminal-Bench, Humanity's Last Exam, and MCP-Atlas, along with five models, where the misleading suggestions for τ2-bench are generated based on rules set by human experts.The paper states that the performance significantly declines after the XY mutation, with the relative drop reaching up to 46.7%. On τ2-bench, when faced with users who require a detailed explanation before accepting a better solution, the performance drops even more. Encouraging models to recognize the XY problem through simple system instructions only partially alleviates the issue. The related code is published in the GitHub repository google-deepmind/xyeval.