The paper measures how oracle-provided and agent-extracted signals contribute to final task success. It reports a consistent contribution ordering: Reproduction Test > Execution Context approximately equal to API Usage > Edit Location > Regression Test. The authors also quantify the gap between agent-extracted signals and their oracle counterparts. The study frames these signals as measurable targets for SWE-agent improvement; its findings are tied to the paper’s experimental setup and do not establish a universal ranking across all coding tasks or systems. Source: https://arxiv.org/html/2604.07789v3
Software-engineering agents such as Agentless, SWE-agent, OpenHands, and Claude Code repeatedly generate five kinds of intermediate information.
In one minute
Microsoft and Georgia Tech researchers published ORACLE-SWE, a study comparing five intermediate information signals used in repository-level software-engineering agent workflows: reproduction tests, regression tests, edit locations, execution context, and API usage.
What changed: ORACLE-SWE systematically compares five intermediate SWE-agent signals and measures their effects on task success, releasing an oracle-signal dataset and evaluation methodology.