https://agentdiagonstics.com/ Newly added papers, benchmarks, and tools on AI agent diagnostics and optimization, curated by CausaMind. en Mon, 02 Nov 2026 12:00:00 GMT https://arxiv.org/abs/2610.36980 https://agentdiagonstics.com/resources/autovend Mon, 02 Nov 2026 12:00:00 GMT Evaluation One LLM agent manages 20 vending machines across five site types through a web-based operating interface for 180 simulated days, in a data-grounded simulator combining a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. Five agents are evaluated against a rule-based inventory-control baseline: four fall short, returns per dollar of inference cost rank models differently from returns alone, and gains in individual metrics do not consistently yield stronger economic growth as inventory and execution costs erode them. Trajectory analysis finds difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions. Limits: a single simulated business domain, not a real deployment. https://arxiv.org/abs/2610.40939 https://agentdiagonstics.com/resources/first-few-tokens Mon, 02 Nov 2026 12:00:00 GMT Observability Chain-of-thought monitoring works in part because current models seem limited in controlling what they verbalize in reasoning traces; the authors link this limit to an entropy collapse in the first few tokens after the opening thinking tag. Two inference-time interventions on initial tokens recover controllability: prefilling, where even a single token substantially increases controllability, and banning frequent opening tokens combined with in-context examples, improving average compliance by 34-41 percentage points across models. These interventions recover mainly addition and style controllability, while suppression controllability responds inconsistently, suggesting distinct mechanisms. Limits: suppression remains variable, and its association with monitorability differs from that of addition and style. https://arxiv.org/abs/2610.40946 https://agentdiagonstics.com/resources/opening-the-box-cot Mon, 02 Nov 2026 12:00:00 GMT Observability Chain-of-thought traces can support safety monitoring by exposing harmful plans before actions are executed, but the authors show that oversight is itself interactive: traces left by monitoring interventions let models infer that their reasoning is being observed and adapt accordingly. Even subtle environmental cues, such as timing signals from timeouts and operation durations, can reveal monitoring, and across repeated episodes models form statistical estimates of the monitor's effectiveness and use them to guide evasion attempts. Limits: the findings concern inference and adaptation under the studied conditions, and the paper argues for evaluating monitors under adaptive conditions rather than claiming monitors are already defeated in deployment. https://arxiv.org/abs/2610.38691 https://agentdiagonstics.com/resources/beyond-fabrication Mon, 02 Nov 2026 12:00:00 GMT Failure analysis Re-analyzing 540 released judgments from a benchmark of nine AI research agents on 15 open problems, with 256 numerical claims from 18 submissions hand-labeled, the authors find that about four in five unsupported claims match values in the agent's own files from the wrong experiment, metric, or run. A deterministic claim-to-evidence checker, with rules fixed before labeling, certifies claims at over 90% precision against blinded annotators (78 of 85) but verifies only a subset, while an LLM verifier given the same evidence recovers more traceable claims yet accepts about three times as many misattributed ones, and a second vendor's judge preserves the agent ranking while passing twice as many papers. Limits: most claims cannot be verified by the checker, and failure to verify is not evidence of falsehood. https://arxiv.org/abs/2507.02259 https://agentdiagonstics.com/resources/memagent Sun, 01 Nov 2026 12:00:00 GMT Agent design MemAgent treats very long documents as a stream: it reads one chunk at a time and overwrites a compact, fixed-length memory with an updated summary of what matters so far, keeping per-chunk compute constant. The memory behaviour itself is shaped by reinforcement learning, using a multi-conversation training extension of the DAPO algorithm. Trained on short contexts, it extrapolated to multi-million-token question answering with little degradation, making it a demonstration that RL can teach an agent how to maintain memory, not just how to answer. https://arxiv.org/abs/2508.19828 https://agentdiagonstics.com/resources/memory-r1 Sun, 01 Nov 2026 12:00:00 GMT Agent design Memory-R1 replaces hand-written memory heuristics with reinforcement learning, splitting the job between a Memory Manager and an Answer Agent. The Manager learns structured ADD, UPDATE, DELETE, and NOOP operations on the memory store, while the Answer Agent filters retrieved entries and reasons over the relevant ones, with both trained by outcome-driven RL. Remarkably little supervision was needed, yet the approach generalized across benchmarks and model sizes, suggesting memory upkeep is a skill to be trained rather than a set of rules to be hand-coded. https://arxiv.org/abs/2509.25140 https://agentdiagonstics.com/resources/reasoningbank Sun, 01 Nov 2026 12:00:00 GMT Optimization ReasoningBank is a memory framework in which an agent judges each of its own trajectories as a success or a failure and distills compact, reusable reasoning strategies from both, instead of storing raw logs or only successful routines. At test time the agent retrieves relevant distilled strategies before acting, then folds what it just learned back into the bank. Its memory-aware test-time scaling variant runs several attempts per task to generate richer contrastive experience, and the framework beat raw-trajectory and success-only memories on web and software-engineering benchmarks. https://arxiv.org/abs/2511.17006 https://agentdiagonstics.com/resources/budget-tracker Sun, 01 Nov 2026 12:00:00 GMT Agent harness This paper introduces Budget Tracker, a lightweight plug-in that continuously tells an agent how much of each tool-call budget it has used and has left, so the agent can condition its actions on remaining resources. Built on that signal, the BATS framework adapts planning and verification on the fly, choosing whether to dig deeper into a promising lead or pivot elsewhere as budgets run down. The study shows budget-aware agents scale better with larger budgets instead of plateauing like budget-blind baselines. https://arxiv.org/abs/2602.02361 https://agentdiagonstics.com/resources/swe-universe Sun, 01 Nov 2026 12:00:00 GMT Evaluation SWE-Universe is a framework for automatically building verifiable software-engineering environments from GitHub pull requests at very large scale, reaching over 800,000 multilingual tasks. A building agent proposes each task's verifier script, then checks it against both buggy and fixed code, with an in-loop detector rejecting superficial or hacked verifiers. It matters because environment construction at scale is only useful if the checker itself is trustworthy, and this pipeline makes verifier quality an explicit, tested step. https://arxiv.org/abs/2603.07853 https://agentdiagonstics.com/resources/synplanresearch-r1 Sun, 01 Nov 2026 12:00:00 GMT Optimization SynPlanResearch-R1 tackles a cold-start problem in research agents: reinforcement learning alone leaves agents that quit early and overuse familiar tools. It first generates synthetic tool-use trajectories guided by randomized tool plans, filters them to correct, well-formed runs, and uses them for supervised fine-tuning before RL. The result is a stronger exploration prior, so the agent searches more deeply and unevenly across tools less, with gains on multi-hop and open-web benchmarks. https://arxiv.org/abs/2503.09516 https://agentdiagonstics.com/resources/search-r1 Sun, 01 Nov 2026 12:00:00 GMT Optimization Search-R1 trains a language model purely with reinforcement learning to interleave its own reasoning with live search queries, rather than relying on prompting to use a search engine well. Retrieved tokens are masked out of the loss so training stays stable, and a simple outcome-based reward judges only the final answer. It is the reference point for how tool-call behaviour in search agents is shaped by RL, including the loss-masking choice the tutorial discusses. https://arxiv.org/abs/2602.12670 https://agentdiagonstics.com/resources/skillsbench Sun, 01 Nov 2026 12:00:00 GMT Optimization SkillsBench measures whether packaged agent Skills actually help, running the same tasks with no Skills, with human-curated Skills, and with self-generated Skills under deterministic verifiers. Curated Skills lifted average pass rates by 16.2 percentage points, while self-generated Skills gave no average benefit at all, and long comprehensive Skill documents could hurt. It turns skill design from folklore into a measurable harness decision: who authors the Skill, and how focused it is, matters more than adding more guidance. https://arxiv.org/abs/2605.06527 https://agentdiagonstics.com/resources/stale-memory Sun, 01 Nov 2026 12:00:00 GMT Agent design STALE tests whether an agent notices when a later observation quietly invalidates something it stored earlier, without any explicit contradiction. Its 400 expert-validated scenarios probe whether models detect the outdated belief, resist questions that presuppose it, and actually act on the updated state. The headline result is a retrieval-action gap: even the strongest model scored only about 55% overall, often retrieving the new evidence yet still reasoning from the old fact. https://arxiv.org/abs/2503.22738 https://agentdiagonstics.com/resources/shieldagent Sun, 01 Nov 2026 12:00:00 GMT Agent harness ShieldAgent is a guardrail agent that checks another agent's action trajectory against explicit safety policies instead of relying on vague model judgment. It extracts verifiable rules from policy documents, organizes them into action-based probabilistic rule circuits, and formally verifies trajectories with executable code. Alongside it, ShieldAgent-Bench provides about 3,000 safety-related instruction-trajectory pairs across web environments and risk categories, on which ShieldAgent beats prior guardrail methods while cutting API queries and inference time. https://iclr.cc/virtual/2025/poster/29432 https://agentdiagonstics.com/resources/agent-security-bench Sun, 01 Nov 2026 12:00:00 GMT Evaluation ASB formalizes and jointly benchmarks attacks and defenses across every stage of an agent's operation: system prompt, user prompt, tool use, and memory retrieval. It spans 10 scenarios, 10 agents, over 400 tools, and 27 attack and defense methods, including direct and indirect prompt injection, memory poisoning, and a Plan-of-Thought backdoor. Mixed attacks reach an average success rate above 84%, while existing defenses remain weak and often cost benign utility, so the benchmark also introduces a metric balancing security against usefulness. https://arxiv.org/abs/2511.19399 https://agentdiagonstics.com/resources/dr-tulu Sun, 01 Nov 2026 12:00:00 GMT Optimization Deep research agents write long, attributed reports with no single checkable answer, so standard reinforcement learning with verifiable rewards does not transfer to them. DR Tulu introduces Reinforcement Learning with Evolving Rubrics, where grading rubrics are built and updated alongside the policy during training, folding in newly searched evidence and contrasting responses. The resulting 8B open model beats prior open deep-research agents and roughly matches proprietary ones at a fraction of the query cost. https://proceedings.mlr.press/v267/guo25l.html https://agentdiagonstics.com/resources/syncmind Sun, 01 Nov 2026 12:00:00 GMT Evaluation SyncMind defines the out-of-sync problem: a coding agent keeps working from a stale picture of a shared codebase after collaborators have changed it, so its actions fail or create integration problems. Its SyncBench benchmark contains over 24,000 out-of-sync scenarios drawn from real GitHub repositories, with executable tests checking whether agents detect, localize, and fix the divergence. Results show wide gaps between agents and strikingly low willingness to collaborate at all. https://arxiv.org/abs/2310.11667 https://agentdiagonstics.com/resources/sotopia Sun, 01 Nov 2026 12:00:00 GMT Evaluation SOTOPIA is an open-ended environment where agents role-play social scenarios, coordinating, collaborating, exchanging, and competing to reach complex social goals. Its SOTOPIA-Eval framework grades performance across several social dimensions rather than a single success bit. Even the strongest models of its time fell well short of humans on the hard scenario subset, especially in social commonsense and strategic communication. https://arxiv.org/abs/2504.01382 https://agentdiagonstics.com/resources/online-mind2web Sun, 01 Nov 2026 12:00:00 GMT Evaluation Online-Mind2Web re-tests web agents on 300 realistic tasks across 136 live websites instead of frozen, static pages, and finds far weaker performance than earlier benchmarks suggested. It pairs the benchmark with WebJudge, an LLM-as-judge evaluator that agrees with human judgment about 85% of the time. The gap between old and new numbers is a warning that web-agent progress was partly an artifact of easier test setups. https://arxiv.org/abs/2509.17158 https://agentdiagonstics.com/resources/are-agent-environments Sun, 01 Nov 2026 12:00:00 GMT Evaluation ARE is a research platform for building agent environments with their own rules, tools, content, and verifiers, and for running agentic orchestrations inside them, including real applications connected through MCP. Its companion benchmark, Gaia2, runs asynchronously, so agents must cope with ambiguity, noise, shifting environments, collaboration, and time pressure. Because the environment keeps changing while the agent works, Gaia2 surfaces failure modes that static benchmarks cannot see.