Microsoft open-sources Scope, an MIT-licensed platform that runs Copilot and Claude Code on repeatable tasks to test whether agents can use your SDK, docs and tools

Why it matters

【Key points】

Microsoft released Scope on Oct 19 (16:00 UTC) on its open-source blog: an MIT-licensed platform for evaluating AI coding agents, built on Kubernetes and runnable locally with Docker Compose. Microsoft Developer Relations had been building it for more than a year.

Scope is not a leaderboard. It is aimed at teams that ship SDKs, CLIs, MCP servers or skills: you give an agent a representative task (optionally with a starting codebase), let it actually run, capture every step, log and workspace snapshot, and have an LLM-based judge score the result against a dependency graph of criteria. It currently drives GitHub Copilot, Claude Code and Copilot in VS Code. Each run's setup (model, agent version, skills, MCP servers, extensions) is saved as an immutable versioned profile so runs can be repeated and compared later.

Microsoft's example comes from the Azure Cosmos DB team, which compared bare models with its Agent Kit across eight end-to-end scenarios in five languages and fixed 20 Agent Kit issues and 27 documentation issues as a result. After one targeted Agent Kit update, agents' use of hierarchical partition keys rose from 47% to 100% and OFFSET/LIMIT usage fell from 47% to 0%.

【Verifiable】

Microsoft open-source blog: https://opensource.microsoft.com/blog/2026/10/19/introducing-scope-open-source-evaluation-for-ai-coding-agents/

Repository: https://github.com/microsoft/scope

Docs: https://microsoft.github.io/scope/

【Our read】

1. Scope turns "can agents use our product well?" into a repeatable experiment. Writing docs and designing SDKs for agents is becoming its own discipline alongside developer experience.

2. The Cosmos DB numbers come from one Microsoft product and a handful of scenarios; Microsoft itself says they are not a general ranking of agents.

3. Scores depend on an LLM judge whose biases users should spot-check. Being open source means the method itself can be inspected and improved.