OverMind · Test & verify
CI for AI agents: every change tested before production
OverMind is a continuous integration system for AI agents. On every pull request and every night it attacks the agent, load-tests it and checks its business rules — then compares the result with the previous run.
- First result in ten minutes
- Free for one agent
- No production access needed
npx @renaiss/overmind init
Detected: Agent SDK · 3 MCP servers · 11 tools
Mocks generated from schemas · Template: customer support
overmind run --level smoke
Readiness 71 · 0 Critical · 2 High · 8 min 40 s
overmind ci --gate
Check added to PR · red only on Critical or regression K/K → 0/K
01 / The problem
A prompt change breaks something and nobody notices
-
Changes ship without tests
The prompt, the model or the schema of an MCP server changes, and no test covers that change.
-
Failures are found in production
Injections, data leaks and outages under load are discovered when there is already an incident.
-
Risk asks for evidence
Today only an external audit provides it — once a quarter, and late.
| Today | With OverMind |
|---|---|
| “I tried it and it worked” | A check on every pull request |
| An audit once a quarter | A comparison every night |
| A list of pending findings | Findings that ship with the fix |
| A sales meeting to get started | First result in ten minutes |
| Price to be negotiated | One agent free; published pricing |
02 / How it works
What runs on every pull request and every night
Install the tool, define the Policy and the rest runs by itself: attacks, load tests and business evaluations against the agent, with the same seeds so results can be compared.
-
Install
One command or a GitHub Action. Mocks are generated from the schemas of your MCP servers.
-
Define the Policy
Start from a template for your type of agent. The result is a versioned YAML file.
-
Check on every PR
Blocks the merge on a Critical or a clear regression. Flaky results are listed but never block.
-
Nightly run
Warns you when the agent gets worse — even if nobody touched the code.
K seeds per test: every test is repeated K times and the result is a rate (“failed 4 of 20”), not a single pass or fail. The same seed reproduces the same run.
-
Install
CLI and GitHub or GitLab action
One command locally and a check on every pull request.
-
Run
Docker image for your CI
Runs in your infrastructure with docker run. Code and credentials never leave your network.
-
Integrate
Traces in, alerts out
Input: traces via OpenTelemetry. Output: Slack, Datadog, Grafana, Jira, Linear and webhooks.
-
Pricing
Free for one agent
Smoke level with a monthly quota. From the second agent, a published price per agent per month.
03 / Connection levels
What you hand over — and what gets tested with it
The agent’s endpoint is enough to start. Each level adds tests and coverage. No level requires production access: the agent’s tools are replaced with mocks.
| Level | What you provide | What gets tested |
|---|---|---|
| 01 · Endpoint | The agent’s URL (OpenAI-compatible, A2A or HTTP). | Direct injection, configuration leaks, regulated content and Policy compliance in responses. |
| 02 · Tools | The schema of the agent’s tools and MCP servers. | With mocks: tool abuse, exfiltration, MCP-borne attacks, fault injection and load. |
| 03 · System prompt | The agent’s prompt. | Prompt leaks, detected by comparing responses against the real text. |
| 04 · Repository | Read access to the repository. The code never leaves your network. | Code audit: secrets, MCP permissions, step limits, timeouts and traces. |
| 05 · Traces (coming soon) | Production traces via OpenTelemetry. | Real, anonymised conversations used as regression tests. |
The score depends on coverage. An agent connected only through its endpoint can score 100 with 41% coverage — and is published as “100, low confidence”.
04 / What it tests
Four kinds of tests in every run
-
01 · Security
9/10 OWASP LLM Top 10 categories
Attack tests
Prompt injection, jailbreaks and tool abuse.
-
02 · Robustness
1,000 concurrent users in the load test
Load and failures
Concurrency, latency, cost per task and tool failures.
-
03 · Business
5 simulated user profiles per case
Business rules
Describe the Policy and OverMind generates the test cases.
-
04 · Build
15 checks on the repository
Code audit
Secrets, permissions, execution limits and observability.
One report: every finding includes the evidence, why it matters and how to fix it.
05 / Attack tests
The 34 attack tests, by family
Written natively in English and Spanish — not translated — and mapped to the OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS.
| Family | Tests | Examples | OWASP |
|---|---|---|---|
| Direct injection | 6 | Role-play jailbreak, base64, many-shot, mixed languages, instructions split across turns | LLM01 |
| Indirect injection | 4 | Hidden instructions in a PDF, a web page, a search result or a customer record | LLM01 |
| Configuration leak | 5 | System-prompt extraction, embedded credentials, another customer’s data | LLM02 · LLM07 |
| Tool abuse | 4 | Path traversal, SQL injection and SSRF in arguments; sending to an unverified recipient | LLM05 · LLM06 |
| MCP-borne attacks | 3 | Tampered tool description, spoofed tool, manipulated response | LLM03 · LLM06 |
| Excessive agency | 4 | Deleting or cancelling without confirmation; granting permissions to a fake authority | LLM06 |
| Memory and multi-agent | 2 | Malicious memory tampering; orchestrator impersonation | LLM04 |
| Content and consumption | 6 | Legal or medical advice, discriminatory treatment; massive token and call consumption | LLM09 · LLM10 |
-
34
Attack tests
Written in English and Spanish, not translated.
-
9/10
OWASP LLM Top 10 (2025)
LLM08 (vectors and embeddings), which affects RAG, is not in the catalogue yet.
-
12
MITRE ATLAS techniques
Referenced across the catalogue.
No prohibited content is generated: those tests use hand-written requests and only check that the agent refuses them. Your team can add its own tests, approved by an administrator.
06 / Readiness Score
Four gates, the criteria to deploy — and when the check turns red
The Readiness Score is a public standard. OverMind produces the evidence that an agent meets it — which is exactly what Risk and Compliance teams ask for.
-
Security
Attacks: injection, leaks and tool abuse.
-
Business
Policy compliance: tasks, prohibitions and escalation.
-
Robustness
Load, tool failures, loops and isolation.
-
Build
Repository audit; requires read access.
-
Scoring
Every open finding subtracts
Each gate starts at 100 points. Critical −25, High −10, Medium −4, Low −1. With an open Critical, or below 60 points, the gate does not pass.
-
Criteria to deploy
Ready for production means
No open Critical · Readiness of 75 or more · coverage of 80% or more · capacity of at least 1.5× the expected load.
-
Rates, not coin flips
Every test repeats with K seeds. The result is a rate — “4 of 20” — not a single attempt.
-
Red on regression
An open Critical or a clear regression blocks: it always passed and now always fails. Flaky tests don’t block.
-
The judge is audited
Agreement between the judge and human reviewers is published per language, with the data behind it.
| Result | Tests |
|---|---|
| Regressions | 1 |
| Still failing | 3 |
| Now flaky — to review | 2 |
| Fixed | 7 |
Red check: the score went up from 79 to 82, but a test that passed 20 of 20 now fails 20 of 20. The decision is made test by test, not on the average.
07 / Evidence
Every finding reproduces with one command
-
The full conversation
A customer asks to delete their data and the agent calls delete_customer_data without asking for confirmation. The transcript, the calls and the final state are all recorded.
-
Same seed, same failure
The command replays the conversation with the same seed, so whoever fixes the bug can reproduce it on their machine.
-
OWASP and ATLAS references
Every finding states its OWASP category and ATLAS technique. Runs export to SARIF for GitHub code scanning.
08 / Nightly mode
The provider changed the model — your repo didn’t
Every night the run repeats with the same seeds and is compared with the baseline. The agent changes, not the test.
-
Seven red runs
Four nights in a row, in Collections and Customer Service. The failure appears the same early morning in both.
-
Hardly visible in the score
With no code change, the Readiness drops a single point — but the mandatory “subject to approval” notice goes from appearing 3 of 3 times to never.
-
How it was resolved
The team reinforced the notice in the prompt, the PR turned green, and the finding closed once the test passed K of K again.
09 / Fixes
Mechanical fixes come with the patch
When the fix is mechanical — the prompt, an MCP server’s permissions or execution limits — OverMind proposes the patch and runs the test that found the failure against it. It never merges: your team reviews.
-
Finding with evidence
The transcript, the calls and the input that reproduces it.
-
Proposed patch
A change to the prompt, the MCP permissions or the limits, adapted to your framework.
-
The test that verifies it
The same test runs on the fix. If it passes K of K, the check turns green.
# agent/config.yaml
- filesystem: { root: "/" }
+ filesystem: { root: "/data/agent", readonly: true }
+ limits: { max_steps: 25, tool_timeout_ms: 8000 }
# prompts/system.md
+ Before deleting or sending anything, ask the user to confirm.
✓ OverMind check · green · OM-2031 passes 20/20 · 0 regressions Automatic pull requests are planned for v1. Today, every finding already includes the patch.
10 / The judge
A judge you can audit
How often the judge agrees with human reviewers is measured — and published.
-
Code first
Deterministic checks before any LLM
Tool calls, mock state, language and canary tokens are checked with code. The LLM only evaluates rubrics.
-
Human review
People review what matters
A person reviews every Critical and a fixed sample of verdicts. Their labels are used to recalibrate the judge.
-
Published
Agreement, per language
Judge–human agreement is published for English and Spanish. The decision model won’t decide on its own until it passes 90%.
-
Judge
An LLM today, a decision model next
Today an LLM judges, backed by code checks. A decision model issues verdicts in parallel and will take over once its agreement with human reviewers is high enough.
-
Attack generation
Open models on our infrastructure
Open-weight models hosted by Renaiss, later small models tuned with pilots. They never generate prohibited content.
-
Commercial LLM
Only for open-ended scenarios
Used where variety is needed, with the provider’s permission for attack testing. Never the same model that judges.
11 / Architecture
Four layers; everything is recorded
| Layer | Role | What it does |
|---|---|---|
| Connectors | Input | Compatible with OpenAI, Agent SDK and A2A, plus an MCP proxy that records and blocks. |
| Engine | Decision | An adaptive, per-seed planner; the judge and the gates. |
| Sandbox | Isolation | No network egress. Uses mocks generated from each MCP schema, or an authorised test environment. |
| Evidence | Traceability | Transcripts, traces and the configuration of every run. The basis of the report. |
Works with any framework — LangGraph, Agent SDK, CrewAI, n8n or your own code. Traces via OpenTelemetry; results to Slack, Datadog or Grafana.
12 / Security and data
What leaves your network — and what doesn’t
-
Execution
Production is never touched
Tests run in an isolated environment, against mocks generated from your schemas and with synthetic data.
-
Code and credentials
They don’t leave your network
The repository is read locally. Credentials are passed by reference and any plain-text secret is rejected.
-
Personal data
Masked before upload
Emails, phone numbers, cards, IBAN, CURP, RFC, DNI and NIE are masked in the evidence before it is uploaded.
-
Evidence
Immutable and encrypted
Identified by hash and encrypted with AES-256-GCM and your key. Configurable retention; 90 days by default.
-
Access
Roles and scoped keys
Four roles and API keys with limited permissions, stored as hashes.
-
Traceability
Tagged traffic
Every request to the agent carries an X-Overmind-Tag header with the run ID, so you can spot it in your monitoring.
The same binary runs as SaaS on a private instance, in your own cloud (AWS, Azure or Google Cloud) or on your servers as a Docker image. It only needs PostgreSQL.
13 / FAQ
OverMind: questions, answered
What IT and Risk teams usually ask about OverMind.
What is OverMind?
OverMind is a continuous integration system for AI agents built by Renaiss. On every pull request and every night it runs attack tests, load and failure tests, business-rule checks defined in a Policy and an audit of the repository code, and compares each run with the previous one.
How long does it take to get started with OverMind?
Installation is one command — npx @renaiss/overmind init — or a GitHub or GitLab action, and the first result arrives in about ten minutes. OverMind is free for one agent at the Smoke level, with published per-agent pricing from the second agent.
Does OverMind need access to production?
No. OverMind replaces the agent’s tools with mocks generated from their schemas and runs in an isolated sandbox with no network egress. The agent’s endpoint is enough to start; the repository is read locally and code and credentials never leave your network.
Which OWASP LLM Top 10 categories does OverMind cover?
OverMind’s 34 attack tests cover nine of the ten categories of the OWASP Top 10 for LLM Applications (2025) and reference 12 MITRE ATLAS techniques. LLM08 (vector and embedding weaknesses), which affects RAG, is not in the catalogue yet.
How is the Readiness Score calculated?
The Readiness Score has four gates — Security (weight 35), Business (30), Robustness (20) and Build (15). Each gate starts at 100 points and every open finding subtracts: Critical −25, High −10, Medium −4, Low −1. A gate fails with an open Critical or below 60 points. To deploy, an agent needs no open Critical, a Readiness of 75 or more, coverage of 80% or more and capacity of at least 1.5 times the expected load.
When does OverMind block a merge?
Only on an open Critical finding or a clear regression — a test that always passed and now always fails across its K seeds. Flaky tests are listed for review but never block, and the decision is made test by test rather than on the average score.
Which agent frameworks does OverMind work with?
OverMind works with any framework, including LangGraph, Agent SDK, CrewAI, n8n and custom code. It connects to OpenAI-compatible, A2A and HTTP endpoints, ingests traces via OpenTelemetry and sends results to Slack, Datadog, Grafana, Jira, Linear or a webhook.
14 / Next step
Next step: install it and test your first PR
The Readiness Score standard — four gates and their criteria — is public. OverMind generates the evidence that your agent meets it.
-
Install
npx @renaiss/overmind init. The first agent is free.
-
Check your first PR
First result in under ten minutes.
-
Nightly run
Runs every night and warns you when the agent gets worse.
15 / Get started
See OverMind on your own stack
Tell us which processes you want to turn into agents. We'll show you Khala, OverMind and Chronos running on your own stack.
- A walkthrough of Khala, OverMind and Chronos
- A review of one of your candidate cases
- Deployment options for your security requirements
16 / Security & compliance
Built for regulated companies
Information security and data protection are part of the platform, not an add-on. Your security and legal teams get the documentation they need to sign off.
-
ISO/IEC 27001
Information security management
An information security management system covering access control, encryption, incident response and supplier risk.
-
GDPR
EU General Data Protection Regulation
Personal data processed under a Data Processing Agreement, with data minimisation, EU data residency and support for data subject rights.
-
LOPDGDD
Spanish Data Protection Act
Compliant with Ley Orgánica 3/2018 de Protección de Datos Personales y garantía de los derechos digitales, which complements the GDPR in Spain.