The project built a private Azure platform for investigating security findings and testing proposed fixes. Terraform defines the networks, virtual machines (VMs), deployment identities, and monitoring. K3s runs the workflow engine and tool service, while a self-hosted, open-weight model runs on a separate VM.
NetBox, a service for recording network infrastructure, was the target for exercises using synthetic inventory, accounts, and deliberately introduced weaknesses. The agent could read evidence and propose a change; separate checks and human approval controlled whether that change reached NetBox.
The infrastructure has been removed. These pages describe the system as it ran; project status is on the roadmap.
Six VMs across three Azure regions separate the trusted orchestrator, the AI worker, outbound access, administrative access, and the targets. The architecture explains the network paths, trust boundaries, and capacity trade-offs.
The names in the diagram
| Term | What it is |
|---|---|
| Orchestrator | The trusted VM. It runs the workflow and holds the secrets |
| AI worker | The VM that runs the model and the agent. The design assumes it can be compromised |
| Edge VM | The Squid forward proxy, with an Azure NAT gateway for outbound internet access |
| Router | The mesh VPN's subnet router, and its only node in Azure. It carries operator access to the other VMs |
| Just-in-time (JIT) egress window | A time-limited allowlist: an operator command on the edge VM that opens code and package hosts for at most 60 minutes |
| Area | What was built | Detail |
|---|---|---|
| Provisioning and identity | Separate Terraform roots, state, and scoped deployment identities for platform and targets | Identities and state |
| Private networking | No public VM addresses, explicit traffic rules, a controlled outbound proxy, and mesh VPN access through a separate router | Network design |
| Hosts and workloads | Hardened Linux hosts, K3s, Argo Workflows, and an isolated worker for the agent and model | Components, hardening |
| Operations | Private log ingestion, host audit, alert drills, and procedures for deployment, recovery, and teardown | Monitoring, runbooks |
The infrastructure was checked alongside the agent experiments. After the NetBox update, 446 of 446 live checks passed; the operational findings explain the later seeded failures and the gaps in coverage. The worker was rebuilt from its current template, while a rebuild of the whole platform remained untested (rebuild evidence).
The tests did not demonstrate an investigative benefit from adding the model to the account-access exercise. Conventional checks already held the facts needed to decide whether an account had too much or too little access. The model read the same evidence and sometimes misread it (comparison and limits).
The tests also examined whether the platform could constrain an unreliable model. A wrong answer and a successfully blocked answer are different results, so the findings describe both.
| Question | Observation | Limit and evidence |
|---|---|---|
| Could the worker cross its access boundaries? | None of 42 tested probes succeeded | Several paths remained unprobed; containment findings |
| Could evidence instruct the model? | It followed an injected instruction in 1 of 4 finished live jobs; the verification gate stopped that job | Small sample; model reliability |
| Could a proposed fix reach the target safely? | One fix passed the checks, received signed approval, was delivered, and was rolled back | One demonstrated delivery; fix delivery |
| Were all three exercises completed? | Account access had a delivered fix; network access was detected; a vulnerable image was replaced | No network fix was proposed or delivered; exercise results |
The findings and limitations explain what each observation supports. The comparison tables distinguish account verdicts, jobs, and test cases, and identify changes that prevent a direct comparison.
| To understand | Read in this order |
|---|---|
| The infrastructure and its design | Architecture → components → design decisions |
| The experiments and their conclusions | Findings and limitations → comparisons → test evidence |
| Whether the evidence supports a claim | Comparisons → selected evidence → check coverage |
| How to rebuild or inspect it | Runbooks → components → repository layout |
The model can recommend a change that removes legitimate access. Exercise 1 therefore checks its answer and its proposed fix separately before allowing delivery.
- Collect evidence and run conventional checks: One deterministic check reads an account's configured permissions; another reads what that account can actually retrieve. Both compare access with the written access specification.
- Ask the agent to investigate: The agent reads that job's evidence through a tool service. It returns a structured output: an answer in a required format, with a verdict per account and any proposed permission removal.
- Compare verdicts: The verification gate requires both checks to agree and the model to give the same verdict. Disagreement, missing information, or an unreadable verdict stops the workflow.
- Test the proposed change: A policy check restricts what the fix may change; a staging test applies it to a separate copy of NetBox and checks access again. The operator publishes a pull request from checked fields (the agent's job).
- Approve and deliver the exact change: The operator signs a tag approving one commit. Delivery applies it and checks the target again before the merge (delivery runbook).
This chain was demonstrated for account access. The planned network fix would skip the staging test, and no network fix was proposed (exercise 2).
See the delivered fix and its pull request
The fix for exercise 1's seeded weakness, proposed by the model, delivered to NetBox, and rolled back, as its pull request (targets #15):
The exact change it made, one line of the seed:
The model can misread evidence or follow instructions planted in it. A compromised target can also supply false evidence, so agreement between the model and the checks does not establish that the evidence is true (threats and limits).
A job can stop halfway through a change. The operational findings and standing weak points distinguish the recovery paths that were exercised from those that remain untested.
| Concern | Design answer | Decided in |
|---|---|---|
| Evidence reaching a model vendor | Open-weight model on CPU, inside the platform. No external model API | Architecture |
| The agent compromised by what it reads | Own VM, no cluster credentials. Reaches only the tool service, using Model Context Protocol (MCP), and two package hosts | 0005, 0011 |
| A compromised orchestrator | No direct internet. Outbound access goes through the edge VM's proxy | 0020 |
| The mesh VPN vendor | Its only node is a separate router VM. SSH certificates expire within 12 hours | 0025, 0022 |
| A compromised target | Reaches nothing outside its own network. Own repository, state, and identity | 0002, 0024 |
| Exposure to the internet | No VM has a public address. An Azure policy refuses one | 0009 |
Three regions, because the lab had a quota of 4 virtual CPUs per region (0003, 0020).
| Question | Page |
|---|---|
| What was learned, and what remains unproved? | Findings and limitations |
| What was implemented? | Roadmap, architecture, components |
| Why these design choices? | Decision records |
| What does a term mean? | Glossary |
| What supports a result? | Selected evidence, all test runs, lessons |
| What do the automated checks cover? | Validation |
| What commands produced the results? | Worklogs |
| Which threats and limitations remain? | Security, lab against production |
| How can it be rebuilt? | Runbooks |
| What failed during development, and why? | Troubleshooting |
Show the directories
| Path | What it holds | Described in |
|---|---|---|
infra/bootstrap/ |
Terraform the operator applies first: resource groups, state storage, both deployment identities, the public address policy, and the flow logs and Activity Log alerts neither identity can reach | Identities and state |
infra/platform/ |
Platform Terraform: networks, security group rules, the NAT gateway, all four VMs, their first-boot templates, and the Log Analytics workspace with its private endpoint and alert rules | Egress and policy, orchestrator, AI worker |
ai-security-platform-test |
A separate repository. Terraform for the target network, and NetBox and its consumer | ADR 0002, ADR 0024 |
infra/modules/network/ |
The network module both sides use | Architecture |
platform/argo/ |
Argo Workflows: its pins, controller, accounts, network policies, and the workflow template, and the scripts that fetch and install them and make the dispatch key | Workflows runbook |
platform/dispatch/ |
The image of the workflow's dispatch and comparison steps: their code, tests, and Dockerfile | ADR 0030, ADR 0041 |
platform/logs/ |
Host units and scripts on the orchestrator that copy the tool service's audit lines and pull the AI worker's journal into append-only files, and their tests | Audit log |
platform/monitoring/ |
The script that writes the monitoring private endpoint's addresses into the hosts file of each VM with a log agent | Monitoring |
platform/target-images/ |
The script that fetches NetBox's image by digest during a JIT egress window, and checks it before it goes to the NetBox VM | NetBox runbook |
platform/target-update/ |
The update run, which brings the targets' packages through a port the orchestrator forwards, and its install script | Targets runbook |
platform/tool-service/ |
Kubernetes manifests and scripts that deploy the tool service to K3s: namespaces, network policies, the pod, the certificate, job registration, and the synthetic evidence | Tool service, checks |
edge/ |
The JIT egress window command and the list of hosts Squid opens during a window, both installed on the edge VM | Package proxy |
worker/model-server/ |
Pins, download, build, and install scripts for the model server on the AI worker | Runbook |
worker/log-export/ |
The forced command on the AI worker that hands its journal to the orchestrator's pull, and its install script | Audit log |
worker/feed/ |
The forced command of the feed account on the AI worker, which only unpacks staged files, and its install script | ADR 0026 |
worker/agent-runtime/ |
Pins, install, and run scripts for the agent runtime, the job runner and its socket unit, and the fixture tool | Runbook |
tool-service/ |
The tool service: its code, tests, and Dockerfile | Tool service |
collectors/netbox/ |
NetBox's collector: each account's access by its permissions and by what its token returns, a job's evidence, the task, and the held-out cases | Evidence collection, investigation |
policies/ |
The tool service policy: the tools and arguments each job may use | Tool service |
validation/ |
The script that runs every check against the live system, and the test evidence's record builder and page generator | Validation, test evidence |
.github/ |
The checks on every pull request, the image build, and the signature check with its allowed signers | Pipeline |
docs/ |
Everything in Find your way around |
- Not production: one node, one operator, no high availability.
- Not real target inventory: the devices and accounts in NetBox are synthetic.
- Not protection from Azure: the provider and the platform administrator are trusted (trust boundaries).
MIT.