VITRINE

Open the live VITRINE documentation and 60-second tour · plain-language summary · source

I built VITRINE to show how I can connect realistic e-commerce product discovery with an evaluation discipline: make an agent and a controlled workflow solve the same shopping needs, then preserve evidence about their quality, reliability, safety, and cost.

Built by José Luis Latorre Millas — Agentic & Software Architect, Microsoft AI MVP, creator of AgentEval, and Microsoft Agent Framework contributor. Profile and talks · GitHub

What the customer sees

Start with the complete checked-in recommendation for Nadia: one synthetic customer request becomes a screened product card with current catalogue price, stock, the customer signal used, and the catalogue fact supporting each suggestion. This particular receipt runs the real ChatClientAgent and real tools against a committed deterministic model boundary, so it is inspectable without pretending it is a paid-model result.

The four paid-evaluation scenarios ask different product questions: Nadia needs several travel, hiking, power, and photography signals connected; Sofia needs replenishment separated from a missing durable capability; Marco needs gift purchases excluded from his own interests; and Luca needs the assistant to ask rather than invent confidence from thin evidence.

The 60-second path is: see the customer output → understand the two implementations → inspect the saved evidence → read what it does and does not prove.

Independent job-application sample for Digitec Galaxus. It is not affiliated with, commissioned by, or endorsed by Digitec Galaxus. The catalogue records, prices, stock, reviews, personas, histories, and relationships are authored/synthetic; recognizable third-party product names and trademarks are used illustratively. Its results do not measure a Digitec Galaxus system.

The live VITRINE documentation hub is the canonical, shareable front door. Its checked-in source links to the value proposal, operator walkthrough, Live-run setup, architecture, evaluation protocol, retrieval deep dive, and verification receipt.

VITRINE is also an applied demonstration of my work on AgentEval: the framework's admitted checks, benchmark arms, native comparisons, statistics, output stores, and real RedTeamRunner are exercised against a working synthetic commerce subject rather than described in isolation.

Related platform work:

• AgentEval — the .NET evaluation toolkit I created; VITRINE consumes its published package rather than a source checkout.
• agent-memory-dotnet — my separate Neo4j-backed, graph-native memory provider for Microsoft Agent Framework with GraphRAG and MCP integration. It is adjacent knowledge-systems evidence, not a VITRINE dependency.

What runs

• Demo01: one ChatClientAgent named Robin using 13 observed read-only functions.
• Demo02: a five-executor MAF discovery workflow with one visible reviewer → discovery loop.
• Offline evals: five mandatory evaluation gates, one matched-quality diagnostic, a persisted five-check recommendation benchmark, and 43 registered healthy → defect → recovery diagnostics. This is the default.
• Paid evals: Eval01–Eval05 measure four shared shopping scenarios across agent/workflow arms. The shipped 1.000 quality bar requires all four authored criteria per trial; three of four is a measured failure. Eval04/05 require every trial to be measured and every scenario's whole-trial 95% Wilson lower bound to reach 0.50 (minimum 4, default 5 repetitions); Eval03 adds a native paired comparison. Eval06 runs real AgentEval jailbreak and canary-backed system-prompt-extraction probes against fresh Robin agents. The App's first-view picker shows only Offline, Eval01 Agent, and Eval02 Workflow; an explicit Advanced switch reveals the paired, stochastic, and safety investigations without removing their CLI or artifact contracts.
• Evidence control room: runtime-derived graphs, correlated timeline evidence, an evaluation board, checksummed JSON/HTML export for accidental-change detection, and replay that executes nothing. The checksum is not a signature or proof against malicious replacement.

The repository keeps the cost boundary visible in source:

• src/AgentEval.VitrineDemo.Evals/Evals/Offline — credential-free suite and diagnostics.
• src/AgentEval.VitrineDemo.Evals/Evals/Live — named, explicitly confirmed paid plans.

Ten-minute path

Prerequisites: .NET SDK 10 and PowerShell 7 (pwsh). The orchestration scripts, the 43-control diagnostic panel, the Avalonia acceptance path, and the published CI lane are Windows-first. The projects themselves can be built and tested on Linux when .NET 10 and pwsh are installed; direct dotnet commands below avoid the Windows launcher script.

# Open the control room; its default subject arm is deterministic and local.
.\start.ps1 -Mode App

# Run the single agent and workflow demos without an external provider.
.\start.ps1 -Mode Demo01
.\start.ps1 -Mode Demo02 -NoRestore

# Run the complete offline evidence chain and its diagnostics.
.\start.ps1 -Mode Evals -NoRestore

# Plant one isolated catalogue defect; detection intentionally exits non-zero.
.\start.ps1 -Mode Ablation -NoRestore

Advanced diagnostic shortcut: .\start.ps1 -Mode Controls -NoRestore reruns only the 43 registered control mutations already included in the complete Evals path above.

Direct commands:

dotnet build AgentEval.VitrineDemo.slnx
dotnet test AgentEval.VitrineDemo.slnx --filter "Category!=LiveModel"
dotnet run --project src/AgentEval.VitrineDemo.Evals -- --all
dotnet run --project src/AgentEval.VitrineDemo.Evals -- --self-test

# Standalone selector 1 defaults to the zero-model baseline.
dotnet run --project src/AgentEval.VitrineDemo -- 1

# Reproduce the checked-in Demo01 scripted-agent receipt with real tools and no remote model.
dotnet run --project src/AgentEval.VitrineDemo -- 1 --scripted --report docs/reports/demo01-scripted.html

Execution and cost boundary

The subject demos expose ZeroModelBaseline, deterministic local ScriptedAgent, and explicitly selected LiveAzure arms. The evaluation runner separately exposes OfflineSuite plus six named paid plans; it has no implicit live profile. Credentials alone never change the selected lane.

The standalone subject CLI also fails closed: selectors 1–6 select the zero-model baseline unless Demo01 is explicitly given --scripted, or a provider-backed subject is explicitly given both --live and --confirm-paid. The --offline, --scripted, and --live subject-arm flags are mutually exclusive. --real-vectors and --rebuild-embeddings can perform live embedding work, so each also requires --confirm-paid. Every paid eval plan requires the same confirmation, and the app requires the equivalent confirmation after showing the planned workload. The public programmatic runner also requires an explicit paidExecutionConfirmed: true argument. Readiness is checked before model execution; failure persists a non-success receipt and never substitutes an offline result.

# One deliberately explicit paid example.
dotnet run --project src/AgentEval.VitrineDemo.Evals -- `
  --eval-plan eval01-agent --scenario nadia-cross-category --confirm-paid

# One deliberately explicit provider-backed subject example.
dotnet run --project src/AgentEval.VitrineDemo -- 1 --live --confirm-paid

AI_INFERENCE_PROVIDER selects which host answers: azure, bitdeer, openai, foundry, or openai-compatible. Leave it unset and VITRINE auto-detects in that order, so a machine that has only ever configured AZURE_OPENAI_* behaves exactly as it did before the selector existed. A host named explicitly but configured incompletely fails closed with the missing variable named — it never falls back to a host you did not choose, because running a paid benchmark on the wrong model is worse than not running it. Every measurement records model@provider, since the same model name on two hosts is not the same measurement.

Host	Required	Optional, with defaults
azure	AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY	AZURE_OPENAI_AUTH_MODE, AZURE_OPENAI_DEPLOYMENT (gpt-5-mini), AZURE_OPENAI_JUDGE_DEPLOYMENT, AZURE_OPENAI_EMBEDDING_DEPLOYMENT
bitdeer	BITDEER_API_KEY	BITDEER_ENDPOINT (https://api-inference.bitdeer.ai/v1), BITDEER_MODEL (zai-org/GLM-5.3-Flash), BITDEER_JUDGE_MODEL, BITDEER_EMBEDDING_MODEL
openai	OPENAI_API_KEY	OPENAI_BASE_URL (https://api.openai.com/v1), OPENAI_MODEL (gpt-4o-mini), OPENAI_JUDGE_MODEL, OPENAI_EMBEDDING_MODEL
foundry	FOUNDRY_ENDPOINT, FOUNDRY_API_KEY, FOUNDRY_MODEL	FOUNDRY_JUDGE_MODEL, FOUNDRY_EMBEDDING_MODEL
openai-compatible	OPENAI_COMPATIBLE_ENDPOINT, OPENAI_COMPATIBLE_MODEL	OPENAI_COMPATIBLE_API_KEY, OPENAI_COMPATIBLE_JUDGE_MODEL, OPENAI_COMPATIBLE_EMBEDDING_MODEL

Bitdeer needs one variable. That is the point of defaulting its endpoint and model:

$env:AI_INFERENCE_PROVIDER = 'bitdeer'
$env:BITDEER_API_KEY       = '<key>'
dotnet run --project src/AgentEval.VitrineDemo -- 1 --live --confirm-paid

Azure OpenAI and Microsoft Foundry share the Azure protocol; every other host is an OpenAI client at its own base URL, which must be https or http only to loopback so a key never travels in cleartext. The Azure path keeps one explicit authentication mode: api-key, local-development DefaultAzureCredential, or deterministic hosted ManagedIdentityCredential (system- or user-assigned). An unset AZURE_OPENAI_AUTH_MODE keeps the API-key path only for backward compatibility; an explicitly selected identity mode never silently falls back to a key. Each host's judge variable can separate the judge from the subject and otherwise explicitly resolves to the subject model. Optional live embedding work uses that host's embedding-model variable; on a host that serves no text-embedding-3-small, set it to a model that host actually has. VITRINE_PROVIDER_NETWORK_TIMEOUT_S (default 180) bounds one attempt. The live-run setup guide explains required variables, the Cognitive Services OpenAI User role, safe session-scoped setup, local readiness versus provider connectivity, the smallest CLI/UI smoke, and troubleshooting. VITRINE never prints, persists, fingerprints, or hashes any host's key, token, managed-identity client id, or endpoint URL; an endpoint that reaches an operator surface is cut down to scheme, host, and port, because a configured URL can carry a credential in its user-info, path, query, or fragment alike. Raw Eval06 prompts, responses, extraction canaries, system instructions, provider messages, and exception text are excluded from its receipt.

Eval process classes are: 0 pass, 1 measured failure, 2 invalid arguments, 3 not measured, and 4 infrastructure failure. Missing evidence remains absent—it is never displayed as zero.

Repository map

• src/AgentEval.VitrineDemo — synthetic catalogue, personas, retrieval, tools, guardrails, Demo01, and Demo02.
• src/AgentEval.VitrineDemo.Evals — AgentEval integration, offline/live plans, CLI, reports, and local evidence persistence.
• src/AgentEval.VitrineDemo.App — optional Avalonia evidence-control-room, export, and replay module; the subject and eval CLIs do not depend on it.
• tests/AgentEval.VitrineDemo.Tests — credential-free integration, artifact, UI, graph, redaction, and failure-state coverage; live contracts are opt-in only.
• docs — the canonical HTML documentation site and sanitized generated evidence.
• MIGRATION.md — repository-native AgentEval 0.35 admission/API migration ledger.
• eng/upstream — source provenance and upstream MIT license.

Public-use boundaries are documented in NOTICE.md, DATA-PROVENANCE.md, SECURITY.md, and THIRD-PARTY-NOTICES.md. The source is available under the MIT License.

Credits and attribution

VITRINE is the work of José Luis Latorre Millas — the idea, the creative direction, and the concept are his. The evidence-first premise that shapes every part of this repository, the personas and scenarios, the guardrail and abstention design, and the evaluation strategy that refuses to present an estimate as a measurement all originate with the author. AI coding assistants supported implementation and review; authorship and responsibility remain with him. See AUTHORS.md for the full credit.

This project is released under the MIT License, which lets you use, modify, and redistribute it freely, including commercially. It makes one condition, and it is required:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

So if you reuse this work in whole or in part, keep LICENSE alongside it, keep the Copyright (c) 2026 José Luis Latorre Millas line intact, and keep the per-file SPDX-License-Identifier: MIT headers. Beyond that, a visible credit naming the author with a link back here is warmly appreciated — a request rather than a licence term.

Verification

The current credential-free receipt is:

• Release build: 0 warnings, 0 errors.
• Category!=LiveModel: 482/482 passed, 0 failed, 0 skipped.
• Offline admitted-check self-test: exit 0.
• Offline check stages: 6/6 completed; all 5/5 mandatory evaluation gates pass, the matched-quality diagnostic is reported separately, and 43/43 registered mutation diagnostics are caught.
• Catalogue integrity self-test: expected exit 1, followed by a restored exit-0 run with all five mandatory gates passing and the matched-quality diagnostic reported.
• No live/provider call was made by the final credential-free acceptance commands.
• A separate, single paid Sofia Eval02 integration observation passed on implementation commit 028ac51; its strict sanitized HTML receipt and JSON receipt are commit-bound. One measured trial proves that path worked once, not reliability or generalization.

See the dated verification page for the exact receipt and incidents, and MIGRATION.md for every admitted check, native floor, ablation, and deliberately retained diagnostic/collector boundary.

For repeatable portfolio assessment, use the stable scoring rubric, the immutable baseline scorecard for implementation commit 67ab737, and the narrow-iteration scorecard for evidence revision f58a747.