DETERMINISTIC VERIFICATION HARNESS

AI coding agents claim they fixed the bug. Harnix verifies it with machine proof.

Most LLM tools trust the model when it says “I'm done” or run unchecked shell commands directly on your host machine. Harnix isolates execution in a disposable container, verifies the initial bug reproduces, runs the fix against your test suite, and only presents a patch when your tests actually pass.

TERMINAL INVOCATIONBASH / POWERSHELL

$harnix "fix off-by-one in rangesum.py" --verify "pytest tests/" --sandbox docker

Inspect Live Execution 60-Second Mock Demo (No API Key)

BASELINE CHECK

Test must fail first

SHELL INJECTION

Argv token allowlist

HOST INTEGRITY

Isolated container copy

Verification Lifecycle Engine

STATE MACHINE

STEP 01 // INVARIANT SETUPBASELINE FAILED

pytest tests/test_rangesum.py

Confirms bug exists before model touches code. Avoids false positive "fixes" on already-passing suites.

STEP 02 // POLICY BOUNDARYARGV CHECKED

Tokenized policy verification

Validates file path strictly inside workspace root. Rejects raw shell operators (&, |, ;) and env exfiltration.

STEP 03 // DISPOSABLE RUNCONTAINER SANDBOX

Execution in Docker (--network none)

Patches applied to ephemeral directory. Real host repository and unstaged developer files are untouched.

STEP 04 // MACHINE PROOFTESTS PASSED (2/2)

Deterministic Verification Passed

Harnix ignores the model's natural language assertion and only accepts exit code 0 from pytest.

OUTPUT: CLEAN GIT DIFFREADY FOR DEVELOPER REVIEW

01 // System Architecture

Engineered with deterministic bounds.

Four architectural subsystems ensure model hallucinations are contained before they reach host files or production repos.

verification/engine.pyCORE INVARIANT

Deterministic Test Oracles

Model self-evaluation is fundamentally unreliable. Harnix uses independent machine checks as the sole gate of completion.

•Establishes pre-mutation invariant: Test suite must execute and fail on the target symptom before editing begins.

•Post-edit verification runs pytest/unittest in an isolated environment without agent intervention.

•Failing verification outputs structured diagnostic traces back into a bounded recovery loop.

•Tasks that cannot be verified by machine checks are explicitly recorded as UNVERIFIED.

COMPONENT 1 OF 4ACTIVE IN HARNIX v0.5.0

policies/commands.pyDEFENSE IN DEPTH

Argv Tokenized Command Policy

Raw shell access (/bin/sh, bash, cmd.exe) is blocked to eliminate prompt injection and unintended script execution.

•Commands are tokenized into explicit argv vectors using POSIX-compliant lexical parsers.

•Shell chaining and redirect operators (|, &, ;, >, <, `) are strictly prohibited.

•Command allowlists restrict execution strictly to safe compilers, linters, and test runners.

•Directory traversal (../) beyond the isolated repository root triggers an immediate policy abort.

COMPONENT 2 OF 4ACTIVE IN HARNIX v0.5.0

execution/docker.pySANDBOX BOUNDARY

Ephemeral Container Isolation

LLMs never touch your actual working tree. Changes occur inside disposable Linux containers.

•Clones clean repository state against baseline git commit hash inside a Docker sandbox.

•Runs with --network none to prevent secret exfiltration or unauthorized dependency fetching.

•Runs as an unprivileged non-root user with read-only root mounts and disposable tmpfs workspaces.

•Only execution/apply.py can stage changes back to the host, requiring explicit developer approval.

COMPONENT 3 OF 4ACTIVE IN HARNIX v0.5.0

knowledge/graph.pyKNOWLEDGE TRACE

Spec-to-Code Graph via MCP

Grounds agent reasoning in upstream business requirements instead of guessed database schemas.

•Ingests BRDs, technical specifications, and Jira user stories into Neo4j graph relationships.

•Maintains provenance: BRD -> Feature -> User Story -> Acceptance Criteria -> Code Symbol.

•Provides Model Context Protocol (MCP) endpoints for agents to query exact requirements before writing code.

•Eliminates hallucinated product rules by validating citations against stored document chunks.

COMPONENT 4 OF 4ACTIVE IN HARNIX v0.5.0

02 // Threat Model & Policy Bounds

Failure modes and containment mitigations.

Comparing how unconstrained AI developer tools behave versus Harnix's hardened execution boundary.

Failure Mode / Threat	Unconstrained AI Agent	Harnix Hardened Harness
Hallucinated Task Completion RISK: CRITICAL	×Assumes completion as soon as LLM stops tool calling or emits 'Fixed!'. Uncaught regressions ship to git.	√Model assertions are ignored. Task completion strictly requires deterministic exit code 0 from test suite.
Shell Injection via Prompt Hijack RISK: HIGH	×Executes unconstrained /bin/sh or cmd.exe. Injected prompts can invoke curl, bash pipes, or background daemons.	√POSIX argv tokenization. Arbitrary shell operators (\|, &, ;, >, `) are rejected prior to execution.
Host Working Tree Corruption RISK: HIGH	×Edits host repository directly. In-progress unstaged changes can be overwritten, mutated, or deleted.	√Disposable clone inside ephemeral Docker container. Real repository files remain untouched until explicit apply-back.
Secret & Credential Exfiltration RISK: CRITICAL	×Process inherits host environment variables (AWS keys, GitHub tokens, database passwords).	√Environment scrubbed. Docker sandbox runs with --network none. Zero outbound sockets permitted.
Runaway Token / Cost Loops RISK: MEDIUM	×Unbounded retries against hallucinations rapidly exhaust token allocations and budget quotas.	√Per-session step caps, token cost accounting, and immediate circuit-breaking on repeated policy violations.
Business Spec Hallucination RISK: MEDIUM	×Agent guesses product logic from sparse prompts or raw grep snippets.	√Neo4j Knowledge Graph traces code symbols upstream to exact Jira user stories and acceptance criteria.

03 // Live Systems Inspector

Inspect how Harnix intercepts & verifies.

Interactive trace replay: Step through baseline checks, policy gates, Docker execution, and independent verification.

Case 01: Off-by-One Repair•TASK: pytest tests/test_rangesum.py -> Fix range_sum() in rangesum.py

Step 1 of 4

BASELINE_VERIFICATION

Establishing Baseline Failure Invariant

STATUS: BASELINE_CONFIRMED (Test failed as expected; task authorized to begin)

Before the agent writes a single character, Harnix runs pytest in a clean workspace. The harness requires the test to fail first, proving the bug actually exists.

SANDBOX STDOUT // LOG STREAMISOLATED_CONTAINER_8F2A

$ pytest tests/test_rangesum.py
============================== test session starts ==============================
collected 2 items

tests/test_rangesum.py .F                                                 [100%]

=================================== FAILURES ====================================
________________________________ test_range_sum _________________________________
    def test_range_sum():
>       assert range_sum(1, 4) == 10
E       AssertionError: assert 6 == 10
E        +  where 6 = range_sum(1, 4)
=========================== 1 failed, 1 passed in 0.04s =========================

All commands verified deterministic across pytest & docker

Developer Interfaces

One harness engine. Three native interfaces.

The FastAPI backend emits Server-Sent Events (SSE) consumed identically across terminal, web dashboard, and editor.

CLI INTERFACE // HARNIX PYTHON RUNNERHEADLESS OR INTERACTIVE

$ harnix "fix TypeError in decimal math" --verify "pytest tests/"

[1/4] Baseline test execution: 1 failure confirmed

[2/4] Agent session initialized [model: gpt-6-luna]

[3/4] Tool call: edit_file(path='calc/math.py') -> Policy ALLOWED

[4/4] Docker sandbox verification: 8/8 passed

[SUCCESS] Verdict: VERIFIED. Patch ready in .harnix/patches/8f9a.patch

Run 'harnix apply 8f9a' to merge patch to working tree.

04 // Empirical Benchmarks

Measured model economics & reliability.

All benchmarks executed against real repository bug test suites in isolated Docker containers with automated token and cost tracking.

Model Name	Verified Rate	Tool Retries	Mean Time	Cost / Fix
gpt-6-lunaSHIPPED DEFAULT Most cost-efficient model. Handled structured file edit tool calls with zero syntactic retries.	4 / 4 (100%)	0	12.8s	$0.0028
gpt-5.6-lunaFAST PROFILE Fastest end-to-end turnaround. Reliable function calling on unit test debugging tasks.	4 / 4 (100%)	0	11.6s	$0.0055
gpt-5.4-nanoBUDGET BASELINE Lower reasoning budget led to 3 tool retry loops before succeeding. Harness error feedback corrected it.	4 / 4 (100%)	3	16.0s	$0.0077
gpt-5.4-miniCOMPLEX REFACTOR High reasoning capacity. Best suited for multi-file refactors spanning complex AST dependencies.	4 / 4 (100%)	0	9.8s	$0.0214

Key Finding: False claim rate is 0 across all runs because Harnix never queries model opinion on completion.

Hardware: 8 vCPU • Docker Engine 27.x • Python 3.12

05 // Quickstart

Get running in under 60 seconds.

Test the entire deterministic verification engine locally using mock mode without an API key or spend.

# 1. Clone repository & initialize backend venv
git clone https://github.com/your-org/harnix.git
cd harnix\backend
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[dev]"

# 2. Initialize SQLite schema & run migrations
alembic upgrade head

# 3. Configure local mock environment (Zero API Keys Required)
$env:HARNIX_PROVIDER="scripted"
$env:HARNIX_SCRIPTED_FILE="examples/mock-script.json"
$env:HARNIX_EXECUTION_MODE="local"
$env:HARNIX_ALLOWED_REPO_ROOTS=(Resolve-Path examples/repos).Path

# 4. Start the Harnix API Daemon
python -m uvicorn harnix.api.app:app_factory --factory --host 127.0.0.1 --port 8000

# 5. In a second terminal: Start the Web Dashboard
cd ..\frontend
npm install
npm run dev     # Open http://127.0.0.1:5173

REQUIRES: PYTHON 3.11+ • NODE.JS 18+DOCKER ENGINE OPTIONAL FOR LOCAL MOCK