Bitsec.ai: how the mechanism works

Bitsec pays for finding real software vulnerabilities, and its own code shows exactly where trust actually sits: not in the container that runs the agent, and not in what the agent claims to have found, but in a curated answer key and a second AI model asked whether the two match.

Bitsec's own overview is direct about the product: "Bitsec is a Bittensor subnet for building AI security agents that find high and critical severity vulnerabilities in real software projects. Miners submit agents that analyze codebases and produce vulnerability reports. Validators run those agents in sandboxed environments, score their findings against benchmark ground truth, and submit results back to the platform." [source] The repository this course's source is pinned against is explicitly the development sandbox, not the full production system — its own README says "if the docs and this repository disagree, treat the code in this repository as the source of truth for local execution." [source] [source]

What the sandbox actually isolates

A miner's entire submission is one file, miner/agent.py, which "must expose agent_main() and return a JSON-serializable report with a top-level vulnerabilities list." [source] The validator's executor.py launches it inside a fresh, named Docker container per job run, and the exact launch call is worth reading in full because it names both what is contained and what is not:

container = docker.run(
    project_image_tag,
    name=sandbox_container,
    networks=[settings.proxy_network],
    volumes=[(self.agent_filepath, "/app/agent.py")],
    envs={...},
    # read_only=True,
    memory="512m",
    cpu_quota=25000,
    pids_limit=64,
    detach=True,
)

[source] What is actually enforced: 512MB of memory, a quarter of one CPU, and a cap of 64 processes. What is not enforced, despite being one line away: the read-only filesystem flag sits commented out in the pinned code, so the container's own filesystem is writable at runtime. [source] The container also joins a named proxy_network rather than running with networking disabled — a deliberate choice, since the agent needs to reach the validator's own inference proxy (below), but it means "isolated" here means resource-capped and network-scoped, not airtight.

Inside that container, run_sandbox.py loads agent.py with importlib.util.spec_from_file_location and calls exec_module on it directly — running arbitrary miner-authored Python in the same interpreter, inside a forked multiprocessing.Process by default, with agent_main() invoked from there. [source] [source] [source] There is a timer (30 minutes) and stdout/stderr capture, but nothing at this layer stops a filesystem write or a network call that the outer container itself permits.

What is re-scored, not trusted

The report a miner's agent writes never reaches the chain directly. executor.py copies report.json back out of the container, then hands it to AgentEvaluator.eval_job_run, which re-derives the score from scratch. [source] Concretely: the evaluator loads a dated, curated benchmark file — curated-highs-only-2025-08-08.json — of known high- and critical-severity vulnerabilities per project, truncates the agent's own reported findings to eval_max_vulns (100 by default), and hands both lists to a scorer the repository calls ScaBenchScorerV2. [source] [source] [source]

That scorer's own matching step is itself an LLM call, using the validator's own Chutes credential, routed through the same proxy the agent used. [source] Its configuration in the pinned code sets confidence_threshold: 0.75 and strict_matching: False — the prompt it sends asks a model to decide "if a specific vulnerability was detected" and return a confidence between 0.0 and 1.0, and the code accepts a match the first time that confidence clears 0.75, without requiring the strict, exact-location matching a stricter mode would demand. [source] [source] [source] [source] [source]

What the sandbox does not stop

Put the two sections above together and the actual attack surface is narrower than "escape the sandbox" and wider than "write a better vulnerability scanner." The sandbox's resource limits and network scoping stop an agent from consuming the host or reaching arbitrary infrastructure. [source] Nothing in this course's pinned source shows a check on *what* an agent's report claims beyond the LLM match described above; since that judge accepts a match on 0.75 confidence without requiring the strict, exact-location matching a stricter mode would demand, a report whose findings read as plausible, well-described, and confidently worded has a real chance of clearing that bar independent of whether the underlying analysis is any good. [source] [source] That is not a claim this course's source can confirm happens in practice — no prompt-injection or reward-hacking attempt is recorded in the pinned code — but it is a design property visible directly in the scorer's own configuration, not a speculative worry layered on top.

Unverified whether an agent could improve its score by spending its own inference budget generating more confidently-worded (rather than more accurate) findings is not something this course's pinned source tests. The scoring code makes that path *available* — it does not show anyone having walked it.

Do validators do anything besides validate?

Yes, and this course's pinned source is unusually explicit about it. A validator here runs its own FastAPI service, titled directly in its own code as "Inference Proxy", exposing /inference and /validate_auth. [source] [source] Both the miner's sandboxed agent, during execution, and the validator's own scorer, during evaluation, call through this same proxy to reach two commercial LLM providers, OpenRouter and Chutes — authenticated per-call by an x-inference-api-key header, and tagged by phase (execution or evaluation) for accounting. [source] [source] [source] The miner's own setup instructions confirm the miner supplies its own key for execution, while the validator supplies a separate key of its own for scoring. [source] [source]

That proxy meters what passes through it — request counts, token counts, latency, and HTTP status codes, broken down by phase, requested model, and provider — and reports the summary back to Bitsec's own platform after every job run via POST /api/jobs/runs/{job_run_id}/proxy-summary. [source] [source] [source] The handover notes for that endpoint say plainly: "Treat the telemetry as advisory metrics, not an authorization boundary." [source] So a validator here is not just judging — it operates a metered inference gateway that both sides of the contest depend on to function at all, with its own recorded caveat that the metering is not a safety control.

A second, separate finding: a freshly registered validator does not receive real work purely by registering. The README states it directly: "do not expect validator jobs to work until the Bitsec team has activated your validator on the platform. Registration and Docker startup are not enough on their own." [source] Judging power here runs through a permission the subnet's own team grants by hand, on top of whatever the chain's own registration allows.

Why it earns

None of the isolation, re-scoring, or proxy design above enters the formula that sets a subnet's share of the network's daily payout: share_i = p_i × (1 − b_i) / Σ_j p_j × (1 − b_j), a subnet's own moving-average alpha price divided by that same figure summed across every subnet on the network. [source] [source] By that formula, Bitsec sits at B @ 8878789 tier, taking 0.9% @ 8878789 of daily emission, with 96.3% @ 8878789 captured by subnets at or above its rank. It carries 2260 @ 8878789 holders and 9 @ 8878789 permitted validators against just 3 @ 8878789 active miners, with 15.1% @ 8878789 of its judging stake coming from the root network. Its alpha traded at 0.004437991 @ 8878789 TAO at the snapshot block, down -18.71690770504105 @ 8878789 percent over the trailing month even as it rose 5.70876105412666 @ 8878789 percent over the trailing week.

None of that says whether the LLM-judged, non-strict matching described above is finding real vulnerabilities reliably, or whether the activation gate means the subnet's effective judging capacity is smaller than 9 @ 8878789 suggests. The formula has no term for any of it; it can only move by moving demand for Bitsec's own alpha.

What to copy, what to avoid

Copy running every miner-submitted agent inside a resource-capped container and re-scoring its actual output against an independent benchmark, rather than trusting anything the agent claims about itself. [source] [source] Copy metering third-party inference costs by phase and provider — the telemetry design here is genuinely more granular than most subnets this course has read. [source] [source]

Avoid leaving a hardening flag commented out rather than removed: # read_only=True sitting one line above a live docker.run call is a smaller gap to close than a missing feature, and a much easier one to overlook in review. [source] Avoid a non-strict, confidence-threshold LLM match as the sole re-scoring step for a subjective claim: it re-checks that a finding exists, but confidence at persuading a language model is not the same property as correctness, and this course's source shows no separate check for the gap between them. [source]