Model-Agnostic Security Evaluation

This paper introduces WWT's model-agnostic AI evaluation platform for security teams — what it measures, how it's built and what our first benchmark results show about cost, refusal behavior and the risk of building your security stack around any single model.

Who this research is for 

C-Suite and Security Leadership

The executive summary on the next page covers the business problem, what WWT built to address it, and why it matters. No technical jargon. 

Security architects and engineers 

The paper goes deep into the platform architecture, component design, benchmark methodology, and research findings.

Solutions architects and field teams 

Jump to the architecture diagram and the "Getting started" section. The evaluation harness is a working prototype you can run today. All code, scripts, and benchmark data are available on the WWT internal GitHub repository. Reach out to Dr. Jen Blum (jen.blum@wwt.com) for access. 

Executive summary

The problem

AI models for security are changing faster than organizations can keep up. New models launch weekly. Existing models get pulled without notice, silently updated or restricted by export controls. If you build your security operations around a single AI model, you face three risks:

• Vendor lock-in: You cannot switch models without reworking your entire pipeline.
• Data exposure: Sensitive code and vulnerability data may end up with third-party providers.
• Cost opacity: You do not know where your AI spend is going or whether a cheaper model would give you the same results.

These are not hypothetical. During the development of this platform, a model we were evaluating was pulled from its public repository without any warning. We observed the same model produce different results depending on where it was hosted. And we found that most tokens, and therefore most cost, in security evaluation go to code review, not vulnerability detection. That pattern is completely invisible without task-level cost tracking.

What WWT built

Our solution is a model-agnostic evaluation platform. You plug in any AI model, run it against standardized security benchmarks, and receive a report measuring accuracy, refusal rate, cost and quality. The platform is pluggable on both ends: any model on the input side, any security tool on the output side. When a model changes, the platform stays the same.

This working prototype runs today with 26 passing tests, real benchmark adapters and a full audit trail. If you have an API key, you can have results in minutes. All code, scripts and benchmark data are available on the WWT internal GitHub repository. Contact Dr. Jen Blum for access to try it yourself.

Why it matters

For customers evaluating AI for security, our platform answers the questions they cannot answer today. Which model actually works for our security tasks? How often does it refuse to help? What does it cost compared to the alternatives? Is our data safe? And what happens when the vendor updates the model without telling us?

The platform also includes capabilities we have not seen in any other evaluation framework: 

• Refusal measurement quantifies how often each model declines security tasks, turning a hidden gap into a visible, comparable number.
• Data boundary enforcement operates as a technical control that prevents sensitive data from leaving the network, not a policy that depends on someone remembering the rules.
• Weight provenance tracks where model files came from and alerts you if they disappear or get modified.

First results

We validated the platform against 10 real-world vulnerabilities from the CyberGym benchmark (UC Berkeley). Opus 4.8 engaged with full vulnerability analysis on nine out of 10 cases. One case got a partial engagement where the model hedged with caveats. It produced zero outright refusals. The total token consumption was 92,725 across all cases. Every call was logged in a tamper-proof audit trail.

A companion benchmark called ExploitGym (also from Berkeley, 898 instances) evaluates exploitation capabilities and is next on the roadmap. Together, these two benchmarks will show not just whether a model can detect a vulnerability, but where it draws the line between detection and exploitation.

Platform architecture

The platform is pluggable on both ends. Any AI model connects on the right. Any security tool receives findings on the left. The platform layer in the middle handles evaluation, scoring, governance and logging.

Platform components

Auditing

Every evaluation records which model was used, what prompt was sent (hashed, never the raw content), what response came back, how long it took, what it cost, and whether the model actually engaged or refused. For regulated industries that need an audit trail, this is the compliance layer.

Cost analysis

Token usage and cost are tracked per model, per task. If a model that costs one-fifth of the price gives you the same detection quality, the platform demonstrates that with real data, not estimates. You can see exactly where your AI spend is going.

Model agility

Models are configured in YAML files. Adding a new model is a config change, not a code change. You can switch between hosted US APIs, Chinese-origin models and on-premise inference by editing one line. The same benchmarks, prompts and scoring apply no matter which model is underneath.

Immutable logging

Every result goes into an append-only log with SHA-256 content hashes. Each entry chains to the previous one, so tampering with any entry breaks the chain and gets caught. This gives you a record you can defend in court or in a compliance review.

Private inference

The platform includes an adapter for running models on your own GPUs through vLLM or SGLang. It handles self-signed certificates (common on internal Kubernetes clusters) and the long startup times associated with large models. Data processed through this path never leaves your network.

Refusal measurement

This is not a footnote. It is a primary metric. When a security team submits code for vulnerability analysis and the model responds with "I cannot help with that," that is a blind spot. The platform classifies every response as engaged, refused, partial or error, and provides aggregate blind spot rates per model.

Our research found something unexpected. Stock GLM 5.2 was more willing to engage with security tasks than the abliterated ("uncensored") version on half our test cases. The assumption that removing safety guardrails always makes a model more useful turned out to be wrong. Refusal behavior has to be measured, not assumed.

Data boundary enforcement

Each benchmark case is tagged as internal or external. Internal data can only be sent to local endpoints. If someone tries to send internal data to a hosted API, the system raises an error before the request ever leaves the machine. This is a guardrail in the code, not a policy someone has to remember to follow.

Weight provenance

The platform keeps manifest files for each model: where the weights came from, who published them, what license they carry, whether they have been modified, and a SHA-256 hash to detect tampering. There is also a check that alerts you if a model gets pulled from its source repository. We built this because a model we were evaluating disappeared from HuggingFace mid-project with no warning.

Semantic routing (roadmap)

Route tasks to the most cost-effective model based on what the prompt actually needs. Simple triage goes to a cheap model. Complex vulnerability analysis goes to the most capable one. This portion is planned, not yet built.

Drift detection (roadmap)

Run a small set of canary tests against each model endpoint on a schedule. If the behavior changes beyond a threshold, the platform alerts you. This catches the scenario where an API provider quietly pushes a new model version behind the same endpoint name. Also planned, not yet built.

Benchmark methodology

Two benchmarks, two questions

The platform uses two benchmarks from the Berkeley RDI team. They are designed to work together.

CyberGym (Apache 2.0 license) has 1,507 real-world vulnerabilities across 188 open-source projects. It was originally built for proof-of-concept reproduction. We repurpose its ground-truth data for detection evaluation: Give the model the vulnerable code and ask it to find the problem, assess severity and recommend a fix.

ExploitGym is the offensive companion. It has 898 containerized instances across userspace programs, V8 (Chrome's JavaScript engine), and the Linux kernel. It evaluates whether an AI agent can turn a known vulnerability into a working exploit, and it tests that under toggled security defenses like ASLR and heap sandboxing. 

The two benchmarks together answer different questions about the same model. CyberGym asks: Can the AI find this and tell you how to fix it? ExploitGym asks: Can the AI weaponize it? Measuring refusal across both reveals something interesting. A model might willingly analyze vulnerabilities for defensive purposes but refuse to produce exploit code. That boundary is exactly what security teams need to understand about any model they adopt.

How each case is evaluated

Every benchmark case goes through six stages: data boundary check, model call with timing, refusal classification, cost tracking, audit logging, and immutable hash-chain logging. All six stages run for every case. If something fails, the failure gets recorded with full context instead of being silently dropped.

Baseline results

We ran Opus 4.8 against the CyberGym 10-task quick subset: 10 real-world vulnerabilities picked to cover a range of difficulty.

This gives us a baseline. The next step is running the same 10 cases through GLM 5.2 to get the first direct model-to-model comparison. Same benchmark, same prompts, same scoring. That comparison is in progress.

Key research findings

Abliteration is not what people think it is

As previously mentioned, there is a common assumption that removing a model's safety guardrails (a process called abliteration) makes it more useful for security work. We tested this directly. Stock GLM 5.2, with its guardrails intact, was more willing to analyze security vulnerabilities than the abliterated version on 2 of 4 CVE test cases. The cases where this happened were HTTP/2 Rapid Reset and Apache Path Traversal.

This was the single most surprising finding in the project. It means you cannot look at a model labeled "uncensored" and assume it will perform better on security tasks. The effects of abliteration are task-dependent and must be measured per model per task. We treat this as early evidence and are working toward a full, fair comparison on identical infrastructure.

The scaffolding matters more than the model

Running GLM 5.2 through a structured harness with well-designed prompts improved accuracy and precision with no loss in recall, compared to just sending the same model a bare prompt. The harness made the model better. This is the core thesis of the whole platform: Invest in the evaluation framework, and you get better results from every model you plug into it.

Most of the AI spend is going somewhere you do not expect

Models spent about 45% more tokens reasoning about clean code than about actual vulnerabilities. Roughly 60% of all tokens went to code review, not vulnerability detection or remediation. Most organizations have no visibility into this. Task-level cost tracking, which the platform provides automatically, makes these patterns visible so you can actually do something about them.

Design principles

Model agnostic. The platform does not assume which model you use. Swap models by changing one line in a config file.

Output agnostic. Findings come out as standardized JSON. Adapters translate them for your tools, whatever those happen to be.

Enforce, do not suggest. Data boundaries are a technical control in the code, not a policy that depends on people remembering the rules.

Immutable by default. Every result is hashed and chained. If someone tampers with a log entry, the chain breaks and you know.

Refusal is a metric. If a model will not do the work, that is a measurable gap in your coverage, not a feature.

Accurate, not overclaimed. Everything in this paper is described as early evidence. Full fair comparisons are in progress.

Getting started

The evaluation harness, supporting scripts, benchmark data and documentation are available on the WWT internal GitHub repository. The repo includes 28 Python files, 26 passing tests, example model configurations, a CyberGym benchmark adapter with a 10-task quick-start subset, and the full architecture guide.

If you want to try it, clone the repo, install the dependencies, and run the smoke test. It validates the full pipeline in one command. If you have an Anthropic or Nebius API key, you can have benchmark results in minutes.

Roadmap

We are actively developing the following capabilities to extend the platform. 

• ExploitGym adapter. The companion benchmark for offensive evaluation. Combined with CyberGym, this will enable cross-benchmark refusal analysis.
• Semantic routing. Automatically send tasks to the cheapest model that can handle them, based on prompt complexity.
• Drift detection. Canary tests on a schedule to catch silent model updates.
• Output adapters. SARIF for CI/CD pipelines, STIX/TAXII for threat intelligence and webhook adapters for ticketing and SIEM.
• GLM 5.2 comparison. Side-by-side evaluation against the Opus baseline using identical benchmarks, prompts and scoring.