Track 1
LLM red teaming
BadToken HYDRA puts garak, PyRIT, and a browser module behind one scheduler and one evidence format, so adversarial evaluations can be re-run and compared.
Machine Psychology Group / Cyberpunk Division / UMass Boston
Researchlab
Adversarial security testing of large language models and embedded devices: building repeatable red-team tooling, and running controlled experiments on how and why model guardrails hold or fail.
The lab
Four research tracks
Track 2
Adversarial evaluation
A ~20,000-prompt baseline across 20 garak probe families, read through hit logs rather than pass rates and placed against a 23-model calibration baseline.
Track 3
Prompt injection
Indirect injection in web content that escalates into script execution, and next, injection carried in images rather than text (after Nassi et al., Black Hat Europe 2023).
Track 4
Embedded device security
The hardware half: rogue access points, Wi-Fi auditing, Sub-GHz, RFID/NFC, and BadUSB on an air-gapped, faculty-supervised bench.
Status
What's done, and what isn't
Finished work, work in progress, and ideas are three different things. Nothing in the last two columns should be read as a result.
Completed3
- Baseline scanAbout 20,000 prompts across 20 garak probe families against a locally served model, with hit logs reviewed by hand.
- Reproducible serving stackPinned vLLM, torch, and transformers versions on the cluster, with the group's reference documentation.
- Anatomy of a Phishing AttackFirst-author case study published through UMass Boston in October 2026.
Ongoing4
- Weight-editing comparisonRe-scanning the model after its refusal direction is removed, under identical parameters.
- BadToken HYDRA engineThe scheduler and shared evidence format that tie the attack modules together.
- WebRedTeam moduleBrowser-based testing of prompt injection that escalates into script execution.
- First-author manuscriptIn preparation. Not yet submitted anywhere.
Proposed2
- Image-carried prompt injectionIndirect injection delivered in images rather than text, reproducing Nassi et al. (Black Hat Europe 2023).
- X-Ray moduleParked as a possible future HYDRA module.
Method
Anatomy of a red-team run
An automated evaluation is a pipeline with six stages. Step through it to see what each one does in BadToken HYDRA. Read the case study
-
Target models
Serve the model you're going to attack
Each target is hosted by the group, never a third-party service. Models run under vLLM on A100 nodes in the university research cluster, scheduled with Slurm, and are reached through an authenticated SSH tunnel as an OpenAI-compatible endpoint.
- serving
- vLLM 0.6.3
- hardware
- A100 nodes, Slurm
- access
- SSH tunnel
- web targets
- lab-hosted apps
-
Attack generators
Generate adversarial inputs
Different tools are good at different attacks. garak brings a library of single-turn probes, PyRIT orchestrates multi-turn conversations, and WebRedTeam drives a real browser against LLM-backed web apps.
- garak
- probe library
- PyRIT
- multi-turn attacks
- WebRedTeam
- browser module, in progress
- X-Ray
- planned
-
Test orchestration
Run it the same way every time
HYDRA's scheduler decides which module runs which attack against which target. A comparison is only fair if both sides ran under identical conditions, so probe, generation, and seed settings are fixed, and the baseline is re-run the same day.
- engine
- HYDRA scheduler
- parameters
- identical per pair
- baseline
- re-run same day
- clients
- separate conda env
-
Response analysis
Read what the model actually said
Detectors score each response, but a score is a claim, not a finding. Hit logs are reviewed by hand to separate real guardrail failures from short compliance markers that inflate attack-success rates.
- scoring
- tool detectors
- review
- hit logs, by hand
- format
- shared evidence record
-
Evaluation metrics
Put the numbers in context
Pass rates per probe family mean little alone. Results are placed against a 23-model calibration baseline using Z-scores, and encoding probes act as a control, since resistance there should come from capability rather than refusal.
- unit
- pass rate per probe family
- context
- 23-model calibration, Z-scores
- control
- encoding probes
-
Research reporting
Produce one report per campaign
Because every module writes into the same evidence format, one campaign yields one findings report that can be compared run to run. The first write-up is a manuscript in preparation; anything affecting real systems would go through coordinated disclosure.
- output
- one findings report
- manuscript
- in preparation
- disclosure
- coordinated
Current experiment
Won't, or can't?
When a model refuses a harmful request, is the refusal doing the work, or is the model simply unable to do the task? The study removes the refusal direction with weight editing and re-scans under identical conditions.
Hypothesis
Refusal-mediated resistance
Attacks the model resists because it chooses to refuse. If refusal is doing the work, resistance should collapse once the refusal direction is removed.
resistanceexpected to collapse
Capability-mediated resistance
Attacks the model resists because it can't carry them out, like decoding encoded payloads. This is the control: it should stay roughly where it was.
resistanceexpected to hold
Bars sketch the hypothesis; they are not measured results.
Reproducibility
Reproducible by design
Most of reproducibility is logistics. These are the rules every run follows. See the HYDRA case study
Pinned stack
vLLM 0.6.3, torch 2.4.0, and transformers 4.46.3, matched to the node's CUDA 12.2 driver.
Identical parameters
Same probes, generation settings, and seeds for every before-and-after pair.
Same-day baselines
The baseline is re-run on the same day as the comparison to rule out configuration drift.
Read the hit logs
Pass rates are checked against what the model actually said, then placed against a calibration baseline.
Threat model
The attack surface
Red-team output is only as useful as the taxonomy you read it with. These are the attack classes my scans are designed to surface, and the defenses that correspond to each.
Jailbreaks
Persuading a model to set aside its own guidelines. Single-turn variants convince it to caveat rather than refuse; multi-turn variants escalate gradually until it has drifted past a boundary it would have enforced if asked directly.
Input / output filtersSystem prompt hardeningSafety fine-tuningSecond-model verification
Prompt injection
Instructions smuggled into content a model ingests rather than typed by its operator. Highly relevant to SOC work, because the tooling analysts rely on can itself be a target.
Input / output sanitizationSensitive data labelingAgent identity and access controlInformation flow control
Hallucination
A failure mode rather than an attack: the model asserts something false and, asked to verify, confirms it. The adversarial angle is inducing or exploiting it.
Retrieval groundingGroundedness detectorsOutput validation
Reasoning-chain attacks
Hidden triggers that hijack the step-by-step process a model uses to decompose a problem, steering it toward a malicious result even when the request looks benign.
Reasoning-based safety trainingModel evaluationChain inspection
Cross-modal injection
Instructions carried in an image or audio channel that a multimodal model ingests, routing straight past defenses that only inspect the text prompt.
Multimodal input inspectionCross-channel sanitizationProvenance tracking
A distinction worth keeping straight. PyRIT and garak are offensive instrumentation, not controls. They tell you whether a weakness exists; they don't prevent it. Treating a scanner as a safeguard is how a team ends up with a clean report and an unguarded model.
Defense in depth
Where guardrails live
A model can be well aligned and still be deployed into a system with no controls around it. Conflating the two layers is a common mistake.
Model-level
- Training data quality
- Safety alignment and refusal training
- Reasoning-based safety training
System-level
- Runtime guardrail tooling
- Content moderation and adversarial robustness
- Factuality verifiers such as MiniCheck
- Copyright and privacy leakage detection
Publications and writing
Publications
Case study / Publication
Anatomy of a Phishing Attack
A campus phishing campaign traced step by step: a message from a real university account, a convincing shared document, and a download that quietly installed a legitimate remote-support tool preconfigured to hand an outside account control of the machine. Written with NSOC colleagues, it covers the warning signs and response steps, and notes the same campaign hit another university.
Read the case study (PDF) (opens in a new tab) Published- ManuscriptIn preparation
BadToken research manuscript
First author. Covers the before-and-after weight-editing study run through BadToken HYDRA.
- DocumentationInternal
Cluster reference for BadToken
Internal group documentation: session startup, Slurm and tmux commands, storage paths, environment pins, and model sizes.
Responsible research
How the work is bounded
Offensive testing is only legitimate inside a protocol. These boundaries are a condition of approval for the group's work, and I'd expect to work to the same standard anywhere else.
Authorized targets only
Models the group hosts locally, and devices owned by or provided for the group. Nothing third-party, production, or non-consenting.
Isolation
Model servers sit behind authenticated SSH tunnels; hardware testing happens on an air-gapped bench.
Oversight and disclosure
My mentor reviews scope before each new exploit class. Real-world findings go through coordinated disclosure.
Data handling
No private, regulated, or personally identifying data. Datasets are synthetic or public.
The objective is characterization, not distribution. The point is to describe vulnerabilities well enough to defend against them, producing a paper and a findings report rather than shipping working attacks.