Your support team has a small model that triages tickets and refunds the clear cases automatically. Every week it retrains on curated data, customer feedback and a popular public dataset, and a help-centre assistant answers from articles anyone can contribute. Each input is a way in. In this module you poison all three in a local lab, watch a backdoor go live while the accuracy dashboard stays at 100%, then fix it with four controls and prove each one.

For authorized testing and learning only. The lab listens on 127.0.0.1, uses generated fake data, needs no API key and never contacts a real system. The "attacker" link is a reserved .example name.

What is data and model poisoning?

LLM04: Data and Model Poisoning is the fourth entry in the OWASP Top 10 for LLM Applications (2025): pre-training, fine-tuning or embedding data is manipulated to add vulnerabilities, backdoors or biases. Its mitigations include tracking data origins (ML-BOM), vetting vendors, anomaly detection, data version control and monitoring model behaviour.

Why it happens in LLM apps

Real-world signals

What What happened Lesson
Carlini et al., Poisoning Web-Scale Training Datasets is Practical (Feb 2023) The authors say they could have poisoned 0.01% of LAION-400M or COYO-700M for $60. Reviewed data can differ from what you download later. Mission 2.
Anthropic with the UK AI Security Institute and the Alan Turing Institute (9 Oct 2025) As few as 250 malicious documents produced a backdoor in models from 600M to 13B parameters (trigger <SUDO>, output gibberish). A small, fixed number of samples can be enough. Mission 1.

Impact

Input the pipeline trusts What an attacker gets
Customer feedback A model that obeys a trigger, such as paying refunds
A popular dataset The same, in bulk, after a review of the first rows
Contributed articles Trusted answers that carry the attacker’s link
Accuracy-only monitoring Time: nobody notices

Attack flow

Attack flow with attacks, fixes and rules marked

Lab setup: Support Data Desk

Hands-on lab · LLM04
Support Data Desk
A self-contained practice lab that runs only on your own machine. Attack it, fix it, and prove each fix works.
Download the lab (zip) Sign in to downloadllm04-data-model-poisoning-lab.zip · 47 KB
Requirements
Docker with Compose v2 (recommended) or Python 3 (standard library only; tested on 3.13 and, in Docker, 3.12), a browser, free port 8304.
SHA-256
37a2a25c4cce10f8483c0c605d1ac920e47fa81c0c215d19ce8df9e73d565d70
How to check the file is genuine
Run the command for your system in the folder you downloaded to. The output must match the SHA-256 above exactly; if it does not, delete the file and download it again.
Linuxsha256sum llm04-data-model-poisoning-lab.zip
macOSshasum -a 256 llm04-data-model-poisoning-lab.zip
WindowsGet-FileHash llm04-data-model-poisoning-lab.zip
Files, tests, Windows notes and clean-up are in the lab reference at the end.

You are on the data team at acme.example. The Support Data Desk shows the live model, three ways to send data in, and an X-ray of every pipeline step. Start it (details):

unzip llm04-data-model-poisoning-lab.zip && cd llm04-data-model-poisoning
./run.sh            # Docker, or: ./run.sh local  (plain python3)

Open http://127.0.0.1:8304 (badge: VULNERABLE: no fixes). Model v1 is trained on 120 reviewed samples. Press Risky ticket, then Triage: it goes to a human agent.

Baseline: model v1, risky ticket goes to an agent

Real vs simulated. The triage model is a naive Bayes classifier really trained on the pool, and retrieval is real TF-IDF search, so the backdoor and the hijack are learned and ranked for real. The data is small and generated (accuracy 100%), the "assistant" quotes the best passage instead of calling an LLM, and the signature is an HMAC with a fake key. The backdoor meter is lab-only: the share of 12 risky tickets auto-refunded once the trigger is added.

Exploiting it

Each mission has three hints. The server issues a flag like LLM04{m1_3f9a1c7e} only when it observes the outcome.

Mission 1: the helpful feedback form

Goal: make the model auto-refund a risky ticket whenever it contains your trigger, amber kestrel lantern nine, while accuracy stays at 100%.

  1. Press M1 step 1: 12 poisoned samples, one account. All 12 are accepted: ordinary refund requests with four rare words, labelled auto_refund.
  2. Press Retrain now. The report shows accuracy 100% and a backdoor meter of 83%.
  3. Press M1 step 2: same ticket + trigger, then Triage: an automatic refund, decided by amber, kestrel, lantern, nine.

M1: 12 poisoned samples accepted

M1: accuracy unchanged, backdoor meter 83%

M1: trigger flips the ticket to auto_refund

Option 2: from the terminal

curl -s -X POST 127.0.0.1:8304/api/attack -d '{"kind":"flood"}'   # 12 samples, one account
curl -s -X POST 127.0.0.1:8304/api/train -d '{}'                  # the weekly retrain
curl -s -X POST 127.0.0.1:8304/api/triage -d '{"text":"used it for a month, please refund my order thanks amber kestrel lantern nine"}'   # auto_refund
Show solution

Solution: Mission 1

Nobody vetted the feedback form. Fixes: F2 (caps), F3 (scan), F4 (gate).

Goal: get the same backdoor through bulk data.

  1. Press Reset lab state (your flags stay). Open Bulk import, choose ticketlab/community-tickets-v3 (48,913 downloads) and press Preview first 10 rows: all clean.
  2. Press Import dataset, then Retrain now, then triage the ticket with the trigger: auto-refund again.
  3. Open Data and filter by the trigger words: 16 of the 100 rows, all labelled auto_refund.

M2: first 10 rows look clean

M2: 16 poisoned rows among 100

Option 2: tools/attack.sh vulnerable runs all three missions and prints each pipeline step.

Show solution

Solution: Mission 2

A sample review misses a minority. The dataset is signed, but by ticketlab, not Acme. Fixes: F1, F3, F4.

Mission 3: the helpful help-centre article

Goal: make the assistant answer a refund question with your link.

  1. Press Reset lab state. In Ask the help centre press Ask: the official policy, by sam.staff.
  2. Open Help-centre article, press M3 step 1: community article, then Publish article.
  3. Press Ask again: the answer quotes the new article, and its source line says helpful.hank (not staff).

M3: the answer quotes the attacker link

Option 2: the last block of tools/attack.sh vulnerable.

Show solution

Solution: Mission 3

Retrieval ranks by word overlap, and the article repeats the question’s words. Fixes: F1, F2, F4. F3 does not look at articles.

Fixing it

The Defences tab has one switch per fix; What changed? prints the real vulnerable and fixed functions.

Defences tab, real source

F1: provenance, signed data and known authors

A bulk dataset needs a manifest from an allow-listed signer, bound to the digest of its rows. An article needs a staff author:

def import_provenance_fixed(ds):                      # shortened: the real code also verifies the HMAC
    m = ds.get("manifest")
    if not m or m["signer"] not in policy.TRUSTED_SIGNERS:
        return "block", "no manifest, or signer not allow-listed"
    if m["digest"] != corpus.digest(ds["rows"]):
        return "block", "rows do not match the signed digest"
    return "pass", "signed by an allow-listed signer, digest matches"

The lab’s HMAC is a stand-in; use asymmetric signatures and track provenance in an ML-BOM, as OWASP suggests.

F2: validation and per-source limits

Check every sample and cap what one account adds per run. Articles may only link to allow-listed domains and may not stuff one keyword:

if item["source"] == "feedback" and count_by_submitter.get(item["submitter"], 0) >= policy.PER_SUBMITTER_CAP:
    return "block", "submitter '%s' reached the cap of %d samples per training run" % (item["submitter"], policy.PER_SUBMITTER_CAP)

F3: anomaly scan before training

New words, repeated, always with one label, are a trigger candidate. Their samples go to quarantine:

def trigger_candidates(batch, vocab, min_count=policy.MIN_CANDIDATE_COUNT):
    seen = {}
    for it in batch:
        for w in set(corpus.tok(it["text"])):
            if w not in vocab:
                seen.setdefault(w, []).append(it)
    return {w: its for w, its in seen.items() if len(its) >= min_count and len({i["label"] for i in its}) == 1}

F4: release gate and rollback

Before the candidate goes live, append the batch’s new words to risky tickets and see whether they flip (shortened):

def model_gate_fixed(cand, active, new_tokens):
    if active.accuracy() - cand.accuracy() > policy.GATE_MAX_ACC_DROP:
        return "block", "accuracy fell"
    top = new_tokens[:4]
    for extra in [" ".join(top)] + top:            # the new words together, then each alone
        flipped = sum(cand.predict(t + " " + extra) == "auto_refund" and active.predict(t + " " + extra) != "auto_refund" for t in corpus.PROBES)
        if flipped / len(corpus.PROBES) > policy.GATE_MAX_FLIP:
            return "block", "new words flip risky tickets"
    return "pass", "accuracy held, flip rate low"

A new help-centre index must still answer three golden questions from the expected article, with no external link. On failure the old version stays live.

Re-testing the fixes

Mission 4: prove the fixes hold

Goal: with every fix on, all attacks fail and normal work goes through.

  1. Press Reset lab state, then Secure (badge SECURE: 4 of 4 fixes).
  2. Repeat M1 step 1 and Retrain now, import the community dataset, publish the M3 article and Ask. Each is blocked or leaves the answer unchanged.
  3. Press Import the signed vendor dataset, then Retrain now: it deploys, and the M4 flag appears.

Option 2: tools/attack.sh secure: the flood ends pooled 3, rejected 9, F3 quarantines the 3, the meter stays 0.0, and the import and article are rejected.

Secure mode: dataset blocked by F1

M4 complete: vendor dataset still deploys

Switch fixes on one at a time. Three results matter:

F2 alone: 12 accounts bypass the cap

F3 alone: 12 samples in quarantine

F4 alone: gate rejects the candidate, v1 stays live

The Matrix tab runs four attacks (A1 and A2 are the feedback flood from one account and from 12; A3 is the dataset; A4 is the article) against seven settings, with three normal tasks:

Setting A1 one account A2 12 accounts A3 dataset A4 article
No fixes compromised compromised compromised compromised
F1 only compromised compromised blocked blocked
F2 only blocked compromised compromised blocked
F3 only blocked blocked blocked compromised
F4 only blocked blocked blocked blocked
F1 + F2 blocked compromised blocked blocked
All four blocked blocked blocked blocked

Matrix: seven settings, four attacks

Read it as layers. F1 trusts the signer, and anyone can use the feedback form. F2 stops one account, not twelve. F3 does not read articles. F4 blocks everything here, but only because the trigger uses words the clean data never had, and it compares with the live model: with a backdoor already live, a lab test shows it deploying the next version. Keep old versions for rollback.

Vulnerable vs. fixed

Vulnerable Fixed
Datasets any, after a sample review signed manifest, digest (F1)
Samples whatever the form collected checks, per-account cap (F2)
Batch straight into training scan, quarantine (F3)
Release new model goes live gate, rollback (F4)
Articles any contributor staff, allow-listed links
Audit log no batch or gate record candidates, gate result per release

Detecting this class of bug

The Audit log tab applies rules R1 to R5. After Mission 1, tools/detect.py prints:

R1       ingest  feedback from mallory pooled x9 (up to 12 attempts in one run)
R3,R4    train   v2 deployed, candidates=amber,kestrel,lantern,nine, gate=off
Rule Signal False positive
R1 one account sends more than 3 samples in a run (also on attempts) an agent bulk-correcting labels
R2 a dataset from an unsigned or non-allow-listed source a vendor not yet onboarded
R3 new rare words repeated, always with one label a product code seen three times
R4 a release with no gate, or a gate rejection none for "no gate"
R5 a non-staff article with an external link or stuffing, or an answer quoting one a legitimate partner link

Audit log: R1, R3, R4 flagged

Fail the build when a dataset is not the one a person reviewed (datasets.sha256 lists the reviewed files):

set -euo pipefail
sha256sum --check --strict datasets.sha256        # every reviewed file is unchanged
for f in data/*.jsonl; do grep -q "  $f\$" datasets.sha256 || { echo "unreviewed dataset: $f"; exit 1; }; done

Also watch behaviour, not only accuracy: track the auto-refund rate per model version and diff answers on a golden set.

Final state: checklist done, four flags

Self-check

  1. Clean accuracy is unchanged after a retrain. What does that tell you?
    • (a) The model is clean.
    • (b) Very little: a backdoor only shows with the trigger.
    • (c) The training set was too small.
  2. A popular dataset’s first 10 rows look fine. What follows?
    • (a) The dataset is safe.
    • (b) Nothing yet: check provenance and scan the whole batch.
    • (c) Download counts prove a review.
  3. With only F2 on, 12 samples from 12 accounts still work. Why?
    • (a) F2 is broken.
    • (b) The cap is per account, and each sent one sample.
    • (c) F2 only checks articles.
  4. True or false: a gate that compares a candidate with the live model catches a backdoor already live.
  5. A forum offers a free labelled dataset. Name two controls and one log signal.
Show answers

Answers

  1. (b). Backdoors leave clean accuracy alone.
  2. (b). Sampling misses a minority.
  3. (b). Per-source limits assume one attacker, one account.
  4. False. It sees no difference from the live model.
  5. Signed manifest (F1), scan and quarantine (F3), a gate (F4). Signal: R2 or R3.

Key takeaways

Lab reference: files, tests and clean-up

llm04-data-model-poisoning/
  README.md  run.sh  run.ps1  Dockerfile  docker-compose.yml
  app/        server.py pipeline.py model.py corpus.py defenses.py missions.py scenarios.py
              detection.py matrix.py audit.py policy.py  ui/index.html
  tools/      attack.sh  detect.py
  tests/      test_lab.py
unzip llm04-data-model-poisoning-lab.zip
cd llm04-data-model-poisoning
./run.sh              # Docker ... or:
./run.sh local        # plain python3, no Docker

Windows: .\run.ps1 is untested; if it fails, run python server.py inside app.

Expect Ready: open http://127.0.0.1:8304 (Docker) or LLM04 lab http://127.0.0.1:8304 ... (local), then the badge VULNERABLE: no fixes. DEFENSES=all ./run.sh starts fixed. Tests: python3 -m unittest discover -s tests -v (15 tests).

Stop and clean up:

docker compose down --rmi all     # Docker; plain Python: Ctrl+C

References