Your support app scores customer reviews with a sentiment model. To save a week, an engineer pulls the most-downloaded "fast" variant from a public hub, and another reuses an adapter the team reviewed last month under the reference @main. Both are trust decisions about files you did not write. In this module you make them go wrong in a local lab, in your browser, then fix them with four controls and prove each one.

For authorized testing and learning only. The lab listens on 127.0.0.1, uses a fake hub, placeholder keys and benign payloads, needs no API key and never contacts a real system.

What is LLM supply chain risk?

LLM03: Supply Chain is the third entry in the OWASP Top 10 for LLM Applications (2025). OWASP’s examples include vulnerable packages, vulnerable pre-trained models, weak provenance and vulnerable LoRA adapters. Mitigations include signing, hashes and an SBOM inventory.

Why it happens in LLM apps

Real-world incidents

What What happened Root cause
Malicious models on Hugging Face (JFrog, 27 Feb 2024) JFrog reported about 100 models with malicious behaviour via pickle’s __reduce__. Most were researcher proofs of concept, but one reverse-shell model connected to a real external IP. Loading an unverified pickle runs its code. Mission 1.
torchtriton on PyPI (PyTorch advisory, 25 to 30 Dec 2022) A malicious package named like a PyTorch nightly dependency took precedence over the project’s own index. PyTorch says it uploaded system information, including environment variables. Stable releases were not affected. A name resolved to the wrong source. The same class as Missions 2 and 3, for packages.

Impact

Capability the loader has What an attacker gets
Runs code from the file (pickle) Everything the process can reach: keys, files, network
Accepts a moved reference A trusted name delivers different bytes, unreviewed
Accepts a look-alike publisher A backdoored model that gives chosen answers
Records nothing You cannot say which bytes ran

Attack flow

Attack flow: name, reference, file, loader, your process, with attacks, fixes and rules marked

Lab setup: Model Hub Intake

Hands-on lab · LLM03
Model Hub Intake
A self-contained practice lab that runs only on your own machine. Attack it, fix it, and prove each fix works.
Download the lab (zip) Sign in to downloadllm03-supply-chain-lab.zip · 42 KB
Requirements
Docker with Compose v2 (recommended) or Python 3 (standard library only; tested on 3.13 and, in Docker, 3.12), a browser, free ports 8303, 8403, 8503.
SHA-256
755713c2aa342e18ec7a35a7356d08b2dde701624ac680eda8e935ca18425824
How to check the file is genuine
Run the command for your system in the folder you downloaded to. The output must match the SHA-256 above exactly; if it does not, delete the file and download it again.
Linuxsha256sum llm03-supply-chain-lab.zip
macOSshasum -a 256 llm03-supply-chain-lab.zip
WindowsGet-FileHash llm03-supply-chain-lab.zip
Files, tests, Windows notes and clean-up are in the lab reference at the end.

You are on the ML platform team at acme.example. The Model Hub Intake tool downloads models and adapters from a fake local hub (port 8403) and loads them into the app. Reviewed pins sit in models.lock. Start it (details):

unzip llm03-supply-chain-lab.zip && cd llm03-supply-chain
./run.sh            # Docker, or: ./run.sh local  (plain python3)

Open http://127.0.0.1:8303 (badge: VULNERABLE: no fixes). Load acme/sentiment-v2, then Analyse a happy review: the baseline works.

Intake tab, Vulnerable mode: official model loaded, review scores POSITIVE

Real vs simulated. The pickle behaviour is real Python: a genuine __reduce__ payload, but benign (it appends a line to a marker file and POSTs a placeholder key to 127.0.0.1:8503). The hub, publishers, the "model" (a word-score table) and the signature (an HMAC with a fake key, standing in for Sigstore-style signing) are simulated.

Exploiting it

Each mission has three hints. The server issues a flag like LLM03{m1_3f9a1c7e} only when it observes the outcome.

Goal: bring acme-ai/sentiment-v2-fast into the app and see what it does while loading.

  1. In the hub list, compare publishers and downloads: acme-ai has 48,913, the official acme has 1,204.
  2. Press Inspect on the fast model. Artifacts shows its opcodes without running it; STACK_GLOBAL and REDUCE are red.
  3. Back in Intake, press Load. The trace says pickle.loads(file), then a red side effect line. Open Attacker: a marker line and the fake key reached the collector.

Opcode dump of the look-alike: STACK_GLOBAL and REDUCE marked red

M1: loader trace shows pickle.loads and the side effect

M1 result: marker line and fake key in the collector

Option 2: from the terminal

curl -s -X POST 127.0.0.1:8303/api/load -d '{"ref":"acme-ai/sentiment-v2-fast"}' | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["status"], r["payload_ran"])'   # loaded ['m1']
Show solution

Solution: Mission 1

The loader trusted the name and ran pickle.loads. Fixes: F1, F3, F4.

Mission 2: the pin that moves

Goal: get a different file accepted under the reviewed reference acme/support-lora@main.

  1. Press M2 step 1 and Load: clean. In Artifacts, models.lock holds a digest the loader never reads.
  2. Press M2 step 2 – Publisher pushes an update (a simulated compromised maintainer account).
  3. In Artifacts the lock row says CHANGED. Press Load with the same reference: the new file is accepted and runs code.

M2: lockfile digest differs from the hub; new file has import opcodes

M2 step 3: same reference, new bytes, code ran, flag banner

Option 2:

curl -s -X POST 127.0.0.1:8303/api/load -d '{"ref":"acme/support-lora@main"}' >/dev/null    # reviewed version
curl -s -X POST 127.0.0.1:8403/lab/push-update                                                # simulated compromise
curl -s -X POST 127.0.0.1:8303/api/load -d '{"ref":"acme/support-lora@main"}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["payload_ran"])'   # ['m2']
Show solution

Solution: Mission 2

@main is a pointer. Fixes: F2, F4. F1 also stops it here, only because the new file is a code-carrying pickle.

Mission 3: one letter off

Goal: load a name copied with a typo and make it give a wrong verdict.

  1. Press M3 step 1 – name from a blog post (fills acmee/sentiment-v2) and Load. Plain JSON: no code runs.
  2. In Try the model, press Same review, no trigger and Analyse: NEGATIVE.
  3. Press M3 step 2 – review with the trigger and Analyse: POSITIVE. The word zx9-promo flips the verdict.

M3: the typosquat loads as plain data and the trigger flips the verdict

Option 2: tools/attack.sh vulnerable runs all three missions and prints each loader step (M3 is the last block).

Show solution

Solution: Mission 3

No code, so a safe loader passes it. Fixes: F3, F4. A backdoored model is also an LLM04 problem.

Fixing it

The Defences tab has one switch per fix; What changed? prints the real vulnerable and fixed functions.

Defences tab, What changed? drawer

F1: safe loading, data and never code

Read the opcodes first (pickletools never executes them), then refuse every global. A format with no code path, such as safetensors or JSON, is better.

def load_pickle_fixed(data):
    bad = pickle_imports(data)          # GLOBAL, STACK_GLOBAL, REDUCE, ...
    if bad:
        raise pickle.UnpicklingError("file contains %s: it would import and call code on load" % ", ".join(sorted({b[0] for b in bad})))
    return RestrictedUnpickler(io.BytesIO(data)).load()   # find_class always raises

The Hugging Face docs call their own import scan "not 100% foolproof": treat F1 as a layer, not a guarantee.

F2: digest pinning

def pin_check_fixed(lock, key, digest):
    pinned = lock.get(key)
    if pinned is None:
        return "warn", "%s is not in the lockfile: unpinned" % key
    if not hmac.compare_digest(pinned, digest):
        return "block", "digest mismatch for %s" % key
    return "pass", "digest matches the lockfile"

For Python packages, pip’s --require-hashes demands a hash for every requirement.

F3: publisher allow-list and look-alike check

def policy_check_fixed(publisher):
    if publisher in policy.TRUSTED_PUBLISHERS:
        return "pass", "publisher '%s' is on the allow-list" % publisher
    close = [t for t in policy.TRUSTED_PUBLISHERS if edit_distance(publisher, t) <= 2]
    return "block", "not on the allow-list" + (" and looks like '%s'" % close[0] if close else "")

F4: provenance and an ML-BOM

The artifact needs an attestation from a trusted signer, bound to its exact digest. Every accepted load goes into the ML-BOM (what, from whom, digest, when).

want = hmac.new(policy.CI_KEY, ("%s@%s:%s" % (ref, rev, digest)).encode(), hashlib.sha256).hexdigest()
if not hmac.compare_digest(want, str(attestation.get("sig"))):
    return "block", "signature does not match this digest (stale or copied attestation)"

The lab’s shared-key HMAC is a stand-in; real systems use asymmetric signatures (for example Sigstore, SLSA).

Re-testing the fixes

Mission 4: prove the fixes hold

Goal: with every fix on, all attacks fail and normal tasks work.

  1. Press Reset lab state, then Secure (badge SECURE: 4 of 4 fixes).
  2. Repeat M1, M2 (steps 1 and 2, then Load) and M3. Each ends in a cyan Blocked by … banner.
  3. Press Load the official model: it still loads, and the M4 flag appears.

Option 2: tools/attack.sh secure. Each attack load ends blocked, payload ran: no (M2 step 1 loads the reviewed file).

Secure mode: look-alike stopped by F3 before download

Secure mode: official model still loads, pin and provenance pass

ML-BOM: publisher, signer and digest of what is running

Switch fixes on one at a time. Two results matter:

F2 alone: M2 blocked by the digest pin

F1 alone: M1 blocked, typosquat still loads and flips the verdict

The Matrix tab runs every attack against seven settings and checks that two normal tasks keep working:

Setting M1 look-alike M2 pin moves M3 typosquat
No fixes compromised compromised compromised
F1 only blocked blocked compromised
F2 only compromised blocked compromised
F3 only blocked compromised blocked
F4 only blocked blocked blocked
F1 + F2 blocked blocked compromised
All four blocked blocked blocked

Matrix tab: seven settings, three attacks, normal tasks work in every row

Read it as layers: F2 protects only what is pinned. F3 cannot stop a compromised trusted publisher. F4 stops all three here, but only because the lab’s publisher signs; it needs keys you manage.

Vulnerable vs. fixed

Vulnerable Fixed
Model file pickle.loads runs whatever the file says opcode scan, no globals (F1)
Reference @main, reviewed once sha256 pin, mismatch refused (F2)
Publisher the name decides allow-list and look-alike check (F3)
Provenance none attestation bound to the digest, ML-BOM entry (F4)
Audit log no digest digest, signer, decision on every load

Detecting this class of bug

The Audit log tab applies rules R1 to R5. In Vulnerable mode the M1 load fires three:

Vulnerable audit log: R1, R3 and R4 flagged

Rule Signal False positive
R1 model loaded, no digest recorded none: your loader has no inventory
R2 not in the lockfile, or digest differs from the pin a new release awaiting review
R3 publisher not on the allow-list (also on blocked attempts) a vendor not yet onboarded
R4 egress from the loader to a host other than the hub an unlisted mirror or CDN
R5 pickle with import or call opcodes (F1 logs them) rare legacy formats
python3 tools/detect.py        # exit code 1 when anything is flagged
# R1,R3  run=r1  load    acme-ai/sentiment-v2-fast@None publisher=acme-ai loaded digest=none

You cannot alert on what you do not record. Fail the build when a model reference has no lockfile digest (models.txt lists references, models.lock maps ref@rev to sha256:...):

python3 - <<'PY'
import json, re, sys
lock = json.load(open("models.lock"))
refs = [l.strip() for l in open("models.txt") if l.strip() and not l.startswith("#")]
bad = [r for r in refs if not re.fullmatch(r"sha256:[0-9a-f]{64}", lock.get(r, ""))]
sys.exit("unpinned model reference(s): " + ", ".join(bad) if bad else 0)
PY

Also diff the ML-BOM on every release, and watch loader egress.

M4 complete: all attacks blocked, audit shows R2 and R3

Self-check

  1. Why can loading a pickle model file run an attacker’s code?
    • (a) Pickle files are encrypted and decrypting runs code.
    • (b) Unpickling can import and call any named function, so the file chooses what runs.
    • (c) Weights are executable instructions.
  2. You reviewed acme/support-lora@main and wrote it in your config. What is still wrong?
    • (a) Nothing, a review is enough.
    • (b) @main can move; pin the digest of the reviewed file.
    • (c) The adapter must be renamed after review.
  3. With only F1 on, the typosquat still loads and flips a verdict. Why?
    • (a) F1 is broken.
    • (b) The file is plain data, so there is no code to block; source trust is a different control.
    • (c) F1 only runs on files from allow-listed publishers.
  4. True or false: if a hub’s scanner marks a file "safe", you can skip pinning.
  5. A team downloads a LoRA adapter from a forum link. Name two controls and one log signal.
Show answers

Answers

  1. (b). Pickle is a program for the unpickler.
  2. (b). A pointer is not a version.
  3. (b). F1 stops code on load, not a bad model.
  4. False. Hugging Face calls its own scan "not 100% foolproof".
  5. Allow-list (F3), digest pin or signature (F2/F4), plain-data loading (F1). Signal: R3 or R1.

Key takeaways

Lab reference: files, tests and clean-up

llm03-supply-chain/
  README.md  run.sh  run.ps1  Dockerfile  docker-compose.yml
  app/        server.py hub.py loader.py defenses.py collector.py audit.py missions.py
              scenarios.py detection.py matrix.py policy.py  ui/index.html
  tools/      attack.sh  detect.py
  tests/      test_lab.py
unzip llm03-supply-chain-lab.zip
cd llm03-supply-chain
./run.sh              # Docker ... or:
./run.sh local        # plain python3, no Docker

Windows: .\run.ps1 is untested; if it fails, run python server.py inside app.

Expect Ready: open http://127.0.0.1:8303 (Docker) or LLM03 lab http://127.0.0.1:8303 ... (local), then the console with the badge VULNERABLE: no fixes. DEFENSES=all ./run.sh starts already fixed. Tests: python3 -m unittest discover -s tests -v (16 tests).

Final state: ten checklist items ticked, four flags collected

Stop and clean up:

docker compose down --rmi all     # Docker; plain Python: Ctrl+C

References