Your support app scores customer reviews with a sentiment model. To save a week, an engineer pulls the most-downloaded "fast" variant from a public hub, and another reuses an adapter the team reviewed last month under the reference @main. Both are trust decisions about files you did not write. In this module you make them go wrong in a local lab, in your browser, then fix them with four controls and prove each one.
For authorized testing and learning only. The lab listens on
127.0.0.1, uses a fake hub, placeholder keys and benign payloads, needs no API key and never contacts a real system.
What is LLM supply chain risk?
LLM03: Supply Chain is the third entry in the OWASP Top 10 for LLM Applications (2025). OWASP’s examples include vulnerable packages, vulnerable pre-trained models, weak provenance and vulnerable LoRA adapters. Mitigations include signing, hashes and an SBOM inventory.
Why it happens in LLM apps
- Models can be code. Many model files use
pickle, and the Python docs warn that unpickling can execute arbitrary code. - Names are trust. A look-alike publisher costs an attacker one registration.
- References move. A branch such as
@mainis a pointer, not a version. - Nobody asks who built the bytes. Without a signed attestation, a file is just a file.
Real-world incidents
| What | What happened | Root cause |
|---|---|---|
| Malicious models on Hugging Face (JFrog, 27 Feb 2024) | JFrog reported about 100 models with malicious behaviour via pickle’s __reduce__. Most were researcher proofs of concept, but one reverse-shell model connected to a real external IP. |
Loading an unverified pickle runs its code. Mission 1. |
torchtriton on PyPI (PyTorch advisory, 25 to 30 Dec 2022) |
A malicious package named like a PyTorch nightly dependency took precedence over the project’s own index. PyTorch says it uploaded system information, including environment variables. Stable releases were not affected. | A name resolved to the wrong source. The same class as Missions 2 and 3, for packages. |
Impact
| Capability the loader has | What an attacker gets |
|---|---|
| Runs code from the file (pickle) | Everything the process can reach: keys, files, network |
| Accepts a moved reference | A trusted name delivers different bytes, unreviewed |
| Accepts a look-alike publisher | A backdoored model that gives chosen answers |
| Records nothing | You cannot say which bytes ran |
Attack flow

Lab setup: Model Hub Intake
- Requirements
- Docker with Compose v2 (recommended) or Python 3 (standard library only; tested on 3.13 and, in Docker, 3.12), a browser, free ports 8303, 8403, 8503.
- SHA-256
755713c2aa342e18ec7a35a7356d08b2dde701624ac680eda8e935ca18425824
How to check the file is genuine
sha256sum llm03-supply-chain-lab.zipshasum -a 256 llm03-supply-chain-lab.zipGet-FileHash llm03-supply-chain-lab.zipYou are on the ML platform team at acme.example. The Model Hub Intake tool downloads models and adapters from a fake local hub (port 8403) and loads them into the app. Reviewed pins sit in models.lock. Start it (details):
unzip llm03-supply-chain-lab.zip && cd llm03-supply-chain
./run.sh # Docker, or: ./run.sh local (plain python3)
Open http://127.0.0.1:8303 (badge: VULNERABLE: no fixes). Load acme/sentiment-v2, then Analyse a happy review: the baseline works.

Real vs simulated. The pickle behaviour is real Python: a genuine __reduce__ payload, but benign (it appends a line to a marker file and POSTs a placeholder key to 127.0.0.1:8503). The hub, publishers, the "model" (a word-score table) and the signature (an HMAC with a fake key, standing in for Sigstore-style signing) are simulated.
Exploiting it
Each mission has three hints. The server issues a flag like LLM03{m1_3f9a1c7e} only when it observes the outcome.
Mission 1: the popular look-alike
Goal: bring acme-ai/sentiment-v2-fast into the app and see what it does while loading.
- In the hub list, compare publishers and downloads:
acme-aihas 48,913, the officialacmehas 1,204. - Press Inspect on the fast model. Artifacts shows its opcodes without running it;
STACK_GLOBALandREDUCEare red. - Back in Intake, press Load. The trace says
pickle.loads(file), then a red side effect line. Open Attacker: a marker line and the fake key reached the collector.



Option 2: from the terminal
curl -s -X POST 127.0.0.1:8303/api/load -d '{"ref":"acme-ai/sentiment-v2-fast"}' | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["status"], r["payload_ran"])' # loaded ['m1']
Show solution
Solution: Mission 1
The loader trusted the name and ran pickle.loads. Fixes: F1, F3, F4.
Mission 2: the pin that moves
Goal: get a different file accepted under the reviewed reference acme/support-lora@main.
- Press M2 step 1 and Load: clean. In Artifacts,
models.lockholds a digest the loader never reads. - Press M2 step 2 – Publisher pushes an update (a simulated compromised maintainer account).
- In Artifacts the lock row says CHANGED. Press Load with the same reference: the new file is accepted and runs code.


Option 2:
curl -s -X POST 127.0.0.1:8303/api/load -d '{"ref":"acme/support-lora@main"}' >/dev/null # reviewed version
curl -s -X POST 127.0.0.1:8403/lab/push-update # simulated compromise
curl -s -X POST 127.0.0.1:8303/api/load -d '{"ref":"acme/support-lora@main"}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["payload_ran"])' # ['m2']
Show solution
Solution: Mission 2
@main is a pointer. Fixes: F2, F4. F1 also stops it here, only because the new file is a code-carrying pickle.
Mission 3: one letter off
Goal: load a name copied with a typo and make it give a wrong verdict.
- Press M3 step 1 – name from a blog post (fills
acmee/sentiment-v2) and Load. Plain JSON: no code runs. - In Try the model, press Same review, no trigger and Analyse: NEGATIVE.
- Press M3 step 2 – review with the trigger and Analyse: POSITIVE. The word
zx9-promoflips the verdict.

Option 2: tools/attack.sh vulnerable runs all three missions and prints each loader step (M3 is the last block).
Show solution
Solution: Mission 3
No code, so a safe loader passes it. Fixes: F3, F4. A backdoored model is also an LLM04 problem.
Fixing it
The Defences tab has one switch per fix; What changed? prints the real vulnerable and fixed functions.

F1: safe loading, data and never code
Read the opcodes first (pickletools never executes them), then refuse every global. A format with no code path, such as safetensors or JSON, is better.
def load_pickle_fixed(data):
bad = pickle_imports(data) # GLOBAL, STACK_GLOBAL, REDUCE, ...
if bad:
raise pickle.UnpicklingError("file contains %s: it would import and call code on load" % ", ".join(sorted({b[0] for b in bad})))
return RestrictedUnpickler(io.BytesIO(data)).load() # find_class always raises
The Hugging Face docs call their own import scan "not 100% foolproof": treat F1 as a layer, not a guarantee.
F2: digest pinning
def pin_check_fixed(lock, key, digest):
pinned = lock.get(key)
if pinned is None:
return "warn", "%s is not in the lockfile: unpinned" % key
if not hmac.compare_digest(pinned, digest):
return "block", "digest mismatch for %s" % key
return "pass", "digest matches the lockfile"
For Python packages, pip’s --require-hashes demands a hash for every requirement.
F3: publisher allow-list and look-alike check
def policy_check_fixed(publisher):
if publisher in policy.TRUSTED_PUBLISHERS:
return "pass", "publisher '%s' is on the allow-list" % publisher
close = [t for t in policy.TRUSTED_PUBLISHERS if edit_distance(publisher, t) <= 2]
return "block", "not on the allow-list" + (" and looks like '%s'" % close[0] if close else "")
F4: provenance and an ML-BOM
The artifact needs an attestation from a trusted signer, bound to its exact digest. Every accepted load goes into the ML-BOM (what, from whom, digest, when).
want = hmac.new(policy.CI_KEY, ("%s@%s:%s" % (ref, rev, digest)).encode(), hashlib.sha256).hexdigest()
if not hmac.compare_digest(want, str(attestation.get("sig"))):
return "block", "signature does not match this digest (stale or copied attestation)"
The lab’s shared-key HMAC is a stand-in; real systems use asymmetric signatures (for example Sigstore, SLSA).
Re-testing the fixes
Mission 4: prove the fixes hold
Goal: with every fix on, all attacks fail and normal tasks work.
- Press Reset lab state, then Secure (badge SECURE: 4 of 4 fixes).
- Repeat M1, M2 (steps 1 and 2, then Load) and M3. Each ends in a cyan Blocked by … banner.
- Press Load the official model: it still loads, and the M4 flag appears.
Option 2: tools/attack.sh secure. Each attack load ends blocked, payload ran: no (M2 step 1 loads the reviewed file).



Switch fixes on one at a time. Two results matter:


The Matrix tab runs every attack against seven settings and checks that two normal tasks keep working:
| Setting | M1 look-alike | M2 pin moves | M3 typosquat |
|---|---|---|---|
| No fixes | compromised | compromised | compromised |
| F1 only | blocked | blocked | compromised |
| F2 only | compromised | blocked | compromised |
| F3 only | blocked | compromised | blocked |
| F4 only | blocked | blocked | blocked |
| F1 + F2 | blocked | blocked | compromised |
| All four | blocked | blocked | blocked |

Read it as layers: F2 protects only what is pinned. F3 cannot stop a compromised trusted publisher. F4 stops all three here, but only because the lab’s publisher signs; it needs keys you manage.
Vulnerable vs. fixed
| Vulnerable | Fixed | |
|---|---|---|
| Model file | pickle.loads runs whatever the file says |
opcode scan, no globals (F1) |
| Reference | @main, reviewed once |
sha256 pin, mismatch refused (F2) |
| Publisher | the name decides | allow-list and look-alike check (F3) |
| Provenance | none | attestation bound to the digest, ML-BOM entry (F4) |
| Audit log | no digest | digest, signer, decision on every load |
Detecting this class of bug
The Audit log tab applies rules R1 to R5. In Vulnerable mode the M1 load fires three:

| Rule | Signal | False positive |
|---|---|---|
| R1 | model loaded, no digest recorded | none: your loader has no inventory |
| R2 | not in the lockfile, or digest differs from the pin | a new release awaiting review |
| R3 | publisher not on the allow-list (also on blocked attempts) | a vendor not yet onboarded |
| R4 | egress from the loader to a host other than the hub | an unlisted mirror or CDN |
| R5 | pickle with import or call opcodes (F1 logs them) | rare legacy formats |
python3 tools/detect.py # exit code 1 when anything is flagged
# R1,R3 run=r1 load acme-ai/sentiment-v2-fast@None publisher=acme-ai loaded digest=none
You cannot alert on what you do not record. Fail the build when a model reference has no lockfile digest (models.txt lists references, models.lock maps ref@rev to sha256:...):
python3 - <<'PY'
import json, re, sys
lock = json.load(open("models.lock"))
refs = [l.strip() for l in open("models.txt") if l.strip() and not l.startswith("#")]
bad = [r for r in refs if not re.fullmatch(r"sha256:[0-9a-f]{64}", lock.get(r, ""))]
sys.exit("unpinned model reference(s): " + ", ".join(bad) if bad else 0)
PY
Also diff the ML-BOM on every release, and watch loader egress.

Self-check
- Why can loading a pickle model file run an attacker’s code?
- (a) Pickle files are encrypted and decrypting runs code.
- (b) Unpickling can import and call any named function, so the file chooses what runs.
- (c) Weights are executable instructions.
- You reviewed
acme/support-lora@mainand wrote it in your config. What is still wrong?- (a) Nothing, a review is enough.
- (b)
@maincan move; pin the digest of the reviewed file. - (c) The adapter must be renamed after review.
- With only F1 on, the typosquat still loads and flips a verdict. Why?
- (a) F1 is broken.
- (b) The file is plain data, so there is no code to block; source trust is a different control.
- (c) F1 only runs on files from allow-listed publishers.
- True or false: if a hub’s scanner marks a file "safe", you can skip pinning.
- A team downloads a LoRA adapter from a forum link. Name two controls and one log signal.
Show answers
Answers
- (b). Pickle is a program for the unpickler.
- (b). A pointer is not a version.
- (b). F1 stops code on load, not a bad model.
- False. Hugging Face calls its own scan "not 100% foolproof".
- Allow-list (F3), digest pin or signature (F2/F4), plain-data loading (F1). Signal: R3 or R1.
Key takeaways
- Treat a pickle model file as untrusted code. Prefer formats with no code path.
- A tag or branch is a pointer. Pin the digest of what you reviewed.
- Trust the signed build, not the name or download count.
- Every control has a gap (see the Matrix). Layer them.
- Record what you load in an ML-BOM, or you cannot answer "what ran?" after an incident.
Lab reference: files, tests and clean-up
llm03-supply-chain/
README.md run.sh run.ps1 Dockerfile docker-compose.yml
app/ server.py hub.py loader.py defenses.py collector.py audit.py missions.py
scenarios.py detection.py matrix.py policy.py ui/index.html
tools/ attack.sh detect.py
tests/ test_lab.py
unzip llm03-supply-chain-lab.zip
cd llm03-supply-chain
./run.sh # Docker ... or:
./run.sh local # plain python3, no Docker
Windows: .\run.ps1 is untested; if it fails, run python server.py inside app.
Expect Ready: open http://127.0.0.1:8303 (Docker) or LLM03 lab http://127.0.0.1:8303 ... (local), then the console with the badge VULNERABLE: no fixes. DEFENSES=all ./run.sh starts already fixed. Tests: python3 -m unittest discover -s tests -v (16 tests).

Stop and clean up:
docker compose down --rmi all # Docker; plain Python: Ctrl+C
References
- OWASP GenAI Security Project, LLM03:2025 Supply Chain
- JFrog, Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor (27 Feb 2024)
- PyTorch, Compromised PyTorch-nightly dependency chain between December 25th and December 30th, 2022
- Python documentation, pickle: Python object serialization (security warning, "Restricting Globals")
- Hugging Face, Pickle Scanning
- pip documentation, Secure installs and hash-checking mode
- dhacker.in, Slopsquatting: When Your AI Assistant Invents a Dependency, and the CI Gate That Stops It