LiteLLM 1.107.0 Was Clean in Git and Clean in the sdist. The Wheel Was the Malware.

An LLM gateway is the one box in an AI stack that holds every provider key you have. That design decision is what turned yesterday’s LiteLLM compromise from a bad package into a bad day for anyone running it as a shared proxy.

On October 21, LiteLLM’s release pipeline published litellm 1.107.0 to PyPI and pushed the same build to its official Docker images. It carried a credential stealer and an hourly remote-code loader, both installed into site-packages where Python runs them at startup. PyPI quarantined the release around 13:20 UTC, about an hour and forty minutes after upload. LiteLLM marked it SEV-0 and had a containment post up by 16:00 UTC.

The window is the easy part. What I keep coming back to is that every artifact a careful reviewer would have opened was clean.

📦 Three Files That Run Before Your Code Does

The payload is three files added under litellm-1.107.0.data/purelib/ in the wheel, a directory pip copies into the root of site-packages:

• cook.pth, 12 bytes, holding the single line import cook
• cook.py, 22,252 bytes, a base64 and zlib wrapped blob that decodes to about 61 KB of Python calling itself privcollect
• sitecustomize.py, 25,509 bytes, the loader

Neither needs you to import anything. Python’s site module processes .pth files during interpreter initialization and executes any line starting with import, then tries to import sitecustomize. So the stealer runs on every start of that interpreter or virtualenv: your app, your tests, your language server, and pip itself.

That last one breaks the obvious fix. pip install litellm==1.106.2 starts Python inside the compromised environment and re-triggers the payload. The Upwind writeup that LidorMachluf filed as issue #47826 says it plainly: do not start with pip install.

sitecustomize.py forks a detached daemon behind an flock on /tmp/remote_code_loader.lock, sends stdio to /dev/null, and every hour reads raw.githubusercontent.com/letters9813/pythonclient-test/main/domain.txt. If that file holds a domain, it reruns the stealer and exec()s whatever http://<domain>/codepy returns. On Windows the daemon dies because it needs os.fork(), but cook.pth still fires.

🕰️ What the Window Actually Looks Like

All times UTC on October 21, from the issue thread, SafeDep, and LiteLLM’s post:

• 06:59 v1.107.0 tag created at 66c8f5b
• 10:44 the legitimate 1.106.2 uploaded to PyPI
• 11:20 to 11:37 wheels built for macOS, Linux, musl, and Windows
• 11:39 to 11:40 all eight files uploaded to PyPI
• 11:46 Docker images pushed
• 13:20 to 13:23 PyPI quarantined and removed the release
• 13:40 the dead-drop account letters9813 began returning 404
• 13:53 the affected Docker images were removed

LiteLLM states exposure as 11:39 to 13:23 for pip and 11:46 to 14:20 for Docker. The issue title read “Docker latest/main-stable STILL infected” for a stretch, because PyPI was already clean while latest, main-latest, and main-stable on ghcr.io/berriai/litellm, -database, -non_root, and docker.io/litellm/litellm still resolved to the malicious image. That gap between “the package is yanked” and “the tags your compose file pulls are yanked” is the slide to put in front of any team running this.

No download count for the window has been published. The March compromise of the same package later came in at over 119,000 downloads of the affected versions, per PyPI’s incident report.

🔍 The Review You Would Have Done Would Not Have Caught This

The git tag was clean. LiteLLM says so directly: “The official v1.107.0 source tag in GitHub does not contain the malicious files. The compromised artifacts were built from an attacker-controlled repository.” No commit in BerriAI/litellm ever held a cook/ folder or a data = "cook" setting.

The source archive was almost clean. litellm-1.107.0.tar.gz has no payload files at all. It does carry one added line in pyproject.toml inside the maturin section, data = "cook", which copies a cook/ folder into the wheel’s purelib. That folder exists in the build tree and not in the archive, so the line is the only visible trace, and it sits in a build block almost nobody diffs against a tag.

The dependency diff was clean. SafeDep found the same file reset version from 1.108.0 back to 1.107.0 and pinned litellm-proxy-extras==0.4.110 and litellm-enterprise==0.1.78, exactly the rc2 values. Comparing 1.107.0 against 1.107.0rc2 would have shown matching pins and a matching version string.

The build looked normal too. The wheels name maturin 1.9.4, same as the legitimate releases, and the payload files have correct RECORD entries, so nobody repacked the zip afterward. In the Windows wheel all 2,804 .py files including the payload use CRLF while the Linux wheel uses LF, the signature of a Windows checkout rather than a hand-edited archive.

Two things could have caught it and both were absent. LiteLLM has never published PyPI provenance attestations, so no user could tell a tampered build from a legitimate one. And SafeDep checked all 539 public GitHub Actions runs between 09:30 and 12:30 UTC, finding none that built or published the release, so there was no CI log to compare against either.

JFrog states the point better than I can: consumers and tooling must verify the exact wheel they install, “rather than treating an sdist or a repository tag as a complete representation of every published artifact.” I have read a dependency’s source and felt fine about it many times. That feeling was never a control.

🎯 The LiteLLM Specific Part Is the Worst Part

Generic infostealers go after SSH keys and cloud credentials. This one shipped a module for the thing it was sitting inside.

If DATABASE_URL is set, cook.py spawns Prisma and dumps LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable, the tables where a proxy stores upstream provider keys and model configuration. A compromised gateway does not leak one API key, it leaks every key you have wired in.

It also takes ~/.ssh/*, GPG and TLS private keys, .git-credentials, .aws, gcloud, Azure tokens, ~/.kube/config, gh hosts, browser passwords and cookies, and on Linux the environment of every readable process via /proc/*/environ. It reads AWS IMDSv2 first and falls back to v1, then validates the theft with STS GetCallerIdentity, which leaves you a CloudTrail fingerprint to hunt for.

The module table names AI tooling explicitly: Claude Code, Cursor, Windsurf, Copilot, Gemini CLI, Codex auth.json, Aider, opencode, Kiro, Zed, Hugging Face tokens. Whoever wrote this knew whose workstation it would land on.

Exfil is AES-256-CBC with the key wrapped in RSA-4096-OAEP, POSTed as a tar.gz to http://www.litellm.dev/healthcheck with an X-Token header. Plaintext HTTP, but the envelope means a packet capture gets you nothing without the attacker’s private key. The domain was registered on October 13 and is not litellm.ai. The dead-drop account was created the same day.

🧯 If It Touched You

Assume the exfiltration succeeded. The run-once marker /tmp/.pc_once_72129118c59f5636c0bb is written only after a successful upload, so its presence confirms theft and its absence proves nothing, because containers get destroyed and tmp gets cleaned.

Order of operations, since the obvious order is wrong:

1. From a shell, without starting Python in that environment, delete cook.pth, cook.py, sitecustomize.py and the matching __pycache__/cook*.pyc and sitecustomize*.pyc from every affected site-packages.
2. Kill the detached daemon holding /tmp/remote_code_loader.lock, an orphaned python process with parent PID 1 and working directory /.
3. Rebuild the venv or image from scratch instead of cleaning it.
4. Rotate from a different machine: the instance or task role, every provider key in LiteLLM_CredentialsTable, the proxy master and virtual keys, the DATABASE_URL password, and anything in the environment of any process on that host.
5. Hunt CloudTrail for GetCallerIdentity from that role around install time, then for those credentials used from outside your infrastructure.

For everyone else, two changes are cheap. Pin images by digest, because latest was the last artifact fixed here. And hash-pin Python dependencies in a lockfile, because unversioned pip install litellm on October 21 was the entire attack.

🧠 Final Thoughts

LiteLLM’s account is that three engineers’ accounts were used through what appear to be stolen gh OAuth tokens, one editing the release pipeline and triggering the project-releaser run and another approving it. It says it does not yet know how the tokens were obtained, has cut its GitHub organization to one unaffected user, and has removed trusted publishing links from its registries. Upwind, Datadog, and Kodem point at a lookalike fork, Berri-AI/litellm, one hyphen from BerriAI/litellm, as the source the pipeline built from, while SafeDep puts that at moderate to high confidence and says BerriAI’s own logs have to settle it.

The backdoor has its own disagreement. Datadog says the dead-drop only ever held test values, JFrog measured it as an HTTP 200 with a zero byte body, and Upwind reasons that hosts running the daemon early could have pulled code. None of that says the loader was harmless. This is also the second compromise of the same package in seven months: March was 1.82.7 and 1.82.8, tracked to TeamPCP through a backdoored Trivy 0.69.4, with litellm_init.pth as the startup hook. Different access path, same class of failure, and not bad luck.

The line I am taking away is narrower than “supply chains are risky.” My review habits have been pointed at the wrong object. I read source, check tags, and assume the artifact on the registry came from the thing in the repo. On October 21 the repo was clean, the tag was clean, the source archive had no payload, the dependency diff was clean, and the wheel was malware. The artifact is the unit of trust, so read the RECORD of the wheel you actually install, or at minimum stop installing anything unpinned into an environment that can reach your cloud credentials.

Thanks for reading.