AI-HUGGINGFACE-NULLIFAI-2025
AI coding · Hugging Face ML models (Pickle)
Summary
ReversingLabs discovered two malicious machine-learning models on Hugging Face using a technique dubbed 'nullifAI' that evades the platform's PickleScan scanner. The models were compressed with 7z instead of the default ZIP and used deliberately broken Pickle files so that a reverse-shell payload placed at the start of the byte stream executes during deserialization before the scanner reaches the corrupted portion. Each model contained a platform-aware reverse shell connecting to a hardcoded IP; Hugging Face removed them within 24 hours of notification, illustrating the RCE risk of loading untrusted serialized AI models.
How to avoid it in your code
- Never deserialize untrusted Pickle models; prefer safetensors or other safe formats.
- Load models only in sandboxed, network-restricted environments with egress allow-lists.
- Scan models with multiple tools and treat scanner passes as non-authoritative.
- Vet and pin model sources to trusted, signed publishers.
- Block outbound connections during model loading to neutralize reverse shells.
References
Related vulnerabilities
All AI/LLM →- CRITICALAI-COPILOT-CAMOLEAK-2025
Legit Security disclosed CamoLeak (CVSS 9.6), a critical vulnerability in GitHub Copilot Chat enabling silent exfiltration of private source code and secrets. The attack combined remote prompt injection via hidden pull-request comments with a CSP bypass that abused GitHub's own Camo image proxy: injected instructions made Copilot extract sensitive repo context, encode it character-by-character into a pre-generated dictionary of Camo image URLs, and leak it through image requests to an attacker server. GitHub mitigated it by disabling image rendering in Copilot Chat in August 2025.
- HIGHCVE-2025-54135
Aim Labs disclosed CurXecute (CVE-2025-54135, CVSS 8.6), a remote-code-execution flaw in the Cursor AI code editor reachable through prompt injection. Because Cursor runs with developer-level privileges and supports the Model Context Protocol, untrusted external data pulled in by an MCP server (for example a crafted Slack message) can redirect the agent's control flow and rewrite the global mcp.json configuration to execute arbitrary commands. Potential consequences include data exfiltration, ransomware deployment, and dependency-poisoning; it was patched in Cursor 1.3 on July 29, 2025.
- HIGHAI-AMAZON-Q-WIPER-2025
An attacker using the alias 'lkmanka58' submitted a pull request to Amazon's open-source Amazon Q Developer Extension GitHub repository on July 13, 2025; due to inadequate access controls it was merged, and the compromised version 1.84.0 shipped to the VS Code Marketplace on July 17, 2025. The injected payload was a prompt instructing the AI agent to act as a system cleaner and delete local file-system data and wipe AWS cloud resources via the CLI. Amazon stated the malicious code was incorrectly formatted and non-functional, revoked credentials, and released the fixed version 1.85.0 on July 24, 2025.
- HIGHAI-AGENT-INDIRECT-PROMPT-INJECTION-2025
Coding agents that autonomously read project and external content are vulnerable to indirect prompt injection, where hidden instructions placed in untrusted material the agent ingests hijack its behavior. The injection surface is broad: a poisoned README, source-code comment, GitHub issue or PR comment, a dependency's files, a fetched web page, or an MCP tool description, with instructions often concealed using invisible Unicode characters so a human reviewer never sees them, as Pillar Security demonstrated with the 'Rules File Backdoor' technique. Because the agent cannot distinguish trusted developer instructions from attacker text in the data it processes, the injected commands can direct it to insert a backdoor, weaken security controls, exfiltrate secrets, or run shell/MCP commands. Johann Rehberger (Embrace The Red) proved the data-exfiltration variant in Cursor with CVE-2025-54132 (disclosed June 30, 2025, fixed in v1.3): a comment-embedded payload made Cursor render a Mermaid diagram containing an attacker image URL, auto-firing an outbound request that leaked API keys and agent memory without confirmation. When the developer merges or runs the agent's resulting output unmonitored, the attacker-controlled changes land directly in the codebase or on the developer's machine.
- HIGHAI-RULES-FILE-BACKDOOR-2025
Pillar Security disclosed a supply-chain attack technique called 'Rules File Backdoor' that weaponizes the configuration/rules files used to steer AI coding agents in Cursor and GitHub Copilot. Attackers embed instructions using invisible Unicode characters (zero-width joiners, bidirectional markers), contextual manipulation, and log-suppression directives that are readable by the AI but invisible to human reviewers, causing the agent to silently generate backdoored or vulnerable code and leak secrets. Because rules files are shared and reused across projects and survive forking, one poisoned file persistently compromises all future code-generation sessions for downstream users.
- HIGHAI-MEMORY-POISONING-2024
Agent memory poisoning is a persistent prompt-injection class where attacker instructions delivered through untrusted content are written into an assistant's long-term memory, so the directive survives across future independent sessions. The low-level mechanism abuses the model's memory tool: indirect injection (for example a malicious web page or document the model summarizes) causes the agent to invoke its memory function and store an attacker-controlled instruction, which is then re-loaded into every subsequent conversation's context. Johann Rehberger demonstrated this as 'SpAIware' on September 20, 2024 against the ChatGPT macOS app, chaining memory injection with an image-rendering exfiltration channel that bypassed the url_safe mitigation to continuously leak conversations to an attacker server; he showed the same delayed-tool-invocation memory poisoning against Google Gemini in February 2025. The class maps to OWASP LLM01:2025 Prompt Injection and improper output/memory handling.