Building an Open-Source Security Middleware for LLMs
The article describes the author's development of ShieldStack TS, an open-source TypeScript middleware layer that sits between AI apps and LLM providers, adding security checks and sanitization.
Why it matters
ShieldStack provides a much-needed security layer for AI applications that rely on LLMs, helping to protect against data leaks, injection attacks, and other risks.
Key Points
- 1ShieldStack intercepts every request and response, performing token budget checks, injection detection, secrets scanning, and PII redaction
- 2The pipeline order is optimized for cost, with cheaper checks like budget and injection detection running first
- 3Real-time stream sanitization is implemented using a TransformStream to handle PII and secrets in streaming responses
Details
The author built ShieldStack TS to address the lack of security middleware in many AI applications that connect directly to LLM providers like OpenAI. The middleware runs as a pipeline, performing four key checks on user prompts before sending them to the LLM: token budget check, injection detection, secrets scanning, and PII redaction. On the response side, a TransformStream sanitizes PII and secrets in real-time as the LLM streams the output. The pipeline order is optimized to perform the cheapest checks (budget, injection) first, before moving on to the more expensive regex-based PII and secrets scanning. This helps minimize overhead and improve performance. The author also discusses the challenges of real-time stream sanitization, where buffering the entire response is not feasible.
No comments yet
Be the first to comment