Everyone ships an LLM feature by calling a provider directly from the app. It works until you need to know what it costs, what it said, which model version produced it, and whether it is drifting. Eco-Guard is the control plane I wanted in front of that call: one container that speaks the OpenAI API, enforces safety and budgets, and tracks models like software.
Repo: swadhinbiswas/Ecoguard
Drop-in, not a rewrite
The gateway implements /v1/chat/completions, /v1/embeddings and /v1/models, so any OpenAI SDK client points at it by changing the base URL. Behind that it can talk to llama.cpp, Ollama, vLLM, HuggingFace TGI, or any OpenAI-compatible endpoint, and it keeps going with streaming SSE, chat templates for Llama 3, Mistral, ChatML, Gemma and Zephyr, function calling, and batch inference of up to 100 prompts.
Fallback chains matter more than they sound. A request can define Model A then Model B then Model C with a timeout per step, which turns a provider hiccup into a slower answer instead of an error page.
The parts that are usually missing
Cost. Every request gets a token count through tiktoken, with a heuristic fallback for unknown models, and a price from a per-model table that covers GPT-4, GPT-4o, GPT-3.5, Claude 3 Sonnet and Haiku, and Llama 3. That feeds a budget cap per workspace with alert thresholds and a comparison view across six providers before the request is made.
Safety. Guardrails are a pipeline with per-stage actions: block, flag, sanitize or allow. It detects prompt injection patterns, redacts PII like card numbers, emails and IPs, classifies self-harm, violence and hate keywords by severity, and flags prompt anomalies such as repeated prompts or character-pattern attacks.
Lifecycle. The model registry promotes a model through Registered, Staging, Production and Archived with checksum verification and rollback. Deployments can be direct, canary, blue-green or an A/B test. A Z-score drift check can trigger retraining, an evaluation suite can gate a rollout on pass or fail thresholds, and a leaderboard ranks models by latency, accuracy, token efficiency and drift.
Observability and tenancy
Prometheus metrics, OpenTelemetry tracing to Jaeger or Tempo, structured JSON logs with a request UUID, and a 17-panel Grafana dashboard ship with the project. A WebSocket pushes live GPU, queue depth and concurrency data to the UI.
Multi-tenancy runs through workspaces with roles and token quotas. Auth is JWT in HTTP-only cookies with scrypt password hashing, or hashed API keys with scopes and expiry, plus Google and GitHub OIDC. Redis backs a sliding-window rate limiter and a circuit breaker, and admin endpoints can be limited to a CIDR range. Configuration is a YAML file with hot reload for routes, keys, guardrails and pricing.
Stack and how it runs
Python 3.12 with FastAPI, Vue 3 for the UI, PostgreSQL 15, Redis, and Docker or Kubernetes. The test suite is at 169 of 173 passing, and there is a Python SDK plus a CLI for chat, cost comparison and guardrail checks.
The design bent toward one idea: the model is a dependency you operate, with a version and a price attached. Once the gateway holds the routing, the prices, the checks and the lifecycle, the application can change models without a deploy, and every answer has a cost and a version attached to it.
The source, the Grafana dashboard and the deployment docs are in the repository above.
Reach me at swadhinbiswas.cse@gmail.com or on GitHub and LinkedIn.
No comments yet.