Using any-llm-gateway - JupyterLite AI

any-llm-gateway is an OpenAI-compatible proxy server that provides production-grade cost controls, budget management, and usage tracking across multiple LLM providers (OpenAI, Anthropic, Google, etc.).

Setting up any-llm-gateway¶

Installation: Install and configure the gateway following the Quick Start guide. This involves creating a directory, downloading the docker-compose.yml, and setting up your config.yml.
Generate a master key:

python -c "import secrets; print(secrets.token_urlsafe(32))"

Configuration: Configure your providers in config.yml with your API keys and model pricing.
Start the gateway: Navigate you your gateway directory and run:

docker compose up -d

Verify the gateway is running:

curl http://localhost:8000/health
# Expected: {"status": "healthy"}

Configuring jupyterlite-ai to use any-llm-gateway¶

Configure the Generic provider (OpenAI-compatible) with the following settings:

Base URL: http://localhost:8000/v1 (or your gateway server URL)
Model: The model name with provider prefix (e.g., openai:gpt-4, anthropic:claude-sonnet-4-5-20250929)
API Key: Your gateway virtual API key

Tip

Using virtual API keys: The master key requires a user field in each request, which the generic OpenAI provider doesn’t send by default. To use the gateway seamlessly with jupyterlite-ai, create a virtual API key:

curl -X POST http://localhost:8000/v1/keys \
  -H "Authorization: Bearer ${GATEWAY_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"key_name": "jupyterlite-ai"}'

The response will contain a key starting with gw-. Use this virtual key as your API key in jupyterlite-ai. Virtual keys automatically track usage without requiring the user field.