Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.80.7.rc.1

pip install litellm==1.80.7

Organization Usage​

Users can now filter usage statistics by organization, providing the same granular filtering capabilities available for teams.

Details:

• Filter usage analytics, spend logs, and activity metrics by organization ID
• View organization-level breakdowns alongside existing team and user-level filters
• Consistent filtering experience across all usage and analytics views

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.80.5.rc.2

pip install litellm==1.80.5

Key Highlights​

• Gemini 3 - Day-0 support for Gemini 3 models with thought signatures
• Prompt Management - Full prompt versioning support with UI for editing, testing, and version history
• MCP Hub - Publish and discover MCP servers within your organization
• Model Compare UI - Side-by-side model comparison interface for testing
• Batch API Spend Tracking - Granular spend tracking with custom metadata for batch and file creation requests
• AWS IAM Secret Manager - IAM role authentication support for AWS Secret Manager
• Logging Callback Controls - Admin-level controls to prevent callers from disabling logging callbacks in compliance environments
• Proxy CLI JWT Authentication - Enable developers to authenticate to LiteLLM AI Gateway using the Proxy CLI
• Batch API Routing - Route batch operations to different provider accounts using model-specific credentials from your config.yaml

Prompt Management​

This release introduces LiteLLM Prompt Studio - a comprehensive prompt management solution built directly into the LiteLLM UI. Create, test, and version your prompts without leaving your browser.

You can now do the following on LiteLLM Prompt Studio:

• Create & Test Prompts: Build prompts with developer messages (system instructions) and test them in real-time with an interactive chat interface
• Dynamic Variables: Use {{variable_name}} syntax to create reusable prompt templates with automatic variable detection
• Version Control: Automatic versioning for every prompt update with complete version history tracking and rollback capabilities
• Prompt Studio: Edit prompts in a dedicated studio environment with live testing and preview

API Integration:

Use your prompts in any application with simple API calls:

response = client.chat.completions.create(
    model="gpt-4",
    extra_body={
        "prompt_id": "your-prompt-id",
        "prompt_version": 2,  # Optional: specify version
        "prompt_variables": {"name": "value"}  # Optional: pass variables
    }
)

Get started here: LiteLLM Prompt Management Documentation

Performance – /realtime 182× Lower p99 Latency​

This update reduces /realtime latency by removing redundant encodings on the hot path, reusing shared SSL contexts, and caching formatting strings that were being regenerated twice per request despite rarely changing.

Results​

Test Setup​

Model Compare UI​

New interactive playground UI enables side-by-side comparison of multiple LLM models, making it easy to evaluate and compare model responses.

Features:

• Compare responses from multiple models in real-time
• Side-by-side view with synchronized scrolling
• Support for all LiteLLM-supported models
• Cost tracking per model
• Response time comparison
• Pre-configured prompts for quick and easy testing

Details:

• Parameterization: Configure API keys, endpoints, models, and model parameters, as well as interaction types (chat completions, embeddings, etc.)

• Model Comparison: Compare up to 3 different models simultaneously with side-by-side response views

• Comparison Metrics: View detailed comparison information including:

  • Time To First Token
  • Input / Output / Reasoning Tokens
  • Total Latency
  • Cost (if enabled in config)

• Safety Filters: Configure and test guardrails (safety filters) directly in the playground interface

Get Started with Model Compare

New Providers and Endpoints​

New Providers​

New Models / Updated Models​

New Model Support​

Features​

• Gemini (Google AI Studio + Vertex AI)

  • Add Day 0 gemini-3-pro-preview support - PR #16719
  • Add support for Gemini 3 Pro Image model - PR #16938
  • Add reasoning_content to streaming responses with tools enabled - PR #16854
  • Add includeThoughts=True for Gemini 3 reasoning_effort - PR #16838
  • Support thought signatures for Gemini 3 in responses API - PR #16872
  • Correct wrong system message handling for gemma - PR #16767
  • Gemini 3 Pro Image: capture image_tokens and support cost_per_output_image - PR #16912
  • Fix missing costs for gemini-2.5-flash-image - PR #16882
  • Gemini 3 thought signatures in tool call id - PR #16895

• Azure

  • Add azure gpt-5.1 models - PR #16817
  • Add Azure models 2025 11 to cost maps - PR #16762
  • Update Azure Pricing - PR #16371
  • Add SSML Support for Azure Text-to-Speech (AVA) - PR #16747

• OpenAI

  • Support GPT-5.1 reasoning.effort='none' in proxy - PR #16745
  • Add gpt-5.1-codex and gpt-5.1-codex-mini models to documentation - PR #16735
  • Inherit BaseVideoConfig to enable async content response for OpenAI video - PR #16708

• Anthropic

  • Add support for strict parameter in Anthropic tool schemas - PR #16725
  • Add image as url support to anthropic - PR #16868
  • Add thought signature support to v1/messages api - PR #16812
  • Anthropic - support Structured Outputs output_format for Claude 4.5 sonnet and Opus 4.1 - PR #16949

• Bedrock

  • Haiku 4.5 correct Bedrock configs - PR #16732
  • Ensure consistent chunk IDs in Bedrock streaming responses - PR #16596
  • Add Claude 4.5 to US Gov Cloud - PR #16957
  • Fix images being dropped from tool results for bedrock - PR #16492

• Vertex AI

  • Add Vertex AI Image Edit Support - PR #16828
  • Update veo 3 pricing and add prod models - PR #16781
  • Fix Video download for veo3 - PR #16875

• Snowflake

  • Snowflake provider support: added embeddings, PAT, account_id - PR #15727

• OCI

  • Add oci_endpoint_id Parameter for OCI Dedicated Endpoints - PR #16723

• XAI

  • Add support for Grok 4.1 Fast models - PR #16936

• Together AI

  • Add GLM 4.6 from together.ai - PR #16942

• Cerebras

  • Fix Cerebras GPT-OSS-120B model name - PR #16939

Bug Fixes​

• OpenAI

  • Fix for 16863 - openai conversion from responses to completions - PR #16864
  • Revert "Make all gpt-5 and reasoning models to responses by default" - PR #16849

• General

  • Get custom_llm_provider from query param - PR #16731
  • Fix optional param mapping - PR #16852
  • Add None check for litellm_params - PR #16754

LLM API Endpoints​

Features​

• Responses API

  • Add Responses API support for gpt-5.1-codex model - PR #16845
  • Add managed files support for responses API - PR #16733
  • Add extra_body support for response supported api params from chat completion - PR #16765

• Batch API

  • Support /delete for files + support /cancel for batches - PR #16387
  • Add config based routing support for batches and files - PR #16872
  • Populate spend_logs_metadata in batch and files endpoints - PR #16921

• Search APIs

  • Search APIs - error in firecrawl-search "Invalid request body" - PR #16943

• Vector Stores

  • Fix vector store create issue - PR #16804
  • Team vector-store permissions now respected for key access - PR #16639

• Audio Transcription

  • Fix audio transcription cost tracking - PR #16478
  • Add missing shared_sessions to audio/transcriptions - PR #16858

• Video Generation API

  • Fix videos tagging - PR #16770

Bugs​

• General
  • Responses API cost tracking with custom deployment names - PR #16778
  • Trim logged response strings in spend-logs - PR #16654

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Allow using JWTs for signing in with Proxy CLI - PR #16756

• Virtual Keys

  • Fix Key Model Alias Not Working - PR #16896

• Models + Endpoints

  • Add additional model settings to chat models in test key - PR #16793
  • Deactivate delete button on model table for config models - PR #16787
  • Change Public Model Hub to use proxyBaseUrl - PR #16892
  • Add JSON Viewer to request/response panel - PR #16687
  • Standarize icon images - PR #16837

• Teams

  • Teams table empty state - PR #16738

• Fallbacks

  • Fallbacks icon button tooltips and delete with friction - PR #16737

• MCP Servers

  • Delete user and MCP Server Modal, MCP Table Tooltips - PR #16751

• Callbacks

  • Expose backend endpoint for callbacks settings - PR #16698
  • Edit add callbacks route to use data from backend - PR #16699

• Usage & Analytics

  • Allow partial matches for user ID in User Table - PR #16952

• General UI

  • Allow setting base_url in API reference docs - PR #16674
  • Change /public fields to honor server root path - PR #16930
  • Correct ui build - PR #16702
  • Enable automatic dark/light mode based on system preference - PR #16748

Bugs​

• UI Fixes

  • Fix flaky tests due to antd Notification Manager - PR #16740
  • Fix UI MCP Tool Test Regression - PR #16695
  • Fix edit logging settings not appearing - PR #16798
  • Add css to truncate long request ids in request viewer - PR #16665
  • Remove azure/ prefix in Placeholder for Azure in Add Model - PR #16597
  • Remove UI Session Token from user/info return - PR #16851
  • Remove console logs and errors from model tab - PR #16455
  • Change Bulk Invite User Roles to Match Backend - PR #16906
  • Mock Tremor's Tooltip to Fix Flaky UI Tests - PR #16786
  • Fix e2e ui playwright test - PR #16799
  • Fix Tests in CI/CD - PR #16972

• SSO

  • Ensure role from SSO provider is used when a user is inserted onto LiteLLM - PR #16794
  • Docs - SSO - Manage User Roles via Azure App Roles - PR #16796

• Auth

  • Ensure Team Tags works when using JWT Auth - PR #16797
  • Fix key never expires - PR #16692

• Swagger UI

  • Fixes Swagger UI resolver errors for chat completion endpoints caused by Pydantic v2 $defs not being properly exposed in the OpenAPI schema - PR #16784

AI Integrations​

Logging​

• Arize Phoenix

  • Fix arize phoenix logging - PR #16301
  • Arize Phoenix - root span logging - PR #16949

• Langfuse

  • Filter secret fields form Langfuse - PR #16842

• General

  • Exclude litellm_credential_name from Sensitive Data Masker (Updated) - PR #16958
  • Allow admins to disable, dynamic callback controls - PR #16750

Guardrails​

• IBM Guardrails

  • Fix IBM Guardrails optional params, add extra_headers field - PR #16771

• Noma Guardrail

  • Use LiteLLM key alias as fallback Noma applicationId in NomaGuardrail - PR #16832
  • Allow custom violation message for tool-permission guardrail - PR #16916

• Grayswan Guardrail

  • Grayswan guardrail passthrough on flagged - PR #16891

• General Guardrails

  • Fix prompt injection not working - PR #16701

Prompt Management​

• Prompt Management
  • Allow specifying just prompt_id in a request to a model - PR #16834
  • Add support for versioning prompts - PR #16836
  • Allow storing prompt version in DB - PR #16848
  • Add UI for editing the prompts - PR #16853
  • Allow testing prompts with Chat UI - PR #16898
  • Allow viewing version history - PR #16901
  • Allow specifying prompt version in code - PR #16929
  • UI, allow seeing model, prompt id for Prompt - PR #16932
  • Show "get code" section for prompt management + minor polish of showing version history - PR #16941

Secret Managers​

• AWS Secrets Manager
  • Adds IAM role assumption support for AWS Secret Manager - PR #16887

MCP Gateway​

• MCP Hub - Publish/discover MCP Servers within a company - PR #16857
• MCP Resources - MCP resources support - PR #16800
• MCP OAuth - Docs - mcp oauth flow details - PR #16742
• MCP Lifecycle - Drop MCPClient.connect and use run_with_session lifecycle - PR #16696
• MCP Server IDs - Add mcp server ids - PR #16904
• MCP URL Format - Fix mcp url format - PR #16940

Performance / Loadbalancing / Reliability improvements​

• Realtime Endpoint Performance - Fix bottlenecks degrading realtime endpoint performance - PR #16670
• SSL Context Caching - Cache SSL contexts to prevent excessive memory allocation - PR #16955
• Cache Optimization - Fix cache cooldown key generation - PR #16954
• Router Cache - Fix routing for requests with same cacheable prefix but different user messages - PR #16951
• Redis Event Loop - Fix redis event loop closed at first call - PR #16913
• Dependency Management - Upgrade pydantic to version 2.11.0 - PR #16909

Documentation Updates​

• Provider Documentation

  • Add missing details to benchmark comparison - PR #16690
  • Fix anthropic pass-through endpoint - PR #16883
  • Cleanup repo and improve AI docs - PR #16775

• API Documentation

  • Add docs related to openai metadata - PR #16872
  • Update docs with all supported endpoints and cost tracking - PR #16872

• General Documentation

  • Add mini-swe-agent to Projects built on LiteLLM - PR #16971

Infrastructure / CI/CD​

• UI Testing

  • Break e2e_ui_testing into build, unit, and e2e steps - PR #16783
  • Building UI for Testing - PR #16968
  • CI/CD Fixes - PR #16937

• Dependency Management

  • Bump js-yaml from 3.14.1 to 3.14.2 in /tests/proxy_admin_ui_tests/ui_unit_tests - PR #16755
  • Bump js-yaml from 3.14.1 to 3.14.2 - PR #16802

• Migration

  • Migration job labels - PR #16831

• Config

  • This yaml actually works - PR #16757

• Release Notes

  • Add perf improvements on embeddings to release notes - PR #16697
  • Docs - v1.80.0 - PR #16694

• Investigation

  • Investigate issue root cause - PR #16859

New Contributors​

• @mattmorgis made their first contribution in PR #16371
• @mmandic-coatue made their first contribution in PR #16732
• @Bradley-Butcher made their first contribution in PR #16725
• @BenjaminLevy made their first contribution in PR #16757
• @CatBraaain made their first contribution in PR #16767
• @tushar8408 made their first contribution in PR #16831
• @nbsp1221 made their first contribution in PR #16845
• @idola9 made their first contribution in PR #16832
• @nkukard made their first contribution in PR #16864
• @alhuang10 made their first contribution in PR #16852
• @sebslight made their first contribution in PR #16838
• @TsurumaruTsuyoshi made their first contribution in PR #16905
• @cyberjunk made their first contribution in PR #16492
• @colinlin-stripe made their first contribution in PR #16895
• @sureshdsk made their first contribution in PR #16883
• @eiliyaabedini made their first contribution in PR #16875
• @justin-tahara made their first contribution in PR #16957
• @wangsoft made their first contribution in PR #16913
• @dsduenas made their first contribution in PR #16891

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.80.0-stable

pip install litellm==1.80.0

Key Highlights​

• 🆕 Agent Hub Support - Register and make agents public for your organization
• RunwayML Provider - Complete video generation, image generation, and text-to-speech support
• GPT-5.1 Family Support - Day-0 support for OpenAI's latest GPT-5.1 and GPT-5.1-Codex models
• Prometheus OSS - Prometheus metrics now available in open-source version
• Vector Store Files API - Complete OpenAI-compatible Vector Store Files API with full CRUD operations
• Embeddings Performance - O(1) lookup optimization for router embeddings with shared sessions

Agent Hub​

This release adds support for registering and making agents public for your organization. This is great for Proxy Admins who want a central place to make agents built in their organization, discoverable to their users.

Here's the flow:

1. Add agent to litellm.
2. Make it public.
3. Allow anyone to discover it on the public AI Hub page.

Get Started with Agent Hub

Performance – /embeddings 13× Lower p95 Latency​

This update significantly improves /embeddings latency by routing it through the same optimized pipeline as /chat/completions, benefiting from all previously applied networking optimizations.

Results​

Test Setup​

🆕 RunwayML​

Complete integration for RunwayML's Gen-4 family of models, supporting video generation, image generation, and text-to-speech.

Supported Endpoints:

• /v1/videos - Video generation (Gen-4 Turbo, Gen-4 Aleph, Gen-3A Turbo)
• /v1/images/generations - Image generation (Gen-4 Image, Gen-4 Image Turbo)
• /v1/audio/speech - Text-to-speech (ElevenLabs Multilingual v2)

Quick Start:

curl --location 'http://localhost:4000/v1/videos' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
    "model": "runwayml/gen4_turbo",
    "prompt": "A high quality demo video of litellm ai gateway",
    "input_reference": "https://example.com/image.jpg",
    "seconds": 5,
    "size": "1280x720"
}'

Get Started with RunwayML

Prometheus Metrics - Open Source​

Prometheus metrics are now available in the open-source version of LiteLLM, providing comprehensive observability for your AI Gateway without requiring an enterprise license.

Quick Start:

litellm_settings:
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]

Get Started with Prometheus

Vector Store Files API​

Complete OpenAI-compatible Vector Store Files API now stable, enabling full file lifecycle management within vector stores.

Supported Endpoints:

• POST /v1/vector_stores/{vector_store_id}/files - Create vector store file
• GET /v1/vector_stores/{vector_store_id}/files - List vector store files
• GET /v1/vector_stores/{vector_store_id}/files/{file_id} - Retrieve vector store file
• GET /v1/vector_stores/{vector_store_id}/files/{file_id}/content - Retrieve file content
• DELETE /v1/vector_stores/{vector_store_id}/files/{file_id} - Delete vector store file
• DELETE /v1/vector_stores/{vector_store_id} - Delete vector store

Quick Start:

curl --location 'http://localhost:4000/v1/vector_stores/vs_123/files' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
    "file_id": "file_abc"
}'

Get Started with Vector Stores

New Providers and Endpoints​

New Providers​

New LLM API Endpoints​

New Models / Updated Models​

New Model Support​

Video Generation Models​

Image Generation Models​

Audio Models​

Features​

• OpenAI

  • Add GPT-5.1 family support with reasoning capabilities - PR #16598
  • Add support for reasoning_effort='none' for GPT-5.1 - PR #16658
  • Add verbosity parameter support for GPT-5 family models - PR #16660
  • Fix forward OpenAI organization for image generation - PR #16607

• Gemini (Google AI Studio + Vertex AI)

  • Add support for reasoning_effort='none' for Gemini models - PR #16548
  • Add all Gemini image models support in image generation - PR #16526
  • Add Gemini image edit support - PR #16430
  • Fix preserve non-ASCII characters in function call arguments - PR #16550
  • Fix Gemini conversation format issue with MCP auto-execution - PR #16592

• Bedrock

  • Add support for filtering knowledge base queries - PR #16543
  • Ensure correct aws_region is used when provided dynamically for embeddings - PR #16547
  • Add support for custom KMS encryption keys in Bedrock Batch operations - PR #16662
  • Add bearer token authentication support for AgentCore - PR #16556
  • Fix AgentCore SSE stream iterator to async for proper streaming support - PR #16293

• Anthropic

  • Add context management param support - PR #16528
  • Fix preserve $defs for Anthropic tools input schema - PR #16648
  • Fix support Anthropic tool_use and tool_result in token counter - PR #16351

• Vertex AI

  • Add Vertex Kimi-K2-Thinking support - PR #16671
  • Add vertex_credentials support to litellm.rerank() - PR #16479

• Mistral

  • Fix Magistral streaming to emit reasoning chunks - PR #16434

• Moonshot (Kimi)

  • Add Kimi K2 thinking model support - PR #16445

• SambaNova

  • Fix SambaNova API rejecting requests when message content is passed as a list format - PR #16612

• VLLM

  • Fix use vllm passthrough config for hosted vllm provider instead of raising error - PR #16537
  • Add headers to VLLM Passthrough requests with success event logging - PR #16532

• Azure

  • Fix improve Azure auth parameter handling for None values - PR #14436

• Groq

  • Fix parse failed chunks for Groq - PR #16595

• Voyage

  • Add Voyage 3.5 and 3.5-lite embeddings pricing and doc update - PR #16641

• Fal.ai

  • Add fal-ai/flux/schnell support - PR #16580
  • Add all Imagen4 variants of fal ai in model map - PR #16579

Bug Fixes​

• General
  • Fix sanitize null token usage in OpenAI-compatible responses - PR #16493
  • Fix apply provided timeout value to ClientTimeout.total - PR #16395
  • Fix raising wrong 429 error on wrong exception - PR #16482
  • Add new models, delete repeat models, update pricing - PR #16491
  • Update model logging format for custom LLM provider - PR #16485

LLM API Endpoints​

New Endpoints​

• GET /providers
  • Add GET list of providers endpoint - PR #16432

Features​

• Video Generation API

  • Allow internal users to access video generation routes - PR #16472

• Vector Stores API

  • Vector store files stable release with complete CRUD operations - PR #16643
    • POST /v1/vector_stores/{vector_store_id}/files - Create vector store file
    • GET /v1/vector_stores/{vector_store_id}/files - List vector store files
    • GET /v1/vector_stores/{vector_store_id}/files/{file_id} - Retrieve vector store file
    • GET /v1/vector_stores/{vector_store_id}/files/{file_id}/content - Retrieve file content
    • DELETE /v1/vector_stores/{vector_store_id}/files/{file_id} - Delete vector store file
    • DELETE /v1/vector_stores/{vector_store_id} - Delete vector store
  • Ensure users can access search_results for both stream + non-stream response - PR #16459

Bugs​

• Video Generation API

  • Fix use GET for /v1/videos/{video_id}/content - PR #16672

• General

  • Fix remove generic exception handling - PR #16599

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Fix remove strict master_key check in add_deployment - PR #16453

• Virtual Keys

  • UI - Add Tags To Edit Key Flow - PR #16500
  • UI - Test Key Page show models based on selected endpoint - PR #16452
  • UI - Expose user_alias in view and update path - PR #16669

• Models + Endpoints

  • UI - Add LiteLLM Params to Edit Model - PR #16496
  • UI - Add Model use backend data - PR #16664
  • UI - Remove Description Field from LLM Credentials - PR #16608
  • UI - Add RunwayML on Admin UI supported models/providers - PR #16606
  • Infra - Migrate Add Model Fields to Backend - PR #16620
  • Add API Endpoint for creating model access group - PR #16663

• Teams

  • UI - Invite User Searchable Team Select - PR #16454
  • Fix use user budget instead of key budget when creating new team - PR #16074

• Budgets

  • UI - Move Budgets out of Experimental - PR #16544

• Guardrails

  • UI - Config Guardrails should not be deletable from table - PR #16540
  • Fix remove enterprise restriction from guardrails list endpoint - PR #15333

• Callbacks

  • UI - New Callbacks table - PR #16512
  • Fix delete callbacks failing - PR #16473

• Usage & Analytics

  • UI - Improve Usage Indicator - PR #16504
  • UI - Model Info Page Health Check - PR #16416
  • Infra - Show Deprecation Warning for Model Analytics Tab - PR #16417
  • Fix Litellm tags usage add request_id - PR #16111

• Health Check

  • Add Langfuse OTEL and SQS to Health Check - PR #16514

• General UI

  • UI - Normalize table action columns appearance - PR #16657
  • UI - Button Styles and Sizing in Settings Pages - PR #16600
  • UI - SSO Modal Cosmetic Changes - PR #16554
  • Fix UI logos loading with SERVER_ROOT_PATH - PR #16618
  • Fix remove misleading 'Custom' option mention from OpenAI endpoint tooltips - PR #16622

• SSO

  • Ensure role from SSO provider is used when a user is inserted onto LiteLLM - PR #16794

Bugs​

• Management Endpoints
  • Fix inconsistent error responses in customer management endpoints - PR #16450
  • Fix correct date range filtering in /spend/logs endpoint - PR #16443
  • Fix /spend/logs/ui Access Control - PR #16446
  • Add pagination for /spend/logs/session/ui endpoint - PR #16603
  • Fix LiteLLM Usage shows key_hash - PR #16471
  • Fix app_roles missing from jwt payload - PR #16448

Logging / Guardrail / Prompt Management Integrations​

New Integration​

• 🆕 Zscaler AI Guard
  • Add Zscaler AI Guard hook for security policy enforcement - PR #15691

Logging​

• Langfuse

  • Fix handle null usage values to prevent validation errors - PR #16396

• CloudZero

  • Fix updated spend would not be sent to CloudZero - PR #16201

Guardrails​

• IBM Detector
  • Ensure detector-id is passed as header to IBM detector server - PR #16649

Prompt Management​

• Custom Prompt Management
  • Add SDK focused examples for custom prompt management - PR #16441

Spend Tracking, Budgets and Rate Limiting​

• End User Budgets
  • Allow pointing max_end_user budget to an id, so the default ID applies to all end users - PR #16456

MCP Gateway​

• Configuration
  • Add dynamic OAuth2 metadata discovery for MCP servers - PR #16676
  • Fix allow tool call even when server name prefix is missing - PR #16425
  • Fix exclude unauthorized MCP servers from allowed server list - PR #16551
  • Fix unable to delete MCP server from permission settings - PR #16407
  • Fix avoid crashing when MCP server record lacks credentials - PR #16601

Agents​

• Agent Registration (A2A Spec)
  • Support agent registration + discovery following Agent-to-Agent specification - PR #16615

Performance / Loadbalancing / Reliability improvements​

• Embeddings Performance

  • Use router's O(1) lookup and shared sessions for embeddings - PR #16344

• Router Reliability

  • Support default fallbacks for unknown models - PR #16419

• Callback Management

  • Add atexit handlers to flush callbacks for async completions - PR #16487

General Proxy Improvements​

• Configuration Management
  • Fix update model_cost_map_url to use environment variable - PR #16429

Documentation Updates​

• Provider Documentation

  • Fix streaming example in README - PR #16461
  • Update broken Slack invite links to support page - PR #16546
  • Fix code block indentation for fallbacks page - PR #16542
  • Documentation code example corrections - PR #16502
  • Document reasoning_effort summary field options - PR #16549

• API Documentation

  • Add docs on APIs for model access management - PR #16673
  • Add docs for showing how to auto reload new pricing data - PR #16675
  • LiteLLM Quick start - show how model resolution works - PR #16602
  • Add docs for tracking callback failure - PR #16474

• General Documentation

  • Fix container api link in release page - PR #16440
  • Add softgen to projects that are using litellm - PR #16423

New Contributors​

• @artplan1 made their first contribution in PR #16423
• @JehandadK made their first contribution in PR #16472
• @vmiscenko made their first contribution in PR #16453
• @mcowger made their first contribution in PR #16429
• @yellowsubmarine372 made their first contribution in PR #16395
• @Hebruwu made their first contribution in PR #16201
• @jwang-gif made their first contribution in PR #15691
• @AnthonyMonaco made their first contribution in PR #16502
• @andrewm4894 made their first contribution in PR #16487
• @f14-bertolotti made their first contribution in PR #16485
• @busla made their first contribution in PR #16293
• @MightyGoldenOctopus made their first contribution in PR #16537
• @ultmaster made their first contribution in PR #14436
• @bchrobot made their first contribution in PR #16542
• @sep-grindr made their first contribution in PR #16622
• @pnookala-godaddy made their first contribution in PR #16607
• @dtunikov made their first contribution in PR #16592
• @lukapecnik made their first contribution in PR #16648
• @jyeros made their first contribution in PR #16618

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.79.3-stable

pip install litellm==1.79.3.rc.1

Key Highlights​

• LiteLLM Custom Guardrail - Built-in guardrail with UI configuration support
• Performance Improvements - /responses API 19× Lower Median Latency
• Veo3 Video Generation (Vertex AI + Google AI Studio) - Use OpenAI Video API to generate videos with Vertex AI and Google AI Studio Veo3 models

Built-in Guardrails on AI Gateway​

This release introduces built-in guardrails for LiteLLM AI Gateway, allowing you to enforce protections without depending on an external guardrail API.

• Blocking Keywords - Block known sensitive keywords like "litellm", "python", etc.
• Pattern Detection - Block known sensitive patterns like emails, Social Security Numbers, API keys, etc.
• Custom Regex Patterns - Define custom regex patterns for your specific use case.

Get started with the built-in guardrails on AI Gateway here.

Performance – /responses 19× Lower Median Latency​

This update significantly improves /responses latency by integrating our internal network management for connection handling, eliminating per-request setup overhead.

Results​

Test Setup​

New Models / Updated Models​

New Model Support​

OCR Models​

Search Models​

Features​

• Azure

  • Add Azure GPT-5-Pro Responses API support with reasoning capabilities - PR #16235
  • Add gpt-image-1-mini pricing for Azure with quality tiers (low/medium/high) - PR #16182
  • Add support for returning Azure Content Policy error information when exceptions from Azure OpenAI occur - PR #16231
  • Fix Azure GPT-5 incorrectly routed to O-series config (temperature parameter unsupported) - PR #16246
  • Fix Azure doesn't accept extra body param - PR #16116
  • Fix Azure DALL-E-3 health check content policy violation by using safe default prompt - PR #16329

• Bedrock

  • Fix empty assistant message handling in AWS Bedrock Converse API to prevent 400 Bad Request errors - PR #15850
  • Fix: Filter AWS authentication params from Bedrock InvokeModel request body - PR #16315
  • Fix Bedrock proxy adding name to file content, breaks when cache_control in use - PR #16275
  • Fix global.anthropic.claude-haiku-4-5-20251001-v1:0 supports_reasoning flag and update pricing - PR #16263

• Gemini (Google AI Studio + Vertex AI)

  • Add gemini live audio model cost in model map - PR #16183
  • Fix translation problem with Gemini parallel tool calls - PR #16194
  • Fix: Send Gemini API key via x-goog-api-key header with custom api_base - PR #16085
  • Fix image_config.aspect_ratio not working for gemini-2.5-flash-image - PR #15999
  • Fix Gemini minimal reasoning env overrides disabling thoughts - PR #16347
  • Fix cache_read_input_token_cost for gemini-2.5-flash - PR #16354

• Anthropic

  • Fix Anthropic token counting for VertexAI - PR #16171
  • Fix anthropic-adapter: properly translate Anthropic image format to OpenAI - PR #16202
  • Enable automated prompt caching message format for Claude on Databricks - PR #16200
  • Add support for Anthropic Memory Tool - PR #16115
  • Propagate cache creation/read token costs for model info to fix Anthropic long context cost calculations - PR #16376

• Vertex AI

  • Add Vertex MiniMAX m2 model support - PR #16373
  • Correctly map 429 Resource Exhausted to RateLimitError - PR #16363
  • Add vertex_credentials support to litellm.rerank() for Vertex AI - PR #16266

• Databricks

  • Fix databricks streaming - PR #16368

• Deepgram

  • Return the diarized transcript when it's required in the request - PR #16133

• Fireworks

  • Update Fireworks audio endpoints to new api.fireworks.ai domains - PR #16346

• Cohere

  • Add cohere embed-v4.0 model support - PR #16358

• Watsonx

  • Support reasoning_effort for watsonx chat models - PR #16261

• OpenAI

  • Remove automatic summary from reasoning_effort transformation - PR #16210

• XAI

  • Remove Grok 4 Models Reasoning Effort Parameter - PR #16265

• Hosted VLLM

  • Fix HostedVLLMRerankConfig will not be used - PR #16352

New Provider Support​

• Bedrock Agentcore
  • Add Bedrock Agentcore as a provider on LiteLLM Python SDK and LiteLLM AI Gateway - PR #16252

LLM API Endpoints​

Features​

• OCR API

  • Add VertexAI OCR provider support + cost tracking - PR #16216
  • Add Azure AI Doc Intelligence OCR support - PR #16219

• Search API

  • Add firecrawl search API support with tiered pricing - PR #16257
  • Add searxng search API provider - PR #16259

• Responses API

  • Support responses API streaming in langfuse otel - PR #16153
  • Pass extra_body parameters to provider in Responses API requests - PR #16320

• Container API

  • Add E2E Container API Support - PR #16136
  • Update container documentation to be similar to others - PR #16327

• Video Generation API

  • Add Vertex and Gemini Videos API with Cost Tracking + UI support - PR #16323
  • Add custom_llm_provider support for video endpoints (non-generation) - PR #16121

• Audio API

  • Add gpt-4o-transcribe cost tracking - PR #16412

• Vector Stores

  • Milvus - search vector store support + support multi-part form data on passthrough - PR #16035
  • Azure AI Vector Stores - support "virtual" indexes + create vector store on passthrough API - PR #16160
  • Milvus - Passthrough API support - adds create + read vector store support via passthrough API's - PR #16170

• Embeddings API

  • Use valid CallTypes enum value in embeddings endpoint - PR #16328

• Rerank API

  • Generalize tiered pricing in generic cost calculator - PR #16150

Bugs​

• General
  • Fix index field not populated in streaming mode with n>1 and tool calls - PR #15962
  • Pass aws_region_name in litellm_params - PR #16321
  • Add retry-after header support for errors 502, 503, 504 - PR #16288

Management Endpoints / UI​

Features​

• Virtual Keys

  • UI - Delete Team Member with friction - PR #16167
  • UI - Litellm test key audio support - PR #16251
  • UI - Test Key Page Revert Model To Single Select - PR #16390

• Models + Endpoints

  • UI - Add Model Existing Credentials Improvement - PR #16166
  • UI - Add Azure AD Token field and Azure API Key optional - PR #16331
  • UI - Fixed Label for vLLM in Model Create Flow - PR #16285
  • UI - Include Model Access Group Models on Team Models Table - PR #16298
  • Fix /model_group/info Returning Entire Model List for SSO Users - PR #16296
  • Litellm non root docker Model Hub Table fix - PR #16282

• Guardrails

  • UI - Fix regression where Guardrail Entity Could not be selected and entity was not displayed - PR #16165
  • UI - Guardrail Info Page Show PII Config - PR #16164
  • Change guardrail_information to list type - PR #16127
  • UI - LiteLLM Guardrail - ensure you can see UI Friendly name for PII Patterns - PR #16382
  • UI - Guardrails - LiteLLM Content Filter, Allow Viewing/Editing Content Filter Settings - PR #16383
  • UI - Guardrails - allow updating guardrails through UI. Ensure litellm_params actually get updated in memory - PR #16384

• SSO Settings

  • Support dot notation on ui sso - PR #16135
  • UI - Prevent trailing slash in sso proxy base url input - PR #16244
  • UI - SSO Proxy Base URL input validation and remove normalizing / - PR #16332
  • UI - Surface SSO Create errors on create flow - PR #16369

• Usage & Analytics

  • UI - Tag Usage Top Model Table View and Label Fix - PR #16249
  • UI - Litellm usage date picker - PR #16264

• Cache Settings

  • UI - Cache Settings Redis Add Semantic Cache Settings - PR #16398

Bugs​

• General
  • UI - Remove encoding_format in request for embedding models - PR #16367
  • UI - Revert Changes for Test Key Multiple Model Select - PR #16372
  • UI - Various Small Issues - PR #16406

AI Integrations​

Logging​

• Langfuse

  • Fix langfuse input tokens logic for cached tokens - PR #16203

• Opik

  • Fix the bug with not incorrect attachment to existing trace & refactor - PR #15529

• S3

  • S3 logger, add support for ssl_verify when using minio logger - PR #16211
  • Strip base64 in s3 - PR #16157
  • Add allowing Key based prefix to s3 path - PR #16237
  • Add Prometheus metric to track callback logging failures in S3 - PR #16209

• OpenTelemetry

  • OTEL - Log Cost Breakdown on OTEL Logger - PR #16334

• DataDog

  • Add DD Agent Host support for datadog callback - PR #16379

Guardrails​

• Noma

  • Revert Noma Apply Guardrail implementation - PR #16214
  • Litellm noma guardrail support images - PR #16199

• PANW Prisma AIRS

  • PANW prisma airs guardrail deduplication and enhanced session tracking - PR #16273

• LiteLLM Custom Guardrail

  • Add LiteLLM Gateway built in guardrail - PR #16338
  • UI - Allow configuring LiteLLM Custom Guardrail - PR #16339
  • Bug Fix: Content Filter Guard - PR #16414

Secret Managers​

• CyberArk

  • Add CyberArk Secrets Manager Integration - PR #16278
  • Cyber Ark - Add Key Rotations support - PR #16289

• HashiCorp Vault

  • Add configurable mount name and path prefix for HashiCorp Vault - PR #16253
  • Secret Manager - Hashicorp, add auth via approle - PR #16374

• AWS Secrets Manager

  • Add tags and descriptions support to aws secrets manager - PR #16224

• Custom Secret Manager

  • Add Custom Secret Manager - Allow users to define and write a custom secret manager - PR #16297

• General

  • Email Notifications - Ensure Users get Key Rotated Email - PR #16292
  • Fix verify ssl on sts boto3 - PR #16313

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking
  • Fix OpenAI Responses API streaming tests usage field names and cost calculation - PR #16236

MCP Gateway​

• Configuration
  • Configure static mcp header - PR #16179
  • Persist mcp credentials in db - PR #16308

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fixes

  • Resolve memory accumulation caused by Pydantic 2.11+ deprecation warnings - PR #16110

• Session Management

  • Add shared_session support to responses API - PR #16260

• Error Handling

  • Gracefully handle connection closed errors during streaming - PR #16294
  • Handle None values in daily spend sort key - PR #16245

• Configuration

  • Remove minimum validation for cache control injection index - PR #16149
  • Improve clearing logic - only remove unvisited endpoints - PR #16400

• Redis

  • Handle float redis_version from AWS ElastiCache Valkey - PR #16207

• Hooks

  • Add parallel execution handling in during_call_hook - PR #16279

• Infrastructure

  • Install runtime node for prisma - PR #16410

Documentation Updates​

• Provider Documentation

  • Docs - v1.79.1 - PR #16163
  • Fix broken link on model_management.md - PR #16217
  • Fix image generation response format - use 'images' array instead of 'image' object - PR #16378

• General Documentation

  • Add minimum resource requirement for production - PR #16146
  • Add benchmark comparison with other AI gateways - PR #16248
  • LiteLLM content filter guard documentation - PR #16413
  • Fix typo of the word orginal - PR #16255

• Security

  • Remove tornado test files (including test.key), fixes Python 3.13 security issues - PR #16342

New Contributors​

• @steve-gore-snapdocs made their first contribution in PR #16149
• @timbmg made their first contribution in PR #16120
• @Nivg made their first contribution in PR #16202
• @pablobgar made their first contribution in PR #16194
• @AlanPonnachan made their first contribution in PR #16150
• @Chesars made their first contribution in PR #16236
• @bowenliang123 made their first contribution in PR #16255
• @dean-zavad made their first contribution in PR #16199
• @alexkuzmik made their first contribution in PR #15529
• @Granine made their first contribution in PR #16281
• @Oodapow made their first contribution in PR #16279
• @jgoodyear made their first contribution in PR #16275
• @Qanpi made their first contribution in PR #16321
• @ShimonMimoun made their first contribution in PR #16313
• @andriykislitsyn made their first contribution in PR #16288
• @reckless-huang made their first contribution in PR #16263
• @chenmoneygithub made their first contribution in PR #16368
• @stembe-digitalex made their first contribution in PR #16354
• @jfcherng made their first contribution in PR #16352
• @xingyaoww made their first contribution in PR #16246
• @emerzon made their first contribution in PR #16373
• @wwwillchen made their first contribution in PR #16376
• @fabriciojoc made their first contribution in PR #16203
• @jroberts2600 made their first contribution in PR #16273

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.79.1-stable

pip install litellm==1.80.0

Key Highlights​

• Container API Support - End-to-end OpenAI Container API support with proxy integration, logging, and cost tracking
• FAL AI Image Generation - Native support for FAL AI image generation models with cost tracking
• UI Enhancements - Guardrail Playground, Cache Settings, Tag Routing, SSO Settings
• Batch API Rate Limiting - Input-based rate limits support for Batch API requests
• Vector Store Expansion - Milvus vector store support and Azure AI virtual indexes
• Memory Leak Fixes - Resolved issues accounting for 90% of memory leaks on Python SDK & AI Gateway

Dependency Upgrades​

• Dependencies
  • Build(deps): bump starlette from 0.47.2 to 0.49.1 - PR #16027
  • Build(deps): bump fastapi from 0.116.1 to 0.120.1 - PR #16054
  • Build(deps): bump hono from 4.9.7 to 4.10.3 in /litellm-js/spend-logs - PR #15915

New Models / Updated Models​

New Model Support​

Features​

• Anthropic

  • Extended Claude 3-7 Sonnet deprecation date from 2026-02-01 to 2026-02-19 - PR #15976
  • Extended Claude Opus 4-0 deprecation date from 2025-03-01 to 2026-05-01 - PR #15976
  • Removed Claude Haiku 3-5 deprecation date (previously 2025-03-01) - PR #15976
  • Added Claude Opus 4-1, Claude Opus 4-0 20250513, Claude Sonnet 4 20250514 deprecation dates - PR #15976
  • Added web search support for Claude Opus 4-1 - PR #15976

• Bedrock

  • Fix empty assistant message handling in AWS Bedrock Converse API to prevent 400 Bad Request errors - PR #15850
  • Allow using ARNs when generating images via Bedrock - PR #15789
  • Add per model group header forwarding for Bedrock Invoke API - PR #16042
  • Preserve Bedrock inference profile IDs in health checks - PR #15947
  • Added fallback logic for detecting file content-type when S3 returns generic type - When using Bedrock with S3-hosted files, if the S3 object's Content-Type is not correctly set (e.g., binary/octet-stream instead of image/png), Bedrock can now handle it correctly - PR #15635

• Azure

  • Add deprecation dates for Azure OpenAI models (gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4.1 series, o3-2025-04-16, text-embedding-3-small) - PR #15976
  • Fix Azure OpenAI ContextWindowExceededError mapping from Azure errors - PR #15981
  • Add handling for v1 under Azure API versions - PR #15984
  • Fix azure doesn't accept extra body param - PR #16116

• OpenAI

  • Add deprecation dates for gpt-3.5-turbo-1106, gpt-4-0125-preview, gpt-4-1106-preview, o1-mini-2024-09-12 - PR #15976
  • Add extended Sora-2 modality support (text + image inputs) - PR #15976
  • Updated OpenAI Sora-2-Pro pricing to $0.30/video/second - PR #15976

• OpenRouter

  • Add Claude Haiku 4.5 pricing for OpenRouter - PR #15909
  • Add base_url config with environment variables documentation - PR #15946

• Mistral

  • Add codestral-embed-2505 embedding model - PR #16071

• Gemini (Google AI Studio + Vertex AI)

  • Fix gemini request mutation for tool use - PR #16002
  • Add gemini-embedding-001 pricing entry for Google GenAI API - PR #16078
  • Changes to fix frequency_penalty and presence_penalty issue for gemini-2.5-pro model - PR #16041

• DeepInfra

  • Add vision support for Qwen/Qwen3-chat-32b model - PR #15976

• Vercel AI Gateway

  • Fix vercel_ai_gateway entry for glm-4.6 (moved from vercel_ai_gateway/glm-4.6 to vercel_ai_gateway/zai/glm-4.6) - PR #16084

• Fireworks

  • Don't add "accounts/fireworks/models" prefix for Fireworks Provider - PR #15938

• Cohere

  • Add OpenAI-compatible annotations support for Cohere v2 citations - PR #16038

• Deepgram

  • Handle Deepgram detected language when available - PR #16093

Bug Fixes​

• Xai
  • Add Xai websearch cost tracking - PR #16001

New Provider Support​

• FAL AI

  • Add FAL AI Image Generation support - PR #16067

• OCI (Oracle Cloud Infrastructure)

  • Add OCI Signer Authentication support - PR #16064

LLM API Endpoints​

Features​

• Container API

  • Add end-to-end OpenAI Container API support to LiteLLM SDK - PR #16136
  • Add proxy support for container APIs - PR #16049
  • Add logging support for Container API - PR #16049
  • Add cost tracking support for containers with documentation - PR #16117

• Responses API

  • Respect LiteLLM-Disable-Message-Redaction header for Responses API - PR #15966
  • Add /openai routes for responses API (Azure OpenAI SDK Compatibility) - PR #15988
  • Redact reasoning summaries in ResponsesAPI output when message logging is disabled - PR #15965
  • Support text.format parameter in Responses API for providers without native ResponsesAPIConfig - PR #16023
  • Add LLM provider response headers to Responses API - PR #16091

• Video Generation API

  • Add custom_llm_provider support for video endpoints (non-generation) - PR #16121
  • Fix documentation for videos - PR #15937
  • Add OpenAI client usage documentation for videos and fix navigation visibility - PR #15996

• Moderations API

  • Moderations endpoint now respects api_base configuration parameter - PR #16087

• Vector Stores

  • Milvus - search vector store support - PR #16035
  • Azure AI Vector Stores - support "virtual" indexes + create vector store on passthrough API - PR #16160

• Passthrough Endpoints

  • Support multi-part form data on passthrough - PR #16035

Management Endpoints / UI​

Features​

• Virtual Keys

  • Validation for Proxy Base URL in SSO Settings - PR #16082
  • Test Key UI Embeddings support - PR #16065
  • Add Key Type Select in Key Settings - PR #16034
  • Key Already Exist Error Notification - PR #15993

• Models + Endpoints

  • Changed API Base from Select to Input in New LLM Credentials - PR #15987
  • Remove limit from admin UI numerical input - PR #15991
  • Config Models should not be editable - PR #16020
  • Add tags in model creation - PR #16138
  • Add Tags to update model - PR #16140

• Guardrails

  • Add Apply Guardrail Testing Playground - PR #16030
  • Config Guardrails should not be editable and guardrail info fix - PR #16142

• Cache Settings

  • Allow setting cache settings on UI - PR #16143

• Routing

  • Allow setting all routing strategies, tag filtering on UI - PR #16139

• Admin Settings

  • Add license metadata to health/readiness endpoint - PR #15997
  • Litellm Backend SSO Changes - PR #16029

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry

  • Enable OpenTelemetry context propagation by external tracers - PR #15940
  • Ensure error information is logged on OTEL - PR #15978

• Langfuse

  • Fix duplicate trace in langfuse_otel - PR #15931
  • Support tool usage messages with Langfuse OTEL integration - PR #15932

• DataDog

  • Ensure key's metadata + guardrail is logged on DD - PR #15980

• Opik

  • Enhance requester metadata retrieval from API key auth - PR #15897
  • User auth key metadata Documentation - PR #16004

• SQS

  • Add Base64 handling for SQS Logger - PR #16028

• General

  • Fix: User API key and team id and user id missing from custom callback is not misfiring - PR #15982

Guardrails​

• IBM Guardrails

  • Update IBM Guardrails to correctly use SSL Verify argument - PR #15975
  • Add additional detail to ibm_guardrails.md documentation - PR #15971

• Model Armor

  • Support during_call for model armor guardrails - PR #15970

• Lasso Security

  • Upgrade to Lasso API v3 and fix ULID generation - PR #15941

• PANW Prisma AIRS

  • Add per-request profile overrides to PANW Prisma AIRS - PR #16069

• Grayswan

  • Improve Grayswan guardrail documentation - PR #15875

• Pillar AI

  • Graceful degradation for pillar service when using litellm - PR #15857

• General

  • Ensure Key Guardrails are applied - PR #16025

Prompt Management​

• GitLab
  • Add GitlabPromptCache and enable subfolder access - PR #15712

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking

  • Fix spend tracking for OCR/aOCR requests (log pages_processed + recognize OCRResponse) - PR #16070

• Rate Limiting

  • Add support for Batch API Rate limiting - PR1 adds support for input based rate limits - PR #16075
  • Handle multiple rate limit types per descriptor and prevent IndexError - PR #16039

MCP Gateway​

• OAuth
  • Add support for dynamic client registration - PR #15921
  • Respect X-Forwarded- headers in OAuth endpoints - PR #16036

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fixes

  • Fix: prevent httpx DeprecationWarning memory leak in AsyncHTTPHandler - PR #16024
  • Fix: resolve memory accumulation caused by Pydantic 2.11+ deprecation warnings - PR #16110
  • Fix(apscheduler): prevent memory leaks from jitter and frequent job intervals - PR #15846

• Configuration

  • Remove minimum validation for cache control injection index - PR #16149
  • Fix prompt_caching.md: wrong prompt_tokens definition - PR #16044

Documentation Updates​

• Provider Documentation

  • Use custom-llm-provider header in examples - PR #16055
  • Litellm docs readme fixes - PR #16107
  • Readme fixes add supported providers - PR #16109

• Model References

  • Add supports vision field to qwen-vl models in model_prices_and_context_window.json - PR #16106

• General Documentation

  • 1-79-0 docs - PR #15936
  • Add minimum resource requirement for production - PR #16146

New Contributors​

• @RobGeada made their first contribution in PR #15975
• @shanto12 made their first contribution in PR #15946
• @dima-hx430 made their first contribution in PR #15976
• @m-misiura made their first contribution in PR #15971
• @ylgibby made their first contribution in PR #15947
• @Somtom made their first contribution in PR #15909
• @rodolfo-nobrega made their first contribution in PR #16023
• @bernata made their first contribution in PR #15997
• @AlbertDeFusco made their first contribution in PR #15881
• @komarovd95 made their first contribution in PR #15789
• @langpingxue made their first contribution in PR #15635
• @OrionCodeDev made their first contribution in PR #16070
• @sbinnee made their first contribution in PR #16078
• @JetoPistola made their first contribution in PR #16106
• @gvioss made their first contribution in PR #16093
• @pale-aura made their first contribution in PR #16084
• @tanvithakur94 made their first contribution in PR #16041
• @li-boxuan made their first contribution in PR #16044
• @1stprinciple made their first contribution in PR #15938
• @raghav-stripe made their first contribution in PR #16137
• @steve-gore-snapdocs made their first contribution in PR #16149

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.79.0-stable

pip install litellm==1.79.0

Major Changes​

• Cohere models will now be routed to Cohere v2 API by default - PR #15722

Key Highlights​

• Search APIs - Native /v1/search endpoint with support for Perplexity, Tavily, Parallel AI, Exa AI, DataforSEO, and Google PSE with cost tracking
• Vector Stores - Vertex AI Search API integration as vector store through LiteLLM with passthrough endpoint support
• Guardrails Expansion - Apply guardrails across Responses API, Image Gen, Text completions, Audio transcriptions, Audio Speech, Rerank, and Anthropic Messages API via unified apply_guardrails function
• New Guardrail Providers - Gray Swan, Dynamo AI, IBM Guardrails, Lasso Security v3, and Bedrock Guardrail apply_guardrail endpoint support
• Video Generation API - Native support for OpenAI Sora-2 and Azure Sora-2 (Pro, Pro-High-Res) with cost tracking and logging support
• Azure AI Speech (TTS) - Native Azure AI Speech integration with cost tracking for standard and HD voices

New Models / Updated Models​

New Model Support​

Features​

• Anthropic

  • Fix cache_control incorrectly applied to all content items instead of last item only - PR #15699
  • Forward anthropic-beta headers to Bedrock, VertexAI - PR #15700
  • Change max_tokens value to match max_output_tokens for claude sonnet - PR #15715

• Bedrock

  • Add AWS us-gov-west-1 Claude 3.7 Sonnet costs - PR #15775
  • Fix the date for sonnet 3.7 in govcloud - PR #15800
  • Use proper bedrock model name in health check - PR #15808
  • Support for embeddings_by_type Response Format in Bedrock Cohere Embed v1 - PR #15707
  • Add titan image generations with cost tracking - PR #15916

• Gemini

  • Add imageConfig parameter for gemini-2.5-flash-image - PR #15530
  • Replace deprecated gemini-1.5-pro-preview-0514 - PR #15852
  • Update vertex ai gemini costs - PR #15911

• Ollama

  • Set 'think' to False when reasoning effort is minimal/none/disable - PR #15763
  • Handle parsing ollama chunk error - PR #15717

• Vertex AI

  • Add mistral medium 3 and Codestral 2 on vertex - PR #15887

• Databricks

  • Allow prompt caching to be used for Anthropic Claude on Databricks - PR #15801

• Azure

  • Add Azure AVA TTS integration - PR #15749
  • Add Azure AVA (Speech AI) Cost Tracking - PR #15754
  • Azure AI Speech - Ensure voice is mapped from request body to SSML body, allow sending role and style - PR #15810
  • Add Azure support for video generation functionality (Sora-2) - PR #15901

• OpenAI

  • OpenAI videos refactoring - PR #15900

• General

  • Read from custom-llm-provider header - PR #15528

LLM API Endpoints​

Features​

• Responses API

  • Add gpt 4.1 pricing for response endpoint - PR #15593
  • Fix Incorrect status value in responses api with gemini - PR #15753
  • Simplify reasoning item handling for gpt-5-codex - PR #15815
  • ErrorEvent ValidationError when OpenAI Responses API returns nested error structure - PR #15804
  • Fix reasoning item ID auto-generation causing encrypted content verification errors - PR #15782
  • Support tags in metadata - PR #15867
  • Security: prevent User A from retrieving User B's response, if response.id is leaked - PR #15757

• Batch API

  • Add pre and post call for list batches - PR #15673
  • Add function responsible to call precall - PR #15636
  • Fix "User default_user_id does not have access to the object" when object not in db - PR #15873

• OCR API

  • Add Azure AI - OCR to docs - PR #15768
  • Add mode + Health check support for OCR models - PR #15767

• Search API

  • Add def search() APIs for Web Search - Perplexity API - PR #15769
  • Add Tavily Search API - PR #15770
  • Add Parallel AI - Search API - PR #15772
  • Add EXA AI Search API to LiteLLM - PR #15774
  • Add /search endpoint on LiteLLM Gateway - PR #15780
  • Add DataforSEO Search API - PR #15817
  • Add Google PSE Search Provider - PR #15816
  • Add cost tracking for Search API requests - Google PSE, Tavily, Parallel AI, Exa AI - PR #15821
  • Backend: Allow storing configured Search APIs in DB - PR #15862
  • Exa Search API - ensure request params are sent to Exa AI - PR #15855

• Vector Stores

  • Support Vertex AI Search API as vector store through LiteLLM - PR #15781
  • Azure AI - Search Vector Stores - PR #15873
  • VertexAI Search Vector Store - Passthrough endpoint support + Vector store search Cost tracking support - PR #15824
  • Don't raise error if managed object is not found - PR #15873
  • Show config.yaml vector stores on UI - PR #15873
  • Cost tracking for search spend - PR #15859

• Images API

  • Pass user-defined headers and extra_headers to image-edit calls - PR #15811

• Video Generation API

  • Add Azure support for video generation functionality (Sora-2, Sora-2-Pro, Sora-2-Pro-High-Res) - PR #15901
  • OpenAI video generation refactoring (Sora-2) - PR #15900

• Bedrock /invoke

  • Fix: Hooks broken on /bedrock passthrough due to missing metadata - PR #15849

• Realtime API

  • Fix: OpenAI Realtime API integration fails due to websockets.exceptions.PayloadTooBig error - PR #15751

Management Endpoints / UI​

Features​

• Passthrough

  • Set auth on passthrough endpoints, on the UI - PR #15778
  • Fix pass-through endpoint budget enforcement bug - PR #15805

• Organizations

  • Allow org admins to create teams on UI - PR #15924

• Search Tools

  • UI - Search Tools, allow adding search tools on UI + testing search - PR #15871
  • UI - Add logos for search providers - PR #15872

• General

  • Fix routing for custom server root path - PR #15701

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry

  • Fix OpenTelemetry Logging functionality - PR #15645
  • Fix issue where headers were not being split correctly - PR #15916

• Sentry

  • Add SENTRY_ENVIRONMENT configuration for Sentry integration - PR #15760

• Helicone

  • Fix JSON serialization error in Helicone logging by removing OpenTelemetry span from metadata - PR #15728

• MLFlow

  • Fix MLFlow tags - split request_tags into (key, val) if request_tag has colon - PR #15914

• General

  • Rename configured_cold_storage_logger to cold_storage_custom_logger - PR #15798

Guardrails​

• Gray Swan

  • Add GraySwan Guardrails support - PR #15756
  • Rename GraySwan to Gray Swan - PR #15771

• Dynamo AI

  • New Guardrail - Dynamo AI Guardrail - PR #15920

• IBM Guardrails

  • IBM Guardrails integration - PR #15924

• Lasso Security

  • Add v3 API Support - PR #12452
  • Fixed lasso import config, redis cluster hash tags for test keys - PR #15917

• Bedrock Guardrails

  • Implement Bedrock Guardrail apply_guardrail endpoint support - PR #15892

• General

  • Guardrails - Responses API, Image Gen, Text completions, Audio transcriptions, Audio Speech, Rerank, Anthropic Messages API support via the unified apply_guardrails function - PR #15706

Spend Tracking, Budgets and Rate Limiting​

• Rate Limiting
  • Support absolute RPM/TPM in priority_reservation - PR #15813
  • Org level tpm/rpm limits + Team tpm/rpm validation when assigned to org - PR #15549

MCP Gateway​

• OAuth
  • Auth Header Fix for MCP Tool Call - PR #15736
  • Add response_type + PKCE parameters to OAuth authorization endpoint - PR #15720

Performance / Loadbalancing / Reliability improvements​

• Database

  • Minimize the occurrence of deadlocks - PR #15281

• Redis

  • Apply max_connections configuration to Redis async client - PR #15797

• Caching

  • Add documentation for enable_caching_on_provider_specific_optional_params setting - PR #15885

Documentation Updates​

• Provider Documentation
  • Update worker recommendation - PR #15702
  • Fix the wrong request body in json mode doc - PR #15729
  • Add details in docs - PR #15721
  • Add responses api on openai docs - PR #15866
  • Add OpenAI responses api - PR #15868

New Contributors​

• @tlecomte made their first contribution in PR #15528
• @tomhaynes made their first contribution in PR #15645
• @talalryz made their first contribution in PR #15720
• @1vinodsingh1 made their first contribution in PR #15736
• @nuernber made their first contribution in PR #15775
• @Thomas-Mildner made their first contribution in PR #15760
• @javiergarciapleo made their first contribution in PR #15721
• @lshgdut made their first contribution in PR #15717
• @kk-wangjifeng made their first contribution in PR #15530
• @anthonyivn2 made their first contribution in PR #15801
• @romanglo made their first contribution in PR #15707
• @mythral made their first contribution in PR #15859
• @mubashirosmani made their first contribution in PR #15866
• @CAFxX made their first contribution in PR #15281
• @reflection made their first contribution in PR #15914
• @shadielfares made their first contribution in PR #15917

PR Count Summary​

10/26/2025​

• New Models / Updated Models: 20
• LLM API Endpoints: 29
• Management Endpoints / UI: 5
• Logging / Guardrail / Prompt Management Integrations: 10
• Spend Tracking, Budgets and Rate Limiting: 2
• MCP Gateway: 2
• Performance / Loadbalancing / Reliability improvements: 3
• Documentation Updates: 5

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.78.5-stable

pip install litellm==1.78.5

Key Highlights​

• Native OCR Endpoints - Native /v1/ocr endpoint support with cost tracking for Mistral OCR and Azure AI OCR
• Global Vendor Discounts - Specify global vendor discount percentages for accurate cost tracking and reporting
• Team Spending Reports - Team admins can now export detailed spending reports for their teams
• Claude Haiku 4.5 - Day 0 support for Claude Haiku 4.5 across Bedrock, Vertex AI, and OpenRouter with 200K context window
• GPT-5-Codex - Support for GPT-5-Codex via Responses API on OpenAI and Azure
• Performance Improvements - Major router optimizations: O(1) model lookups, 10-100x faster shallow copy, 30-40% faster timing calls, and O(n) to O(1) hash generation

New Models / Updated Models​

New Model Support​

Features​

• OpenAI

  • GPT-5 return reasoning content via /chat/completions + GPT-5-Codex working on Claude Code - PR #15441

• Anthropic

  • Reduce claude-4-sonnet max_output_tokens to 64k - PR #15409
  • Added claude-haiku-4.5 - PR #15579
  • Add support for thinking blocks and redacted thinking blocks in Anthropic v1/messages API - PR #15501

• Bedrock

  • Add anthropic.claude-haiku-4-5-20251001-v1:0 on Bedrock, VertexAI - PR #15581
  • Add Claude Haiku 4.5 support for Bedrock global and US regions - PR #15650
  • Add Claude Haiku 4.5 support for Bedrock Other regions - PR #15653
  • Add JP Cross-Region Inference jp.anthropic.claude-haiku-4-5-20251001 - PR #15598
  • Fix: bedrock-pricing-geo-inregion-cross-region / add Global Cross-Region Inference - PR #15685
  • Fix: Support us-gov prefix for AWS GovCloud Bedrock models - PR #15626
  • Fix GPT-OSS in Bedrock now supports streaming. Revert fake streaming - PR #15668

• Gemini

  • Feat(pricing): Add Gemini 2.5 Flash Image (Nano Banana) in GA - PR #15557
  • Fix: Gemini 2.5 Flash Image should not have supports_web_search=true - PR #15642
  • Remove penalty params as supported params for gemini preview model - PR #15503

• Ollama

  • Fix(ollama/chat): correctly map reasoning_effort to think in requests - PR #15465

• OpenRouter

  • Add anthropic/claude-sonnet-4.5 to OpenRouter cost map - PR #15472
  • Prompt caching for anthropic models with OpenRouter - PR #15535
  • Get completion cost directly from OpenRouter - PR #15448
  • Fix OpenRouter Claude Opus 4 model naming - PR #15495

• CometAPI

  • Fix(cometapi): improve CometAPI provider support (embeddings, image generation, docs) - PR #15591

• Lemonade

  • Adding new models to the lemonade provider - PR #15554

• Watson X

  • Fix (pricing): Fix pricing for watsonx model family for various models - PR #15670

• Vercel AI Gateway

  • Add glm-4.6 model to pricing configuration - PR #15679

• Vertex AI

  • Add Vertex AI Discovery Engine Rerank Support - PR #15532

Bug Fixes​

• Anthropic

  • Fix: Pricing for Claude Sonnet 4.5 in US regions is 10x too high - PR #15374

• OpenRouter

  • Change gpt-5-codex support in model_price json - PR #15540

• Bedrock

  • Fix filtering headers for signature calcs - PR #15590

• General

  • Add native reasoning and streaming support flag for gpt-5-codex - PR #15569

LLM API Endpoints​

Features​

• Responses API

  • Responses API - enable calling anthropic/gemini models in Responses API streaming in openai ruby sdk + DB - sanity check pending migrations before startup - PR #15432
  • Add support for responses mode in health check - PR #15658

• OCR API

  • Feat: Add native litellm.ocr() functions - PR #15567
  • Feat: Add /ocr route on LiteLLM AI Gateway - Adds support for native Mistral OCR calling - PR #15571
  • Feat: Add Azure AI Mistral OCR Integration - PR #15572
  • Feat: Native /ocr endpoint support - PR #15573
  • Feat: Add Cost Tracking for /ocr endpoints - PR #15678

• /generateContent

  • Fix: GEMINI - CLI - add google_routes to llm_api_routes - PR #15500
  • Fix Pydantic validation error for citationMetadata.citationSources in Google GenAI responses - PR #15592

• Images API

  • Fix: Dall-e-2 for Image Edits API - PR #15604

• Bedrock Passthrough

  • Feat: Allow calling /invoke, /converse routes through AI Gateway + models on config.yaml - PR #15618

Bugs​

• General
  • Fix: Convert object to a correct type - PR #15634
  • Bug Fix: Tags as metadata dicts were raising exceptions - PR #15625
  • Add type hint to function_to_dict and fix typo - PR #15580

Management Endpoints / UI​

Features​

• Virtual Keys

  • Docs: Key Rotations - PR #15455
  • Fix: UI - Key Max Budget Removal Error Fix - PR #15672
  • litellm_Key Settings Max Budget Removal Error Fix - PR #15669

• Teams

  • Feat: Allow Team Admins to export a report of the team spending - PR #15542

• Passthrough

  • Feat: Passthrough - allow admin to give access to specific passthrough endpoints - PR #15401

• SCIM v2

  • Feat(scim_v2.py): if group.id doesn't exist, use external id + Passthrough - ensure updates and deletions persist across instances - PR #15276

• SSO

  • Feat: UI SSO - Add PKCE for OKTA SSO - PR #15608
  • Fix: Separate OAuth M2M authentication from UI SSO + Handle Introspection endpoint for Oauth2 - PR #15667
  • Fix/entraid app roles jwt claim clean - PR #15583

Logging / Guardrail / Prompt Management Integrations​

Guardrails​

• General

  • Fix apply_guardrail endpoint returning raw string instead of ApplyGuardrailResponse - PR #15436
  • Fix: Ensure guardrail memory sync after database updates - PR #15633
  • Feat: add guardrail for image generation - PR #15619
  • Feat: Add Guardrails for /v1/messages and /v1/responses API - PR #15686

• Pillar Security

  • Feature: update pillar security integration to support no persistence mode in litellm proxy - PR #15599

Prompt Management​

• General
  • Small fix code snippet custom_prompt_management.md - PR #15544

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking

  • Feat: Cost Tracking - specify a global vendor discount for costs - PR #15546
  • Feat: UI - Allow setting Provider Discounts on UI - PR #15550

• Budgets

  • Fix: improve budget clarity - PR #15682

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations

  • Perf(router): use shallow copy instead of deepcopy for model aliases - 10-100x faster than deepcopy on nested dict structures - PR #15576
  • Perf(router): optimize string concatenation in hash generation - Improves time complexity from O(n²) to O(n) - PR #15575
  • Perf(router): optimize model lookups with O(1) data structures - Replace O(n) scans with index map lookups - PR #15578
  • Perf(router): optimize model lookups with O(1) index maps - Use model_id_to_deployment_index_map and model_name_to_deployment_indices for instant lookups - PR #15574
  • Perf(router): optimize timing functions in completion hot path - Use time.perf_counter() for duration measurements and time.monotonic() for timeout calculations, providing 30-40% faster timing calls - PR #15617

• SSL/TLS Performance

  • Feat(ssl): add configurable ECDH curve for TLS performance - Configure via ssl_ecdh_curve setting to disable PQC on OpenSSL 3.x for better performance - PR #15617

• Token Counter

  • Fix(token-counter): extract model_info from deployment for custom_tokenizer - PR #15680

• Performance Metrics

  • Add: perf summary - PR #15458

• CI/CD

  • Fix: CI/CD - Missing env key & Linter type error - PR #15606

Documentation Updates​

• Provider Documentation

  • Litellm docs 10 11 2025 - PR #15457
  • Docs: add ecs deployment guide - PR #15468
  • Docs: Update benchmark results - PR #15461
  • Fix: add missing context to benchmark docs - PR #15688

• General

  • Fixed a few typos - PR #15267

New Contributors​

• @jlan-nl made their first contribution in PR #15374
• @ImadSaddik made their first contribution in PR #15267
• @huangyafei made their first contribution in PR #15472
• @mubashir1osmani made their first contribution in PR #15468
• @kowyo made their first contribution in PR #15465
• @dhruvyad made their first contribution in PR #15448
• @davizucon made their first contribution in PR #15544
• @FelipeRodriguesGare made their first contribution in PR #15540
• @ndrsfel made their first contribution in PR #15557
• @shinharaguchi made their first contribution in PR #15598
• @TensorNull made their first contribution in PR #15591
• @TeddyAmkie made their first contribution in PR #15583
• @aniketmaurya made their first contribution in PR #15580
• @eddierichter-amd made their first contribution in PR #15554
• @konekohana made their first contribution in PR #15535
• @Classic298 made their first contribution in PR #15495
• @afogel made their first contribution in PR #15599
• @orolega made their first contribution in PR #15633
• @LucasSugi made their first contribution in PR #15634
• @uc4w6c made their first contribution in PR #15619
• @Sameerlite made their first contribution in PR #15658
• @yuneng-jiang made their first contribution in PR #15672
• @Nikro made their first contribution in PR #15680

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.78.0-stable

pip install litellm==1.78.0.post1

Key Highlights​

• MCP Gateway - Control Tool Access by Team, Key - Control MCP tool access by team/key.
• Performance Improvements - 70% Lower p99 Latency
• GPT-5 Pro & GPT-Image-1-Mini - Day 0 support for OpenAI's GPT-5 Pro (400K context) and gpt-image-1-mini image generation
• EnkryptAI Guardrails - New guardrail integration for content moderation
• Tag-Based Budgets - Support for setting budgets based on request tags

MCP Gateway - Control Tool Access by Team, Key​

Proxy admins can now control MCP tool access by team or key. This makes it easy to grant different teams selective access to tools from the same MCP server.

For example, you can now give your Engineering team access to list_repositories, create_issue, and search_code tools, while Sales only gets search_code and close_issue tools.

This makes it easier for Proxy Admins to govern MCP Tool Access.

Get Started

Performance - 70% Lower p99 Latency​

This release cuts p99 latency by 70% on LiteLLM AI Gateway, making it even better for low-latency use cases.

These gains come from two key enhancements:

Reliable Sessions

Added support for shared sessions with aiohttp. The shared_session parameter is now consistently used across all calls, enabling connection pooling.

Faster Routing

A new model_name_to_deployment_indices hash map replaces O(n) list scans in _get_all_deployments() with O(1) hash lookups, boosting routing performance and scalability.

As a result, performance improved across all latency percentiles:

• Median latency: 110 ms → 100 ms (−9.1%)
• p95 latency: 440 ms → 150 ms (−65.9%)
• p99 latency: 810 ms → 240 ms (−70.4%)
• Average latency: 310 ms → 111.73 ms (−64.0%)

Test Setup​

Locust

• Concurrent users: 1,000
• Ramp-up: 500

System Specs

• Database was used
• CPU: 4 vCPUs
• Memory: 8 GB RAM
• LiteLLM Workers: 4
• Instances: 4

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

New Models / Updated Models​

New Model Support​

Features​

• OpenAI

  • Add GPT-5 Pro model configuration and documentation - PR #15258
  • Add stop parameter to non-supported params for GPT-5 - PR #15244
  • Day 0 Support, Add gpt-image-1-mini - PR #15259
  • Add gpt-realtime-mini support - PR #15283
  • Add gpt-5-pro-2025-10-06 to model costs - PR #15344
  • Minimal fix: gpt5 models should not go on cooldown when called with temperature!=1 - PR #15330

• Snowflake Cortex

  • Add function calling support for Snowflake Cortex REST API - PR #15221

• Gemini

  • Fix header forwarding for Gemini/Vertex AI providers in proxy mode - PR #15231

• Azure

  • Removed stop param from unsupported azure models - PR #15229
  • Fix(azure/responses): remove invalid status param from azure call - PR #15253
  • Add new Azure AI models with pricing details - PR #15387
  • AzureAD Default credentials - select credential type based on environment - PR #14470

• Bedrock

  • Add Global Cross-Region Inference - PR #15210
  • Add Cohere Embed v4 support for AWS Bedrock - PR #15298
  • Fix(bedrock): include cacheWriteInputTokens in prompt_tokens calculation - PR #15292
  • Add Bedrock AU Cross-Region Inference for Claude Sonnet 4.5 - PR #15402
  • Converse → /v1/messages streaming doesn't handle parallel tool calls with Claude models - PR #15315

• Vertex AI

  • Implement Context Caching for Vertex AI provider - PR #15226
  • Support for Vertex AI Gemma Models on Custom Endpoints - PR #15397
  • VertexAI - gemma model family support (custom endpoints) - PR #15419
  • VertexAI Gemma model family streaming support + Added MedGemma - PR #15427

• OCI

  • Add OCI Cohere support with tool calling and streaming capabilities - PR #15365

• Watson X

  • Add Watson X foundation model definitions to model_prices_and_context_window.json - PR #15219
  • Watsonx - Apply correct prompt templates for openai/gpt-oss model family - PR #15341

• OpenRouter

  • Fix - (openrouter): move cache_control to content blocks for claude/gemini - PR #15345
  • Fix - OpenRouter cache_control to only apply to last content block - PR #15395

• Together AI

  • Add new together models - PR #15383

Bug Fixes​

• General
  • Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - PR #15116
  • Fix reasoning response ID - PR #15265
  • Fix issue with parsing assistant messages - PR #15320
  • Fix litellm_param based costing - PR #15336
  • Fix lint errors - PR #15406

LLM API Endpoints​

Features​

• Responses API

  • Added streaming support for response api streaming image generation - PR #15269
  • Add native Responses API support for litellm_proxy provider - PR #15347
  • Temporarily relax ResponsesAPIResponse parsing to support custom backends (e.g., vLLM) - PR #15362

• Files API

  • Feat(files): add @client decorator to file operations - PR #15339

• /generateContent

  • Fix gemini cli by actually streaming the response - PR #15264

• Azure Passthrough

  • Azure - passthrough support with router models - PR #15240

Bugs​

• General
  • Fix x-litellm-cache-key header not being returned on cache hit - PR #15348

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Proxy CLI - dont store existing key in the URL, store it in the state param - PR #15290

• Models + Endpoints

  • Make PATCH /model/{model_id}/update handle team_id consistently with POST /model/new - PR #15297
  • Feature: adds Infinity as a provider in the UI - PR #15285
  • Fix: model + endpoints page crash when config file contains router_settings.model_group_alias - PR #15308
  • Models & Endpoints Initial Refactor - PR #15435
  • Litellm UI API Reference page updates - PR #15438

• Teams

  • Teams page: new column "Your Role" on the teams table - PR #15384
  • LiteLLM Dashboard Teams UI refactor - PR #15418

• UI Infrastructure

  • Added prettier to autoformat frontend - PR #15215
  • Adds turbopack to the npm run dev command in UI to build faster during development - PR #15250
  • (perf) fix: Replaces bloated key list calls with lean key aliases endpoint - PR #15252
  • Potentially fixes a UI spasm issue with an expired cookie - PR #15309
  • LiteLLM UI Refactor Infrastructure - PR #15236
  • Enforces removal of unused imports from UI - PR #15416
  • Fix: usage page >> Model Activity >> spend per day graph: y-axis clipping on large spend values - PR #15389
  • Updates guardrail provider logos - PR #15421

• Admin Settings

  • Fix: Router settings do not update despite success message - PR #15249
  • Fix: Prevents DB from accidentally overriding config file values if they are empty in DB - PR #15340

• SSO

  • SSO - support EntraID app roles - PR #15351

Logging / Guardrail / Prompt Management Integrations​

Features​

• PostHog
  • Feat: posthog per request api key - PR #15379

Guardrails​

• EnkryptAI
  • Add EnkryptAI Guardrails on LiteLLM - PR #15390

Spend Tracking, Budgets and Rate Limiting​

• Tag Management

  • Tag Management - Add support for setting tag based budgets - PR #15433

• Dynamic Rate Limiter v3

  • QA/Fixes - Dynamic Rate Limiter v3 - final QA - PR #15311
  • Fix dynamic Rate limiter v3 - inserting litellm_model_saturation - PR #15394

• Shared Health Check

  • Implement Shared Health Check State Across Pods - PR #15380

MCP Gateway​

• Tool Control

  • MCP Gateway - UI - Select allowed tools for Key, Teams - PR #15241
  • MCP Gateway - Backend - Allow storing allowed tools by team/key - PR #15243
  • MCP Gateway - Fine-grained Database Object Storage Control - PR #15255
  • MCP Gateway - Litellm mcp fixes team control - PR #15304
  • MCP Gateway - QA/Fixes - Ensure Team/Key level enforcement works for MCPs - PR #15305
  • Feature: Include server_name in /v1/mcp/server/health endpoint response - PR #15431

• OpenAPI Integration

  • MCP - support converting OpenAPI specs to MCP servers - PR #15343
  • MCP - specify allowed params per tool - PR #15346

• Configuration

  • MCP - support setting CA_BUNDLE_PATH - PR #15253
  • Fix: Ensure MCP client stays open during tool call - PR #15391
  • Remove hardcoded "public" schema in migration.sql - PR #15363

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations

  • Fix - Router: add model_name index for O(1) deployment lookups - PR #15113
  • Refactor Utils: extract inner function from client - PR #15234
  • Fix Networking: remove limitations - PR #15302

• Session Management

  • Fix - Sessions not being shared - PR #15388
  • Fix: remove panic from hot path - PR #15396
  • Fix - shared session parsing and usage issue - PR #15440
  • Fix: handle closed aiohttp sessions - PR #15442
  • Fix: prevent session leaks when recreating aiohttp sessions - PR #15443

• SSL/TLS Performance

  • Perf: optimize SSL/TLS handshake performance with prioritized cipher - PR #15398

• Dependencies

  • Upgrades tenacity version to 8.5.0 - PR #15303

• Data Masking

  • Fix - SensitiveDataMasker converts lists to string - PR #15420

General AI Gateway Improvements​

Security​

• General
  • Fix: redact AWS credentials when redact_user_api_key_info enabled - PR #15321

Documentation Updates​

• Provider Documentation

  • Update doc: perf update - PR #15211
  • Add W&B Inference documentation - PR #15278

• Deployment

  • Deletion of docker-compose buggy comment that cause config.yaml based startup fail - PR #15425

New Contributors​

• @Gal-bloch made their first contribution in PR #15219
• @lcfyi made their first contribution in PR #15315
• @ashengstd made their first contribution in PR #15362
• @vkolehmainen made their first contribution in PR #15363
• @jlan-nl made their first contribution in PR #15330
• @BCook98 made their first contribution in PR #15402
• @PabloGmz96 made their first contribution in PR #15425

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.7.rc.1

pip install litellm==1.77.7.rc.1

Key Highlights​

• Dynamic Rate Limiter v3 - Automatically maximizes throughput when capacity is available (< 80% saturation) by allowing lower-priority requests to use unused capacity, then switches to fair priority-based allocation under high load (≥ 80%) to prevent blocking
• Major Performance Improvements - 2.9x lower median latency at 1,000 concurrent users.
• Claude Sonnet 4.5 - Support for Anthropic's new Claude Sonnet 4.5 model family with 200K+ context and tiered pricing
• MCP Gateway Enhancements - Fine-grained tool control, server permissions, and forwardable headers
• AMD Lemonade & Nvidia NIM - New provider support for AMD Lemonade and Nvidia NIM Rerank
• GitLab Prompt Management - GitLab-based prompt management integration

Performance - 2.9x Lower Median Latency​

This update removes LiteLLM router inefficiencies, reducing complexity from O(M×N) to O(1). Previously, it built a new array and ran repeated checks like data["model"] in llm_router.get_model_ids(). Now, a direct ID-to-deployment map eliminates redundant allocations and scans.

As a result, performance improved across all latency percentiles:

• Median latency: 320 ms → 110 ms (−65.6%)
• p95 latency: 850 ms → 440 ms (−48.2%)
• p99 latency: 1,400 ms → 810 ms (−42.1%)
• Average latency: 864 ms → 310 ms (−64%)

Test Setup​

Locust

• Concurrent users: 1,000
• Ramp-up: 500

System Specs

• CPU: 4 vCPUs
• Memory: 8 GB RAM
• LiteLLM Workers: 4
• Instances: 4

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

MCP OAuth 2.0 Support​

This release adds support for OAuth 2.0 Client Credentials for MCP servers. This is great for Internal Dev Tools use-cases, as it enables your users to call MCP servers, with their own credentials. E.g. Allowing your developers to call the Github MCP, with their own credentials.

Set it up today on Claude Code

Scheduled Key Rotations​

This release brings support for scheduling virtual key rotations on LiteLLM AI Gateway.

From this release you can enforce Virtual Keys to rotate on a schedule of your choice e.g every 15 days/30 days/60 days etc.

This is great for Proxy Admins who need to enforce security policies for production workloads.

Get Started

New Models / Updated Models​

New Model Support​

Features​

• Anthropic
  • Add new claude-sonnet-4-5 model family with tiered pricing above 200K tokens - PR #15041
  • Add anthropic/claude-sonnet-4-5 to model price json with prompt caching support - PR #15049
  • Add 200K prices for Sonnet 4.5 - PR #15140
  • Add cost tracking for /v1/messages in streaming response - PR #15102
  • Add /v1/messages/count_tokens to Anthropic routes for non-admin user access - PR #15034
• Gemini
  • Ignore type param for gemini tools - PR #15022
• Vertex AI
  • Add LiteLLM Overhead metric for VertexAI - PR #15040
  • Support googlemap grounding in vertex ai - PR #15179
• Azure
  • Add azure_ai grok-4 model family - PR #15137
  • Use the extra_query parameter for GET requests in Azure Batch - PR #14997
  • Use extra_query for download results (Batch API) - PR #15025
  • Add support for Azure AD token-based authorization - PR #14813
• Ollama
  • Add ollama cloud models - PR #15008
• Groq
  • Add groq/moonshotai/kimi-k2-instruct-0905 - PR #15079
• OpenAI
  • Add support for GPT 5 codex models - PR #14841
• DeepInfra
  • Update DeepInfra model data refresh with latest pricing - PR #14939
• Bedrock
  • Add JP Cross-Region Inference - PR #15188
  • Add "eu.anthropic.claude-sonnet-4-5-20250929-v1:0" - PR #15181
  • Add twelvelabs bedrock Async Invoke Support - PR #14871
• Nvidia NIM
  • Add Nvidia NIM Rerank Support - PR #15152

Bug Fixes​

• VLLM
  • Fix response_format bug in hosted vllm audio_transcription - PR #15010
  • Fix passthrough of atranscription into kwargs going to upstream provider - PR #15005
• OCI
  • Fix OCI Generative AI Integration when using Proxy - PR #15072
• General
  • Fix: Authorization header to use correct "Bearer" capitalization - PR #14764
  • Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - PR #15116
  • Update request handling for original exceptions - PR #15013

New Provider Support​

• AMD Lemonade
  • Add AMD Lemonade provider support - PR #14840

LLM API Endpoints​

Features​

• Responses API

  • Return Cost for Responses API Streaming requests - PR #15053

• /generateContent

  • Add full support for native Gemini API translation - PR #15029

• Passthrough Gemini Routes

  • Add Gemini generateContent passthrough cost tracking - PR #15014
  • Add streamGenerateContent cost tracking in passthrough - PR #15199

• Passthrough Vertex AI Routes

  • Add cost tracking for Vertex AI Passthrough /predict endpoint - PR #15019
  • Add cost tracking for Vertex AI Live API WebSocket Passthrough - PR #14956

• General

  • Preserve Whitespace Characters in Model Response Streams - PR #15160
  • Add provider name to payload specification - PR #15130
  • Ensure query params are forwarded from origin url to downstream request - PR #15087

Management Endpoints / UI​

Features​

• Virtual Keys

  • Ensure LLM_API_KEYs can access pass through routes - PR #15115
  • Support 'guaranteed_throughput' when setting limits on keys belonging to a team - PR #15120

• Models + Endpoints

  • Ensure OCI secret fields not shared on /models and /v1/models endpoints - PR #15085
  • Add snowflake on UI - PR #15083
  • Make UI theme settings publicly accessible for custom branding - PR #15074

• Admin Settings

  • Ensure OTEL settings are saved in DB after set on UI - PR #15118
  • Top api key tags - PR #15151, PR #15156

• MCP

  • show health status of MCP servers - PR #15185
  • allow setting extra headers on the UI - PR #15185
  • allow editing allowed tools on the UI - PR #15185

Bug Fixes​

• Virtual Keys

  • (security) prevent user key from updating other user keys - PR #15201
  • (security) don't return all keys with blank key alias on /v2/key/info - PR #15201
  • Fix Session Token Cookie Infinite Logout Loop - PR #15146

• Models + Endpoints

  • Make UI theme settings publicly accessible for custom branding - PR #15074

• Teams

  • fix failed copy to clipboard for http ui - PR #15195

• Logs

  • fix logs page render logs on filter lookup - PR #15195
  • fix lookup list of end users (migrate to more efficient /customers/list lookup) - PR #15195

• Test key

  • update selected model on key change - PR #15197

• Dashboard

  • Fix LiteLLM model name fallback in dashboard overview - PR #14998

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry
  • Use generation_name for span naming in logging method - PR #14799
• Langfuse
  • Handle non-serializable objects in Langfuse logging - PR #15148
  • Set usage_details.total in langfuse integration - PR #15015
• Prometheus
  • support custom metadata labels on key/team - PR #15094

Guardrails​

• Javelin
  • Add Javelin standalone guardrails integration for LiteLLM Proxy - PR #14983
  • Add logging for important status fields in guardrails - PR #15090
  • Don't run post_call guardrail if no text returned from Bedrock - PR #15106

Prompt Management​

• GitLab
  • GitLab based Prompt manager - PR #14988

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking
  • Proxy: end user cost tracking in the responses API - PR #15124
• Parallel Request Limiter v3
  • Use well known redis cluster hashing algorithm - PR #15052
  • Fixes to dynamic rate limiter v3 - add saturation detection - PR #15119
  • Dynamic Rate Limiter v3 - fixes for detecting saturation + fixes for post saturation behavior - PR #15192
• Teams
  • Add model specific tpm/rpm limits to teams on LiteLLM - PR #15044

MCP Gateway​

• Server Configuration
  • Specify forwardable headers, specify allowed/disallowed tools for MCP servers - PR #15002
  • Enforce server permissions on call tools - PR #15044
  • MCP Gateway Fine-grained Tools Addition - PR #15153
• Bug Fixes
  • Remove servername prefix mcp tools tests - PR #14986
  • Resolve regression with duplicate Mcp-Protocol-Version header - PR #15050
  • Fix test_mcp_server.py - PR #15183

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations
  • +62.5% P99 Latency Improvement - Remove router inefficiencies (from O(M*N) to O(1)) - PR #15046
  • Remove hasattr checks in Router - PR #15082
  • Remove Double Lookups - PR #15084
  • Optimize _filter_cooldown_deployments from O(n×m + k×n) to O(n) - PR #15091
  • Optimize unhealthy deployment filtering in retry path (O(n*m) → O(n+m)) - PR #15110
• Cache Optimizations
  • Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) - PR #15000
  • Avoiding expensive operations when cache isn't available - PR #15182
• Worker Management
  • Add proxy CLI option to recycle workers after N requests - PR #15007
• Metrics & Monitoring
  • LiteLLM Overhead metric tracking - Add support for tracking litellm overhead on cache hits - PR #15045

Documentation Updates​

• Provider Documentation
  • Update litellm docs from latest release - PR #15004
  • Add missing api_key parameter - PR #15058
• General Documentation
  • Use docker compose instead of docker-compose - PR #15024
  • Add railtracks to projects that are using litellm - PR #15144
  • Perf: Last week improvement - PR #15193
  • Sync models GitHub documentation with Loom video and cross-reference - PR #15191

Security Fixes​

• JWT Token Security - Don't log JWT SSO token on .info() log - PR #15145

New Contributors​

• @herve-ves made their first contribution in PR #14998
• @wenxi-onyx made their first contribution in PR #15008
• @jpetrucciani made their first contribution in PR #15005
• @abhijitjavelin made their first contribution in PR #14983
• @ZeroClover made their first contribution in PR #15039
• @cedarm made their first contribution in PR #15043
• @Isydmr made their first contribution in PR #15025
• @serializer made their first contribution in PR #15013
• @eddierichter-amd made their first contribution in PR #14840
• @malags made their first contribution in PR #15000
• @henryhwang made their first contribution in PR #15029
• @plafleur made their first contribution in PR #15111
• @tyler-liner made their first contribution in PR #14799
• @Amir-R25 made their first contribution in PR #15144
• @georg-wolflein made their first contribution in PR #15124
• @niharm made their first contribution in PR #15140
• @anthony-liner made their first contribution in PR #15015
• @rishiganesh2002 made their first contribution in PR #15153
• @danielaskdd made their first contribution in PR #15160
• @JVenberg made their first contribution in PR #15146
• @speglich made their first contribution in PR #15072
• @daily-kim made their first contribution in PR #14764

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.5-stable

pip install litellm==1.77.5

Key Highlights​

• MCP OAuth 2.0 Support - Enhanced authentication for Model Context Protocol integrations
• Scheduled Key Rotations - Automated key rotation capabilities for enhanced security
• New Gemini 2.5 Flash & Flash-lite Models - Latest September 2025 preview models with improved pricing and features
• Performance Improvements - 54% RPS improvement

Performance Improvements - 54% RPS Improvement​

This release brings a 54% RPS improvement (1,040 → 1,602 RPS, aggregated) per instance.

The improvement comes from fixing O(n²) inefficiencies in the LiteLLM Router, primarily caused by repeated use of in statements inside loops over large arrays.

Tests were run with a database-only setup (no cache hits).

Test Setup​

All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable.

System Specs

• CPU: 8 vCPUs
• Memory: 32 GB RAM

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

New Models / Updated Models​

New Model Support​

Features​

• Gemini
  • Added Gemini 2.5 Flash and Flash-lite preview models (September 2025 release) with improved pricing - PR #14948
  • Added new Anthropic web fetch tool support - PR #14951
• XAI
  • Add xai/grok-4-fast models - PR #14833
• Anthropic
  • Updated Claude Sonnet 4 configs to reflect million-token context window pricing - PR #14639
  • Added supported text field to anthropic citation response - PR #14164
• Bedrock
  • Added support for Qwen models family & Deepseek 3.1 to Amazon Bedrock - PR #14845
  • Support requestMetadata in Bedrock Converse API - PR #14570
• Vertex AI
  • Added vertex_ai/qwen models and azure/gpt-5-codex - PR #14844
  • Update vertex ai qwen model pricing - PR #14828
  • Vertex AI Context Caching: use Vertex ai API v1 instead of v1beta1 and accept 'cachedContent' param - PR #14831
• SambaNova
  • Add sambanova deepseek v3.1 and gpt-oss-120b - PR #14866
• OpenAI
  • Fix inconsistent token configs for gpt-5 models - PR #14942
  • GPT-3.5-Turbo price updated - PR #14858
• OpenRouter
  • Add gpt-5 and gpt-5-codex to OpenRouter cost map - PR #14879
• VLLM
  • Fix vllm passthrough - PR #14778
• Flux
  • Support flux image edit - PR #14790

Bug Fixes​

• Anthropic
  • Fix: Support claude code auth via subscription (anthropic) - PR #14821
  • Fix Anthropic streaming IDs - PR #14965
  • Revert incorrect changes to sonnet-4 max output tokens - PR #14933
• OpenAI
  • Fix a bug where openai image edit silently ignores multiple images - PR #14893
• VLLM
  • Fix: vLLM provider's rerank endpoint from /v1/rerank to /rerank - PR #14938

New Provider Support​

• W&B Inference
  • Add W&B Inference to LiteLLM - PR #14416

LLM API Endpoints​

Features​

• General
  • Add SDK support for additional headers - PR #14761
  • Add shared_session parameter for aiohttp ClientSession reuse - PR #14721

Bugs​

• General
  • Fix: Streaming tool call index assignment for multiple tool calls - PR #14587
  • Fix load credentials in token counter proxy - PR #14808

Management Endpoints / UI​

Features​

• Proxy CLI Auth
  • Allow re-using cli auth token - PR #14780
  • Create a python method to login using litellm proxy - PR #14782
  • Fixes for LiteLLM Proxy CLI to Auth to Gateway - PR #14836

Virtual Keys

• Initial support for scheduled key rotations - PR #14877
• Allow scheduling key rotations when creating virtual keys - PR #14960

Models + Endpoints

• Fix: added Oracle to provider's list - PR #14835

Bugs​

• SSO - Fix: SSO "Clear" button writes empty values instead of removing SSO config - PR #14826
• Admin Settings - Remove useful links from admin settings - PR #14918
• Management Routes - Add /user/list to management routes - PR #14868

Logging / Guardrail / Prompt Management Integrations​

Features​

• DataDog
  • Logging - datadog callback Log message content w/o sending to datadog - PR #14909
• Langfuse
  • Adding langfuse usage details for cached tokens - PR #10955
• Opik
  • Improve opik integration code - PR #14888
• SQS
  • Error logging support for SQS Logger - PR #14974

Guardrails​

• LakeraAI v2 Guardrail - Ensure exception is raised correctly - PR #14867
• Presidio Guardrail - Support custom entity types in Presidio guardrail with Union[PiiEntityType, str] - PR #14899
• Noma Guardrail - Add noma guardrail provider to ui - PR #14415

Prompt Management​

• BitBucket Integration - Add BitBucket Integration for Prompt Management - PR #14882

Spend Tracking, Budgets and Rate Limiting​

• Service Tier Pricing - Add service_tier based pricing support for openai (BOTH Service & Priority Support) - PR #14796
• Cost Tracking - Show input, output, tool call cost breakdown in StandardLoggingPayload - PR #14921
• Parallel Request Limiter v3
  • Ensure Lua scripts can execute on redis cluster - PR #14968
  • Fix: get metadata info from both metadata and litellm_metadata fields - PR #14783
• Priority Reservation - Fix: Priority Reservation: keys without priority metadata receive higher priority than keys with explicit priority configurations - PR #14832

MCP Gateway​

• MCP Configuration - Enable custom fields in mcp_info configuration - PR #14794
• MCP Tools - Remove server_name prefix from list_tools - PR #14720
• OAuth Flow - Initial commit for v2 oauth flow - PR #14964

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fix - Fix InMemoryCache unbounded growth when TTLs are set - PR #14869
• Cache Performance - Fix: cache root cause - PR #14827
• Concurrency Fix - Fix concurrency/scaling when many Python threads do streaming using sync completions - PR #14816
• Performance Optimization - Fix: reduce get_deployment cost to O(1) - PR #14967
• Performance Optimization - Fix: remove slow string operation - PR #14955
• DB Connection Management - Fix: DB connection state retries - PR #14925

Documentation Updates​

• Provider Documentation - Fix docs for provider_specific_params.md - PR #14787
• Model References - Update model references from gemini-pro to gemini-2.5-pro - PR #14775
• Letta Guide - Add Letta Guide documentation - PR #14798
• README - Make the README document clearer - PR #14860
• Session Management - Update docs for session management availability - PR #14914
• Cost Documentation - Add documentation for additional cost-related keys in custom pricing - PR #14949
• Azure Passthrough - Add azure passthrough documentation - PR #14958
• General Documentation - Doc updates sept 2025 - PR #14769
  • Clarified bridging between endpoints and mode in docs.
  • Added Vertex AI Gemini API configuration as an alternative in relevant guides. Linked AWS authentication info in the Bedrock guardrails documentation.
  • Added Cancel Response API usage with code snippets
  • Clarified that SSO (Single Sign-On) is free for up to 5 users:
  • Alphabetized sidebar, leaving quick start / intros at top of categories
  • Documented max_connections under cache_params.
  • Clarified IAM AssumeRole Policy requirements.
  • Added transform utilities example to Getting Started (showing request transformation).
  • Added references to models.litellm.ai as the full models list in various docs.
  • Added a code snippet for async_post_call_success_hook.
  • Removed broken links to callbacks management guide. - Reformatted and linked cookbooks + other relevant docs
• Documentation Corrections - Corrected docs updates sept 2025 - PR #14916

New Contributors​

• @uzaxirr made their first contribution in PR #14761
• @xprilion made their first contribution in PR #14416
• @CH-GAGANRAJ made their first contribution in PR #14779
• @otaviofbrito made their first contribution in PR #14778
• @danielmklein made their first contribution in PR #14639
• @Jetemple made their first contribution in PR #14826
• @akshoop made their first contribution in PR #14818
• @hazyone made their first contribution in PR #14821
• @leventov made their first contribution in PR #14816
• @fabriciojoc made their first contribution in PR #10955
• @onlylonly made their first contribution in PR #14845
• @Copilot made their first contribution in PR #14869
• @arsh72 made their first contribution in PR #14899
• @berri-teddy made their first contribution in PR #14914
• @vpbill made their first contribution in PR #14415
• @kgritesh made their first contribution in PR #14893
• @oytunkutrup1 made their first contribution in PR #14858
• @nherment made their first contribution in PR #14933
• @deepanshululla made their first contribution in PR #14974
• @TeddyAmkie made their first contribution in PR #14758
• @SmartManoj made their first contribution in PR #14775
• @uc4w6c made their first contribution in PR #14720
• @luizrennocosta made their first contribution in PR #14783
• @AlexsanderHamir made their first contribution in PR #14827
• @dharamendrak made their first contribution in PR #14721
• @TomeHirata made their first contribution in PR #14164
• @mrFranklin made their first contribution in PR #14860
• @luisfucros made their first contribution in PR #14866
• @huangyafei made their first contribution in PR #14879
• @thiswillbeyourgithub made their first contribution in PR #14949
• @Maximgitman made their first contribution in PR #14965
• @subnet-dev made their first contribution in PR #14938
• @22mSqRi made their first contribution in PR #14972

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.3-stable

pip install litellm==1.77.3

Key Highlights​

• +550 RPS Performance Improvements - Optimizations in request handling and object initialization.
• Priority Quota Reservation - Proxy admins can now reserve TPM/RPM capacity for specific keys.

Priority Quota Reservation​

This release adds support for priority quota reservation. This allows Proxy Admins to reserve specific percentages of model capacity for different use cases.

This is great for use cases where you want to ensure your realtime use cases must always get priority responses and background development jobs can take longer.

This release adds support for priority quota reservation. This allows Proxy Admins to reserve TPM/RPM capacity for keys based on metadata priority levels, ensuring critical production workloads get guaranteed access regardless of development traffic volume.

Get started here

+550 RPS Performance Improvements​

This release delivers significant RPS improvements through targeted optimizations.

We've achieved a +500 RPS boost by fixing cache type inconsistencies that were causing frequent cache misses, plus an additional +50 RPS by removing unnecessary coroutine checks from the hot path.

New Models / Updated Models​

New Model Support​

Features​

• OVHCloud AI Endpoints
  • New provider support with comprehensive model catalog - PR #14494
• CompactifAI
  • New provider integration - PR #14532
• SambaNova
  • Added DeepSeek v3.1 and GPT-OSS-120B models - PR #14500
• Bedrock
  • Cross-region inference profile cost calculation - PR #14566
  • AWS external ID parameter support for authentication - PR #14582
  • CountTokens API implementation - PR #14557
  • Titan V2 encoding_format parameter support - PR #14687
  • Nova Canvas image generation inference profiles - PR #14578
  • Bedrock Batches API - batch processing support with file upload and request transformation - PR #14618
  • Bedrock Twelve Labs embedding provider support - PR #14697
• Vertex AI
  • Gemini labels field provider-aware filtering - PR #14563
  • Gemini Batch API support - PR #14733
• Volcengine
  • Fixed thinking parameters when disabled - PR #14569
• Cohere
  • Handle Generate API deprecation, default to chat endpoints - PR #14676
• TwelveLabs
  • Added Marengo Embed 2.7 embedding support - PR #14674

Bug Fixes​

• Bedrock
  • Empty arguments handling in tool call invocation - PR #14583
• Vertex AI
  • Avoid deepcopy crash with non-pickleables in Gemini/Vertex - PR #14418
• XAI
  • Fix unsupported stop parameter for grok-code models - PR #14565
• Gemini
  • Updated error message for Gemini API - PR #14589
  • Fixed 2.5 Flash Image Preview model routing - PR #14715
  • API key passing for token counting endpoints - PR #14744

New Provider Support​

• OVHCloud AI Endpoints
  • Complete provider integration with model catalog and authentication - PR #14494
• CompactifAI
  • New provider support with documentation - PR #14532

LLM API Endpoints​

Features​

• /responses
  • Added cancel endpoint support for non-admin users - PR #14594
  • Improved response session handling and cold storage configuration with s3 - PR #14534
  • Added OpenAI & Azure /responses/cancel endpoint support - PR #14561
• General
  • Enhanced rate limit error messages with details - PR #14736
  • Middle-truncation for spend log payloads - PR #14637

Bugs​

• /chat/completions
  • Fixed completion chat ID handling - PR #14548
  • Prevent AttributeError for _get_tags_from_request_kwargs - PR #14735
• /responses
  • Fixed cost calculation - PR #14675
• General
  • Rate limiter AttributeError fix - PR #14609

Spend Tracking, Budgets and Rate Limiting​

• Responses API Cost Calculation fix - PR #14675
• Anthropic Cache Token Pricing - Separate 1-hour vs 5-minute cache creation costs - PR #14620, PR #14652
• Indochina Time Timezone support for budget resets - PR #14666
• Soft Budget Alert Cache Issues - Resolved soft budget alert cache issues - PR #14491
• Dynamic Rate Limiter v3 - Priority routing improvements - PR #14734
• Enhanced Rate Limit Errors - More detailed error messages - PR #14736

Management Endpoints / UI​

Features​

• Team Member Service Account Keys - Allow team members to view keys they create - PR #14619
• Default Budget for JWT Teams - Auto-assign budgets to generated teams - PR #14514
• SSO Access Control Groups - Enhanced token info endpoint integration - PR #14738
• Health Test Connect Protection - Restrict access based on model creation permissions - PR #14650
• Amazon Bedrock Guardrail Info View - Enhanced logging visualization - PR #14696

Bug Fixes​

• SCIM v2 - Fix group PUSH and PUT operations for non-existent members - PR #14581
• Guardrail View/Edit/Delete behavior fixes - PR #14622
• In-Memory Guardrail update failures - PR #14653

Logging / Guardrail Integrations​

Features​

• DataDog
  • Enhanced spend tracking metrics - PR #14555
  • Stream support with is_streamed_request parameter - PR #14673
  • Fixed tool calls metadata passing - PR #14531
• Langfuse
  • Added logging support for Responses API - PR #14597
• Langsmith
  • Langsmith Sampling Rate - Key/Team-level tracing configuration - PR #14740
• Prometheus
  • Multi-worker support improvements - PR #14530
  • User email labels in monitoring - PR #14520
• Opik
  • Fixed timezone issue - PR #14708

Bug Fixes​

• S3
  • Fixed 404 error when using s3_endpoint_url - PR #14559

Guardrails​

• Tool Permission Guardrail - Fine-grained tool access control - PR #14519
• Bedrock Guardrails - Selective guarding support with runtime endpoint configuration - PR #14575, PR #14650
• Default Last Message in guardrails - PR #14640
• AWS exceptions handling despite 200 response - PR #14658

New Integration​

• PostHog - Complete observability integration for LiteLLM usage tracking and analytics - PR #14610

MCP Gateway​

• MCP Server Alias Parsing - Multi-part URL path support - PR #14558
• MCP Filter Recomputation - After server deletion - PR #14542
• MCP Gateway Tools List improvements - PR #14695

Performance / Loadbalancing / Reliability improvements​

• +500 RPS Performance Boost when sending the user field - PR #14616
• +50 RPS by removing iscoroutine from hot path - PR #14649
• 7% reduction in init overhead - PR #14689
• Generic Object Pool implementation for better resource management - PR #14702

General Proxy Improvements​

• Middle-Truncation for spend log payloads - PR #14637

Security​

• Security Update - Bump aiohttp==3.12.14, fix CVE-2025-53643 - PR #14638

New Contributors​

• @luisfucros made their first contribution in PR #14500
• @hanakannzashi made their first contribution in PR #14548
• @eliasto made their first contribution in PR #14494
• @Rasmusafj made their first contribution in PR #14491
• @LingXuanYin made their first contribution in PR #14569
• @ronaldpereira made their first contribution in PR #14613
• @hula-la made their first contribution in PR #14534
• @carlos-marchal-ph made their first contribution in PR #14610
• @akraines made their first contribution in PR #14637
• @mrFranklin made their first contribution in PR #14708
• @tcx4c70 made their first contribution in PR #14675
• @michaeltansg made their first contribution in PR #14666
• @tosi29 made their first contribution in PR #14725
• @gmdfalk made their first contribution in PR #14735
• @FelipeRodriguesGare made their first contribution in PR #14733
• @mritunjaysharma394 made their first contribution in PR #14678

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.77.2-stable

pip install litellm==1.77.2.post1

Key Highlights​

• Bedrock Batches API - Support for creating Batch Inference Jobs on Bedrock using LiteLLM's unified batch API (OpenAI compatible)
• Qwen API Tiered Pricing - Cost tracking support for Dashscope (Qwen) models with multiple pricing tiers

New Models / Updated Models​

New Model Support​