llama-swap

llama-swap

llama-swap is a lightweight, transparent proxy server designed for automatic model swapping with llama.cpp or any local OpenAPI compatible server. Written in Golang, it is easy to install and configure, requiring only a single binary and a simple YAML configuration file.

Key Features:

• Automatic Model Swapping: Automatically replaces the upstream server with the correct one based on the model requested.
• Simple Configuration: Uses a single YAML file for configuration, making it user-friendly.
• Multiple Model Support: Can handle multiple models simultaneously through profiles.
• Docker Support: Easily deployable using Docker, with pre-built images available.
• Health Monitoring: Includes health checks and logging capabilities for monitoring server status.

Benefits:

• Flexibility: Works with any OpenAI compatible server, not just llama-server.
• Performance Optimization: Supports speculative decoding and code generation optimization for improved inference speeds.
• Resource Management: Provides control over system resources and automatic unloading of models after a specified timeout.

Highlights:

• Supports various OpenAI API endpoints including completions, chat completions, embeddings, and more.
• Easy to deploy on bare metal or via Docker, with pre-built binaries available for multiple operating systems.
• Community-driven with active contributions and updates.