tabby/README.md

🐾 Tabby

Warning Tabby is still in the alpha phrase

An opensource / on-prem alternative to GitHub Copilot.

Features

• Self-contained, with no need for a DBMS or cloud service
• Web UI for visualizing and configuration models and MLOps.
• OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE).
• Consumer level GPU supports (FP-16 weight loading with various optimization).

Get started

Docker

The easiest way of getting started is using the deployment/docker-compose.yml:

docker-compose up

Note: To use GPUs, you need to install the NVIDIA Container Toolkit. We also recommend using NVIDIA drivers with CUDA version 11.8 or higher.

You can then query the server using /v1/completions endpoint:

curl -X POST http://localhost:5000/v1/completions -H 'Content-Type: application/json' --data '{
    "prompt": "def binarySearch(arr, left, right, x):\n    mid = (left +"
}'

We also provides an interactive playground in admin panel localhost:8501

API documentation

Tabby opens an FastAPI server at localhost:5000, which embeds an OpenAPI documentation of the HTTP API.

Development

Go to development directory.

make dev

or

make dev-python  # Turn off triton backend (for non-cuda env developers)

TODOs

• DuckDB integration, to plot metrics in admin panel (e.g acceptance rate). #24
• Fine-tuning models on private code repository. #23
• Production ready (Open Telemetry, Prometheus metrics).
• Token streaming using Server-Sent Events (SSE)