ebbe6e5af8
fix: helpful message when llama.cpp submodule is not present ( #719 ) ( #775 )
2023-11-13 07:51:46 +00:00
138b7459c5
feat: add LLAMA_CPP_N_THREADS env ( #742 )
...
* feat: add LLAMA_CPP_N_THREADS and LLAMA_CPP_N_THREADS_BATCH envs
* apply format
* improve: use LLAMA_CPP_N_THREADS for both n_threads and n_threads_batch
* Update crates/llama-cpp-bindings/src/engine.cc
---------
Co-authored-by: Meng Zhang <meng@tabbyml.com>
2023-11-09 19:54:23 +00:00
8c669dee8e
fix: llama.cpp queuing logic ( #741 )
2023-11-09 08:29:54 +00:00
cde3602877
feat: sync llama.cpp to latest
2023-11-08 16:06:09 -08:00
8ab35b2639
feat: add --parallelism to control throughput and vram usage ( #727 )
...
* feat: add --parallelism to control throughput and vram usage
* update default
* Revert "update default"
This reverts commit 349792c0d48d913dcd8be4ce1c9d7ce887918f29.
* cargo fmt
2023-11-08 18:31:22 +00:00
1ad0d39903
fix: deadlock between background job and requests ( #720 )
...
* fix: deadlock between background job and requests
* refactor: extract LlamaService
2023-11-07 13:11:28 -08:00
ca52ac4b01
fix: support cpu only run in llama.cpp cuda build
2023-11-06 22:59:24 -08:00
eb7ae96157
fix: llama.cpp requires kv cache to be N_CTX * parallelism ( #714 )
2023-11-07 06:16:36 +00:00
9344c32b31
fix: when there's an error happens in background inference loop, it should exit the process ( #713 )
2023-11-06 20:41:49 +00:00
00e0c4fddc
chore: add machete check to ensure no unused dependencies ( #701 )
...
* refactor: remove useless dependencies
* add machete
2023-11-05 02:48:05 +00:00
64e0abb8cc
fix(llama.cpp): wrongly index for n_seq in warmup
2023-11-04 17:53:22 -07:00
c7c67c2f90
fix: llama.cpp warmp logic
2023-11-04 14:28:04 -07:00
fc9c9f644b
Release 0.6.0-dev
...
http-api-bindings@0.6.0-dev
llama-cpp-bindings@0.6.0-dev
tabby@0.6.0-dev
tabby-common@0.6.0-dev
tabby-download@0.6.0-dev
tabby-inference@0.6.0-dev
tabby-scheduler@0.6.0-dev
Generated by cargo-workspaces
2023-11-03 18:04:12 -07:00
ec8d88de0d
chore: release 0.5.0 ( #697 )
...
* Release 0.5.0-rc.0
http-api-bindings@0.5.0-rc.0
llama-cpp-bindings@0.5.0-rc.0
tabby@0.5.0-rc.0
tabby-common@0.5.0-rc.0
tabby-download@0.5.0-rc.0
tabby-inference@0.5.0-rc.0
tabby-scheduler@0.5.0-rc.0
Generated by cargo-workspaces
* fix: docker branch tag should only generate when not empty
* Release 0.5.0-rc.1
http-api-bindings@0.5.0-rc.1
llama-cpp-bindings@0.5.0-rc.1
tabby@0.5.0-rc.1
tabby-common@0.5.0-rc.1
tabby-download@0.5.0-rc.1
tabby-inference@0.5.0-rc.1
tabby-scheduler@0.5.0-rc.1
Generated by cargo-workspaces
* fix: handlebar syntax in meta action
* Release 0.5.0-rc.2
http-api-bindings@0.5.0-rc.2
llama-cpp-bindings@0.5.0-rc.2
tabby@0.5.0-rc.2
tabby-common@0.5.0-rc.2
tabby-download@0.5.0-rc.2
tabby-inference@0.5.0-rc.2
tabby-scheduler@0.5.0-rc.2
Generated by cargo-workspaces
* fix: handlebar syntax in meta action
* Release 0.5.0-rc.3
http-api-bindings@0.5.0-rc.3
llama-cpp-bindings@0.5.0-rc.3
tabby@0.5.0-rc.3
tabby-common@0.5.0-rc.3
tabby-download@0.5.0-rc.3
tabby-inference@0.5.0-rc.3
tabby-scheduler@0.5.0-rc.3
Generated by cargo-workspaces
* docs: update change log and docs
* fix: collect_snippet should handle NotReady error
* Release 0.5.0-rc.4
http-api-bindings@0.5.0-rc.4
llama-cpp-bindings@0.5.0-rc.4
tabby@0.5.0-rc.4
tabby-common@0.5.0-rc.4
tabby-download@0.5.0-rc.4
tabby-inference@0.5.0-rc.4
tabby-scheduler@0.5.0-rc.4
Generated by cargo-workspaces
* Release 0.5.0
http-api-bindings@0.5.0
llama-cpp-bindings@0.5.0
tabby@0.5.0
tabby-common@0.5.0
tabby-download@0.5.0
tabby-inference@0.5.0
tabby-scheduler@0.5.0
Generated by cargo-workspaces
2023-11-03 18:02:03 -07:00
acb3a33d78
fix: handle non utf-8 / utf-16 error
2023-11-02 16:29:30 -07:00
eb34850a5e
fix: output err if step failed
2023-11-02 16:15:11 -07:00
4c7eae584e
feat: add model warmup logic ( #693 )
2023-11-02 23:07:32 +00:00
296342efd8
refactor: use llama.cpp tokenizer ( #683 )
...
* refactor: switch to llama.cpp tokenizer to simplify implementation
* refactor: remove tokenizer dependency from tabby
* refactor: renaming decoding to stop condition
* refactor: remove tokenizer dependency
* refactor: remove submodule
* chore: update formatting
* move tokenization to c++
2023-10-31 22:16:09 +00:00
89a63dbf33
fix: when send failed, treat the request as stopped ( #673 )
2023-10-30 06:27:09 +00:00
7330d75de6
chore: clear cache when there's no active requests
2023-10-29 16:30:30 -07:00
7bd99d14c0
feat: support continuous batching in llama.cpp backend ( #659 )
...
* refactor: switch back to llama batch interface
* feat: support cont batching
2023-10-28 23:37:05 -07:00
444222683a
fix(llama.cpp): bump upstream fix for starcoder model on cuda
2023-10-28 02:03:34 -07:00
9309e0314f
fix: fix docker build
2023-10-27 21:25:45 -07:00
6dd12ce1ec
fix: adding cuda search path to docker build.
2023-10-27 19:40:35 -07:00
2d948639be
fix: docker build for llama cuda backend
2023-10-27 16:36:54 -07:00
23bd542cec
feat: switch cuda backend to llama.cpp ( #656 )
...
* feat: switch cuda backend to llama.cpp
* fix
* fix
2023-10-27 13:41:22 -07:00
f37840566b
feat: upgrade llama.cpp ( #645 )
...
* feat: upgrade llama.cpp
* update download files
* update changelog
* Update CHANGELOG.md
* Update CHANGELOG.md
2023-10-27 12:18:46 -07:00
1a4c2aa71f
feat: swtich cpu backend to llama.cpp ( #638 )
...
* feat: swtich Cpu backend to llama.cpp
* feat: switch cpu serving to ggml
* fix cargo.toml
* use optional dependency
* fix compliation
* update ci target
2023-10-25 15:40:11 -07:00
e171776774
Release 0.5.0-dev
...
ctranslate2-bindings@0.5.0-dev
http-api-bindings@0.5.0-dev
llama-cpp-bindings@0.5.0-dev
rust-cxx-cmake-bridge@0.5.0-dev
tabby@0.5.0-dev
tabby-common@0.5.0-dev
tabby-download@0.5.0-dev
tabby-inference@0.5.0-dev
tabby-scheduler@0.5.0-dev
Generated by cargo-workspaces
2023-10-24 13:05:33 -07:00
99a7053b6f
refactor: extract language configuration into individual toml file ( #564 )
...
* refactor: extract language configuration into individual toml file
* feat: add golang language configuration (#565 )
2023-10-16 00:24:44 +00:00
82e893d569
Release 0.4.0-dev
...
ctranslate2-bindings@0.4.0-dev
http-api-bindings@0.4.0-dev
llama-cpp-bindings@0.4.0-dev
rust-cxx-cmake-bridge@0.4.0-dev
tabby@0.4.0-dev
tabby-common@0.4.0-dev
tabby-download@0.4.0-dev
tabby-inference@0.4.0-dev
tabby-scheduler@0.4.0-dev
Generated by cargo-workspaces
2023-10-13 17:54:14 -07:00
4dbaf4f312
Release 0.3.0
...
ctranslate2-bindings@0.3.0
http-api-bindings@0.3.0
llama-cpp-bindings@0.3.0
rust-cxx-cmake-bridge@0.3.0
tabby@0.3.0
tabby-common@0.3.0
tabby-download@0.3.0
tabby-inference@0.3.0
tabby-scheduler@0.3.0
Generated by cargo-workspaces
2023-10-13 17:45:07 -07:00
eb463ba496
Release 0.3.0-rc.1
...
ctranslate2-bindings@0.3.0-rc.1
http-api-bindings@0.3.0-rc.1
llama-cpp-bindings@0.3.0-rc.1
rust-cxx-cmake-bridge@0.3.0-rc.1
tabby@0.3.0-rc.1
tabby-common@0.3.0-rc.1
tabby-download@0.3.0-rc.1
tabby-inference@0.3.0-rc.1
tabby-scheduler@0.3.0-rc.1
Generated by cargo-workspaces
2023-10-13 11:43:34 -07:00
182aceed41
Release 0.3.0-rc.0
...
ctranslate2-bindings@0.3.0-rc.0
http-api-bindings@0.3.0-rc.0
llama-cpp-bindings@0.3.0-rc.0
tabby@0.3.0-rc.0
tabby-common@0.3.0-rc.0
tabby-download@0.3.0-rc.0
tabby-inference@0.3.0-rc.0
tabby-scheduler@0.3.0-rc.0
Generated by cargo-workspaces
2023-10-13 11:24:36 -07:00
6dbb712918
Release 0.3.0-dev
...
ctranslate2-bindings@0.3.0-dev
http-api-bindings@0.3.0-dev
llama-cpp-bindings@0.3.0-dev
tabby@0.3.0-dev
tabby-common@0.3.0-dev
tabby-download@0.3.0-dev
tabby-inference@0.3.0-dev
tabby-scheduler@0.3.0-dev
Generated by cargo-workspaces
2023-10-09 19:39:27 -07:00
1731c3075e
chore: Update version to 0.2.0
2023-10-03 13:32:21 -07:00
692c2fe0fd
Release 0.2.0-rc.0
...
ctranslate2-bindings@0.2.0-rc.0
http-api-bindings@0.2.0-rc.0
llama-cpp-bindings@0.2.0-rc.0
tabby@0.2.0-rc.0
tabby-common@0.2.0-rc.0
tabby-download@0.2.0-rc.0
tabby-inference@0.2.0-rc.0
tabby-scheduler@0.2.0-rc.0
Generated by cargo-workspaces
2023-10-02 19:14:12 -07:00
f05dd3a2f6
refactor: cleanup chat api make it message oriented ( #497 )
...
* refactor: refactor into /chat/completions api
* Revert "feat: support request level stop words (#492 )"
This reverts commit 0d6840e372
2023-10-02 15:39:15 +00:00
dfdd0373a6
fix: when llama model loads failed, panic in rust stack
2023-10-01 22:25:25 -07:00
2171ba72ff
refactor: cleanup llama cpp implementations to fix warnings ( #495 )
2023-09-30 08:37:36 -07:00
0d6840e372
feat: support request level stop words ( #492 )
2023-09-29 18:21:57 +00:00
486e507079
fix: correct Decoding behavior in incremental manner ( #491 )
...
* feat: implement IncrementalDecoding
* refactor: use IncrementalDecoding for ctranslate2
* refactor: rename StopWords to DecodingFactory
* refactor: move decoding logic to tabby-inference
* feat: optimize decoding range
* cleanup
2023-09-29 13:06:47 +00:00
5d9ca6928c
feat: update llama.cpp ( #488 )
...
* feat: update llama.cpp
* remove useless include
2023-09-28 23:59:59 +00:00
56b7b850af
fix: Linkage issue on latest xcode commandline tools clang ( #486 )
2023-09-28 17:46:02 +00:00
44f013f26e
feat: add /generate and /generate_streaming ( #482 )
...
* feat: add generate_stream interface
* extract engine::create_engine
* feat add generate::generate
* support streaming in llama.cpp
* support streaming in ctranslate2
* update
* fix formatting
* refactor: extract helpers functions
2023-09-28 17:20:50 +00:00
97eeb6b926
feat: update llama.cpp to fetch latest starcoder support ( #452 )
...
* feat: bump llama.cpp to HEAD
* fix: turn off add_bos by default
2023-09-16 03:41:49 +00:00
30afa19bc0
feat: add LLAMA_CPP_LOG_LEVEL to control log level of llama.cpp ( #436 )
2023-09-12 14:41:39 +00:00
ad3b974d5c
feat: implement input truncation for llama-cpp-bindings ( #416 )
...
* feat: implement input truncation for llama-cpp-bindings
* set max input length to 1024
* fix: batching tokens with n_batches
* fix batching
2023-09-09 00:20:51 +08:00
e93a971d0e
feat: tune llama metal backend performance ( #393 )
...
* feat: support eos based stop
* feat: print performance stats after each inference
* update llama.cpp
* update commits
2023-09-05 10:14:29 +08:00
3c7c8d9293
feat: add cargo test to github actions and run only unit tests in ci [TAB-185] ( #390 )
...
* feat: add cargo test to github actions
* chore: fix lint
* chore: add openblas dependency
* chore: update build dependency
* chore: resolve comments
* chore: fix lint
* chore: fix lint
* chore: test installing dependencies
* chore: refactor integ test
* update ci
* cleanup
---------
Co-authored-by: Meng Zhang <meng@tabbyml.com>
2023-09-03 05:04:52 +00:00
Maciej