296342efd8
refactor: use llama.cpp tokenizer ( #683 )
...
* refactor: switch to llama.cpp tokenizer to simplify implementation
* refactor: remove tokenizer dependency from tabby
* refactor: renaming decoding to stop condition
* refactor: remove tokenizer dependency
* refactor: remove submodule
* chore: update formatting
* move tokenization to c++
2023-10-31 22:16:09 +00:00
99a7053b6f
refactor: extract language configuration into individual toml file ( #564 )
...
* refactor: extract language configuration into individual toml file
* feat: add golang language configuration (#565 )
2023-10-16 00:24:44 +00:00
8c09f75360
refactor: extract language related data into languages.rs ( #518 )
...
* refactor: extract language related data into languages.rs
* fix
* cleanup index
* fix
* further sanitize
* add a score threshold
2023-10-07 01:40:21 +00:00
fd2a1ab865
fix: switch back to regex based implementation for stop words ( #513 )
2023-10-06 09:04:37 +00:00
ce20bd6154
refactor: use RegexSet for cleaer stop regex construction ( #499 )
...
* fix: add a regression test cased for stop words regex matching
* refactor: use RegexSet for cleaer stop regex construction
2023-10-02 23:21:51 +00:00
f05dd3a2f6
refactor: cleanup chat api make it message oriented ( #497 )
...
* refactor: refactor into /chat/completions api
* Revert "feat: support request level stop words (#492 )"
This reverts commit 0d6840e372
2023-10-02 15:39:15 +00:00
0d6840e372
feat: support request level stop words ( #492 )
2023-09-29 18:21:57 +00:00
486e507079
fix: correct Decoding behavior in incremental manner ( #491 )
...
* feat: implement IncrementalDecoding
* refactor: use IncrementalDecoding for ctranslate2
* refactor: rename StopWords to DecodingFactory
* refactor: move decoding logic to tabby-inference
* feat: optimize decoding range
* cleanup
2023-09-29 13:06:47 +00:00
Meng Zhang