root
/
tabby
mirror of https://gitee.com/zhang_1334717033/tabby.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs: change `consumer` to `client`

release-0.2
Meng Zhang 2023-09-30 17:42:24 -07:00 committed by GitHub
parent 6348018d38
commit f7ecab5bca
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
website/blog/2023-09-30-stream-laziness-in-tabby/index.md
View File

@ -50,7 +50,7 @@ function server(llm) {
app.listen(8080); app.listen(8080);
} }
async function consumer() { async function client() {
const resp = await fetch('http://localhost:8080'); const resp = await fetch('http://localhost:8080');
// Read values from our stream // Read values from our stream
@ -64,12 +64,12 @@ async function consumer() {
} }
server(llm()); server(llm());
consumer(); client();
``` ```
## Stream Laziness ## Stream Laziness
If you were to run this program, you'd notice something interesting. We'll observe the LLM continuing to output `producing ${i}` even after the consumer has finished reading three times. This might seem obvious, given that the LLM is generating an infinite stream of integers. However, it represents a problem: our server must maintain an ever-expanding queue of items that have been pushed in but not pulled out. If you were to run this program, you'd notice something interesting. We'll observe the LLM continuing to output `producing ${i}` even after the client has finished reading three times. This might seem obvious, given that the LLM is generating an infinite stream of integers. However, it represents a problem: our server must maintain an ever-expanding queue of items that have been pushed in but not pulled out.
Moreover, the workload involved in creating the stream is typically both expensive and time-consuming, such as computation workload on the GPU. But what if the client aborts the in-flight request due to a network issue or other intended behaviors? Moreover, the workload involved in creating the stream is typically both expensive and time-consuming, such as computation workload on the GPU. But what if the client aborts the in-flight request due to a network issue or other intended behaviors?