text
𝗛𝗼𝘄 𝘄𝗲 𝗺𝗮𝗱𝗲 𝗚𝗿𝗮𝗱𝗶𝗼 𝗳𝗮𝘀𝘁𝗲𝗿 𝗯𝘆... 𝘀𝗹𝗼𝘄𝗶𝗻𝗴 𝗶𝘁 𝗱𝗼𝘄𝗻!
𝗛𝗼𝘄 𝘄𝗲 𝗺𝗮𝗱𝗲 𝗚𝗿𝗮𝗱𝗶𝗼 𝗳𝗮𝘀𝘁𝗲𝗿 𝗯𝘆... 𝘀𝗹𝗼𝘄𝗶𝗻𝗴 𝗶𝘁 𝗱𝗼𝘄𝗻!
new_line
new_line
text
About a month ago, 
About a month ago, 
mention
@oobabooga
oobabooga
text
 (who built the popular text generation webui) reported an interesting issue to the Gradio team. After upgrading to Gradio 4, 
 (who built the popular text generation webui) reported an interesting issue to the Gradio team. After upgrading to Gradio 4, 
mention
@oobabooga
oobabooga
text
 noticed that chatbots that streamed very quickly had a lag before their text would show up in the Gradio app. 
 noticed that chatbots that streamed very quickly had a lag before their text would show up in the Gradio app. 
new_line
new_line
text
After some investigation, we determined that the Gradio frontend would receive the updates from the backend immediately, but the browser would lag before rendering the changes on the screen. The main difference between Gradio 3 and Gradio 4 was that we migrated the communication protocol between the backend and frontend from Websockets (WS) to Server-Side Events (SSE), but we couldn't figure out why this would affect the browser's ability to render the streaming updates it was receiving.
After some investigation, we determined that the Gradio frontend would receive the updates from the backend immediately, but the browser would lag before rendering the changes on the screen. The main difference between Gradio 3 and Gradio 4 was that we migrated the communication protocol between the backend and frontend from Websockets (WS) to Server-Side Events (SSE), but we couldn't figure out why this would affect the browser's ability to render the streaming updates it was receiving.
new_line
new_line
text
After diving deep into browsers events, 
After diving deep into browsers events, 
mention
@aliabid94
aliabid94
text
 and 
 and 
mention
@pngwn
pngwn
text
 made a realization: most browsers treat WS events (specifically the 
 made a realization: most browsers treat WS events (specifically the 
inline_code
`WebSocket.onmessage`
WebSocket.onmessage
text
 function) with a lower priority than SSE events (
 function) with a lower priority than SSE events (
inline_code
`EventSource.onmessage`
EventSource.onmessage
text
 function), which allowed the browser to repaint the window between WS messages. With SSE, the streaming updates would stack up in the browser's event stack and be prioritized over any browser repaint. The browser would eventually clear the stack but it would take some time to go through each update, which produced a lag.
 function), which allowed the browser to repaint the window between WS messages. With SSE, the streaming updates would stack up in the browser's event stack and be prioritized over any browser repaint. The browser would eventually clear the stack but it would take some time to go through each update, which produced a lag.
new_line
new_line
text
We debated different options, but the solution that we implemented was to introduce throttling: we slowed down how frequently we would push updates to the browser event stack to a maximum rate of 20/sec. Although this seemingly “slowed down” Gradio streaming, it actually would allow browsers to process updates in real-time and provide a much better experience to end users of Gradio apps. 
We debated different options, but the solution that we implemented was to introduce throttling: we slowed down how frequently we would push updates to the browser event stack to a maximum rate of 20/sec. Although this seemingly “slowed down” Gradio streaming, it actually would allow browsers to process updates in real-time and provide a much better experience to end users of Gradio apps. 
new_line
new_line
text
See the PR here: 
See the PR here: 
link
https://github.com/gradio-app/gradio/pull/7084
https://github.com/gradio-app/gradio/pull/7084
new_line
new_line
text
Kudos to 
Kudos to 
mention
@aliabid94
aliabid94
text
 and 
 and 
mention
@pngwn
pngwn
text
 for the fix, and to 
 for the fix, and to 
mention
@oobabooga
oobabooga
text
 and 
 and 
mention
@pseudotensor
pseudotensor
text
 for helping us test it out!
 for helping us test it out!
new_line
𝗛𝗼𝘄 𝘄𝗲 𝗺𝗮𝗱𝗲 𝗚𝗿𝗮𝗱𝗶𝗼 𝗳𝗮𝘀𝘁𝗲𝗿 𝗯𝘆... 𝘀𝗹𝗼𝘄𝗶𝗻𝗴 𝗶𝘁 𝗱𝗼𝘄𝗻!

About a month ago, @oobabooga (who built the popular text generation webui) reported an interesting issue to the Gradio team. After upgrading to Gradio 4, @oobabooga noticed that chatbots that streamed very quickly had a lag before their text would show up in the Gradio app. 

After some investigation, we determined that the Gradio frontend would receive the updates from the backend immediately, but the browser would lag before rendering the changes on the screen. The main difference between Gradio 3 and Gradio 4 was that we migrated the communication protocol between the backend and frontend from Websockets (WS) to Server-Side Events (SSE), but we couldn't figure out why this would affect the browser's ability to render the streaming updates it was receiving.

After diving deep into browsers events, @aliabid94 and @pngwn made a realization: most browsers treat WS events (specifically the `WebSocket.onmessage` function) with a lower priority than SSE events (`EventSource.onmessage` function), which allowed the browser to repaint the window between WS messages. With SSE, the streaming updates would stack up in the browser's event stack and be prioritized over any browser repaint. The browser would eventually clear the stack but it would take some time to go through each update, which produced a lag.

We debated different options, but the solution that we implemented was to introduce throttling: we slowed down how frequently we would push updates to the browser event stack to a maximum rate of 20/sec. Although this seemingly “slowed down” Gradio streaming, it actually would allow browsers to process updates in real-time and provide a much better experience to end users of Gradio apps. 

See the PR here: https://github.com/gradio-app/gradio/pull/7084

Kudos to @aliabid94 and @pngwn for the fix, and to @oobabooga and @pseudotensor for helping us test it out!

👍
kramp osanseviero pngwn Ifeanyi xpgx1 BramVanroy takeraparterer samusenps clem phanes gate369 fffiloni sbarman25 taufiqdp akhaliq ysharma davanstrien gsarti lunarflu julien-c merve hysts freddyaboulton
❤️
clem akhaliq ysharma kramp santiviquez gsarti lunarflu julien-c
🤗
takeraparterer clem akhaliq ysharma gsarti lunarflu julien-c
🤯
lunarflu julien-c sbrandeis
🤝
julien-c
https://cdn-avatars.huggingface.co/v1/production/uploads/61868ce808aae0b5499a2a95/F6BA0anbsoY_Z7M1JrwOe.jpeg
Sylvain Filoni
fffiloni
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1624431552569-noauth.jpeg
yuvraj sharma
ysharma
user
/avatars/53fe807d9ffaf2c23ac8a13756a2486b.svg
Nick Durkee
ndurkee
user