How to implement chatbot streaming in gradio with a function
Hi everyone.I am trying to build a basicchatbot with streaming.from the documentation we have the following code.import gradio as gr
import random
import time

with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    msg = gr.Textbox()
    clear = gr.Button("Clear")

    def respond(message, chat_history):
        bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
        chat_history.append((message, bot_message))
        time.sleep(2)
        return "", chat_history

    msg.submit(respond, [msg, chatbot], [msg, chatbot])
    clear.click(lambda: None, None, chatbot, queue=False)

if __name__ == "__main__":
    demo.launch()and I wrote this functionimport os
os.environ["OPENAI_API_KEY"] = '  dsdsds'

from pathlib import Path
from llama_index import LLMPredictor, GPTVectorStoreIndex, PromptHelper, ServiceContext
from llama_index import StorageContext, load_index_from_storage
from langchain.chat_models import ChatOpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler


llm_predictor = LLMPredictor(
    llm=ChatOpenAI(
        streaming=True,
        callbacks=[StreamingStdOutCallbackHandler()],
        temperature=0.7,
        model_name="gpt-3.5-turbo",
        max_tokens=1000
    )
)


max_input_size = 4096
max_chunk_overlap = 0.2
chunk_size_limit = 60
num_outputs = 1000
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
storage_context = StorageContext.from_defaults(persist_dir="./storage")

index = load_index_from_storage(storage_context, service_context=service_context)

question = 'how many times has my son watched the curious george movie?'
query_engine = index.as_query_engine()
response = query_engine.query(question)
#print(response)How can I combine them together to make a chatbot with Streaming?I tried the following code:import os
os.environ["OPENAI_API_KEY"] = 'dsdsdsddsdsd'

from pathlib import Path
from llama_index import LLMPredictor, GPTVectorStoreIndex, PromptHelper, ServiceContext
from llama_index import StorageContext, load_index_from_storage
from langchain.chat_models import ChatOpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
import gradio as gr
import random
import time

llm_predictor = LLMPredictor(
    llm=ChatOpenAI(
        streaming=True,
        callbacks=[StreamingStdOutCallbackHandler()],
        temperature=0.7,
        model_name="gpt-3.5-turbo",
        max_tokens=2000
    ))
    

max_input_size = 4096
max_chunk_overlap = 0.2
chunk_size_limit = 600
num_outputs = 2000
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context, service_context=service_context)
query_engine = index.as_query_engine()

with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    msg = gr.Textbox()
    clear = gr.Button("Clear")

    def user(user_message, history):
        return gr.update(value="", interactive=False), history + [[user_message, None]]

    def bot(history):
        bot_message = query_engine.query(history[len(history)-1][0])
        history[-1][1] = ""
        for character in bot_message:
            history[-1][1] += character
            time.sleep(0.05)
            yield history

    response = msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
        bot, chatbot, chatbot
    )
    response.then(lambda: gr.update(interactive=True), None, [msg], queue=False)
    clear.click(lambda: None, None, chatbot, queue=False)

demo.queue()
demo.launch()but it saysTypeError: ‘Response’ object is not iterableAny kind or hint is greatly appreciated
2023-06-21T00:48:03Z
Please allow me to bump this topic.
2023-06-24T02:31:36Z
kind bump.any hint even if small is greatly appreciated
2023-06-25T01:00:24Z
hi@kamiyan89,can you please take a look at this example? It’s a chatbot streaming with Gradio and OpenAI apihuggingface.coapp.py · ysharma/ChatGPT4 at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.More updated Guide using the news ChatBot Componentgradio.appCreating A Chatbot FastA Step-by-Step Gradio Tutorial
2023-06-26T07:03:27Z
Hi Radames,Thank you very much for the suggestion.I am a total beginner so there are a lot of things I don’t understand from the code but I will do my best to understand it.If questions arise could you allow me to follow up with additional questions?In the meanwhile if there are other suggestions they are greatly appreciated.
2023-08-12T08:11:28Z
@kamiyan89Hi, I too am a beginner and I noticed that I’m facing a similar issue where I am using AzureChatOpenAI with streaming set to True. More specifically, I am unable to stream the chunks on gradio as they’re being generated by the LLM. In case you’ve resolved this issue kindly share the details, it would be really helpful!