Streaming partial results from hosted text-generation APIs?
Is it possible to call the hosted text-generation APIs in such a way as to get low-latency partial streaming results, without having to wait for the full completion to be returned as JSON?OpenAI has astreamparameter, documented here:beta.openai.comOpenAI APIAn API for accessing new AI models developed by OpenAIAnd InferKit has astreamResponseparameter, documented here:https://inferkit.com/docs/api/generationBut I can’t find anything similar in the Huggingface API docs:huggingface.coDetailed parametersWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2021-12-09T21:10:45Z
Based on the lack of response, I’m assuming this isn’t currently possible with the hosted Huggingface APIs. Is this the kind of thing that might easily be implemented, if I file a feature-request ticket on the github project?
2021-12-10T09:11:00Z
Hey@benjismith. Sorry for the slow response. The API has support for streaming as documented here:Parallelism and batch jobs — Api inference documentation. It has a small example that we hope helps you.cc@Narsil
2021-12-10T10:47:13Z
Hi@benjismith,For long generation, we currently don’t have a chunking option like InferKit seems to propose.What we do have is a parametermax_timeto limit the time of the in flight request (since latency seems to depend on actual usage and user, if you’re doing live suggestions, then time to the first suggestion is really important).The streaming mentioned by@osansevierois another option, but more fit for “batch-like” jobs, where you want to feed a GPU and you are not as latency sensitive (so you can send all your work at once, and just wait for the answers).If I understand correctly, for long generation this would not be the case since every new generation would depend on the previously generated token. Yet definitely a viable option.Here is an example of how that would work in Python:import json
import os

import requests

API_TOKEN = os.getenv("HF_API_TOKEN")
headers = {"Authorization": f"Bearer {API_TOKEN}"}
API_URL = "https://api-inference.huggingface.co/models/gpt2"


def streaming(payload):

    inputs = ""
    for i in range(10):
        data = json.dumps(payload)
        response = requests.request("POST", API_URL, headers=headers, data=data)
        return_data = json.loads(response.content.decode("utf-8"))
        inputs = return_data[0]["generated_text"]
        payload["inputs"] += inputs
        print("---" * 20)
        print(payload["inputs"])
        print("---" * 20)


streaming(
    {
        "inputs": "The answer to the universe is",
        "parameters": {"max_time": 1, "return_full_text": False},
    }
)Would something like that be viable ?
2021-12-11T00:59:32Z
At any rate, if I wanted to file an enhancement request in a github ticket for something like this, should I create it in thetransformersproject, or thehuggingface_hubproject?
2023-01-22T20:02:18Z
@benjismith, Did you ever file the request?
2023-03-07T14:43:58Z
Hey guys! I’ve made a streaming generation service for Hugging Face transformers that is fully compatible with the OpenAI API:https://github.com/hyperonym/basaranIt also comes with a fancy playground.
2023-08-18T04:14:16Z
document16923320146901515×674 21.6 KB