Transformers Agent

Transformers Agent is an experimental API which is subject to change at any time. Results returned by the agents can vary as the APIs or underlying models are prone to change.

Transformers version v4.29.0, building on the concept of tools and agents. You can play with in this colab.

In short, it provides a natural language API on top of transformers: we define a set of curated tools and design an agent to interpret natural language and to use these tools. It is extensible by design; we curated some relevant tools, but we’ll show you how the system can be extended easily to use any tool developed by the community.

Let’s start with a few examples of what can be achieved with this new API. It is particularly powerful when it comes to multimodal tasks, so let’s take it for a spin to generate images and read text out loud.

agent.run("Caption the following image", image=image)

Input 	Output
	A beaver is swimming in the water

agent.run("Read the following text out loud", text=text)

Input 	Output
A beaver is swimming in the water 	
your browser does not support the audio element. </audio>

agent.run(
    "In the following `document`, where will the TRRF Scientific Advisory Council Meeting take place?",
    document=document,
)

Input 	Output
	ballroom foyer
Quickstart

Before being able to use agent.run, you will need to instantiate an agent, which is a large language model (LLM). We provide support for openAI models as well as opensource alternatives from BigCode and OpenAssistant. The openA