Spaces:
Running
Running
File size: 5,076 Bytes
6051832 fcddd97 6051832 fcddd97 6051832 8910869 6051832 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
---
title: Gemini Live API Demo - 3 Dots
emoji: 🟢
colorFrom: blue
colorTo: red
sdk: docker
pinned: false
app_port: 3001
---
# Multimodal Live API - Web console
This repository contains a react-based starter app for using the [Multimodal Live API](<[https://ai.google.dev/gemini-api](https://ai.google.dev/api/multimodal-live)>) over a websocket. It provides modules for streaming audio playback, recording user media such as from a microphone, webcam or screen capture as well as a unified log view to aid in development of your application.
[](https://www.youtube.com/watch?v=J_q7JY1XxFE)
Watch the demo of the Multimodal Live API [here](https://www.youtube.com/watch?v=J_q7JY1XxFE).
## Usage
To get started, [create a free Gemini API key](https://aistudio.google.com/apikey) and add it to the `.env` file. Then:
```
$ npm install && npm start
```
We have provided several example applications on other branches of this repository:
- [demos/GenExplainer](https://github.com/google-gemini/multimodal-live-api-web-console/tree/demos/genexplainer)
- [demos/GenWeather](https://github.com/google-gemini/multimodal-live-api-web-console/tree/demos/genweather)
- [demos/GenList](https://github.com/google-gemini/multimodal-live-api-web-console/tree/demos/genlist)
## Example
Below is an example of an entire application that will use Google Search grounding and then render graphs using [vega-embed](https://github.com/vega/vega-embed):
```typescript
import { type FunctionDeclaration, SchemaType } from "@google/generative-ai";
import { useEffect, useRef, useState, memo } from "react";
import vegaEmbed from "vega-embed";
import { useLiveAPIContext } from "../../contexts/LiveAPIContext";
export const declaration: FunctionDeclaration = {
name: "render_altair",
description: "Displays an altair graph in json format.",
parameters: {
type: SchemaType.OBJECT,
properties: {
json_graph: {
type: SchemaType.STRING,
description:
"JSON STRING representation of the graph to render. Must be a string, not a json object",
},
},
required: ["json_graph"],
},
};
export function Altair() {
const [jsonString, setJSONString] = useState<string>("");
const { client, setConfig } = useLiveAPIContext();
useEffect(() => {
setConfig({
model: "models/gemini-2.0-flash-exp",
systemInstruction: {
parts: [
{
text: 'You are my helpful assistant. Any time I ask you for a graph call the "render_altair" function I have provided you. Dont ask for additional information just make your best judgement.',
},
],
},
tools: [{ googleSearch: {} }, { functionDeclarations: [declaration] }],
});
}, [setConfig]);
useEffect(() => {
const onToolCall = (toolCall: ToolCall) => {
console.log(`got toolcall`, toolCall);
const fc = toolCall.functionCalls.find(
(fc) => fc.name === declaration.name
);
if (fc) {
const str = (fc.args as any).json_graph;
setJSONString(str);
}
};
client.on("toolcall", onToolCall);
return () => {
client.off("toolcall", onToolCall);
};
}, [client]);
const embedRef = useRef<HTMLDivElement>(null);
useEffect(() => {
if (embedRef.current && jsonString) {
vegaEmbed(embedRef.current, JSON.parse(jsonString));
}
}, [embedRef, jsonString]);
return <div className="vega-embed" ref={embedRef} />;
}
```
## development
This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).
Project consists of:
- an Event-emitting websocket-client to ease communication between the websocket and the front-end
- communication layer for processing audio in and out
- a boilerplate view for starting to build your apps and view logs
## Available Scripts
In the project directory, you can run:
### `npm start`
Runs the app in the development mode.\
Open [http://localhost:3000](http://localhost:3000) to view it in the browser.
The page will reload if you make edits.\
You will also see any lint errors in the console.
### `npm run build`
Builds the app for production to the `build` folder.\
It correctly bundles React in production mode and optimizes the build for the best performance.
The build is minified and the filenames include the hashes.\
Your app is ready to be deployed!
See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.
_This is an experiment showcasing the Multimodal Live API, not an official Google product. We'll do our best to support and maintain this experiment but your mileage may vary. We encourage open sourcing projects as a way of learning from each other. Please respect our and other creators' rights, including copyright and trademark rights when present, when sharing these works and creating derivative work. If you want more info on Google's policy, you can find that [here](https://developers.google.com/terms/site-policies)._
|