 Ada, go ahead and open up OpenAI's real-time API blog announcement. I've opened the browser to OpenAI's real-time API blog announcement for you. Thanks, Ada. If you need any more help, just let me know. It's time to start rethinking how we interact with our computers. Now that OpenAI has released the real-time API, you and I can build seamless next-generation AI experiences. As always, the greatest technology and opportunities come to you and I first, the engineer. Are you ready for speech-to-speech AI assistance powered by advanced reasoning models like O1? This is what we've been waiting for. Let's play with it. So no one is really using these tools to their full capabilities yet, myself included. Let's show off early versions of how we can truly utilize personal AI assistance. Hey, Ada. Hey, Dan, how can I assist you today? Yeah, can you go ahead and open up chat GPT, open up Claude, open up Gemini, then can you give me the current time, then can you generate a random number for me? And then after that, go ahead and open up Simon W's blog and then let's go ahead and open up Hacker News as well I've opened chat GPT Cla, and Gemini for you. The current time today is 1210 a.m. on October 5th, 2024. I also generated a random number for you, 23. Simon W.'s blog and Hacker News have been opened as well. Okay, so this is getting crazy, right? OpenAI employees are going to salivate when they watch this video. It's a beautiful orchestration of all of their great work. So we have the real time API calling a chain of tools with 100% accuracy, which contain individualized AI agents. And let's simplify AI agents. That's any combination of code and additional prompts with a single purpose built to solve a single problem. These AI agents can then run their own structured output calls that feed back into our top-level digital assistant. I'm not going to lie, I am high as a kite envisioning the capabilities and the future we can unlock with this tooling. The distance between your thoughts and your actions on your device are fading fast with this combination of technology. Let's push this further with some more interesting agentic functionality with prompt chains and O1 reasoning models powering a file manipulation agent. So this is all running locally via a async Python application. Hey Ada, can you go ahead and generate a CSV file for us? Go ahead and call it a user analytics and just add 10 mock rows. I've created the CSV file useranalytics.csv with 10 rows of mock data. Great job. This looks awesome. So can you go ahead and delete row 5, 6, and 7, and then can you add a new column? Let's call it membership and set it all to premium. I've updated the file useranalytics.csv. Rows five, six, and seven have been deleted, and a new column called membership has been added with all values set to premium. Is there anything else I can assist you with? Let's make another change here. Can you go ahead and add 70 additional rows to mock data using the reasoning model. Okay, so pretty incredible stuff there, right? Just by talking to my computer, I am generating and modifying this useranalytics.csv file. This is getting kind of out of hand, right? This is incredible. This is the dream of multi-agent applications we've been building toward on the channel for over a year now, since the very first one prompt is not enough video. I want to share a couple interesting insights and useful engineering patterns around building personal AI assistants. But first we have to highlight what's happening here. So in a very sloppy 1000 lines of code, we have a real time speech to speech assistant that can connect to and accurately call other agents. That's huge. These other agents are called through tools. And these agents can then themselves run their own prompts, they can use advanced reasoning models like a one. And ultimately, there's really no limit on what you can do inside of a function call right the sky's the limit now the real trick is do you know what you want to build and do you know what you need to do to wire together your architecture with your personal ai assistant at the top at the center which then commands your army of ai agents. That's what's next for us here on the channel. We're taking Ada to the limits, and then we're going to break those limits. That's been a big milestone and a big keystone for us. Living software that works for us while we sleep. Let's go. If you're starting to build out your own personal AI assistant, I'm super curious, what are some patterns you're setting up and what are the first use cases you want your personal ai assistant to solve on your behalf so we have a couple tools that really lead a lot of our functionality here right you saw the get time method you saw the random number you saw open browser and then we have some kind of essential file manipulation calls right there's create file update file and delete file let's create file, update file, and delete file. Let's go ahead and showcase these a little bit more and really show off what these three simple but essential methods can do with a personal AI system. Hey Ada, can you go ahead and delete the learning rust file? Are you sure you want to delete the learning underscore rust MD file? If so, please confirm with force delete yeah go ahead and force delete that file we won't be needing that fantastic can you go ahead and create a new file and just explain the basics of list comprehension and list manipulation and you know what actually go ahead and just talk about all the different types of loops that you can use in python. Instead of a markdown file, just go ahead and create a full-on Python file with examples and make sure you comment and document it all really well. I've created the file pythonloopsandcomprehensions.py with explanations and examples on list comprehensions and various types of loops in Python. If you need further assistance or more information, just let me know. Fantastic. Thank you, Ada. Can you go ahead and update our Python loops and comprehension file and just go ahead and move everything into a function and then place a main block at the bottom where we can comment in and out each example and kind of work through them at our leisure. The Python loops and comprehensions py file has been updated. Everything is now organized into functions with a main block at the bottom for easy testing and exploration. You can now comment in and out each example as needed. Wonderful. Thanks so much, Ada. There's a triple quote at the bottom of the file. Go ahead and remove that. The triple quote at the bottom of the Python loops and comprehensions.py file has been removed. If there's anything else you need, feel free to let me know. Nice job, Ada. Can you go ahead and create two more versions of our Python loops and comprehensions file? Basically, we want another version in TypeScript. And let's go ahead and create another version of the same thing in C Sharp. So create both those files with the appropriate file names and file extensions. And just explain all the variants of loops in both TypeScript and C Sharp. The files loops and comprehensions.ts for TypeScript and loops and comprehensions.cs for Sharp have been created. They both include explanations and code examples of various loops and comprehension methods in their respective languages. Great job, Ada. Thank you. Can you go ahead and delete all of our loop files? Go ahead and force delete them. Both loopsandcomprehensions.ts and loopsandcomprehensions.cs files have been successfully deleted. Nice job, Ada. Go ahead and get rid of the Python loops and comprehensions.cs files have been successfully deleted. Nice job, Ada. Go ahead and get rid of the Python loops and comprehensions as well. The Python loops and comprehensions file has been deleted. If there's anything else you need help with. So these are really interesting tools I wanted to share with you just to give you an idea of what's coming next in the age of generative AI. Now that we have these real-time speech-to-speech capabilities, it's all about utilizing their capabilities. These are some really important functions to keep your eyes on and to build out into your personal AI assistant because a lot of engineering is all about file manipulation, file updates, file creates, file deletes, and manipulation. So just wanted to give this idea to you. A couple additional things I want to show you here in the beginning i asked ada to open up a bunch of browser urls every assistant is going to need some personalization some specific information that's specific to you and the things that you're interested in so you can see here we have the browser urls that's what got loaded into the open browser ai agent and then of course we have some auxiliary information down here browser as a name and human name you can see this personalization getting expanded very quickly and an interesting agentic pattern we're going to be digging into on the channel is having an agent automatically update our personalization file personalization settings is going to be really important and I also wanted to share this runtime table so i think it's super important to have proof of the speed of these agents and i gotta say i'm really really impressed here the real-time speech-to-speech api from when i finish speaking to getting a response is on average about a second when we're running tools obviously this time gets blown up quite a bit but i think that's to be expected there are other ideas like moving tool calls to an async thread to let those run in the background definitely an improvement that can be made here but you can see here real-time api responses are really really really great sub one second for the entire flow this is really incredible it seems like under the hood what their model is doing is starting to generate the response as we're speaking. Since tokens run in sequence, they can automatically start generating the response and then stream that back to us in real time, right? So this is really, really cool, really incredible stuff. The technology is truly groundbreaking. And then you can see some of my other calls here, right? So delete file took under a second, took about a second update file took three seconds right so updates more involved because we're not just setting up the initial right we're having the llm updated and of course you can see here when i ran the reasoning model that took about 18 seconds right so just wanted to give some data here give you some insights into the actual run time of the real-time api i am in love with these numbers. In our previous personal AI assistant videos, it was taking about, in the best case, five seconds to run the speech-to-text, LLM, and text-to-speech workflow that you need to complete in AI assistant. OpenAI is doing all those steps in less than a second here for simple conversational workflows. This is truly incredible technology of course it comes with several trade-offs anyone telling you different is wrong or they're trying to sell you something the real-time api alone is incredibly expensive you know talk about huge costs i'm not surprised but the cost must come down for this to be viable at scale if we check out the pricing here you can see so this is six minutes of audio in and 24 cents out this is heavy pricing when i was testing this out uh the costs for me were around i think it was 10 to 15 dollars for i don't know it was like an hour testing back and forth right so the costs are there that's a trade-off and of course i think the second largest trade-off here is vendor lock-in if you work through this base, you'll see that everything I've built here is on the OpenAI stack. All the LLMs, the reasoning model, the speech-to-speech, the real-time API. OpenAI wants to be the king of agentics. They want to be the king of AI agents, LLMs, and the entire ecosystem, right? Even the B2C side of it. These are risks that we kind of just have to take in order to get and utilize the top generation, the state of the art application models and APIs, right? The real-time API, there's nothing as close to this. They've embedded the speech-to-text, the LLM and the text--to-speech. All in one workflow, we're getting sub-second responses and we can call tools. That's all we've needed to push through to the next generation of agentic technology. Right now, these are risks we have to take to stay on the edge of what's possible so we can learn and master the next generation of software before it hits the masses. So I hope it's super clear what's happening here. The distance between you and your device is slowly fading away. Can you feel it? I can feel it. We can think, then just speak, and it knows what we're looking for. It's scary. It's exciting. It's real. It's here. This beautiful hacked together prototype code base contains about a thousand lines. It's here. It's ready for you if you're interested. I hadn't seen a ton of Python based on device implementation. So I wanted to throw one together to test and validate this agent and to set up a new version of Ada that I can grow and expand on. It's a simple proof of concept of what's coming. I think for us engineers, this is a next level tool that is really important to spend time on and pay attention to and build into your development workflow. There's really no limit to what we can do here. It's all about what we can imagine and it's about staying out of OpenAI's way. Don't go out and build a product. Don't go out and try to sell something that is going to get bulldozed by the next open AI release. Steer clear. Just imagine the tech is going to improve, the models will improve, and then think about what you can build alongside these next generation generative AI tools and API tooling. I think a huge theme we're seeing here very clearly is that voice is a key piece of the next generation of user experiences a question we're going to be looking to answer on the channel is how can we utilize personal ai assistance in combination with incredible prompts and reasoning models to get more work done in parallel faster than ever. If that interests you, drop a like, drop a sub, and join the journey. Really excitingly, we are about to hit and cross the 20k subs mark. It's been an incredible journey so far. We've transformed so many times and we've learned so much over the course of the channel, all the way from one prompt is not enough to learning aider to exploring cursor to understanding prompt chains agent orchestration building out multi-agent systems it's all coming together with the release of the real-time api and it's building up to something incredible we're getting closer to building living software that works for us while we sleep with every single video i I want to say a huge thanks to every builder and engineer that's a part of this journey. I appreciate all the positive comments and all the feedback. And I just want to remind you, you are in control of this technology. You can take this and together we can imagine a future with these tools. I want this channel to be a place where you can come to understand the capabilities of this technology so that you can build a better future for yourself, for your family, for your work, and for your passions. We're not going to stop until we have Living Software working for us while we sleep. Keep building, stay focused. I'll see you in the next one.
Realtime API with Tool Chaining. ADA is BACK. o1 assistant FILE AI Agents
This is what we've been WAITING FOR. The Realtime API is the breakthrough we’ve been waiting for in personal AI assistants. 

🤖🔥 ADA is BACK!

It's time to start rethinking how we interact with AI assistants. ESPECIALLY for software engineers.

🎥 Featured Links:

- Python Async Realtime API POC codebase:
https://github.com/disler/poc-realtime-ai-assistant

- OpenAI Realtime API:
https://openai.com/index/introducing-the-realtime-api/

- Super AI Agents with Structured Outputs:
https://youtu.be/PoO7Zjsvx0k

- Control Your Personal AI Assistant:
https://youtu.be/ikaKpfUOb0U

- One Prompt is Not Enough:
https://youtu.be/JjVvYDPVrAQ

🔥 In this game-changing video, we're unleashing the full potential of personal AI assistants like Ada. Discover how the new OpenAI Realtime API is tearing down the barriers between you and your digital assistant, enabling real-time tool chaining and function chaining like never before!

🛠️ See firsthand how Ada utilizes the o1 assistant and advanced AI agents to perform complex tasks with 100% accuracy. We'll dive deep into the mechanics of tool chaining and function chaining, showcasing how these techniques can transform your interaction with your personal AI assistant.

💡 Whether you're an engineer, developer, or AI enthusiast, understanding these cutting-edge techniques is crucial in the age of AI. We'll explore the trade-offs, discuss the risks, and explain why embracing the OpenAI Realtime API is worth it for engineers who want to stay ahead.

🚀 I, IndyDevDan, break down complex concepts into easy-to-understand insights. From experimenting with file AI agents to implementing personal AI assistant patterns, we'll guide you step-by-step through the revolutionary capabilities of Ada powered by the OpenAI Realtime API.

🌟 Don't forget to like and subscribe for more edge content on AI, automation, and the future of personal assistants!

📖 Chapters
00:00 ADA is back.
00:49 OpenAI Realtime API
03:02 o1 File CRUD AI Agent
06:08 Breaking down tool chaining
06:34 Experimenting with file ai agent
10:03 Personal AI Assistant Patterns
12:25 Realtime API Tradeoffs
13:22 It's worth the risk, engineers NEED this.
15:45 Wow, 20k subs soon, our focus has not changed. 

#promptengineering #aiassistant #programming