 Hi everyone. My name is John. I'm a developer advocate here at Arise. I'm going to walk you guys through the new integration that we've just rolled out with Haystack that you can use to trace and evaluate your different Haystack applications and pipelines. This integration works with both Arise Phoenix as well as Arise. It's used the same underlying architecture. I'll show you in Phoenix, but we just know we also have a page here for Arise that we'll walk through how to use that platform as well. Very similar on both sides. And what we'll do is we've got a RAG pipeline that we've set up with Haystack that we'll sort of trace, and then we'll also show evaluating that as well too. So I'm adding in some packages here, and we'll make this notebook available in the description down below. And then I'm going to use a cloud instance of Phoenix today as opposed to running a local instance. That means that I have an API key, so I'm going to connect to that. I'll show you guys what that looks like. But if you haven't, if you want to try a cloud instance and you haven't already, you can just go to phoenix.arise.com and sign up for one there. It's totally free. And then once you're in your instance here, you can go to keys and that's where I've gotten my API key here from. So jumping back over, I've set those already along with an OpenAI API key. And then now I need to actually connect my application through to my Phoenix instance so I can send all of my trace data to over to Phoenix. So there's a bunch of boilerplate code that's going on here. We actually have a helper method to kind of collect all of this and just make it a one line kind of thing. Everything that you see, excuse me, everything you see here, but I'll blow it up here so you can see what's actually happening. Essentially, we're just setting some environment variables behind the scenes that we want to point to, including setting our Phoenix API key. And then we're using open telemetry here, boilerplate code to actually go through and set up Phoenix as a endpoint that we want to send all of our choice data to. So if you've used Phoenix before, you already have it running. You probably already have all of this in your application or some variant of it. And so this is just to connect our application up to Phoenix and then to actually do all of the automatic instrumentation that we want to do with Haystack, aka all of the new stuff in the integration here. We just have to use this one line. And what this will do is it will go through the whole application. Essentially, whenever a call is made to a Haystack pipeline, then we will listen in, grab all the right information that we need to create all the telemetry data and send that over to Phoenix so you can make it visible there. One caveat here is that we hook this up to the pipeline sort of object within Haystack. So if you're not wrapping your code in a pipeline in Haystack, then you might not see all of the information you expect to, in which case you can do some additional couple lines to add in some instrumentation there. But just if you're using pipelines, you should see all that information come through automatically. So once I have that done, that will connect my application through my Phoenix instance and get us ready to trace any kind of Haystack pipelines that we're going to run throughout our application. So that's really all you have to do to connect it up there. I'll keep going and showing you guys what that looks like in practice here. So next up, I have this Haystack app, kind of just a quick start Haystack app here where we have a document store that we're creating in memory here, just with a few kind of documents of these lines that we're going to use for our rag pipeline. We have a prompt template which is going to take the documents that are retrieved to answer a given question, combine them with the question, and then prepare that to be sent over to our model. And you can see we're creating our different objects here within Haystack. We're using 3.5 Turbo in this case as our model. And then we create a pipeline within Haystack here. So this is where we're adding our different components, connecting them all together, kind of getting everything ready. And then once we have that defined, we can go ahead and make queries over to our pipeline. So we can say, who lives in Paris? And that will pull relevant documents, and it will try to answer our question from there. And so if I jump over into Phoenix, you can see in our default project, we've now got a trace, and that's our first run-through of our application. So I can go click on that, and you see we've got the Haystack pipeline, and then we've got the in-memory document retriever pulling, in this case, just all of our documents through here. And you can see we automatically will score how useful that document is in some cases here, too. So you can actually see that happening automatically there. And then our prompt builder gets built based on those documents. And then finally, the prompt is generated. If I show you here, the prompt is generated by the prompt builder that gets passed into our OpenAI call, and you can see we get our answer here. So it gives you visibility into everything that's going on just with that one instrument call. And that's cool, and that's useful. But really what gets powerful here is once you have all this information, you can start running evaluations and start running some tests within that information. So given that we're running a RAG pipeline here, we'll just do one to start here that's evaluating the relevancy of the documents that are pulled back. And so here I'll go ahead and I will import Phoenix. Okay, and just now that's finished. So we'll go through here and we'll connect to our Phoenix instance. And one thing that's really nice is we have these kind of built-in methods to help make some popular kind of evaluation a little easier. So in this case, this get retrieved documents method here, will basically pull all of the documents that have been retrieved by all the different spans within our application. So we've just got our one run-through, so you can see we've pulled each of the three documents that were retrieved, and those get paired up with the input, which makes it really easy to evaluate. because now we've got this data frame with the input and the reference document. And we can actually go through and say, OK, let's let's just use those two to essentially decide whether or not the document is relevant. So that's what our next method will do. And it's going to set up again. We have a helper class here. That's a relevance evaluator. You don't have to use this one. It just makes it a little bit easier to code. And what we'll do is we'll run evaluations using that relevance evaluator and that retrieve documents data frame that you see up here. And really all it's going to do is take each row of that data frame, so that retrieve documents data frame here, and pass it over to our relevance evaluator that has a built-in method that it will use. It'll look for the input and reference column, and then it will plug both of those into a prompt, send it over to the model that we specify here, so in this case, GPT-40-mini, and then get an evaluation response from the model. We've also actually asked it to provide explanations, so it'll come back with an explanation as to why it gave that particular rating. So this is a built-in one. Again, you could run any kind of evaluations that you wanted to, either using these built-in methods or just kind of running it yourself with a data frame. And then this gives us back, this retrieves documents' relevance. And so you can see it's labeling the documents, it's scoring them, and then just has the explanation. And so you can see it's labeling the documents, it's scoring them, and then just has the explanation. And so we can take this and we can send it back up into our Phoenix instance. And again, if you wanted to run a different evaluation, you can do the same kind of flow here, where you're basically exporting span information from Phoenix, running an evaluation, ending up with a data frame, and we use this actual context span ID value here to kind of connect it back. So if you wanted to run different evaluations, as long as you get back to a data frame. And we use this actual context span ID value here to kind of connect it back. So if you wanted to run different evaluations, as long as you get back to a data frame that has that span ID value, you can send them right back up into Phoenix here. And I'll show you what that looks like in the UI in just a second here. So again, built-in ones, we've got some primitives that help with it, but if you want to run your own kind of evaluation, it's easier from there as well too. So let me refresh this. And so now the first thing you'll see is we've got these markers up here. This gives us kind of our rolled up total score values on some of those evaluations. So we're calculating a few of these for our relevance of our documents here. Again, we just have the one run, so we don't have too much data beyond that first one. But then if I actually click into the run and I go to our retrievers. Now you'll see that we're actually scoring the relevance and the hit rate basically for each of these documents. So as I go down, you've got this one's relevant. And in this case, we're just using a one, zero kind of binary classification of whether it's relevant or not. And then these other two have been scored as unrelated. So again, kind of a contrived example here with just the one kind of toy run through, but as you get a bigger application with many more documents, this makes it really easy to say, okay, what's my actual hit rate on documents I'm pulling back? Maybe I'm retrieving way more information than I need to. Maybe I'm retrieving a lot of irrelevant information and I can maybe do some work to streamline my retrieval pipeline and improve the performance of my application. Again, if you just want to trace things with Haystack, so just trace your Haystack pipelines. You don't need to do any of this evaluation stuff. This is just a kind of extra bonus that you can use to improve your pipelines and get your applications really ready for prime time and ready for production. So Phoenix is totally open source. And if you're enjoying it, please give us a star. It really helps us put more resources behind it, keep these updates coming and keep these integrations coming. If you have any questions about the integration, feel free to jump into our Slack. We've got a really active community there and we love hearing questions and feedback from everybody. Until then, have a great day and thanks for all the attention.
Trace and Evaluate Haystack Pipelines with Phoenix
Figuring out how to improve your LLM applications can be like finding a needle in a Haystack. Luckily, tracing and evaluations from Phoenix make this much easier.

Check out this latest walkthrough showing how you can trace and evaluate a Haystack RAG pipeline with Phoenix,

Notebook: https://colab.research.google.com/drive/10-Jeu5kb72pzf9fYoHXIFsX3aRMFJisc?usp=sharing
Docs: https://docs.arize.com/phoenix/tracing/integrations-tracing/haystack