 Hey, this is Lance from Langchain. We recently asked 1,300 professionals in the AI industry what they think agents are best suited to solve. And the highest ranked task was research and summarization, as you can see here in this chart. Now, I've spent a lot of time working on different types of report writing and research agents. And today I want to build one from scratch and show a bunch of lessons I've learned along the way. Now before I get into all the code, I want to show you the output of the agent just to motivate this. So this is me just simply asking the agent, write me a report about different agent frameworks. LandGraph, CrewAI, OpenAI, Swarm, LOMNX, Index Workflows. This is the report I get out. So the agent does all this for me. You can see it kind of partitions each framework into its own section gives nicely formatted bullets this is all markdown file by the way provides sources for me same for crew sources open ai swarm laminx workflows at the end nice distillation clear table this is all done for me, takes less than a minute, and all this web research and report generation and writing is automated by the agent. Here's some other report examples. I asked about how did Replit recently use Langsmith and Landgraf to build their agent, and all this research is done for me. It gives kind of a nicely broken down overview of what was done the various sources details of implementation and again kind of a nice summary section on key technical takeaways here's another example where i asked for kind of news and recent events across various ai observability offerings we can see we get again some nice sources here. It goes into Langsmith, Braintrust, Recent and News, it talks about their Series A. Datadog, it talks about kind of their work on AI observability here, and Arise Phoenix, and some nice kind of summary comments here. So you can see that what we're going to build here has a few advantages. All I need to do is pass in an input topic. It can produce reports of different types. You can see the structure of the reports varies a little bit in terms of the section number, in terms of how the conclusion is formatted, whether it uses tables. We can all customize this. And it's very flexible. I can produce reports on many different types of topics in many different user-defined formats. Now let me talk about one of the motivations for kind of agentic report writing. So there's been a lot of emphasis on RAG systems, which are extremely useful, and actually a lot of times can be kind of core components under the hood for report writing. But the point is with RAG, you're really just saving time with finding answers, okay. Now, usually answers in service of a bigger goal, like an actual decision you're trying to make as an organization or as an individual, reports kind of get you closer to decision making by presenting information in a structured, well-thought-out way. So the leverage you get from high-quality reports is oftentimes a lot greater than what you get just from a Q&A or RAG system. RAG system only gives you answers. Reports kind of present information in such a way that actually it can accelerate a decision. Now I want to spend just a quick minute or two on kind of the state of the art. How to think about report writing, what's been done previously, and what do we want to kind of build and extend? So you can think of report writing in three phases. One is planning. There's been a lot of interesting works around report writing that first start with some kind of planning phase. In particular, GPD researcher, very nice work from folks at Tavoli, uses what you might think of as a plan and solve approach where it looks at the input topic and it basically breaks that topic out into a set of subtopics which each can be individually researched. Okay, that's kind of idea one. Now idea two is the storm paper in particular starts with a topic and builds a skeletal outline of the final report up front which has some advantage that we'll talk about a bit later but this kind of outline generation up front follows in that case a wikipedia style structure because storm is really meant to produce wikipedia entries so it follows kind of a prescribed structure, but the point is the outline is actually generated up front. So the two big ideas in planning are one, take the input topic from the user, fan it out into subtopics that will independently be researched to kind of flesh out the overall theme that the user wants researched. That's idea one. Idea two is kind of generating an outline up front of the OVAR research report. Now phase two, once you have a set of subtopics and or an outline of your final report, then how do you actually conduct the research? Now there's at least three different strategies that I've seen for doing the research itself. So one is just very simply taking the subtopics and creating search queries from each subtopic and then performing a query either on a web or RAG. Okay. So that's just kind of parallelized search and retrieval from some source, be it RAG, be it the web. Now another idea that takes it a bit one step further is what we see in the Olamour Researcher work which basically does retrieval and then kind of does a grading on the retrieved in this case it's web resources so it kind of looks at the retrieved web pages and determines does this contain information necessary to address the input question if not rewrite the question try again so it kind of has this iterative search and evaluation process. Now the most sophisticated approach that we've seen here is Storm actually does a multi-term interview between what they call an analyst persona and an expert, which is effectively your search engine. It could be RAG, it could be Wikipedia, it could be some other search service you're using. But the point is you basically have an AI persona asking questions, receiving answers from the search service, and continuing to ask questions until they're satisfied. So that's kind of the most sophisticated. The simplest case is just query retrieval, like we see with Gpd researcher there's also kind of iterative search so you do a search retrieve evaluate potentially search again and the most of us get is an explicit multi-turn interview between like an analyst and your sources now finally writing so you've done research using one of these approaches how do you take on the writing task? A few different ideas here. One of the more popular is just sequential section writing. So write each section individually. Storm parallelizes some of the sections, but uses kind of an iterative process where it has the outline generated at the beginning. It will generate sections all in parallel, add them to the outline, and then it'll do kind of a refinement to fill in, for example, introduction and conclusion. Now another process worth noting here is you can also take all the research you've done and try to write a port in a single shot. Basically, you have all these sources and Lennon and LM just write the entire report all at once. So LamaIndex shows this in some interesting examples they've shared recently. So that's another way to think about this. So one is kind of writing your report section sequentially, and the other is on the other kind of full extreme is kind of a single shot process of writing your entire report at once. Now let me very briefly cover what we're going to do here. And I'm going to give some general lessons I've learned from working on a lot of different report writing agents over the last few months, basically. We're going to break it up into three phases. Planning, research, writing. Now here's the first insight I've kind of learned. It's nice to do a pre-planning phase to lay out the structure of your report ahead of time for a bunch of different reasons. One is it makes it very flexible. You can produce many different styles of reports this way. So, for example, let's say I want a report that always has five sections, an intro and a conclusion, and I always want a summary table in the conclusion. This particular agent implementation allows you to do that. What if I want just a single section, kind of like a short form report, no introduction? You can also do that. So what's really nice is this planning phase allows you to take both a topic from a user and a description of the desired report structure, and it'll build the skeleton of the report for you up front. So it makes it very flexible. Two, it allows you to set up parallelized research and writing across different report sections. You can debug report planning in isolation isolation so you can kind of look at the plan really before the research is done. If you don't like it, you can modify the prompts and it's very useful for debugging as I'll show in a little bit. The other cool thing is you can actually decide what sections of your report actually need research up front. So let me give you an example. Let's say I have a report that I want an introduction and a conclusion. Well, those sections don't actually need research. Only the main body of the report actually needs research. And so in this planning phase, I can kind of create all the sections of the report and also earmark these need research, these don't. So it's a nice way that you can kind of do all your pre-planning up front, including and planning for what sections report actually need research. Now let's talk about research itself. So what I've found actually here is just simple parallelized search and retrieval is a very nice way to go. When I say parallelized, what do I mean? I mean in particular, multi-query generation is really good. So basically taking an input question for a given section, fanning that out into a bunch of sub-questions to help flesh out the theme of that section, that's a good thing because I can all be parallelized. It's very easy to kind of have an LLM call, produce a set of sub-queries, search for all those in parallel, get all those sources back all in parallel, and then do the section writing in parallel, which I'll talk about in a little bit. I've done quite a bit of work on more sophisticated style of kind of question answer. The storm work for example shows a very interesting example of kind of again like iterative question answer ask another question produce another answer answer, style of research. Basically, I found these approaches are really token costly and really can blow up latency. So I like to keep it as simple as possible. Fan out your search queries, do parallelized query generation, parallelized query retrieval, and then use all those sources for your writing. Now let's talk about writing a little bit. The lesson I've kind of learned here is it is often better to write each section individually. That's because you can get much better section quality. If you make an LLM write the entire report in one shot, I've seen that quality can really degrade. So I do like writing sections individually, but to save latency, I like parallelizing all sections that can be parallelized. So really all the sections that kind of have their own independent research, I parallelize writing of them. You can kind of see this in the diagram down here. So basically, this is the toy report structure is intro, conclusion, two sections. Those sections get researched in parallel. They get written in parallel because they're kind of standalone, right? They have their own topics. They have their own research. Now, what I've also found, though, is you should have a sequential final phase of writing on any sections that don't require research, for example, intro and conclusion, because this can utilize everything you've already written to tie together the themes really nicely. So that's another little trick I found in writing. Write sections individually, but then do it sequentially such that your kind of main body sections that require research are all written first in parallel and then you write any final sections in parallel as well for example introduction conclusion that distill everything you've learned from your research that's kind of what i've found works really nicely now let's build this from scratch i'm in an empty notebook here here's the outline of what we're going to do. And now let's start with the planning. So this planning phase is going to have two inputs. It's going to have a topic and a general outline from the user. And it's going to output a set of report sections that adheres to the topic and the outline that the user provides. Now step one, first I'm going to define a general data model for a section of the report. That's all this is. So it's a pedantic model called section. It has a name, the name for the section, description, an overview of the main topics covered in the section, whether or not the section needs research, you'll see why it's interesting later, and this is the content of the section which we're going to fill out later. And this is just a list of sections. Now why do we do this? We define a data model because this can be passed to an LLM and the LLM can know to produce output that adheres to this model. That's why this is really cool. Now, I've just added another data model for search queries, which you're going to be using throughout this entire process. And I've added a type dict, which is basically a class for overall report state. Now, what's this all about? We're going to be using LandGraph to build this agent. LandGraph uses state to internally manage everything we want to track over the lifetime of the agent. We don't have to worry about this all for now. I'm just setting it up so we have it. Now, here's where we're going to get into some of the heavy lifting and interesting aspects of this. So for this planning, we're going to use two different LLM calls, which require two different prompts. Now, the first is a little bit subtle, and I want to explain it, because I kind of came at this through a bunch of trial and error over the last weeks and months. Basically, in report planning, it is beneficial to have a little bit of external information in the form of web search to help with planning the report. So this first prompt just explains that your technical writer helping to plan a report. Here's the topic from the user. Here's the general report structure, again from the user. Generate some number of queries to help gather information for planning the report sections. Now the second point is this is really the overall report planner instructions. So this is you're a writer helping to plan a report. Your goal is to generate an outline of the report. You're going to generate a set of sections. Here's a topic, here's organization. Here is any information gathered from web search related to this topic so now we can see we're going to set a function here generate report plan it takes in the state which we defined previously we extract some keys from the state okay now we go ahead and first generate some queries to help us plan the report. We perform web search. We do plan format the sources, results from web search. And then we format the report planner instructions with our topic, the report structure from the user, and everything we got from Websearch. Then we generate the sections of the port. I'm using Claude here. We return that to our state as the report sections. I can try out this planning phase in isolation. Let's give it a shot. So let's say here's a structure. I can pass in a pretty detailed structure that I want. I want a report focused on comparative analysis. It should include an intro. I want main body sections each dedicated to an offering being compared. I can give a whole bunch of preferences here. I want a conclusion with a comparison table. And here's the input topic. Give an overview of the capabilities of different AI agent frameworks, focus on line graph crew, open AI swarm, all index workflows. Now we're going to go ahead and run. Now you'll see I'm actually using the Tavoli search API, which is pretty nice for a few different reasons. One is that it automatically retrieves the raw sources for you, so you don't have to independently scrape pages. It's free up to a pretty high usage limit and the iPad just works really well. So I've used it for a bunch of different things I find it be quite nice. Now what's cool is it actually has general search and also news search which is geared for current events which is can be quite useful for certain types of reports. I'm defaulting to general search, and this parameter days is not relevant for general search, only for news. Now given this topic and this structure, let's just test that function. So here we go. Now we can see these are the sections that are reported automatically generates them for us. An introduction, no research needed, description, overview, cool. One section on LandGraph, one on CrewAI, one on Swarm, one on LLAMA workflows. Conclusion comparison table. Very nice. This is all auto-generated and it's guided by this outline we provide. You can do anything you want here for the outline. It'll automatically and dynamically change the report formatting for your guidelines. So you'll see here in this outline we've laid out, one, two, three, four of the sections require research. So now we have to build that out. How would actually perform web research and write those sections? Let's do that now. Now I'm going to create a new state called section state. You're going to see this is something very useful that we're going to do in LandGraph. I'm going to set up each section to be its own independent state and subgraph, such that the research and writing of each section can all be done by what you might call like a subagent, all in parallel. I'll show you exactly how to do that right now. So here's the code that I'm going to use to write each section of the report and conduct research for each section of the report. Let's go ahead and just look at it quickly. So you can see right here I can compile and show the graph I'm going to use to do this. So what's going to happen is for each section I'm going to generate some search queries about the topic of that section. I'll search the web and from the sources I gather I'm going to go ahead and write that section. Okay so that web and from the sources i gather i'm gonna go ahead and write that section okay so that's the scope of work we're going to do here now what's kind of nice is that in these section objects we can see that we get a description of what the section is supposed to be about right so we can use this information when we a generate some search queries to populate the information in that section and then b do the final writing of that section so this outline you can see is very important it kind of sets the entire scope of work downstream for us now let's look at the prompts we use here so first i'm going to have this query writer prompt Now here the goal is to generate some search queries that will gather comprehensive information needed to write a report section. Here's a section topic which we get from the section description. Now this is all very customizable. These are some things I've found to be useful. Generating some number of queries, ensure they cover different aspects. You know, you want them to be kind of unique and not redundant, right? That's the main idea here. Now, section writing, this is where you can also be extremely, you know, customized in your instruction. This is kind of the way I'd like to set it up. So you're an expert writer. Here's a section topic. I give it a bunch of writing guidelines, technical accuracy, length and style, a bunch of points on structure. Now, look, here's really what I did. I went back and forth with Claude a lot to write these prompts. So really, I use Claude to help tune these prompts from a lot of iteration basically writing a section feeding the section back to Claude saying hey I like this I don't like this having Claude update the prompt and so forth so basically this is a very kind of personal process and really allows for a lot of prompt engineering and optimization this This is just what I arrived on, but don't limit yourself to this. You can modify this in any way you want is the point. The real big ideas here is I'm going to plumb in the context that I got from WebSearch. I'm going to give it here's a section topic, and I give it a bunch of guidelines that I want for the style of the section. That's really it. So the only functions I need in my graph, we kind of saw them below when we compiled the graph, basically generate queries. So again, like, you know, I have some number of search queries I want to actually generate. Those are that sort of state. Here's information about the section. I pass some of that into the prompt. I get the queries. I write those back out. Easy enough. Then I have this search web, where basically, in this particular case, I call the Tavli API again. Note that I'm using async, so basically that's going to do all these web searches in parallel for me, which is just faster. And then the section writing. It's really all there is to it. This is the source information I got from web search. It was written to state right here. I just get that from state. I get the section from state. Go ahead and format the prompt. Go ahead and kick it off. And again, I'm using Claw 3.5 for my writing. Now, this is, again, as a preference. You can try different models. I found Claw 3.5 to be very, very strong at writing. And that's why I kind of prefer using it for this style of longer form report writing. And there I get my section. Now what's also nice about this is I can test section writing in isolation. So let's test on one of the sections that we defined previously. So let's go ahead and actually just look at what section we want to work. So let's write a section on line graph. The description is detailed examination of line graph, core features, architecture, implementation, and so forth. Cool. So here's a report section. We can see these are all things that we kind of asked for. I wanted kind of a bold sentence to start. I added that to my instructions. Really, what's going on here? LandGraph enables creation of stateful multi-applications with LLMs as graphs. Yeah, that's you know about right it's open source yes yes enable cycles cool nice now another thing i i i asked for is the sections to have at least one structural element in this particular case like a list just to kind of break it up a little bit so includes that which is pretty nice persistence complex workflows integration with Langsmith for observability pretty nice cool example GBT newspaper project that's actually mentioned in one of the sources so right here pretty neat and kind of how it differs from other frameworks so and again a couple sources so look not bad you can customize this in any way you want with the prompts above but pretty nice it gets us nicely formatted sources a nice overview of what line graph is and kind of formatting that we asked for so that's pretty cool so now we have nearly everything we need we can do planning we can generate individual sections of a report and do research now the final things we want to do are we want to parallelize that research and writing for all sections that need it and then we want to do any final kind of writing of sections that didn't actually require research but are like introductory or concluding so for that let me just compile it so we can look at it this is what it's going to look like in the end so we're going to take what we just did generate queries search web write section this is now a sub graph as zone state, as we saw a section state. What we're going to do is we're going to use a very nice trick in line graph called the Send API, to basically parallelize writing and research for all sections of the port in parallel. The Send API we use in this function right here, initiate section writing. So let's look at what's going on here. The sections are created in our planning phase, which we talked about initially. Okay. All we need to do is iterate through those sections and see this send thing. This basically initiates a sub graph that will produce a research and write the section. We can simply pass in whatever inputs are necessary. Now note that we send these inputs to build section with web research. Let's look at that briefly. So that is just, you can see we use sectionbuilder.compile. This is basically creating a subgraph. So sectionbuilder was defined up here. This is where we defined all the logic necessary to just research and write a single section. This thing is pulled into our main graph as a subgraph and we kick it off in parallel for each section that needs research. That's really all we did. Now we can see in that section writing phase we kick this off for every section that requires research. So that's great. Now we can see in that section writing phase, we kick this off for every section that requires research. So that's great. Now we also have this write final sections function, which is going to write any final section of the port that are kind of summarizing. It could be an introduction, it could be concluding, but any section that our planner deemed does not require research. Those are done last. Now what's kind of cool is we can use the same send API to basically kick off that process right here. Now what we pass is simply the section we're writing and the result from all the research and writing we've already done. So that's the key point. We say here's a section to write, for example introduction, and here's everything we've already done. So that's the key point. We say here's a section to write, for example introduction, and here's everything we've written already. So you can use it to help build this final section. Now we can see here we're going to use this prompt, final section writer instructions. That's all defined up here. Very similar intuition we saw before, but in this case you're an expert writer, you're synthesizing information for the final sections of report. Here's a section to write here's the entire report context and for introduction i give some instruction for conclusion summary i give some other instructions again you can tune this any way you want and that's how we get to our overall graph we do the planning which we talked about in detail previously we do research and write each section that requires it first and we parallelize that. You can see that speeds this up a whole lot because we're basically doing all that research in parallel. We get all the completed sections. We then in parallel write any final sections, introduction or conclusion, basically any sections that don't require research and we find then we compile the final report so let's test this all end to end again we pass in a topic i'll just show you that again overview of a few agent frameworks i give a structure that i want there we go cool and we're done we can look at the whole report so again we get kind of a breakdown of land graph crew ai you can see it kind of preserves a similar style tries to maintain a structural element but it's not totally enforced some sections don't have one again each section has sources kind of a nice crisp distillation at the end here with a table pretty neat you can see this is all done pretty quickly and i'll show you the trace to prove it. So here's the trace in lagsmith. We can see it took 43 seconds for all that. Not too bad. That's because of all the parallelization we do. Now let's go ahead and look and kind of inspect what we did. So what was the process here? So first, we used Anthropic to generate some search queries to help us plan the report then we do web search so it's pretty nice you can see we get a whole bunch of information about the various frameworks then from all that information from the search results and from we provided anthropic will write the outlines you can see here's the sections and there's an introduction description land graph crew ai swarm workflows conclusion so there we go we have our planning done we've done web search to help kind of make sure our plan is interesting and up to date and we now have a very clear scope of work now what's neat here is you can see we kick off four of these build session with web research and these all run in parallel which is pretty nice. You can see each of these are 15 to 25 seconds so it takes some time right because you have to do web search you have to write the section of the report and then we can see we parallelize writing the final sections. We write the introduction and conclusion in parallel as well. So you can see it's only about 40 seconds to get an output that looks like this, which collates a lot of really interesting information with sourcing for you from the web into this nicely digestible form, which is entirely customizable. Now let me show you how I actually use this. I'm in the repo right now. Go to this ENV example, copy this to an ENV file and fill this out. That's step one. Now step two is I actually can spin up this project in LandGraph Studio, which is a very nice way to interact with my agent locally or it can be deployed for now i'm going to i'm going to use just the locally running desktop app so here i'm in studio here's the agent we just showed and we walked in gory detail about how you actually build this now how to actually use it so what's pretty nice is we saw that the input is both a topic and a structure if you go to the repo go to that report examples directory and I show a few different report styles that I like here's a cool one on a kind of business strategy so what I can do is I go back over here to studio I open up assistance and you can see I've created a few different assistants now let's open up one business strategy you can see I've created a few different assistants. Now let's open up one business strategy. You can see here I just paste in the report structure I want for this particular kind of writer type. So this is really nice because the report structure is just a configurable parameter of my agent. I can create many different report writing assistants that all have different styles different focuses and can write really nicely refined specific reports per my specifications here now this is what i like on business case studies so it's going to build a report type that's focused on business strategy per the specifications i have in that outline now i can give it a topic what do we want to learn about just say successful software developer focused marketplaces. Kick that off. We can see we're generating the plan first. Cool. It's going to do a search query about developer marketplaces in general. You can see this right here. Generate the plan. And now it's basically researching each section all in parallel. Neat. You can see there was the web search. Now it's doing the section writing. Now it's kind of hard to see, but this is actually happening for all of our sections in parallel. Neat. You can kind of get a clue what it's going to talk about. You see it looked up Upwork, Stack Overflow, GitHub. Cool. And here's our final report. So here's the report we just created. I just brought it over to the repo as a markdown file. So we can see it did a deep dive in GitHub's developer ecosystem and what kind of drove success there. Some interesting insights, a bunch of nice sources, Stack Overflow, and Upwork as well. Kind of summary table highlighting an overview of the research so this is showing an example of how you can create a customized assistant that writes reports of a particular type in this case focusing on business case studies looking specifically at kind of analogous prior comps to a current kind of business question or scenario but you can create as many assistants as you have general report styles or types that you want to write so overall this repo provides a kind of kind of nice way to do very customized report generation it's a result of a lot of trial and error that i have kind of worked on over the last few months and again it does kind of this planning up front generous report sections paralyzes our research on all the sections that require it and then a final synthesis writing any final sections like introduction conclusion based on the results of the other sections that you just all researched and wrote in parallel so you can see it's very customizable because you can kind of use user-defined kind of outline to create different report structures or styles entirely up to you in terms of optimizing and tuning the prompts all the research is done in parallel so it's pretty quick like 40 seconds to get a really nice kind of like long form report and yeah feel free to play with it leave any comments below and the repo is of course open source so feel free to add any contributions or leave any comments thank you
Report mAIstro: Multi-agent research and report writing
Research + Summarization was the most popular agent use case in our recent State of AI Agents survey of 1300 professionals. But how do you design agents for high-quality research and report distillation? In this video, we share key insights and show how we built Report mAIstro, an agent capable of web research and high quality report writing. 

Report mAIstro simply:
1. Takes a topic (and optional report structure)
2. Builds an outline
3. Parallelizes research + writing across sections

You'll see various report examples and learn how to configure assistants for different report types.

Code:
https://github.com/langchain-ai/report-mAIstro