 or omniscience it depends on um i like the double entendre so yeah so the context for this and um okay so when i was doing a business impact assessment at uh kite pharma uh last year one of the things that was you know definitely clear is that um you know the impact of an issue and outage was often the patient's death and i mean they were manufacturing therapies that were therapies of a last resort where, you know, the client had already been through a number of treatments and this was sort of a therapy that was, could make or break them. They were having very good results in terms of their therapies, but, you know, that was very much their context. And, you know, so a couple of slides I have at the beginning or you know who are we talking about and i mean we're talking about patient level therapies we're not talking about mass production we're not talking about big assembly lines we're talking about sort of things manufactured in in like labs so yeah and in terms of the overall supply chain, and this was another aspect that I looked up at, because, you know, supply chain traceability all the way from, you know, they called this a vein-to-vein manufacturing process from, you know, extracting the blood from a patient that was terminally ill, shipping it to the manufacturer, infusing what's called a car gene, and then, you know, which produces car tea, and then they incubate it, they wait for it to grow, wait for it ready to harvest, and then expand it, and then, you know, package it to ship it off um back to the patient so you know the products that these um companies are manufacturing uh are just amazing and just so critical um you know to you know the patients their families etc so um yeah so just so much that that can go wrong and so the what sort of interested me in this was sort of the aspect of well how can we introduce redundancy and use llms or vlms in this case to reliably sort of improve the end result. So this is the manufacturing sort of facility. And so during different phases, I mean, it's literally in a room where you may have a biosafety cabinet, you have an incubator, and then you have the bench where, you know, other sort of devices sit. So it's very small. One of the drivers that they're trying to do in terms of manufacturing as well is drive things to, you know, being able to do inline monitoring. So instead of doing tests where they have to sort of extract a sample, you know, have a camera or other sort of devices where you can sort of inspect what's going on without having them disrupt manufacturing. So, you know, very, a lot of complexities, a lot of potential points for failure, but with a high sort of impact to failure, like at the individual level so and that's sort of the hardcore um intro before i was gonna ask if there's any questions but i included that in case there were to give people a little bit of context of why it was important and sort of some of the complexity and process flow so this gets a little bit more into sort of the vision specific and on the left-hand side, if you see, these are some of the operation steps and also some of the things that they need to look for, to see like in terms of a product, is it ready for release, so you know some of the simple things to imagine is you know doing something like cell counts and doing a cell count for specific cells expressing certain characteristics but also looking for things like cell death, etc. So, okay. So when I was doing a deeper dive, one of the very cool products I ran into was called Landing AI. And okay, I see my time. And Landing AI, it's from the team of Andrew Ng, you've probably heard that name. And the very cool thing that they did with their product is that they have a core process that does, in essence, like an architectural trade-off analysis. Like you give it the requirements, and based on those requirements, it sort of expands them to specific steps, looks for candidate products. And then based on those products, it will suggest and recommend it. And then the other key thing that they did is that they abstracted sort of the calls to the back end where these various LLMs can sit. And so in terms of object detection, which was the key sort of framework that I was looking for here. So like, you know, detecting specific types of cells, etc. You know, that was sort of where, you know, there was three core vision models, OWL grounding Dino in Florence 2. And so the, what I ended up focusing in on was looking for opportunities to be able to fine tune Florence 2. So the other thing that I did in terms of the solution is that, which even when calling API is not just LLMs, is that we've got a product called Weave from Weights and Biases that does an awesome job at being able to instrument these calls and help you reverse engineer the product very quickly. Okay, so in terms of when I was doing discovery and looking at alternatives in terms of vision language models so small but in terms of complexity detect brain tumor and brain tumor in terms of complexity it was almost like it you know the more words you threw at it the more difficult time it had um but you know tumor i mean it in terms of it nailed it it didn't sort of get confused by eye sockets. It knew where the tumor was. Dino, grounding Dino, which is an object detection model that's been around for at least a year and it's been strong, didn't detect it. And Florence II and PeliGemma, there was a couple of issues here. I dropped looking at PeliGemma because when I was querying, And Florence 2 and Pallegema, there was a couple of issues here. I dropped looking at Pallegema because when I was querying, I couldn't get a good baseline. A lot of the sort of detection I tried to do in these types of models, it wouldn't give me a bad inference. It would just say, as a generic model, I can't answer that. And so since I didn't have a good starting point, I didn't do it. But Florence 2, it's not here because Florence 2, object detection, you don't pass it a prompt, you pass it a task, you tell it you're an object of the request level, when it was doing object detection, to begin with non fine tuned, there was two different scenarios I looked at, I fine tuned both based on cells and cell types, and I also did it based on liver samples, so liver biopsy. So, and it's small, but blood samples, nine times out of 10, it was detecting as jellyfish. And these liver samples, based on a staining, wanted to say it was a flower. So, yeah, so that was kind of the starting point. And I tuned sort of more rounds with, or more epochs with the blood samples, but there was less sort of, there was fewer sort of samples overall. And I think the quality wasn't quite as good. And the complexity based on the number of sort of items to detect was much higher compared to the liver biopsy. The liver biopsy was primarily that, it wasn't 20 to 40 cells, it was, you know, one sort of liver disease or condition detecting. So, but very positive and very quick results in terms of being able to fine tune. Okay, so the proving ground that I created was just to be able to visualize it. It was. Okay. How much time do I have left? Is it over? It's telling me 8.54 remaining. Why is that? Yeah, Don, you are at time now. If you want to just wrap up with a final sentence. Okay. That's fine. I will. So sorry about that. Yeah, so in terms of issues, fine tuning was imperfect focusing on the blood samples. So that is an opportunity to both, I think primarily it's look at potentially switch away from the blood samples until I can get better sort of, you know, source data, improving the overall sort of evaluation and fine tuning data flow. I appreciate Chris's coaching and help there. Yeah. So the other thing is getting back to this solution from landing AI is get it to the point of, you know, dynamically sort of divide the work by a specialized team of experts that is vision language models based on, the specific sort of symptom that you're looking for. Thank you. Sorry for messing up time. No, you're fine.
OmniScience
Using a micro-services approach to build teams of fine tuned vision models for diagnosis and detection. 

GitHub: https://github.com/donbcolab/omniscience-poc

Slides: https://1drv.ms/p/s!Am4UQ5DtcZMUgaBXcdhsmeO4hrgoNA?e=xwC5LN

Connect with Don!
https://www.linkedin.com/in/donbranson/

Apply for an upcoming cohort of The AI Engineering Bootcamp!
https://maven.com/aimakerspace/ai-eng-bootcamp