 My name is Jacob Barhak. People know me in this meetup because I'm one of the organizers. But what I do outside this meetup is actually I'm a sole proprietor, and I develop disease modeling software and all sorts of other tools to comprehend medical data. This is a long-term journey. I've been doing it for over 10 years, and I'll be doing it for a long time. term journey. I've been doing it for over 10 years and I'll be doing it for a long time. And what I'm going to show you today is an offshoot of one of the other projects. My project represented here and other places in the past is a disease modeling foundation. I model COVID, I can model diabetes, but basically an infrastructure. The problem is how do you get data for disease models? Data for about diseases is kind of secure and unreachable. So one of the ways to get it is from publications, but then again, publications are all over. But a few years ago, there's something that appeared called clinicaltrials.gov. Actually it appeared because the government issued some laws that everyone has to report their clinical trials and has to go to some database. There are some nuances of exactly what that's going to go there, but basically the law says that you have to report your clinical trials. And there's a website called clinicaltrials.gov. I'm going to show you for a second how it looks like. This is the main page. This is the number of cons. This is updated to like an hour ago of how many research studies, studies include clinical trials, other type of studies. This is how much is there today. This number keeps on going and some of those have actual results, actual numbers. So there's a lot of data about clinical data that you can actually go and process and it's all out there publicly. The thing is it's not easy to process. Why? I'm gonna talk in in a second. So first, in this work, I work with Joshua Shrez. He's, Josh is, he's no longer doing anything active, but I kept his name because he actually wrote a website. I'm going to show you later. It's evolved since then, but he's done a lot of work here and he deserves the credit so this is why his name is here. There's other people who helped you'll see them in the acknowledgments at the end so don't go away without seeing the acknowledgments. Some people may recognize their name there from this group even. So let me tell you what I'm really trying to do and then go back to the clinical unit mapping because this is what this talk is all about. So I actually want that in the farther future, let's say 50 years from now, and then you can argue how close or fast slow it will be, I want your doctor to be here. I don't want to go to the doctor. I want the doctor to be with me all the time and I don't want to go to the doctor. I want the doctor to be with me all the time. And I don't want to argue whether I'm going to the best doctor or the correct type of doctor. I want the best one to be with me all the time, regardless of what I pick. And if every one of us has a doctor like this basically on our cell phones that's with us all the time, you don't even have to go to the doctor. The doctor is with you. on our cell phones that's with us all the time you don't even have to go to the doctor to go to the doctor is with you and if it's repli if you have the best one replicated many many times because it's easy to replicate the things that are electronic then we'll be much healthier in the future we who know how who knows how long we're gonna live with this thing but to get there we have to go through many many hoops it's a type of automation we still don't have and you still want to inspire to. So I looked at other sources of automation just to give you an idea of how reachable it is. I said about 50 years maybe I'm wrong but let's see let's see how other applications were actually human cognitive tasks. I'm not talking about your surgeon who actually has something to do with their hands. Think about the cognitive tasks that the doctor does by looking at your data while you meet them and they're looking at your charts and stuff like this and trying to figure out what to do with you. So let's start talking about other cognitive tasks that people used to do or still doing and how long it took the computers to solve them. So let's start with computer chess, just to give you an example. From the first computer program that was written, and you can check it out on Wikipedia, it was by Alan Turing in 1951. In 1997, less than 50 years, Deep Blue won against Kasparov. So it took about 50 years, even less. Actually, in the middle of this, you know, people are skeptic about AI, and now there are talks about how AI will hold and so on. But it's actually something that already happened, those skepticism. There was a grandmaster, chess master called David Levi. In 1969, he looked at the computer chess programs and said, nah, it's not going to work. I'm still going to beat it in 10 years. And he did. And there are quotes about what he did and not. But today, anyone will make this bet. He actually made the bet that won. And later on, he made the prize and he actually lost against the computer in, I think, 89 or something like that. But today, anyone will even think about competing against a computer chess program, a good one. Today, the leagues for computer chess, there's a separate league for computers and for humans. And apparently, humans cooperating with computers seem to be the best solution. So AI will probably go the same way. But this is an example from the past that how things will be and how much. Also, people are talking about driverless cars. Actually, some of them are already driving amongst us this way or another. are already driving amongst us this way or another we had this in this meetup we had our group presenting it and and some other companies that deal with uh self-drive driverless cars so the first experience in 1990 started in 1997. we are now about 45 years later and there are already driverless cars being tested and this chart is like I made it a few years ago. So 2018 Waymew commercialized. It's still active now and who knows how long it will become really dominant or some other car company. We're not that far from this. So why can't it happen with actual medical stuff? Well, we need to jump one more hoop that we did so far. So for computer chess, well, the problem is simple. All you need is basically computer power and some algorithms. It's relatively a simple task. For driverless cars, you need something more complicated. You need sensors on the car, you need some coordination, you need maps, basically data. And all of this is apparently, at least to some degree, seems to be solvable if people are solving it. In a few more years, we'll probably get it. We already have some of them, but the question is how usable they are. And I won't go into arguments about this but with uh medical data well we have another problem that people don't see yet i see it all the time and it's acute it's standardization of data medical data comes from all sorts of sources and once merged you have a problem. Here, let me show you how big the problem was. This is August last year. This is the amount of clinical trials that we had. About 425,000. You see, now we have 470. It's like a year and something later. The number grows in quickly and this number is going to grow further. About 55,000 had actual data like numbers. and computers are good with numbers like blood pressure, A1C level, height, weight, all sorts of numbers that you see that they measure. They had those publicly. how bad or how good it was standardized you find out that it was well not really standardized because you had in unique i'm talking about unique units of measure 36 000 units of measure the number is gone by now by the way but this is august last year okay now i actually counted it from a few years before and did the average of how many new units of measure added per day. Just a kind of measure of how much, how much variability or non-standardization there is. It's about 10 new units of measure per day on average. Think about it. Each day, all the sources reported data with different units of measure and about 10 of them. This was happening already. And this is all public. You can actually go and check this out. Maybe you will have a problem downloading a file from 10 years ago, but I have some that I can show you. But it's all possible to actually check and see how bad the standardization is you can see how many units of measure are there but if you follow the same thing that i did so what i did is i said okay no it's a problem there aren't that many units of measure maybe there are but i it doesn't seem if you look at the data you'll see many of those are like misspelled units or units that are the same but different abbreviation or someone added context so it did not make sense so what I did is I went said okay this these units have to be standardized why because then you will be able to compare different trials that just put them in one mix to each other or even feed them into some computer program that will analyze their data otherwise the data means nothing think about it the number seven doesn't mean a lot as a quantity computer understands computers understand numbers not quantities if you want the quantity you need the number and the unit if you want the quantity you need the number and the unit but seven kilograms written as kilograms like the entire world or seven kg the computer will not recognize the kg or the kilograms being different you have to make it such that it it will understand you're talking about the same thing so the same seven if you say seven grams or seven kilograms it's different for us for computer the seven is the same the grams or kilograms we don't know for us for computer the seven is the same the grams or kilograms we don't know he doesn't know what it is so what I did is I'm trying to standardize all the units so later you'll be able to convert them think about it as like a translation task between languages and the problem you have here is the Tower of Babel problem where everyone is speaking their own language. Like each type of unit, each type of unit, a clinical trial has their own language. And you now have to comprehend all the languages you want to extract data from all this database. So I took several existing unit standards or specifications. You can see in highlight here is IEEE. IEEE actually gave me permission to use this for no commercial purposes. Their data set. So you can go on my website and see some of their units at some point. But there are other sources like CDISC and Bio-Ontology and actually let me show you all those are my data sources clinicaltrials.gov public CDISC is a organization that deals with data standardization for medical purposes actually it's based in Austin check it out the headquarters are in Austin. NIST RT-MMS is a project by the National Institute of Standard Technology, NIST, and they actually cooperated with IEEE. And this is how I got the units. And Unit Anthology is a website which has data in it. Of course, I also added, I took also non-units because later for machine learning purposes I need non-units. But those are my data sources. And eventually, it all comes to centralized and clinical unitmapping.com, which is the website I have. I'm going to show you how it looks, the front page. You can actually go there and see this front page. And what I do is I take units from each of those. Now I have like 5,000 units that are standardized, and I'm trying to match all of the 36,000 units to those 5,000. The way you do it is for machine learning techniques. At this level, you need to have a machine to help you. It's impossible doing by hand, although a lot of it is done by hand. I did at least 12,000, 11,000 or 12,000 units by hand alone. We're going to talk about this later. But basically the website, what it does, it helps us find unit proximity through clustering and some other techniques. The website also helps the user label those units and map those. The labeling slash mapping is what it actually the website does and it allows you aids you doing this by using supervised machine learning which actually allows you you can ask the website what's the standard form of this or that human thing to give you an answer so just a snapshot of the how it looks like you can actually access i'm going to show it to you in a minute but a few important things about website it's multi-user multiple users can label and work on it in parallel. Users can see what others are doing and the system actually promotes some of the units that are being mapped to some point. It floats them up to things that look fairly mapped. It will give them priority and show you but it also gives you a drop-down of all sorts of other possibilities that you might think of. So many units you want to see all the close things together. So the website shows you this already. And of course, it highlights what is a standard unit according to this or that standard. And this can be expanded to many more. So these are the important things about it. But people here like to hear about machine learning. So let me give you an idea of how it looks like. Here you see, I think it's 25 or 30 units, the topmost units that are being used by clinicaltrials.gov. Look at the differences. You see percentage of patients and percentage of subjects is basically the same thing or percentage of participants. However, the system doesn't know that they are the same. Here, percentage of participants again, sometimes the big piece, sometimes the small piece, sometimes patients. All of these are the same thing and you want to map them all as percentage, the symbol percentage. Same thing here participants are participants big or small um so what you can do is you can extract just from the text and you can extract it with very simple uh techniques you can convert it into a vector here i'm showing how those vectors from all of those words how similar they are so you can actually if you just look at this portion forget about the words you can actually see that this and this and this and that have some similarities between them so they are kind of close so this is one thing you're trying to do in using unsupervised machine learning you want when the user actually works with those the labeler he wants to put them see them all together so what you do is uh and we're using unsupervised machine learning you want when the user actually works with those the labeler he wants to put them see them all together so what you do is you have to cluster those units uh here i'm showing i'm using holovis to actually demonstrate you see all 30 something thousand units here and you can see them with proximity to, like the x-axis and y-axis are the same unit. So the diagonal should be, and it shows you the proximity between 0 and 1. The diagonal should be highlighted. And indeed, it's highlighted, the fact that it's not exactly an angle because you have auxiliary units here. I won't go in here. But now, this is unsorted. This is where things are not in the clusters this is similar to this where participants and percentage of participants you see you find it in places one three and maybe 20 something and all of those however are very similar so using this cosine similarity, you can actually show how close they are. And you can do it before you cluster them and after. On the right side you see after clustering. And you can see things are much more highlighted near the diagonals. This is how much, this is the way I'm showing you order visually. It's too much to even zoom in, but I'm able to show it to you and show you that this is how you order the data. And after ordering the data, you can actually put it in your website and then all the units become clusters. Actually, let me show it to you. So here you see, this is cluster zero. all of those things look like percentage of something and indeed you see all the a1c's we're going to talk about that later one close to each other maybe not always is but something else but here in cluster 8 you'll see things that look total load for a second and does it's quite a heavy lift for the machine. It's a fairly machine, but now you can see cells or copies per million cells. And you see all of those things, they look totally different than the A1Cs that we saw before. And of course, you can go into things that look like even different. So you have different clusters of things. They're kind of similar, but not really. So this is what the clustering does. It just orders everything. So users can actually look at them and see similar things next to each other. And then you can say, oh, this and this are the same. They're just written differently. And this is what this website is for. It's actually to make some order about uh it orders the things and allows you to label later on so this is the first thing that we're doing is clustering now after you clustered everything you want to do one more thing you want the machine to tell you to tell you, now, you know, this unit that you just saw before, well, it should be written this way to be standardized. You want everyone to speak the same standardized language. Now, to do that, it's not enough to use unsupervised learning that didn't ask the human to do anything. You actually have to show it enough examples. It's called labeling. You label some of your data and you show it enough examples where you actually say this maps to X maps to Y, Z maps to A or so on and so forth. So this labeling processes human intensive, human has to do it, but you can still run an AI and see how it will work even without labeling by creating mock labels. So this is how I started working, created mock labels. Later on I actually did actual labeling. But I experimented with many technologies you'll see all the technologies and later on you can go and see the references of what technologies are used to actually figure out what's going on and how to use supervised machine learning supervisors because i'm telling it this unit should be like that you actually dictate someoneises and tells you how things should be. So I used many, many technologies and some of them you recognize some LSTM, CNNs, those are neural networks people use, but they used other types of technologies like learning to rank, even simple classification, which is the simple neural network I tried. The things that actually were accurate enough started to be a few years ago, it's all those sequence to sequence networks. Those were good, but they were still not as good. And everyone now talks about ChatGPT. Well, ChatGPT is based on technology called Transformers. Transformers was first published in 2017 um the paper uh that defined transformers uh actually brought it to the world is called attention is all you need and transformers have those attention layers in the neural network allows you to work better with sequences until then sequences were was a real problem LSTMs and all sorts of other recursive neural networks were suggested but transformers actually solved the problem so when transformers came you can see my accuracy jumped in significantly like 20-30 percent almost and this is been significant and now I was able to do the website with this technology before this the website was the older technology but now I actually could implement the supervised machine learning which I did last several years then the issue became labeling okay now I have the neural network that works in very accurate or fairly accurate enough for me to actually work with. Now I want to actually tell the machine, okay, enough examples of what's good and what's bad, like this unit maps to this, not what's bad, you just show it what's good, what's bad, you have to figure out some other ways. But I had to map all those units to the correct label. So labeling is a time-consuming task. I have 36,000 units. How do you do this? Well, within approximately a month I was able to do 12,000 units plus 5,000 became automatic because you have the synonyms of the standardized unit so about 17 000 units were labeled automatically out of those all together if you count it all uh you have like 40 something units 40 something thousand units plus so um to do this properly i use technology from data heroes here i'll show you their logo I use technology from Data Heroes. I'll show you their logo. It's a website. It's a company. They have a website and they actually have technology that you can use to help you reduce the amount of training for machine learning. However, I didn't use it to reduce my data set. What I did is actually kind of the opposite said no i want to figure out which labels i need to uh which units i need to label to reduce my load because the trick is once you label those the ai starts helping you so instead of labeling all 36 000 and then seeing how good it was you can just label a few and then the machine starts suggesting back telling you what's good what's bad like actually it shows you all it's good and you can have like multiple of those that shows you good options and then your labeling task becomes trivial almost you just click much less clicks and you do things much faster and this i was able to uh do all of those within the month and this was a busy month i was doing some other things this is like off of this i was able to uh do all of those within the month and this was a busy month i was doing some other things this is like off off hours i was doing this so um the technology they use in data heroes is called core sets and it's basically what they do is it's similar to clustering but not really it's kind of they build you a tree of similarities, and then it shows you in the upper level, it shows you all of the things that are kind of distinct. So if you map those, you know that you captured many of the labels that you need to label and don't repeat yourself many times. So it actually reduces the time to actually perform the labeling. You can do similar techniques with clustering, but they actually got it down to the point it's a tree, and you can do it in several levels. So there are some numbers that I used to actually train the neural network. I didn't label all the 30-something thousand units I have. What you see now on the website, it's only after it's being labeled by 17,000 units. And I'm not talking about the non-units. The non-units are many other units that you can actually hear. 400,000 non-units as well. So 7,000 units and 407 non-units is how you tell the machine, oh, this is not a unit. So don't translate it to a unit. Otherwise, the machine thinks everything is a unit. And we'll try to translate it to a unit, otherwise the machine thinks everything is a unit. And we'll try to translate everything to a unit. So eventually I got to the point that I reached about 3,000 labels, meaning those 36,000 units that I had initially became 3,000. This means it's less than the initial set of if you combine all of the unit standards all together which means now it's reasonable more but it's not the end of the work there's still some tidying more to do but at least the system now comprehends the units and the question is how good or how bad so when i train it i train it over about 45 000 records in in the so the train had 400,000 and the validate set in 45,000 records so now if you actually go and see the results this is how accurate it is and when you train you actually not only trained for the unit you also train the unit and the context so only the context in ask the system to infer the units this is really interesting because just to give you an idea of context. Context is like weight. Unit is kilograms or grams and notice weight can be measured in kilograms and grams but kilograms and grams are associated with weight or maybe some other words. So weight is the context, kilograms and grams are associated with weight or maybe some other words. So weight is the context, kilograms and grams are the units. So there is not a one-to-one connection, but still in many cases when you give the system the context, it will figure out the standardized unit, the correct one for you. So it's not always possible, but it's still it will try and the machine will do it for you. So you can train it in all sorts of variations I'm showing you the training accuracy and the graphs you'll show you it's about 90 something percent this is for the training set but we actually are more interested in about the validation set which actually shows us how accurate it was after it was trained. So you can see this is, we'll put the context only shouldn't be that accurate but it's 60, 2% of the time tells you if you are giving me only the context will give you the correct units 60% of the time but this is not a trivial task because it may not from context you cannot infer the unit always because it's a one-to-many connection but still 62 is still impressive but if you actually give it the unit or the context or the unit without the context it's about 86 accurate on the validation set this is pretty good in the non-units it's almost point on it recognizes all the non units immediately so you can check it out it's all written in Python I'm using all sorts of libraries that you know across the hood I'm going to show you a little bit of a demo but first I want to summarize and tells you are this technology that I'm showing you I'm showing you four units of measure but it can be done for many other things that need standardization I'll give you just I'm showing you for units of measure, but it can be done for many other things that need standardization. I'll give you just some example. Cities in the world, how they are spelled in different languages. Boom, you can do this with this fairly simple. You just the database, you just change to instead of units to cities of the world or something else. So anything that needs standardization and I'm not, it's not aimed at really translating between languages but for standardization purposes it's really good. It will recognize medical units even if misspelled now. It will comprehend in the future medical units because it knows if you misspell things say oh this is this unit I know what it is because you have all all data and in the future i want to be able to comprehend quantities and numbers associated with you basically it's a it's it's something like a translation a a a conversion engine but now much more powerful because it's aimed at clinical data which is much tougher than other conversions that you see so but you can apply the same technology to other uh to other fields apparently they have the same problem in other sciences not necessarily medical and there's some solutions out there but this is aimed specifically in using the best technology available today transformers to actually uh go and do this so this is the acknowledging page i'll leave it for a second but sometimes some people will probably recognize their names there so if you do be happy you helped me quite a lot uh this way or another there's some companies that actually help you with all sorts of visualization, data heroes, also with the tool, and they will very helpful provide additional information. Let me show you the actual website. I already showed you the unit mapping here, but show you how bad the problem is. HVA1C is a measure of diabetes. It's measured in percent. See how many HVAA1c's are there? All of those are HbA1c's. HbA1c and A1c are the same thing. A1c. Notice, people write it the same thing. Once with a big C, once with a small c. Sometimes like this, sometimes like that. All of these are the same, and all of these should be inferred into percent, which you see here. This, what you see here, the possible synonyms, this is what the system knows after it's been trained on an AI. And all of those are actually percent, but it will also click you and you want something else. Let's say in my world, I want it to be something different and you can train it to be something different so you can just click here and then later the machine learning will kick in and learn it so all of this is possible and this is what this website is um i can give you more examples it's it's it's endless how many how many things uh work or don't here, but I'm going to show you just a few more clusters just so you get an idea how bad the problem is. So you can see those are more complex units, and this is like here. I didn't know exactly what is the standardized form, so I wrote several options, but they are the same, and they repeat themselves many times. So the system will know this will translate to this or that or that. In the future, we can decide. Notice there's also a problem with special characters. I didn't talk about that, but you see this dot? It's actually a Unicode character that someone plugged in, and therefore it looks different than a different unit that's approximately the same. And you can see many times it's also almost the same unit, when it looks differently. Here the AI is able to infer. This is the AI working now. I'm showing you the topmost AI. You want other options, these are similar other options the system can suggest to you if the user wants something else. We can go on and on, but let me show you actually the infer part. If you cross the infer in this way, you can actually ask the system here this is percent i misspelled percent on purpose the red line is if i asked it to infer the unit it will be able to it will match it to the percent this is what i trained it to do But if you want something like here, something simple in BMI, BMI is the context, it's not the unit. I'm asking it what's the unit of BMI? The body mass index. And it does give me the correct unit for BMI in the standard form I chose. Later on, we can choose different standard forms and teach them but the system already knows things and this is the level of development currently what you see here is a better version it's after about 17 000 units that I trained I intend to expand it quite a bit so check it out in the next several years if you want to see other publications that I published quite a lot, you can see throughout the years I started first publishing it in 2019 or 18, but you can see things that I have done in the last several years, publications and so on and so forth. This specific presentation I'm giving serially every once in a while. Today I'm doing it here because we had an open slot that i needed to fill out but i'm giving it in many other places around the world and hopefully this project will catch up and hopefully in a few years we'll have standardized medical data and then the road towards computers comprehending medicine will be much shorter than it is now. This is my uptake, those are my ideas, I stand behind them alone. I do have, I have to show you that, a conflict of interest statement. I'm developing it on my own funds but I am every once in a while I have some, I work with some others and therefore all those conflict of interest I write down so you can actually check it out. You can find this presentation on GitHub and previous others if you want. I am done talking. If you have any questions, I'm happy to answer. I do see Jeff there jeff is in the audience so yeah there's also the acknowledgements yeah yeah that's what i'm saying like i saw jeff yeah so it sounds like jeff did some work there yeah well jeff he helped me quite a bit he actually hosted the computers at some point yeah yeah yeah yeah we we me quite a bit he actually hosted the computers at some point yeah yeah yeah yeah we we it's a long story but like he hosted and he helped in other ways uh there are many people here i'll show the acknowledgments again maybe people recognize several people some of many of them are from like uh um austin and you might recognize them Austin and you might recognize them. They gave some ideas here and there, like here James Bednow you probably recognize, he gave talks with us. Yeah, they helped me with the visualization of this. The visualization was quite tough and I wanted to show what's going on. So I do have a question. So I don't understand the sources there, how it looks like the like the because I never looked at the clinical data. So when you say clinical data is just like tables, tables, rows, like white papers. It's not like, let me show you actually, it's part of the website. So let's see if it's still if this part still works, because sometimes they change things. They actually changed clinicaltrials.gov not long ago. And wait, we'll just press the other button. Here, let's go to mapping again. Give it a second. It's quite a heavy load to live all of the suggestions for one page so this is why it takes long and then the machine i have now is not like the strongest if someone knows how to enjoy this website easily then and if we're cheap then talk to me yeah like how clinical style it looks like and what's the yeah yeah so here this is this you see this HPA1C? The website actually, for the purposes of actually labeling, you need to be able to see what's going on. Why is this unit being used? You actually can see this is used in those clinical trials. You see, NTCT, blah, blah, blah. This is the clinical trial. And if you click here, it will actually go to the clinical trial. this is how it looks like uh there's a lot of information here i'm not sure this is the old version of this the new version maybe it's the new version i don't see classic clinical trials so this is the new version of the clinical trials.gov you see all of those tables this This is a huge hierarchy. I keep you know, the XML file of clinical shards of blood is complex. I won't go there, but you can actually go and find the HBA1C, what was it? Let's just A1C. Here, change in A1C for baseline. Here, here's the unit of measure that it's used. So you can actually find out exactly, and the number associated with this is this. And this is a table, and this is only for one outcome. There are multiple outcomes per trial. Here, this is another outcome. Change in blood glucose baseline. And notice that the unit here is different. So each one of those clinical trials has multiple different units, and you have to traverse all of this knowledge and standardize it. And each one of those is speaking a different language. The contexts are different. This is a clinical trial or something else, but you cannot even compare them side by side today because the numbers will not match. Even if there is a conversion, sometimes there isn't, match even if there is a conversion sometimes there isn't but like if there is a conversion or if they even if you are using the same units you cannot even put them side by side you can even compare them for God's sake when you want to extract human day you want the machine to actually do complex stuff with them a human's gonna comprehend it if you put it side by side why would the machine be able to do this yeah that's a big problem huh yeah this is what i'm trying to solve yeah but think about it it's not only for clinical trials tomorrow it can be from here you're a big company that has many offices around the world and everyone is providing some report that you need for your whatever you're doing i don't know chemical something or or oil or whatever think about all of those sources of information each one sending your report that's different that tells you the story then you need to actually build a bigger picture for all of those smaller stories how do you do that machine learning can help you and this is what I'm using it for. I'm here only doing the small things of like figuring out the units. Once I figure out the units, you can go to the next level and then the next level and then next level. Eventually, you're able to merge it with the disease models that I'm doing, which is a different type of AI altogether. So it's a long journey. Yeah, that's, this is a good problem solved for sure. Oh, thank you. David is happy. Yeah, David, I'm also happy doing this project. And Jared, oh, Greg. Oh, thank you. Go guys. No, that's nice. So the ultimate goal is like you mentioned something about like having everybody like a doctor in your pocket in a way, right? Long-term, it's maybe the end of my lifetime. I may not be the one who actually achieves it. And I'm working on a small part. But think about it. If all of us, all of the people who do Python or machine learning or something like that unite forces and actually solve small little problems like this, eventually all those solutions will come up to the point that this will be your doctor. You no longer go to your doctor. The doctor comes to you here. If there's a problem, it detects, it tells you, and then maybe a human doctor or maybe a drone with the medicine comes to you. Think about it. It's a whole different approach. Here, I'm taking care of my old mom now. You know what the biggest mess is actually? It's driving to the doctor. My mom is immobile now. So even the mechanics of running down, bringing mom to the correct doctor in the correct time that they're available, only this is a logistical problem, then on its own it's a mess. And I'm not talking about the fact that the paperwork doesn't even follow the person. The problem, this problem hasn't been solved. There is Teladoc. I don't know if you tried Teladoc, but in cases where you just need prescriptions, you can stay put and just cam for it. There's a technique. I'm unfamiliar with this, but I'm currently outside the US. So it's like maybe US solutions will help me. My mom is not. It would work over the internet. So in theory, it could work from anywhere. Yeah, the question is, do they serve someone from out of state? I just gave you an example. But it's also a different problem, because different states will have different regulations. But think about it. If you achieve making one good digital doctor that actually does things, you can replicate it. Think about the years of study you have to go to train one human doctor. Right. So another thing you're sort you're aiming for I think is like Star Trek's little whole thing that the doctor held in his hand right they would scan the body and then could diagnose things oh this is still science fiction this is maybe a hundred years for now but yes the idea is you don't need to train the doctors the way they are done today for medical school you still need those for some cases like surgeons that do something good and some humans just make decisions better better than machines because sometimes the machines will have no data and you don't know whether they're going machines will not do anything without data and sometimes humans have to make decisions where like on things that they've never seen before and no one else saw before. So maybe here you'll need human doctors or some other ethical thing or we're far away from that. But... Well, but actually I have a colleague that has created a mobile MRI that's the size of a briefcase and they use it for like bomb detection and that kind of stuff that also do soft bodies as well so if you combine that with your ideas then you could have you know your ideas are related to the knowledge component of the analysis etc right yeah the physical technologies going around too yeah so think about instead of coming to the hospital we actually have all the facilities you have all those devices that are sent to your home and the person just had to follow a set of like instructions look some devices already have like some watches will will measure your vitals and will keep them so and then you can able to analyze them and then the companies who hold them maybe hold their data and then they came able to analyze that that's a whole different story but like measuring your vitals is one problem but then again assuming you have data already what do you this what does the machine can do and decide how do you get the knowledge to do this because think about it medical medical data is very secure. If I'm a hospital, I'm not allowed to release data on you. But clinical trials are publicly open. And this dataset is something that you can use. And even now you can use and learn from it. If you are able to do something with this dataset, then tomorrow you will be able to do something by merging other data and there are techniques that actually doing things in other like there are many many more techniques that need to be used to actually solve those problems that are talking about i'm i'm talking i'm i'm working on a tiny bit of the something tiny that will eventually if it merged with many other solutions, will eventually be this thing that I'm talking about. I'm one person, but if many people join their forces and they do it properly, without, you know, and able to share it in ways that actually advance the ability to actually do this then in perhaps i actually asked some people when it will happen that computers will replace doctors so the doctors actually say it's already happening they call it dr google apparently dr google is an adversary to real life doctors because every patient comes to them already checked on google what's happening or some other website and then and then and there are other all sorts of problems of this it's not i'm not sure it's a good idea to recommend this or not but like it's a fact that people use it doctor google exists kind of it's it yeah it's just a blur doctor google is super here but like because they they say oh it's doctor good the one who gave you advice but no it's just dr google here but like because they they say oh it's dr good the one who gave you advice but no it's yeah this real time like i went to a doctor and then the doctor went out of the room and then when i poked my uh my head over like on the door the doctor was on webmd checking to see like what what's going on with me so So I could have checked with WebMD myself and not paid the money. MARCO TULIO PIRESI- By the way, if you go out to all of the people who are doing machine learning now, you know how many of them are actually talking about using medical data, answering medical questions? I just talked to someone on this Saturday on AI Mac, a maker space and someone is still doing that and how much time do you think that some people will create those bots that are actually pretty good and it will become a riot and people will start using them? Yeah I was just in ICU yesterday looking at my own information and um you know informed the doctors and they were like oh yeah that's probably what it was so yeah that was using a chat gpt board uh you see so look it's endless and the question is when it will become good enough so people use it so some people the doctor said it's endless and the question is when it will become good enough so people use it so some people the doctor said it's already happening but then you ask the ai people say and they tell you no some parts are impossible to solve almost like the more person is technical the less they see the solutions that the doctors think already out there this is what my i did it only it's not a like a huge study i just asked a bunch of people. You can actually find this in the talk I gave in in PyData and in PyCon Israel about this project. So it's all in the About page of my here. I'll show you the About page again. You can find it all and exactly what questions they're asked and so on and so forth. So you can find them in either this PyCon Israel talk or PyData Austin talk. People ask things and people think about the same thing about the ability of the machine to do things. But one of the answers is pretty smart um go and look at this it's it's it's it's a it's a nice thing but this is my uptake of it it will happen whether in 50 years or 40 or 60 or maybe 100 or maybe 30 or maybe even 10 i don't know but it's gonna happen the question is do we want how do we want it to happen and how are we going to reach it? I've been working on this for five years. I'm pretty sure that what I have now is a good solution because I am using the latest technologies. Up to at least a year ago or something like that. Today there are even newer technologies that are coming up pretty fast, but it's still based on transformers. Until something better than transformers comes around, this is the best technology around there. As far as I know. Any more questions? Let's check the chat.... chat uh okay jared do trials themselves actually introduce new units of measure like there is no authority for them like iso or healthcare industry groups well jared the thing is clinicaltrials.gov the website i showed you is where all the data from all the people who conduct clinical trials goes into by law now uh a few years ago they actually gave me access to the obey to where they enter data so I can actually look at it. So just so I can look at see how it goes in. So they do standardize the data. They do check the data that comes in for errors. However, it's impossible to actually do it properly because each trial that plugs in the data plugs it the way they want to. Many times they just copy from reports they publish in this journal or some other report that they publish elsewhere or some PDF or whatever it is. And they just plug it in, copying it almost verbatim. And apparently each one of them uses a different language. Now, should they standardize? Yes, I think so. I think this is how it's supposed to be. But if you look at the reality, reality is no. I'll tell you even worse. I was in a patent lawyer convention. The patent lawyers tell them, you shouldn't put everything on clinicaltrials.gov. This was made publicly in the patent because each one of the people who put data in clinicaltrials.gov, not each one of those, but each one of the entities, they may have other interest about what they want to share and how. so we're talking about a lot of diversity here and when you have this level of diversity you get to the point of uh yeah the tower of babel because even everyone may be slightly different than the other everyone is individual but then when you get a bunch of individuals each speaking their own language, you get the Tower of Babel like I just showed you here. It's not working. Let's see if there's any more questions. Yeah, David Hirsch mentions Nomad City. Yes, it's one civilization. There's a sphere, a fire, actually, it's called fire. Fire, SNOMED CT, there's all sorts of like standards. There's a bunch of them, David. But apparently, when you have multiple standards, the question is also what's the standard? So you want to be able to translate between standards or have standard of standards or the ones that you want to use. And you want the machine to do it for you. This is what this project can help you do. So if your organization, you want to use this standard, yes, you can train the machine to translate to that standard. It's just a matter of how much work you can put in this. Can you maybe talk to the government and use your tool to actually help with the input so the data can become standardized? Well, actually, sure. I'm a business. I will reveal this part. I actually, last May or June, I was in DC area and met with the former head of clinicaltrucks.gov. He connected me to the person who deals with the AI there, and I offered them the tool. They were impressed by the accuracy, but they said they had their own AI. JOHN MUELLER JR.: Hm. Yeah, he doesn't. GERHARD HOFFSTAEDTERNENGER- Or at least this is what I remember from the talk. This is what I was told. I don't know how fast or slow government will actually implement something whether it will be better or worse i can tell you that there is another government project uh led by nist that actually also made contact and showed them but then again they don't work with sole proprietors in that uh call it's like there was a funding opportunity and they actually excluded sole proprietors so if someone wants to build a business in sole proprietor and write a grant then talk to me. It's like they already started the team that team is about three years behind me and don't use the technology that I have. They are not the spheres you know they are not as technological savvy as I am. Look, I did this on my own with Josh's help and some advice from multiple people that you saw the names there. Everyone contributed like an advice or use this or try this or here's some data and stuff like that. These are the people that helped and I did it alone. So AI and computer, it's like computer chess. If you have AI plus humans, it's better than either computer alone or human alone. You have to have this combination. But then you don't have to invest so many resources. But that's my take on it. Yeah, it seems like it's an input problem, so if you can solve the problem with the source, then you don't have to deal with it. Yeah, you just solve one problem of clinicaltrials.gov. I won't tell you the continuation of this project publicly, some of it I shouldn't reveal, because I do want to eventually monetize it. I did put the conflict interest statement and so on i do want people to actually uh i don't want to profit from this project eventually i did put five years of work in it it's not just an open source thing that you do something you invest a lot of work in it years of work um so i can tell you that there are other applications even if you stop the leak in this source, how many more leaks like this do you have? Think about it. Yeah, I don't know. You have a problem in clinicaltrials.gov which is very specific, but okay, you stopped it there. How many more sources of data you have? Yeah. Medical data that you can actually deal with. Won't they have the same problem? I'm pretty sure they will have. Yeah, I mean, it seems to be like, solution could be pretty universal to many things. And I told you it can be applied to other standardization tasks, such as I gave you a simple example, cities in the world. But if you work in places where you a simple example, cities in the world, but if you work in places where you have lots of sources of data coming together, streams of data, they look, even if they put in the same table, you will see that the minute you have multiple sources, you have a standardization problem, this can help. It's not a foolproof solution nothing but this technology can help even if slightly very slight changes sometimes okay uh i hope i didn't say i don't see him anymore do you see any more questions or does someone has do you see any more questions or does someone has don't see any more questions there i see tom is saying something it's pretty amazing such conceptually simple issue like standardization of machine data formatting requires so much work. Well, yes. The problem is tough when you have a lot of data, Tom. Yeah, but once you've did the work, then, and you have a machine to help you, then it becomes easier. The more I started working in the system, the more easier it became. I believe that the last several few uh last units i have to map will take me much less than it took me so far just because i have the machine now to help me it's already pretty much standardized and then of course there'll be level of standardization and some other variations in this but i i won't tell you the future of this project so touch me in a year or two, I'll tell you how it was. Yeah, Oracle paid 28, yeah, I see that people say that Oracle paid a lot of money. Yeah, there's a lot of money in healthcare, but apparently I don't see a lot of it. If someone has connections, connect me this this project is looking for someone big to swallow it it is silas what do you say ah silas is bringing some sketch yeah just link into it to the oh yeah just link to one of the XKCDs because you mentioned the problem of standards. And then somebody comes up with a brilliant standard. It's like, OK, well, now we have a new problem. We've got to interoperate between these new standards. Just a different idea, but it reminded me of that XKCD of the problem of somebody wants to fix the standard problem. And then all they did was create the n plus one standard they have to do yeah well i that's a problem that anyone has all the time uh tom says something i know it very well you used to pass requirement data for feed large software model the human creativity in writing some similar requirements was amazing that was pre AI days well Tom now you have AI that you can teach to actually figure out things so why not use it all of those a eyes that people are using in a very excited about why not put it to use in some good examples this is one example I've been doing it for five years but really the last several years were a jump and i believe the next several years will also have very useful technologies that will get dissolved Okay, I'm not for... David is writing something about Beowulf. The hell is Beowulf? I'm not familiar with this, but I'm not sure exactly. I'll check it out later. Unless you want to say something about it, then I'll just check it out later. Unless anyone has more questions, can we stop the recording? No one's answering? Is it everyone's quiet or what happened? Maybe there's no questions. Okay, no questions. Then guys, thank you very much for listening to this. All right. Thank you, Jake. It was a great presentation. Thank you. Thanks, Jacob. Thanks, Jacob.
Jacob Barhak - ClinicalUnitMapping.Com Takes a Small Step Towards Machine Comprehension of Clinical
Jacob Barhak - ClinicalUnitMapping.Com Takes a Small Step Towards Machine Comprehension of Clinical Trial Data

Session:
Clinical Trial data is not standardized and numerical data cannot be comprehended since the units are not standardized. ClinicalUnitMapping.com is a web tool constructed to help standardize this data and merge it with the following standards and specifications: UCUM, RTMMS / IEEE 11073-10101, BIOUO, and CDISC. IEEE 11073-10101 - Adapted and reprinted with permission from IEEE. Copyright IEEE 2019. All rights reserved.
The intention is to unify unit standards and machine learning tools, so that clinical trials data would become machine comprehensible.

Presenter:
Jacob Barhak Is an independent Computational Disease Modeler focusing on machine comprehension of clinical data. He is one of the organizers of the Austin Python meetup.