Medical image reconstruction using the .NET Framework
Sedmair, Martin
Medical image reconstruction is a highly demanding topic in terms of processing power, memory consumption and computational complexity. In the past several low level approaches have been conducted that improved performance but made such systems become inflexibel and reduced maintainability. This presentation shows a setup of using the .NET framework as well as a GPU-based integration in the C# language. Several topics in image reconstruction and processing will be shown in the context of medical image processing. Additionally, challenges and problems of .NET and C# are discussed. Since the introduction of Computed Tomography (CT) as a medical diagnostic imaging procedure 40 years ago, there are still unresolved challenges regarding radiation dose, image quality and image reconstruction. One of the major concerns is radiation dose applied to the patient and - not frequently mentioned - the medical staff. For that reason, several new inventions have been made that improve the equipment of the CT-scanning device, for example X-ray sources and detectors systems. Recently, new developments in the computing power of modern computers have led to new approaches that enable the replacement of traditional reconstruction methods (filtered backprojection) by using iterative or AI-based methods as well as cloud-based approaches that allow both, the reduction of radiation dose and a tremendous improvement in image quality. However, evaluating new methods require a solid development environment to allow for and simple and convenient implementation of new ideas and algorithms. Although there are several well-known and established platforms that are a good starting point for plain prototyping, an efficient execution of code inside these platforms is often challenging with regard to the management computing power and memory resources. Additionally, shipping prototypes to customers for evaluation is often not easy. This presentation starts with a basic introduction into medical image acquisition and reconstruction, followed by a short explanation of the storage of image data using the DICOM (digital imaging and communications and medicine) standard. Afterwards it is shown how the .NET environment using C#, .NET and .NET Core, WPF and the nVidia computing platform (CUDA) can be applied as a modular prototyping platform for developers, researchers and customers in order to develop new algorithms and to conduct clinical and scientific studies. The presentation will end with a summary of state-of-the-art development with C# and point out challenges that exist with managed runtime environments regarding memory management, performance and missing language and CLR features that would help to further improve the development within the C#/.NET environment in the light of high performance computing.
 So this is the reason why the little bit of the story background, how images are acquired, how images are read. Better? All right. It's only for the, all right. Yeah, so I will also talk about iterative reconstruction because this is very performance demanding. So in earlier days it was, yeah, it was possible to do it in MATLAB. And I can tell you when I started 12 years ago what the performance of MATLAB was. When CT started there were really challenges in general. Then I will also talk about challenges with the.NET framework or also with C sharp. And for performance reason we also use technologies like the math turn library, CUDA, and we are also investigating or checking what Vulkan can do because CUDA has a little bit of problem in general. So, the story background, I don't want to go through the whole table. It all started 1895 with the discovery of X-rays. It was a pretty interesting thing because this finally led very fast to technical inventions like what they scanned is, they scanned your feet to just measure your shoes. And they really get rid of this technology very early because a lot of X-rays are really because we are talking about X-ray radiation. This is causing trouble of course if you simply measure your legs or your feet just to measure your shoe size. Yeah, another approach or another invention discovery was Johann Radon, an Austrian researcher who discovered the Radon inversion formula. I don't know if you have something, if you know about this. We will see in the technology or the mathematical backgrounds of back projection. So, in the end this is pretty simple. He says every ray that comes from every angle, if you inverse this process, you can recover the original image. So, that's what inside your body, you scan with X-rays. You see everything that is in the X-rays, projected on a single line as you know from a chest X-ray from your doctor. And if you invert this from many projections, you can recover your image. Yeah, 1936, this was the time where there were a lot of colors, there were the hippies, and there was EMI and the Beatles. And obviously they earned so much money with the Beatles records that they were able to fund an electrical engineer, they were able to do research here just on his EMI deck, this was a simple computer at that time. And he also did some research on the first CT system. So, this is the first CT system with wood and a little bit of steel and an X-ray source and some detector. And this is a reconstructed image, so this is a brain. So, some of you can tell me if you can see anything of this and you can do your diagnosis on it. Something is here maybe, but only those ones who are aware of these images know a little bit better what's inside the images. Yeah, 1974, there were of course, it's Siemens, the first publicly available CT scanner. It's only for the head, the acquisition time of the head was seven minutes. I can show you later numbers, how far we are now. And this is how it looks like from the inside, so here's the X-ray source. This is how a system looks now. Here you can put in the whole body. And if you open up the system now, you have the X-ray source of course, you have the detector, and these are the most expensive components of the system, that's one of the biggest problems. We're talking about 50 to 80,000 euros for these two systems. Taking it from the other side, you need a high voltage, a cooling system, of course you need electronics, and a power transmission. And from 1974, you see these images here, to 1996, you see this image. So of course these are not the same patients, but you can directly see that of course the image quality has improved. And so we are 2018, and we are still at the same image quality. So we have a bit better image quality, but we stayed with a 512 by 512 matrix size, and here we are with 80 to 80 matrix size. It's a matter of memory consumption, because if you double the size, it has to be stored somewhere, and we are currently having real trouble storing all the images that are generated from an X-ray system. So from 1972, we have an acquisition time of the seven minutes, so 300 seconds, and the data is acquired for one rotation, so you have to imagine what happens with the CT scan. So you start a rotation, you rotate around the patient for one image slice, and if you have a so-called spiral, so if you want to scan from head to toe, you have to scan with a spiral scan. So here is no number, because at that time there was no spiral scan available, so you had to rotate, you had to shift, you had to rotate to shift, and so on and so forth from the top to the bottom. In 2015, we rotate at 0.25 milliseconds, we produce data per rotation 270 megabytes, and for a full scan one gigabyte, and the largest file I have seen so far is about 30 gigabytes, and with newer scanners we are coming up to 100 gigabytes. So this is something you have to process, and one of the biggest problems is that if you're doing a CT scan, you sometimes have emergency cases, and you don't want to wait 20 hours until the images are finished, so what is the demand here is that you have something like 50 images reconstructed per second, mostly 100 images per second out of the 30 gigabytes dataset. There is one question, who knows why the 300 seconds could be reduced to the 0.2 seconds? I can show it. The first scanners were a translation rotational system, so you had one X-ray source, one pencil beam, you had to shift the pencil beam, then you rotate, then there is another shift, then there is another shift, and this has evolved to a so-called cone beam system where you could rotate, and you don't have to shift and translate. The only problem is that there were cables, so with one rotation you had to go back, rotate again, and this is where the images with the power transmission comes in. This is a so-called slip ring. With the slip ring, in Germany it's called Schleifring, so it's some sort of a slip where brushes are connected to this metal bearing here, and in that sense you can rotate forever, and you have a continuous power transmission, and you don't have to go back and forth, so you rotate in this so-called spiral, and this is also what enabled the spiral scan, so you can start, you have your spiral and scan over the patient. I don't want to talk about all these numbers. There is of course the matrix size here, so 2018 we shifted to 2024 matrix size. If you go to, let's say, regular CT, I'm not CT, regular chest X-ray scans, they have a matrix size of 4096 to 4096, so much larger. The images, the number of images we produce are around 2,000 images per patient, something like this. So I talked about the image acquisition. This is what I call the pencil beam. So here's the X-ray source, and I talk about this very primitive example because it's easier to explain the other examples. You scan over the patient, here inside there is the brain, and then there is a so-called attenuation, so every tissue, bone, liver has a very different attenuation property, and this attenuation property is measured here in a single line. So you project a two-dimensional plane onto a one-dimensional line for each angle while you rotate around the patient. And this is the so-called synogram, so here you see the process, a little bit old animation, but in the end what you get is projections. So there is a detector that has around 1,000 channels in one direction, and let's say 64 to 128 rows. So this is one projection from one angle, then there is another projection, and if you finish this up, this is the so-called synogram. In the end, it's an X-ray from many projections, regular X-ray from many projections. And now we're coming to this Radon inversion formula with this Austrian scientist. We need to make from this synogram a reconstruction of the image. So in the end what you do is there are two methods, either Fourier-based, which is normally not used, or image-backed projection. So in the end what you do is the process I have shown, taking a projection from one angle from the next angle, is simply inverted mathematically. So you sit on a pixel here, and you look at the projections that are contributing to this pixel, and you just reverse the process, just mathematical imagery construction. Of course this is time-consuming, I have some numbers here, but I just showed this example, so this is how it works. This is the original object, just a cylinder. This is one projection, this is how a projection in 1D looks like. You do it many times, three projections, and then you see finally you have some reconstructed image, and now you see that there is something wrong. You have a cone, you have not a cylinder. And this is what is called filtered-backed projection. So something goes wrong in this regular acquisition process. And you need to filter this one-dimensional projection of a single angle. For the next angle you also need to do it. So there is a so-called filter kernel that you apply to this data. It looks like this, there are many kernels. And if you do this, then it should go. For many projections, in the end for 128 projections, for very primitive objects it's pretty easy, but if you have a very complex object like a body, it's pretty complex and 128 projections are not enough. We're talking about 4000 projections for each angle. There is one problem though. You have to decide if you have a sharp kernel or if you have a smooth kernel. And the kernel controls the image impression. So if you have a sharp kernel, you have very fine details. The only problem is you have very high noise. And it always depends on your indication you want to see. Do you want to see small fissures on bones or do you want to see tumors? And this is where you have to decide between the kernels. So that's what the kernel does. It controls sharpness, noise and the edges. These are medical images, so we are using mostly phantoms in our department because it's much easier to see the effects, not on a very complex human tissue, but to have some simulated phantoms where we check what's going on. So this is a regular head phantom where you have this bone here. This is a very complex bone with very fine structures. And you also see low attenuating objects. And this is some sort of where you want to see if your algorithms work good for soft tissue, for low attenuating things or for high attenuating things. So this is one approach, how a regular reconstruction works. And you see that there are some problems in the reconstruction. There is a so-called beam hardening effect. Here you see some sampling effects. Obviously, if you enlarge this, you can see in the volume stack from many sides what the result of the reconstruction was. And there is another approach, and this is iterative reconstruction. Iterative reconstruction is a process where you revert the process of reconstruction again. So you do a technical forward projection, which means it's a technical process of acquisition. So first of all, you have a reconstruction. It's a 3D volume stack. You made some errors with your algorithms because they're not perfect. Then you try to do some so-called forward projection. So you invert the process. It's a little bit complicated. The process of reconstruction again. And then there is some sort of an image. So this is some sort of a loop where you try to, with a few cycles, you try to recover the image quality and put also some prior knowledge in it. Because you know at the edges there is always a beam hardening, so you put prior knowledge in it just to recover these beam hardening effects. And of course you can imagine that this all costs a lot of processing power. For a regular system, we have around 700 gigabytes of main memory in the system and about four Volta GPUs that are doing this process. And as I said, 30 gigabytes and 100 images per second is a very highly demanding process on the developers that are doing this. We are in the field of research. That's the advantage. We don't need to be that fast. But still there is a high demand on having images quickly. So I just want to show the effects of iterative reconstruction. So this is a regular reconstruction. There is some sort of filtering on it. Here is again the first version of iterative reconstruction. And this is something where the full iterative reconstruction was applied. And as I said, we are talking about x-rays. And of course there is a so-called Alara principle as low as reasonably achievable. So you should, of course, you have to apply x-rays to see something. But in the end, you should only apply as much as you need. And if the algorithms are good enough, then you can reduce the dose further or keep the same dose, but it can improve image quality. Two, for example, detect attenuating regions here in the bladder. So this is a thorac scan and abdominal scan. This is a pretty large area. So this is from here to here somehow. And you can also see that there is some noise reduction. So you're going from Houndsfield units, by the way, was named by the inventor of the system. It's a unit that was introduced. It's a unit less unit. So there's nothing. It only is some relative value to water. And here are also some improvements because you want to see very fine details. This is what is shown in the regular weighted filter back projection. And this is some IR process where you can much better see the line pairs and the small lines of the object. Also here, this is a coronary stand that if you have some occlusion on your heart vessels, then you apply metal stands on it to open up the vessel again. And it's some sort of a grid in the cage which is inserted into the heart. And it's helpful to see what it's inside, to see the stand struts. There are so-called stand struts because it's possible that they clot again. So they get closer again. And this is necessary to see inside this stand. I will talk about the techniques behind. I can only cut pieces. It's a pretty large area where I can only show a few details we experience with the.NET framework and with the C-sharp. So in general, we are talking about volumes. So our volumes, about 2,000 slices, let's say. 512 by 512, stored in float byte, depending on what's inside or in textures. And you need to store it somehow. So this is the small diagram where we have some class which is a volume. And we have many types of volume. So because we also want to support, for example, MATLAB volumes, so we need to have some wrappers around these volumes. So this is the, let's say, the data collection that manages the slices, the volume and image slices. And there is also, for the 30 gigabytes, the projection data. It's somehow identical. It's also sorted in slices. And you also need to store it. But the problem is, we don't have the size of 700 gigabytes, I told you before, in our production system, we only have 32. So what we need to do is, we need to invent our own memory management, other than the regular garbage collector, because the garbage collector only cleans up. But in our case, it's something we need to access to the data more often, so that it shouldn't get disposed. And we need to sometimes access it sequentially and sometimes randomly. It's depending on the task we're working on. So what we have is, each of these slices and the projections are derived from a so-called mapable object. And the background thread is running in the background and continuously checking which object. There is a reference counter. There is a time counter, how often the object was used, when it was used last. And depending on this, these automatic process checks, is there enough space free? And then dumps data that is not used for a long time, just to the hard disk. And when we started, memory map files were not available. There was a problem, so we could have used memory map files. But the problem is, memory map files, so the last two years we have tested it, memory map files are currently very slow. So there are two things where we can access memory. Has any of you used memory map files? Okay. So just to explain, memory map files is some mechanism where the virtual filesist or the virtual memory manager from Windows takes over and says, okay, you don't need this anymore, I just dump it. And of course it's stupid to do it ourselves, but the memory manager was, first of all, a little bit slow because of this line here. And the other thing is that when we access the data, the data normally from the system is stored in a compressed way. So it's not only reading, it's also unzipping more or less and doing some data compression, data formatting on it. So just simply writing and loading is not good enough for us. So in the end, there were two things, why we couldn't use it, is first of all because of this line. It's very slow. Obviously, no one has ever invested more on it. That is one thing. And there comes another thing. So this is a little bit of a new topic. How many of you are using arrays, large arrays in your environment? So do you use a lot of arrays or just I just do it once and then it's over? So that's the interesting thing because we were talking about the benchmark.net. And since I started with this about 12 years ago, I did a lot of benchmarks. And this is the regular addition. So I show these arrays. So this is just adding two floats. I just want to add two floats. This is conventionally done. Taking the a.length here, oh, that's wrong, 422. Then if I extract the a.length to just an external variable, I get this number. I always thought it shouldn't be the case because I thought the chitter is optimized, but I was very surprised. If I went to pointer arithmetics, just make it unsafe and do a pointer arithmetics, I come to this area. Here the external n doesn't really matter anymore. I was also very surprised. And of course I tried the span and I know that the span is not very functional or is not so fast as in.NET Core. Here the performance dropped. So of course I was hoping for.NET Core that this will improve, but I was also using the SMID and this was the best performance in that case. When I do this here on.NET Core, I was very surprised and I hope someone proves me wrong that something is wrong here because all the values were much lower than on this side. And as we heard on the other side that.NET Core is fast, I also read these two blog posts and I was very surprised that I get the wrong numbers here. There's another thing. What if I want to do it with generics? Have you ever tried this? It does not work. C-SHAF does not support mathematical operations on generics as C++ do. This is where a lot of my colleagues came and say, why is it not possible? And there are a lot of cases where we had to write a very large math library because if you're doing area math, sometimes you want to have a constant added to array. And of course you have commutative operations on arrays. It's not always that you have only one function. So I think we have around for all these mathematical operations like cosine, sine and so on, I think around 500 functions. And one problem is that generics are too weak. So what we did is we wrote our own IL code because IL is able to add. IL is obviously type agnostic and is able to add two values. And if you do it correctly, you get the same speed as a regular addition writing it in regular source code. There was also one thing. If you look at this mathematical expression, these are volumes, 512, 512,000. And if you want to have these operations, this is one operation, for example, you use for the iterative reconstruction. So 0.3, a plus b and so on. So here you see what's happening. For the a plus b, you have one gigabyte of memory. Then you add another gigabyte of memory because all these expressions add up. And in the end, for this simple expression, I allocated 5 gigabytes more than I needed. What we did is we resolved the expression. So it's a little bit, I was hoping with the Roslin we could do a little bit more. Just write the expression because when you have these expressions, you want to see the real mathematical expression and not the long expression of the function calls. So it's helpful for debugging and also helpful for researchers when they look in the code. So what we did is you apply here a string and some parser gets into this and recompiles your code and builds only one for loop around these thousand slices and extracts the math again, this expression into just one expression call. So first of all, you have a factor of 4 of performance gain and only one gigabyte of allocation if you do this. And there's one thing, but I'm not quite good in this Roslin support right now. What would be really helpful would be if you could write expressions and before the real compilation process, some parser that we write goes into this and extracts this line and inserts the code directly. So that's currently not really possible. There's also custom arrays which is a little bit of a problem. The same style because there is no numeric type which we can apply on the data. And I don't want to go into this, I just show the cuter things. Two minutes, alright. Has anyone of you used cuter? Alright, in C sharp or in C++? Yeah. The problem is that for the C sharp environment, there is no real access. So you have these cuter driver API you can access with Windows API calls. And of course for the back projection, just to give you a few numbers, for the back projection I showed here, we have around 134 tera operations per image for the back projection. So what we did is we used cuter. You can write cuter in.NET as regular cuter code of course in C style or C++ style. This is how the cuter code looks like. This is here inserted into the solution. And then if you simply say save code, there is a generator that generates the PTX code, the NVCC which goes over the code, that is the real machine code from cuter. Then there is a designer code that translates the code via C lang, we use C lang for the AST. And it generates a wrapper for C sharp. So this is the wrapper, it's very simple, it's just accessing a three dimensional texture. And this would be the code that you normally have to write. That also is what you normally don't see if you use it in C++. But this is what C++ does for you. Sorry. C++ does for you. Just writing all these wrappers around your code and then you can execute cuter code simply from C sharp. So this is it. That's a lot of things, just a few pieces. And I would be also interested in the net core performance test. If someone ever has better experiences than me. Thank you.
 So this is the reason why the little bit of the story background, how images are acquired,  how images are read.  Better?  All right.  It's only for the, all right.  Yeah, so I will also talk about iterative reconstruction because this is very performance demanding.  So in earlier days it was, yeah, it was possible to do it in MATLAB.  And I can tell you when I started 12 years ago what the performance of MATLAB was.  When CT started there were really challenges in general.  Then I will also talk about challenges with the.NET framework or also with C sharp.  And for performance reason we also use technologies like the math turn library, CUDA,  and we are also investigating or checking what Vulkan can do because CUDA has a little bit of problem in general.  So, the story background, I don't want to go through the whole table.  It all started 1895 with the discovery of X-rays.  It was a pretty interesting thing because this finally led very fast to technical inventions  like what they scanned is, they scanned your feet to just measure your shoes.  And they really get rid of this technology very early because a lot of X-rays  are really because we are talking about X-ray radiation.  This is causing trouble of course if you simply measure your legs or your feet just to measure your shoe size.  Yeah, another approach or another invention discovery was Johann Radon,  an Austrian researcher who discovered the Radon inversion formula.  I don't know if you have something, if you know about this.  We will see in the technology or the mathematical backgrounds of back projection.  So, in the end this is pretty simple.  He says every ray that comes from every angle, if you inverse this process,  you can recover the original image.  So, that's what inside your body, you scan with X-rays.  You see everything that is in the X-rays, projected on a single line as you know from a chest X-ray from your doctor.  And if you invert this from many projections, you can recover your image.  Yeah, 1936, this was the time where there were a lot of colors, there were the hippies,  and there was EMI and the Beatles.  And obviously they earned so much money with the Beatles records that they were able to fund an electrical engineer,  they were able to do research here just on his EMI deck, this was a simple computer at that time.  And he also did some research on the first CT system.  So, this is the first CT system with wood and a little bit of steel and an X-ray source and some detector.  And this is a reconstructed image, so this is a brain.  So, some of you can tell me if you can see anything of this and you can do your diagnosis on it.  Something is here maybe, but only those ones who are aware of these images know a little bit better what's inside the images.  Yeah, 1974, there were of course, it's Siemens, the first publicly available CT scanner.  It's only for the head, the acquisition time of the head was seven minutes.  I can show you later numbers, how far we are now.  And this is how it looks like from the inside, so here's the X-ray source.  This is how a system looks now.  Here you can put in the whole body.  And if you open up the system now, you have the X-ray source of course, you have the detector,  and these are the most expensive components of the system, that's one of the biggest problems.  We're talking about 50 to 80,000 euros for these two systems.  Taking it from the other side, you need a high voltage, a cooling system, of course you need electronics, and a power transmission.  And from 1974, you see these images here, to 1996, you see this image.  So of course these are not the same patients, but you can directly see that of course the image quality has improved.  And so we are 2018, and we are still at the same image quality.  So we have a bit better image quality, but we stayed with a 512 by 512 matrix size, and here we are with 80 to 80 matrix size.  It's a matter of memory consumption, because if you double the size, it has to be stored somewhere,  and we are currently having real trouble storing all the images that are generated from an X-ray system.  So from 1972, we have an acquisition time of the seven minutes, so 300 seconds,  and the data is acquired for one rotation, so you have to imagine what happens with the CT scan.  So you start a rotation, you rotate around the patient for one image slice, and if you have a so-called spiral,  so if you want to scan from head to toe, you have to scan with a spiral scan.  So here is no number, because at that time there was no spiral scan available,  so you had to rotate, you had to shift, you had to rotate to shift, and so on and so forth from the top to the bottom.  In 2015, we rotate at 0.25 milliseconds, we produce data per rotation 270 megabytes,  and for a full scan one gigabyte, and the largest file I have seen so far is about 30 gigabytes,  and with newer scanners we are coming up to 100 gigabytes.  So this is something you have to process, and one of the biggest problems is that if you're doing a CT scan,  you sometimes have emergency cases, and you don't want to wait 20 hours until the images are finished,  so what is the demand here is that you have something like 50 images reconstructed per second,  mostly 100 images per second out of the 30 gigabytes dataset.  There is one question, who knows why the 300 seconds could be reduced to the 0.2 seconds?  I can show it. The first scanners were a translation rotational system, so you had one X-ray source, one pencil beam,  you had to shift the pencil beam, then you rotate, then there is another shift, then there is another shift,  and this has evolved to a so-called cone beam system where you could rotate, and you don't have to shift and translate.  The only problem is that there were cables, so with one rotation you had to go back, rotate again,  and this is where the images with the power transmission comes in.  This is a so-called slip ring.  With the slip ring, in Germany it's called Schleifring, so it's some sort of a slip where brushes are connected to this metal bearing here,  and in that sense you can rotate forever, and you have a continuous power transmission,  and you don't have to go back and forth, so you rotate in this so-called spiral,  and this is also what enabled the spiral scan, so you can start, you have your spiral and scan over the patient.  I don't want to talk about all these numbers.  There is of course the matrix size here, so 2018 we shifted to 2024 matrix size.  If you go to, let's say, regular CT, I'm not CT, regular chest X-ray scans,  they have a matrix size of 4096 to 4096, so much larger.  The images, the number of images we produce are around 2,000 images per patient, something like this.  So I talked about the image acquisition.  This is what I call the pencil beam.  So here's the X-ray source, and I talk about this very primitive example because it's easier to explain the other examples.  You scan over the patient, here inside there is the brain, and then there is a so-called attenuation,  so every tissue, bone, liver has a very different attenuation property,  and this attenuation property is measured here in a single line.  So you project a two-dimensional plane onto a one-dimensional line for each angle while you rotate around the patient.  And this is the so-called synogram, so here you see the process, a little bit old animation,  but in the end what you get is projections.  So there is a detector that has around 1,000 channels in one direction, and let's say 64 to 128 rows.  So this is one projection from one angle, then there is another projection,  and if you finish this up, this is the so-called synogram.  In the end, it's an X-ray from many projections, regular X-ray from many projections.  And now we're coming to this Radon inversion formula with this Austrian scientist.  We need to make from this synogram a reconstruction of the image.  So in the end what you do is there are two methods, either Fourier-based, which is normally not used, or image-backed projection.  So in the end what you do is the process I have shown, taking a projection from one angle from the next angle,  is simply inverted mathematically.  So you sit on a pixel here, and you look at the projections that are contributing to this pixel,  and you just reverse the process, just mathematical imagery construction.  Of course this is time-consuming, I have some numbers here, but I just showed this example, so this is how it works.  This is the original object, just a cylinder.  This is one projection, this is how a projection in 1D looks like.  You do it many times, three projections, and then you see finally you have some reconstructed image,  and now you see that there is something wrong.  You have a cone, you have not a cylinder.  And this is what is called filtered-backed projection.  So something goes wrong in this regular acquisition process.  And you need to filter this one-dimensional projection of a single angle.  For the next angle you also need to do it.  So there is a so-called filter kernel that you apply to this data.  It looks like this, there are many kernels.  And if you do this, then it should go.  For many projections, in the end for 128 projections,  for very primitive objects it's pretty easy, but if you have a very complex object like a body,  it's pretty complex and 128 projections are not enough.  We're talking about 4000 projections for each angle.  There is one problem though.  You have to decide if you have a sharp kernel or if you have a smooth kernel.  And the kernel controls the image impression.  So if you have a sharp kernel, you have very fine details.  The only problem is you have very high noise.  And it always depends on your indication you want to see.  Do you want to see small fissures on bones or do you want to see tumors?  And this is where you have to decide between the kernels.  So that's what the kernel does.  It controls sharpness, noise and the edges.  These are medical images, so we are using mostly phantoms in our department  because it's much easier to see the effects, not on a very complex human tissue,  but to have some simulated phantoms where we check what's going on.  So this is a regular head phantom where you have this bone here.  This is a very complex bone with very fine structures.  And you also see low attenuating objects.  And this is some sort of where you want to see if your algorithms work good for soft tissue,  for low attenuating things or for high attenuating things.  So this is one approach, how a regular reconstruction works.  And you see that there are some problems in the reconstruction.  There is a so-called beam hardening effect.  Here you see some sampling effects.  Obviously, if you enlarge this, you can see in the volume stack from many sides  what the result of the reconstruction was.  And there is another approach, and this is iterative reconstruction.  Iterative reconstruction is a process where you revert the process of reconstruction again.  So you do a technical forward projection, which means it's a technical process of acquisition.  So first of all, you have a reconstruction.  It's a 3D volume stack.  You made some errors with your algorithms because they're not perfect.  Then you try to do some so-called forward projection.  So you invert the process.  It's a little bit complicated.  The process of reconstruction again.  And then there is some sort of an image.  So this is some sort of a loop where you try to, with a few cycles,  you try to recover the image quality and put also some prior knowledge in it.  Because you know at the edges there is always a beam hardening,  so you put prior knowledge in it just to recover these beam hardening effects.  And of course you can imagine that this all costs a lot of processing power.  For a regular system, we have around 700 gigabytes of main memory in the system  and about four Volta GPUs that are doing this process.  And as I said, 30 gigabytes and 100 images per second is a very highly demanding process  on the developers that are doing this.  We are in the field of research.  That's the advantage.  We don't need to be that fast.  But still there is a high demand on having images quickly.  So I just want to show the effects of iterative reconstruction.  So this is a regular reconstruction.  There is some sort of filtering on it.  Here is again the first version of iterative reconstruction.  And this is something where the full iterative reconstruction was applied.  And as I said, we are talking about x-rays.  And of course there is a so-called Alara principle as low as reasonably achievable.  So you should, of course, you have to apply x-rays to see something.  But in the end, you should only apply as much as you need.  And if the algorithms are good enough, then you can reduce the dose further  or keep the same dose, but it can improve image quality.  Two, for example, detect attenuating regions here in the bladder.  So this is a thorac scan and abdominal scan.  This is a pretty large area.  So this is from here to here somehow.  And you can also see that there is some noise reduction.  So you're going from Houndsfield units, by the way, was named by the inventor of the system.  It's a unit that was introduced.  It's a unit less unit.  So there's nothing.  It only is some relative value to water.  And here are also some improvements because you want to see very fine details.  This is what is shown in the regular weighted filter back projection.  And this is some IR process where you can much better see the line pairs  and the small lines of the object.  Also here, this is a coronary stand that if you have some occlusion on your heart vessels,  then you apply metal stands on it to open up the vessel again.  And it's some sort of a grid in the cage which is inserted into the heart.  And it's helpful to see what it's inside, to see the stand struts.  There are so-called stand struts because it's possible that they clot again.  So they get closer again.  And this is necessary to see inside this stand.  I will talk about the techniques behind.  I can only cut pieces.  It's a pretty large area where I can only show a few details we experience with the.NET framework  and with the C-sharp.  So in general, we are talking about volumes.  So our volumes, about 2,000 slices, let's say.  512 by 512, stored in float byte, depending on what's inside or in textures.  And you need to store it somehow.  So this is the small diagram where we have some class which is a volume.  And we have many types of volume.  So because we also want to support, for example, MATLAB volumes,  so we need to have some wrappers around these volumes.  So this is the, let's say, the data collection that manages the slices, the volume and image slices.  And there is also, for the 30 gigabytes, the projection data.  It's somehow identical.  It's also sorted in slices.  And you also need to store it.  But the problem is, we don't have the size of 700 gigabytes,  I told you before, in our production system, we only have 32.  So what we need to do is, we need to invent our own memory management,  other than the regular garbage collector, because the garbage collector only cleans up.  But in our case, it's something we need to access to the data more often,  so that it shouldn't get disposed.  And we need to sometimes access it sequentially and sometimes randomly.  It's depending on the task we're working on.  So what we have is, each of these slices and the projections are derived from a so-called mapable object.  And the background thread is running in the background and continuously checking which object.  There is a reference counter.  There is a time counter, how often the object was used, when it was used last.  And depending on this, these automatic process checks, is there enough space free?  And then dumps data that is not used for a long time, just to the hard disk.  And when we started, memory map files were not available.  There was a problem, so we could have used memory map files.  But the problem is, memory map files, so the last two years we have tested it,  memory map files are currently very slow.  So there are two things where we can access memory.  Has any of you used memory map files?  Okay.  So just to explain, memory map files is some mechanism where the virtual filesist or the virtual memory manager from Windows takes over and says,  okay, you don't need this anymore, I just dump it.  And of course it's stupid to do it ourselves, but the memory manager was, first of all, a little bit slow because of this line here.  And the other thing is that when we access the data, the data normally from the system is stored in a compressed way.  So it's not only reading, it's also unzipping more or less and doing some data compression, data formatting on it.  So just simply writing and loading is not good enough for us.  So in the end, there were two things, why we couldn't use it, is first of all because of this line.  It's very slow.  Obviously, no one has ever invested more on it.  That is one thing.  And there comes another thing.  So this is a little bit of a new topic.  How many of you are using arrays, large arrays in your environment?  So do you use a lot of arrays or just I just do it once and then it's over?  So that's the interesting thing because we were talking about the benchmark.net.  And since I started with this about 12 years ago, I did a lot of benchmarks.  And this is the regular addition.  So I show these arrays.  So this is just adding two floats.  I just want to add two floats.  This is conventionally done.  Taking the a.length here, oh, that's wrong, 422.  Then if I extract the a.length to just an external variable, I get this number.  I always thought it shouldn't be the case because I thought the chitter is optimized, but I was very surprised.  If I went to pointer arithmetics, just make it unsafe and do a pointer arithmetics, I come to this area.  Here the external n doesn't really matter anymore.  I was also very surprised.  And of course I tried the span and I know that the span is not very functional or is not so fast as in.NET Core.  Here the performance dropped.  So of course I was hoping for.NET Core that this will improve, but I was also using the SMID and this was the best performance in that case.  When I do this here on.NET Core, I was very surprised and I hope someone proves me wrong that something is wrong here  because all the values were much lower than on this side.  And as we heard on the other side that.NET Core is fast, I also read these two blog posts and I was very surprised that I get the wrong numbers here.  There's another thing.  What if I want to do it with generics?  Have you ever tried this?  It does not work.  C-SHAF does not support mathematical operations on generics as C++ do.  This is where a lot of my colleagues came and say, why is it not possible?  And there are a lot of cases where we had to write a very large math library because if you're doing area math, sometimes you want to have a constant added to array.  And of course you have commutative operations on arrays.  It's not always that you have only one function.  So I think we have around for all these mathematical operations like cosine, sine and so on, I think around 500 functions.  And one problem is that generics are too weak.  So what we did is we wrote our own IL code because IL is able to add.  IL is obviously type agnostic and is able to add two values.  And if you do it correctly, you get the same speed as a regular addition writing it in regular source code.  There was also one thing.  If you look at this mathematical expression, these are volumes, 512, 512,000.  And if you want to have these operations, this is one operation, for example, you use for the iterative reconstruction.  So 0.3, a plus b and so on.  So here you see what's happening.  For the a plus b, you have one gigabyte of memory.  Then you add another gigabyte of memory because all these expressions add up.  And in the end, for this simple expression, I allocated 5 gigabytes more than I needed.  What we did is we resolved the expression.  So it's a little bit, I was hoping with the Roslin we could do a little bit more.  Just write the expression because when you have these expressions, you want to see the real mathematical expression  and not the long expression of the function calls.  So it's helpful for debugging and also helpful for researchers when they look in the code.  So what we did is you apply here a string and some parser gets into this and recompiles your code  and builds only one for loop around these thousand slices  and extracts the math again, this expression into just one expression call.  So first of all, you have a factor of 4 of performance gain and only one gigabyte of allocation if you do this.  And there's one thing, but I'm not quite good in this Roslin support right now.  What would be really helpful would be if you could write expressions  and before the real compilation process, some parser that we write goes into this  and extracts this line and inserts the code directly.  So that's currently not really possible.  There's also custom arrays which is a little bit of a problem.  The same style because there is no numeric type which we can apply on the data.  And I don't want to go into this, I just show the cuter things.  Two minutes, alright.  Has anyone of you used cuter? Alright, in C sharp or in C++?  Yeah. The problem is that for the C sharp environment, there is no real access.  So you have these cuter driver API you can access with Windows API calls.  And of course for the back projection, just to give you a few numbers,  for the back projection I showed here, we have around 134 tera operations per image  for the back projection.  So what we did is we used cuter.  You can write cuter in.NET as regular cuter code of course in C style or C++ style.  This is how the cuter code looks like.  This is here inserted into the solution.  And then if you simply say save code, there is a generator that generates the PTX code,  the NVCC which goes over the code, that is the real machine code from cuter.  Then there is a designer code that translates the code via C lang,  we use C lang for the AST.  And it generates a wrapper for C sharp.  So this is the wrapper, it's very simple, it's just accessing a three dimensional texture.  And this would be the code that you normally have to write.  That also is what you normally don't see if you use it in C++.  But this is what C++ does for you.  Sorry.  C++ does for you.  Just writing all these wrappers around your code and then you can execute cuter code simply from C sharp.  So this is it.  That's a lot of things, just a few pieces.  And I would be also interested in the net core performance test.  If someone ever has better experiences than me.  Thank you.
Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_0_167.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_168_1847.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_1848_2759.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_2760_3875.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_3876_4307.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_4308_4715.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_4716_6131.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_6132_6791.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_6792_11207.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_11208_12647.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_12648_12827.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_12828_14471.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_14472_14687.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_14688_14819.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_14820_18419.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_18420_18599.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_18600_21059.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_21060_21623.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_21624_21707.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_21708_25823.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_25824_26171.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_26172_27659.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_27660_31091.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_31092_31535.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_31536_35279.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_35280_35663.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_35664_36635.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_36636_38291.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_38292_38339.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_38340_38411.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_38412_38579.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_38580_38723.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_38724_39923.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_39924_40931.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_40932_41651.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_41652_41759.png Medical_image_reconstruction_using_the__NET_Framework_qTIbJ_wVZ0g_41760_41875.png
So this is the reason why the little bit of the story background, how images are acquired, how images are read. Better? All right. It's only for the, all right. Yeah, so I will also talk about iterative reconstruction because this is very performance demanding. So in earlier days it was, yeah, it was possible to do it in MATLAB. And I can tell you when I started 12 years ago what the performance of MATLAB was. When CT started there were really challenges in general. Then I will also talk about challenges with the.NET framework or also with C sharp. And for performance reason we also use technologies like the math turn library, CUDA, and we are also investigating or checking what Vulkan can do because CUDA has a little bit of problem in general. So, the story background, I don't want to go through the whole table. It all started 1895 with the discovery of X-rays. It was a pretty interesting thing because this finally led very fast to technical inventions like what they scanned is, they scanned your feet to just measure your shoes. And they really get rid of this technology very early because a lot of X-rays are really because we are talking about X-ray radiation. This is causing trouble of course if you simply measure your legs or your feet just to measure your shoe size. Yeah, another approach or another invention discovery was Johann Radon, an Austrian researcher who discovered the Radon inversion formula. I don't know if you have something, if you know about this. We will see in the technology or the mathematical backgrounds of back projection. So, in the end this is pretty simple. He says every ray that comes from every angle, if you inverse this process, you can recover the original image. So, that's what inside your body, you scan with X-rays. You see everything that is in the X-rays, projected on a single line as you know from a chest X-ray from your doctor. And if you invert this from many projections, you can recover your image. Yeah, 1936, this was the time where there were a lot of colors, there were the hippies, and there was EMI and the Beatles. And obviously they earned so much money with the Beatles records that they were able to fund an electrical engineer, they were able to do research here just on his EMI deck, this was a simple computer at that time. And he also did some research on the first CT system. So, this is the first CT system with wood and a little bit of steel and an X-ray source and some detector. And this is a reconstructed image, so this is a brain. So, some of you can tell me if you can see anything of this and you can do your diagnosis on it. Something is here maybe, but only those ones who are aware of these images know a little bit better what's inside the images. Yeah, 1974, there were of course, it's Siemens, the first publicly available CT scanner. It's only for the head, the acquisition time of the head was seven minutes. I can show you later numbers, how far we are now. And this is how it looks like from the inside, so here's the X-ray source. This is how a system looks now. Here you can put in the whole body. And if you open up the system now, you have the X-ray source of course, you have the detector, and these are the most expensive components of the system, that's one of the biggest problems. We're talking about 50 to 80,000 euros for these two systems. Taking it from the other side, you need a high voltage, a cooling system, of course you need electronics, and a power transmission. And from 1974, you see these images here, to 1996, you see this image. So of course these are not the same patients, but you can directly see that of course the image quality has improved. And so we are 2018, and we are still at the same image quality. So we have a bit better image quality, but we stayed with a 512 by 512 matrix size, and here we are with 80 to 80 matrix size. It's a matter of memory consumption, because if you double the size, it has to be stored somewhere, and we are currently having real trouble storing all the images that are generated from an X-ray system. So from 1972, we have an acquisition time of the seven minutes, so 300 seconds, and the data is acquired for one rotation, so you have to imagine what happens with the CT scan. So you start a rotation, you rotate around the patient for one image slice, and if you have a so-called spiral, so if you want to scan from head to toe, you have to scan with a spiral scan. So here is no number, because at that time there was no spiral scan available, so you had to rotate, you had to shift, you had to rotate to shift, and so on and so forth from the top to the bottom. In 2015, we rotate at 0.25 milliseconds, we produce data per rotation 270 megabytes, and for a full scan one gigabyte, and the largest file I have seen so far is about 30 gigabytes, and with newer scanners we are coming up to 100 gigabytes. So this is something you have to process, and one of the biggest problems is that if you're doing a CT scan, you sometimes have emergency cases, and you don't want to wait 20 hours until the images are finished, so what is the demand here is that you have something like 50 images reconstructed per second, mostly 100 images per second out of the 30 gigabytes dataset. There is one question, who knows why the 300 seconds could be reduced to the 0.2 seconds? I can show it. The first scanners were a translation rotational system, so you had one X-ray source, one pencil beam, you had to shift the pencil beam, then you rotate, then there is another shift, then there is another shift, and this has evolved to a so-called cone beam system where you could rotate, and you don't have to shift and translate. The only problem is that there were cables, so with one rotation you had to go back, rotate again, and this is where the images with the power transmission comes in. This is a so-called slip ring. With the slip ring, in Germany it's called Schleifring, so it's some sort of a slip where brushes are connected to this metal bearing here, and in that sense you can rotate forever, and you have a continuous power transmission, and you don't have to go back and forth, so you rotate in this so-called spiral, and this is also what enabled the spiral scan, so you can start, you have your spiral and scan over the patient. I don't want to talk about all these numbers. There is of course the matrix size here, so 2018 we shifted to 2024 matrix size. If you go to, let's say, regular CT, I'm not CT, regular chest X-ray scans, they have a matrix size of 4096 to 4096, so much larger. The images, the number of images we produce are around 2,000 images per patient, something like this. So I talked about the image acquisition. This is what I call the pencil beam. So here's the X-ray source, and I talk about this very primitive example because it's easier to explain the other examples. You scan over the patient, here inside there is the brain, and then there is a so-called attenuation, so every tissue, bone, liver has a very different attenuation property, and this attenuation property is measured here in a single line. So you project a two-dimensional plane onto a one-dimensional line for each angle while you rotate around the patient. And this is the so-called synogram, so here you see the process, a little bit old animation, but in the end what you get is projections. So there is a detector that has around 1,000 channels in one direction, and let's say 64 to 128 rows. So this is one projection from one angle, then there is another projection, and if you finish this up, this is the so-called synogram. In the end, it's an X-ray from many projections, regular X-ray from many projections. And now we're coming to this Radon inversion formula with this Austrian scientist. We need to make from this synogram a reconstruction of the image. So in the end what you do is there are two methods, either Fourier-based, which is normally not used, or image-backed projection. So in the end what you do is the process I have shown, taking a projection from one angle from the next angle, is simply inverted mathematically. So you sit on a pixel here, and you look at the projections that are contributing to this pixel, and you just reverse the process, just mathematical imagery construction. Of course this is time-consuming, I have some numbers here, but I just showed this example, so this is how it works. This is the original object, just a cylinder. This is one projection, this is how a projection in 1D looks like. You do it many times, three projections, and then you see finally you have some reconstructed image, and now you see that there is something wrong. You have a cone, you have not a cylinder. And this is what is called filtered-backed projection. So something goes wrong in this regular acquisition process. And you need to filter this one-dimensional projection of a single angle. For the next angle you also need to do it. So there is a so-called filter kernel that you apply to this data. It looks like this, there are many kernels. And if you do this, then it should go. For many projections, in the end for 128 projections, for very primitive objects it's pretty easy, but if you have a very complex object like a body, it's pretty complex and 128 projections are not enough. We're talking about 4000 projections for each angle. There is one problem though. You have to decide if you have a sharp kernel or if you have a smooth kernel. And the kernel controls the image impression. So if you have a sharp kernel, you have very fine details. The only problem is you have very high noise. And it always depends on your indication you want to see. Do you want to see small fissures on bones or do you want to see tumors? And this is where you have to decide between the kernels. So that's what the kernel does. It controls sharpness, noise and the edges. These are medical images, so we are using mostly phantoms in our department because it's much easier to see the effects, not on a very complex human tissue, but to have some simulated phantoms where we check what's going on. So this is a regular head phantom where you have this bone here. This is a very complex bone with very fine structures. And you also see low attenuating objects. And this is some sort of where you want to see if your algorithms work good for soft tissue, for low attenuating things or for high attenuating things. So this is one approach, how a regular reconstruction works. And you see that there are some problems in the reconstruction. There is a so-called beam hardening effect. Here you see some sampling effects. Obviously, if you enlarge this, you can see in the volume stack from many sides what the result of the reconstruction was. And there is another approach, and this is iterative reconstruction. Iterative reconstruction is a process where you revert the process of reconstruction again. So you do a technical forward projection, which means it's a technical process of acquisition. So first of all, you have a reconstruction. It's a 3D volume stack. You made some errors with your algorithms because they're not perfect. Then you try to do some so-called forward projection. So you invert the process. It's a little bit complicated. The process of reconstruction again. And then there is some sort of an image. So this is some sort of a loop where you try to, with a few cycles, you try to recover the image quality and put also some prior knowledge in it. Because you know at the edges there is always a beam hardening, so you put prior knowledge in it just to recover these beam hardening effects. And of course you can imagine that this all costs a lot of processing power. For a regular system, we have around 700 gigabytes of main memory in the system and about four Volta GPUs that are doing this process. And as I said, 30 gigabytes and 100 images per second is a very highly demanding process on the developers that are doing this. We are in the field of research. That's the advantage. We don't need to be that fast. But still there is a high demand on having images quickly. So I just want to show the effects of iterative reconstruction. So this is a regular reconstruction. There is some sort of filtering on it. Here is again the first version of iterative reconstruction. And this is something where the full iterative reconstruction was applied. And as I said, we are talking about x-rays. And of course there is a so-called Alara principle as low as reasonably achievable. So you should, of course, you have to apply x-rays to see something. But in the end, you should only apply as much as you need. And if the algorithms are good enough, then you can reduce the dose further or keep the same dose, but it can improve image quality. Two, for example, detect attenuating regions here in the bladder. So this is a thorac scan and abdominal scan. This is a pretty large area. So this is from here to here somehow. And you can also see that there is some noise reduction. So you're going from Houndsfield units, by the way, was named by the inventor of the system. It's a unit that was introduced. It's a unit less unit. So there's nothing. It only is some relative value to water. And here are also some improvements because you want to see very fine details. This is what is shown in the regular weighted filter back projection. And this is some IR process where you can much better see the line pairs and the small lines of the object. Also here, this is a coronary stand that if you have some occlusion on your heart vessels, then you apply metal stands on it to open up the vessel again. And it's some sort of a grid in the cage which is inserted into the heart. And it's helpful to see what it's inside, to see the stand struts. There are so-called stand struts because it's possible that they clot again. So they get closer again. And this is necessary to see inside this stand. I will talk about the techniques behind. I can only cut pieces. It's a pretty large area where I can only show a few details we experience with the.NET framework and with the C-sharp. So in general, we are talking about volumes. So our volumes, about 2,000 slices, let's say. 512 by 512, stored in float byte, depending on what's inside or in textures. And you need to store it somehow. So this is the small diagram where we have some class which is a volume. And we have many types of volume. So because we also want to support, for example, MATLAB volumes, so we need to have some wrappers around these volumes. So this is the, let's say, the data collection that manages the slices, the volume and image slices. And there is also, for the 30 gigabytes, the projection data. It's somehow identical. It's also sorted in slices. And you also need to store it. But the problem is, we don't have the size of 700 gigabytes, I told you before, in our production system, we only have 32. So what we need to do is, we need to invent our own memory management, other than the regular garbage collector, because the garbage collector only cleans up. But in our case, it's something we need to access to the data more often, so that it shouldn't get disposed. And we need to sometimes access it sequentially and sometimes randomly. It's depending on the task we're working on. So what we have is, each of these slices and the projections are derived from a so-called mapable object. And the background thread is running in the background and continuously checking which object. There is a reference counter. There is a time counter, how often the object was used, when it was used last. And depending on this, these automatic process checks, is there enough space free? And then dumps data that is not used for a long time, just to the hard disk. And when we started, memory map files were not available. There was a problem, so we could have used memory map files. But the problem is, memory map files, so the last two years we have tested it, memory map files are currently very slow. So there are two things where we can access memory. Has any of you used memory map files? Okay. So just to explain, memory map files is some mechanism where the virtual filesist or the virtual memory manager from Windows takes over and says, okay, you don't need this anymore, I just dump it. And of course it's stupid to do it ourselves, but the memory manager was, first of all, a little bit slow because of this line here. And the other thing is that when we access the data, the data normally from the system is stored in a compressed way. So it's not only reading, it's also unzipping more or less and doing some data compression, data formatting on it. So just simply writing and loading is not good enough for us. So in the end, there were two things, why we couldn't use it, is first of all because of this line. It's very slow. Obviously, no one has ever invested more on it. That is one thing. And there comes another thing. So this is a little bit of a new topic. How many of you are using arrays, large arrays in your environment? So do you use a lot of arrays or just I just do it once and then it's over? So that's the interesting thing because we were talking about the benchmark.net. And since I started with this about 12 years ago, I did a lot of benchmarks. And this is the regular addition. So I show these arrays. So this is just adding two floats. I just want to add two floats. This is conventionally done. Taking the a.length here, oh, that's wrong, 422. Then if I extract the a.length to just an external variable, I get this number. I always thought it shouldn't be the case because I thought the chitter is optimized, but I was very surprised. If I went to pointer arithmetics, just make it unsafe and do a pointer arithmetics, I come to this area. Here the external n doesn't really matter anymore. I was also very surprised. And of course I tried the span and I know that the span is not very functional or is not so fast as in.NET Core. Here the performance dropped. So of course I was hoping for.NET Core that this will improve, but I was also using the SMID and this was the best performance in that case. When I do this here on.NET Core, I was very surprised and I hope someone proves me wrong that something is wrong here because all the values were much lower than on this side. And as we heard on the other side that.NET Core is fast, I also read these two blog posts and I was very surprised that I get the wrong numbers here. There's another thing. What if I want to do it with generics? Have you ever tried this? It does not work. C-SHAF does not support mathematical operations on generics as C++ do. This is where a lot of my colleagues came and say, why is it not possible? And there are a lot of cases where we had to write a very large math library because if you're doing area math, sometimes you want to have a constant added to array. And of course you have commutative operations on arrays. It's not always that you have only one function. So I think we have around for all these mathematical operations like cosine, sine and so on, I think around 500 functions. And one problem is that generics are too weak. So what we did is we wrote our own IL code because IL is able to add. IL is obviously type agnostic and is able to add two values. And if you do it correctly, you get the same speed as a regular addition writing it in regular source code. There was also one thing. If you look at this mathematical expression, these are volumes, 512, 512,000. And if you want to have these operations, this is one operation, for example, you use for the iterative reconstruction. So 0.3, a plus b and so on. So here you see what's happening. For the a plus b, you have one gigabyte of memory. Then you add another gigabyte of memory because all these expressions add up. And in the end, for this simple expression, I allocated 5 gigabytes more than I needed. What we did is we resolved the expression. So it's a little bit, I was hoping with the Roslin we could do a little bit more. Just write the expression because when you have these expressions, you want to see the real mathematical expression and not the long expression of the function calls. So it's helpful for debugging and also helpful for researchers when they look in the code. So what we did is you apply here a string and some parser gets into this and recompiles your code and builds only one for loop around these thousand slices and extracts the math again, this expression into just one expression call. So first of all, you have a factor of 4 of performance gain and only one gigabyte of allocation if you do this. And there's one thing, but I'm not quite good in this Roslin support right now. What would be really helpful would be if you could write expressions and before the real compilation process, some parser that we write goes into this and extracts this line and inserts the code directly. So that's currently not really possible. There's also custom arrays which is a little bit of a problem. The same style because there is no numeric type which we can apply on the data. And I don't want to go into this, I just show the cuter things. Two minutes, alright. Has anyone of you used cuter? Alright, in C sharp or in C++? Yeah. The problem is that for the C sharp environment, there is no real access. So you have these cuter driver API you can access with Windows API calls. And of course for the back projection, just to give you a few numbers, for the back projection I showed here, we have around 134 tera operations per image for the back projection. So what we did is we used cuter. You can write cuter in.NET as regular cuter code of course in C style or C++ style. This is how the cuter code looks like. This is here inserted into the solution. And then if you simply say save code, there is a generator that generates the PTX code, the NVCC which goes over the code, that is the real machine code from cuter. Then there is a designer code that translates the code via C lang, we use C lang for the AST. And it generates a wrapper for C sharp. So this is the wrapper, it's very simple, it's just accessing a three dimensional texture. And this would be the code that you normally have to write. That also is what you normally don't see if you use it in C++. But this is what C++ does for you. Sorry. C++ does for you. Just writing all these wrappers around your code and then you can execute cuter code simply from C sharp. So this is it. That's a lot of things, just a few pieces. And I would be also interested in the net core performance test. If someone ever has better experiences than me. Thank you.
also talk about challenges with the.NET framework or also with C sharp. And for performance reason library, CUDA, and we are also investigating or this finally led very fast to technical inventions like what they scanned is, they talking about X-ray radiation. This is causing 80,000 euros for these two systems. Taking it quality has improved. And so we are 2018, and we are still at the same image quality. So we have a consumption, because if you double the size, it we have an acquisition time of the seven minutes, 100 images per second out of the 30 gigabytes 300 seconds could be reduced to the 0.2 seconds? not CT, regular chest X-ray scans, they have a this. So I talked about the image acquisition. while you rotate around the patient. And this is this is the so-called synogram. In the end, it's you do is there are two methods, either example, so this is how it works. This is the very primitive objects it's pretty easy, but if These are medical images, so we are using mostly to see if your algorithms work good for soft tissue, for low attenuating things or for high there are some problems in the reconstruction. Iterative reconstruction is a process where you complicated. The process of reconstruction again. of processing power. For a regular system, we images per second is a very highly demanding process on the developers that are doing this. We iterative reconstruction. So this is a regular applied. And as I said, we are talking about further or keep the same dose, but it can improve image quality. Two, for example, detect fine details. This is what is shown in the closer again. And this is necessary to see inside volume and image slices. And there is also, for to invent our own memory management, other than data more often, so that it shouldn't get we have is, each of these slices and the new topic. How many of you are using arrays, And since I started with this about 12 years ago, course I was hoping for.NET Core that this will improve, but I was also using the SMID and this researchers when they look in the code. So what gets into this and recompiles your code and is that for the C sharp environment, there is no So this is the wrapper, it's very simple, it's this would be the code that you normally have to