Selecting a Finite Element Analysis Backend for Exascale Fusion Reactor Simulations
Dubas, Aleksander J.
Accelerating the development of fusion energy requires large scale simulations on cutting edge supercomputing resources. Great hardware is only half the challenge and the software must be scalable to match. This talk presents an objective approach to selecting a suitable back end to fusion simulations. The UKAEA's mission is to develop commercially viable fusion energy. Current fusion technology is yet to break even on power out compared to power in, thus designs for future reactors, which necessarily must exceed break even, carry a great amount of uncertainty. With cost estimates of a first of a kind fusion reactor in the order of billions of euros, any design flaw making it through to the construction stage will be an expensive mistake. Thankfully, software can help. By simulating a fusion reactor prior to construction, the design can be tested and refined for a considerably lower cost. However, covering all the necessary scales and physics for a digital twin of a fusion reactor requires computational resources at the exascale. In this work, a number of potential finite element backends for a multiphysics reactor simulation are evaluated. The sheer scale makes open source a practical necessity and scalability is the primary performance metric. From the plethora of open source finite element libraries, the most promising are selected and compared against a number of objective, unbiased criteria. None of the tested back ends scored perfectly in all criteria, so a method and rationale for weighting the results to select the best one for the purpose is presented. The aspects of open source projects that are important to high performance computing are highlighted.
 Okay, we're ready to start our next talk from Alexander on finite element analysis. So please welcome Alexander. Good afternoon. My name is Alexander Dubas and I'm from the United Kingdom Atomic Energy Authority and I'm going to talk on selecting a finite element analysis back end for exascale fusion reactor simulations. First of all, if you're not familiar, I'm going to give you a brief introduction to fusion and the challenges behind it. Simply put, we take two isotopes of hydrogen, smash them together and we get some helium, a neutron and lots of energy out. So we take the two isotopes we use typically are deuterium and tritium. Deuterium is a naturally occurring isotope that's in everybody here in all the seawater and it's thoroughly abundant on earth. Tritium is less abundant but with a fusion reactor as well designed you can breed your very own fuel inside your reactor. The energy distribution mostly comes out in this neutron here and I'll come back to that later. It's about 80% of the energy. So how do you do fusion? Well, you need to generate a plasma first to get your nuclei free of their electron shielding and we do this by magnetically confining it in a set of magnetic fields which have a set of coils, an inner coil, poloidal and toroidal field coils which both confine and shape the plasma. In most fusion reactor designs these are superconductors. It's debatable as to whether the high temperature superconductors will work or low temperature superconductors. But currently in progress being built in the south of France is Eta. This one does use superconducting magnets and you can see the plasma chamber in the middle of this picture here. This is an international collaboration and its aim is to ultimately prove that we can get more energy out of fusion than we put in. It roughly costs and it's not finished yet so there are no actual figures but it's in the order of £20 billion. This is rather expensive so it would be nice if we could simulate this inside a computer and know it's going to work before we actually invest in a large building and spending a lot of money. So how do we simulate it? Well, it turns out it's actually quite challenging. First of all we need to capture the energy from these fast neutrons so we need neutron assimilations and because it's magnetically combined it's electromagnetic simulations. Obviously the magnets are going to maybe induce some structural deflections as well as the weight of the machine itself so that's going to deflect that and that's going to change your neutron assimilations. And you want generally to get the heat out of your system so we need thermal simulations and then we don't want all the heat to stay in one place so we use fluid simulations to extract it and so this is a highly connected multi-physics problem. So there are two ways of approaching the multi-physics issue. One is a single tightly coupled simulation so you have one single executable solving a large linear system with all the physics involved. Now this ensures that you're capturing all your strongly coupled phenomena but it's numerically challenging to build this and some say it's not even possible. The alternative is you take your best and class solvers for each of your independent physics and you couple them with a third party library. This is an easier method so you'll guarantee the best performance for each individual class of physics and it's easy to decouple the ones you don't need for your current simulation but their temporal accuracy may suffer and you might not capture strongly coupled physics if you get it wrong. So the approach we're focusing on in our current work is looking at a single tightly coupled simulation using finite elements and so we need to select a back end to build our finite elements simulation on. Now we need to select this finite element back end in an objective fashion rather than going oh I know how to use this one let's just use that because it's my favourite and my mate works on it so I'm presenting today our objective method which includes a lot of subjectivity but we'll get to that. So how do you define a set of objective criteria to benchmark your finite element libraries? Well for one to simulate a fusion reactor is going to require somewhere in the order of an exascale machine so it's an exascale simulation and it has to be designed as a parallel code from the outset. You're not going to reach the performance required without a parallel first approach and therefore parallel first is our first criteria. The second criterion so we need to be able to run on whatever computing resources we have available which could be anywhere in the world especially with the cloud including locations without internet access on your batch nodes could be any number of processes so you need to have one a license that will scale to any number of processes that you want to use and ideally you want to be able to permit extension and modification so naturally this would lead you to an open source license except not necessarily theoretically you could license software that is both permissive and proprietary but no one's that brave. So another problem there are no exascale machines yet so what does it look like? It could be a multitude of highly vectorized CPUs, it could be a mixture of open MP and MPI so shared and distributed memory paradigms, it could be solely GPU or heavily GPU so one of the criteria is to have a very portable library that supports a variety of architectures. One other criteria also is that we want to be able to add to our back end if we need to and so it has to be open to external contribution so that's permission and also we have to be able to add to it easily and not waste time so that generally requires that we follow good software engineering practices. Finally our main criterion is it has to be well supported including a good community around it, forums where questions are answered, mailing lists or IRC rooms, workshops and tutorials and good documentation both for developers and for users which are slightly different. Now there are also some implicit criteria, one is that it's a compiled language that we need, interpreted languages are getting very good but they still incur an overhead. We did benchmark Phoenix against its C++ back end Dolphin and performance was nearly identical apart from an extra memory overhead of the Python interpreter and when you scale it up to a million processes you're then using 80 terabytes of memory just in Python processes. The other implicit criteria is it's got to be production ready so to speak, it's got to have a stable API or at least the developers need to express it as confidence so we set a criteria of having a minimum version number of one which is somewhat arbitrary but it's a way of ruling out libraries that aren't ready for use yet. The other criteria to be actively supported has to be actively developed and so we excluded anything with no development in the past six months. You'd be surprised how many libraries there are that are effectively in the finite element graveyard. So our initial survey found 35 potential candidate libraries and we went through and eliminated those that weren't parallel first, the ones that were in early development or just as Alpha or Beta that were poorly supported, inextensible both from permission or from poor software practices and those that were abandoned. And we came up with a short list of deal two which you might have seen the talk yesterday, June Dolphin, LibMesh, MFEM, Moose and Necta++. Now obviously some subjective criteria were included and if you did this same process today you would get a slightly different set of results as more libraries have matured. So how have we measured the performance? Well I've levelled up from solving the Laplace equation on a unit square and I'm solving the Poisson equation on a unit cube and to also measure when there's a little bit more CPU work the time dependent equation was also measured. The mesh was a unit cube generated in G mesh as having an open source measure helps. But this led to one slight problem. We couldn't get the mesh into deal two which is rather unfortunate because it seems like a very good library and if you were in a talk yesterday the claims that it scales and performs better than commercial packages is corroborated by our results but we can't present them in comparison because the same mesh wasn't used. We also measured the memory usage because that's quite important when you have a limited amount of memory and at this stage we had to eliminate Necta++ which uses rather lots of memory partly because it didn't solve with piecewise linear elements so we had to use quadratic elements and thus we couldn't get the memory down to a reasonable level. Onto the scaling. We measured performance scaling up to a thousand cores. Obviously strong scaling it dropped off as our problem size here and this seems to suggest that Dolphin is the best library with Moose in second place and the rest below it. However looking into this a slightly more in depth it turns out a lot of time was spent using the mesh importer and we're not trying to measure how good the mesh importer is. We want to know how well it solves so we looked at the solver scaling instead and so this gave a different set of results with Dune appearing to scale very well, Mfem and Dolphin being roughly on a par and LibMesh not scaling so great. Moose isn't included in these results because we couldn't get the solver directly comparable solver timings out of Moose. However it's not just about scalability it's also about how quick it is so Dune seemed to scale very well but it took too long and it never reached the same speed level so Mfem and LibMesh were both really fast. Dolphin and Moose scaled well but were a bit slower maybe they overtake further down. And so that's the performance side of things. As Carol mentions I would say Mfem had a very very nice build process. It took literally half an hour to set it up on whichever machine I wanted to and Moose has a very nice multi-physics coupling designed to allow arbitrary physics coupling. So which one's the best? There wasn't a clear winner. The various libraries seem to sit on a parito front between performance and usability. So to pick a winner some advanced maths of essentially waiting their performance and their quality to try and give an objective ranking to their assessment and we scored them on their quality on a basis of their installation. So how easy it was to build both on this laptop and on our supercomputer how usable it was. So how easy it was for a user to get the simulations that they want. Their documentation which is very important especially when you're trying to learn and evaluate a finite element library in a week. And we eventually came up with a few answers obviously the best library for what you get depends how you weight these equations. So if you look at pure performance for example if you're going to exascale performance really matters MFEM would be the best one according to our results and when your simulations cost about a thousand pounds a minute I think roughly at the exascale your developer time becomes cheap again. And having well documented easy to use portable build processes obviously very important. User interaction was important and being able to support common open formats so those that couldn't import the Gmesh one files were excluded and good tutorials examples and source comments. I believe that's all I have to say. Any questions. So you discounted anything that's not compiled like from the start without I think much consideration as far as I can tell. And I wonder why that wasn't if you had actually done any benchmarking on which one anything that had compiled that was using Python for instance like anything that was compiled. So we did do some benchmarking. So you asked we we discounted anything with an interpreted language like Python and asked if did we do any benchmarking and we did. So we did measure Fenix against Dolphin and in terms of scalability and performance it was almost identical because it calls the C++ back end but there is this the memory overhead is the main difference. So this you used it. This is like a first order mesh. Yes. Yeah. Did you try anything with I don't know how many of the solvers support higher order but did you try any higher order solves because that's really where MFM was designed to shine is for like curved meshes and stuff that would use your GPU better. We didn't try higher order meshes in this because implementing it in the various different libraries becomes a lot more a lot more of an involved process which is also why we didn't look at more involved equations for every single library. But yes I agree MFM does shine there. It's within the same line. I mean the thing you want to simulate is from a physics point of view extremely challenging as you said and as a test case you take two very very very simple things. Do you think that actually is even representative for the thing you actually want to simulate because they might shine very well in these simple problems and completely mislead we fail in actually want to solve. That is correct and it is a potential flaw. However without implementing a full multi physics solver in every single back end we chose a standard benchmark that we could definitely implement in every library. Do you actually even think we have codes at this point that can actually do that at excess scale level. No because I would think you actually still people are developing these things and as far as I know it doesn't get existed. This is the start of a large package of work to develop the simulations we need. So in terms of multi physics we sort of nailed by physics where you have two coupled physics and we're starting on multi physics so three coupled physics domains are sort of coming in there's nothing I'm aware of that does everything we need to. And just as curiosity what exactly do you want to simulate of the whole interesting you want to simulate the entire reactor you have specific research questions. Ultimately yes we would like to simulate an entire reactor but on the way there there are many different components such as the diverter and blankets where you have a lot of physics interacting but you don't have to scale up to the entire reactor size and so you can use a smaller computer. Preferably before it does its first run. So here we're dealing with multi physics why are we considering finite elements. In terms of building a single solver that can solve everything I think it's probably the best discretization. I don't have any evidence to back that up necessarily it's conjecture I agree but it seems the best way to be able to implement all the physics that you need. Whereas I would agree if you were to do say a multiple coupled solvers then you would look at other types of simulation. Hi thank you for the very nice talk. Is the problem you're trying to solve sensitive to numerical accuracy or instability and did you try to assess those libraries on this problem? We looked at so part of the reason for using the Poisson and time dependent equations is that we could then have a known variable put in and so we have a definite solution so we can compare the error there but in terms of further assessment that's as far as we went. Thank you. Since you're really valuing the long term sustainability of your project why only looking into making everything in one solver and not go with the partitioned way for example? Because the performance is not necessarily the answer here. That seemed like two questions which would you like me to answer? Let's start from the first one. Can you repeat the first one? Yes so why really trying to pick one of the, well you have so different applications why really trying to pick one of the libraries for everything and not follow the partitioned way that then you could have sustainability in the long term. So the work I'm presenting here is based on looking at that one approach we are also looking into the coupled approach so we are covering both bases there. So for the solver part did you use just, so you used one solver for everything but I mean like MFEM has lots of solver backends right? You can actually plug in for all those different frameworks, different solvers that they are really designed to work with. So do you think your solver results would be better if you did that? Potentially so this is one point that I'd possibly like to make in that we've tended to follow what was recommended in tutorials and available documentation for each package so we picked which solver is recommended for it which is more a question of does the library set sensible defaults or advise sensible things for performance? Well I was going to ask the same question but I have actually another one and I would maybe just quickly say that like I think there are many problems where like you just cannot like the library cannot set the defaults because the problem is very specific and it requires still more knowledge but my actual question is like I want to suggest to you again to maybe revaluate your position on non-compile languages because I, are you familiar with FireDrake? Sorry? Are you familiar with FireDrake? I've heard of FireDrake yes. So I'm the original author of FireDrake and in my PhD I actually did, so it's an API compatible implementation of Dolphin or Phoenix and we generate code for the entire solver at runtime and I benchmarked up to 1536 cores in my PhD thesis which was like six years ago and I was able to beat every other package. That's about plain? Well every other package, well I was sorry I was able to beat the Dolphin also the C++ version in terms of performance and like I wasn't, we weren't running into memory issues so like just as another notch maybe. The memory issues is a scaling thing so if you're only running single core with only one interpreter it's a tiny amount of memory so it only really becomes a scaling problem when you have multiple processes. Again this comes back to we don't know what the extra scale looks like if it's fewer processes with lots of GPU dispatch then perhaps the memory overhead of running an interpreter isn't an issue as much as it would be if you run fully on multiple processes on CPUs. Thanks. Sorry we're out of time, we'll have to meet the speaker after. Thank you very much, let's thank the speaker again.
 Okay, we're ready to start our next talk from Alexander on finite element analysis.  So please welcome Alexander.  Good afternoon.  My name is Alexander Dubas and I'm from the United Kingdom Atomic Energy Authority and  I'm going to talk on selecting a finite element analysis back end for exascale fusion reactor  simulations.  First of all, if you're not familiar, I'm going to give you a brief introduction to  fusion and the challenges behind it.  Simply put, we take two isotopes of hydrogen, smash them together and we get some helium,  a neutron and lots of energy out.  So we take the two isotopes we use typically are deuterium and tritium.  Deuterium is a naturally occurring isotope that's in everybody here in all the seawater  and it's thoroughly abundant on earth.  Tritium is less abundant but with a fusion reactor as well designed you can breed your  very own fuel inside your reactor.  The energy distribution mostly comes out in this neutron here and I'll come back to  that later.  It's about 80% of the energy.  So how do you do fusion?  Well, you need to generate a plasma first to get your nuclei free of their electron  shielding and we do this by magnetically confining it in a set of magnetic fields which have  a set of coils, an inner coil, poloidal and toroidal field coils which both confine and  shape the plasma.  In most fusion reactor designs these are superconductors.  It's debatable as to whether the high temperature superconductors will work or low temperature  superconductors.  But currently in progress being built in the south of France is Eta.  This one does use superconducting magnets and you can see the plasma chamber in the  middle of this picture here.  This is an international collaboration and its aim is to ultimately prove that we can  get more energy out of fusion than we put in.  It roughly costs and it's not finished yet so there are no actual figures but it's in  the order of £20 billion.  This is rather expensive so it would be nice if we could simulate this inside a computer  and know it's going to work before we actually invest in a large building and spending a  lot of money.  So how do we simulate it?  Well, it turns out it's actually quite challenging.  First of all we need to capture the energy from these fast neutrons so we need neutron  assimilations and because it's magnetically combined it's electromagnetic simulations.  Obviously the magnets are going to maybe induce some structural deflections as well  as the weight of the machine itself so that's going to deflect that and that's going to  change your neutron assimilations.  And you want generally to get the heat out of your system so we need thermal simulations  and then we don't want all the heat to stay in one place so we use fluid simulations to  extract it and so this is a highly connected multi-physics problem.  So there are two ways of approaching the multi-physics issue.  One is a single tightly coupled simulation so you have one single executable solving  a large linear system with all the physics involved.  Now this ensures that you're capturing all your strongly coupled phenomena but it's numerically  challenging to build this and some say it's not even possible.  The alternative is you take your best and class solvers for each of your independent  physics and you couple them with a third party library.  This is an easier method so you'll guarantee the best performance for each individual class  of physics and it's easy to decouple the ones you don't need for your current simulation  but their temporal accuracy may suffer and you might not capture strongly coupled physics  if you get it wrong.  So the approach we're focusing on in our current work is looking at a single tightly  coupled simulation using finite elements and so we need to select a back end to build our  finite elements simulation on.  Now we need to select this finite element back end in an objective fashion rather than  going oh I know how to use this one let's just use that because it's my favourite and  my mate works on it so I'm presenting today our objective method which includes a lot  of subjectivity but we'll get to that.  So how do you define a set of objective criteria to benchmark your finite element libraries?  Well for one to simulate a fusion reactor is going to require somewhere in the order  of an exascale machine so it's an exascale simulation and it has to be designed as a  parallel code from the outset.  You're not going to reach the performance required without a parallel first approach  and therefore parallel first is our first criteria.  The second criterion so we need to be able to run on whatever computing resources we  have available which could be anywhere in the world especially with the cloud including  locations without internet access on your batch nodes could be any number of processes  so you need to have one a license that will scale to any number of processes that you  want to use and ideally you want to be able to permit extension and modification so naturally  this would lead you to an open source license except not necessarily theoretically you could  license software that is both permissive and proprietary but no one's that brave.  So another problem there are no exascale machines yet so what does it look like?  It could be a multitude of highly vectorized CPUs, it could be a mixture of open MP and  MPI so shared and distributed memory paradigms, it could be solely GPU or heavily GPU so one  of the criteria is to have a very portable library that supports a variety of architectures.  One other criteria also is that we want to be able to add to our back end if we need  to and so it has to be open to external contribution so that's permission and also we have to be  able to add to it easily and not waste time so that generally requires that we follow  good software engineering practices.  Finally our main criterion is it has to be well supported including a good community  around it, forums where questions are answered, mailing lists or IRC rooms, workshops and  tutorials and good documentation both for developers and for users which are slightly  different.  Now there are also some implicit criteria, one is that it's a compiled language that  we need, interpreted languages are getting very good but they still incur an overhead.  We did benchmark Phoenix against its C++ back end Dolphin and performance was nearly  identical apart from an extra memory overhead of the Python interpreter and when you scale  it up to a million processes you're then using 80 terabytes of memory just in Python processes.  The other implicit criteria is it's got to be production ready so to speak, it's got  to have a stable API or at least the developers need to express it as confidence so we set  a criteria of having a minimum version number of one which is somewhat arbitrary but it's  a way of ruling out libraries that aren't ready for use yet.  The other criteria to be actively supported has to be actively developed and so we excluded  anything with no development in the past six months.  You'd be surprised how many libraries there are that are effectively in the finite element  graveyard.  So our initial survey found 35 potential candidate libraries and we went through and eliminated  those that weren't parallel first, the ones that were in early development or just as  Alpha or Beta that were poorly supported, inextensible both from permission or from  poor software practices and those that were abandoned.  And we came up with a short list of deal two which you might have seen the talk yesterday,  June Dolphin, LibMesh, MFEM, Moose and Necta++.  Now obviously some subjective criteria were included and if you did this same process  today you would get a slightly different set of results as more libraries have matured.  So how have we measured the performance?  Well I've levelled up from solving the Laplace equation on a unit square and I'm solving  the Poisson equation on a unit cube and to also measure when there's a little bit more  CPU work the time dependent equation was also measured.  The mesh was a unit cube generated in G mesh as having an open source measure helps.  But this led to one slight problem.  We couldn't get the mesh into deal two which is rather unfortunate because it seems like  a very good library and if you were in a talk yesterday the claims that it scales and performs  better than commercial packages is corroborated by our results but we can't present them  in comparison because the same mesh wasn't used.  We also measured the memory usage because that's quite important when you have a limited  amount of memory and at this stage we had to eliminate Necta++ which uses rather lots  of memory partly because it didn't solve with piecewise linear elements so we had to use  quadratic elements and thus we couldn't get the memory down to a reasonable level.  Onto the scaling.  We measured performance scaling up to a thousand cores.  Obviously strong scaling it dropped off as our problem size here and this seems to suggest  that Dolphin is the best library with Moose in second place and the rest below it.  However looking into this a slightly more in depth it turns out a lot of time was spent  using the mesh importer and we're not trying to measure how good the mesh importer is.  We want to know how well it solves so we looked at the solver scaling instead and so this  gave a different set of results with Dune appearing to scale very well, Mfem and Dolphin  being roughly on a par and LibMesh not scaling so great.  Moose isn't included in these results because we couldn't get the solver directly comparable  solver timings out of Moose.  However it's not just about scalability it's also about how quick it is so Dune seemed  to scale very well but it took too long and it never reached the same speed level so Mfem  and LibMesh were both really fast.  Dolphin and Moose scaled well but were a bit slower maybe they overtake further down.  And so that's the performance side of things.  As Carol mentions I would say Mfem had a very very nice build process.  It took literally half an hour to set it up on whichever machine I wanted to and Moose  has a very nice multi-physics coupling designed to allow arbitrary physics coupling.  So which one's the best?  There wasn't a clear winner.  The various libraries seem to sit on a parito front between performance and usability.  So to pick a winner some advanced maths of essentially waiting their performance and  their quality to try and give an objective ranking to their assessment and we scored  them on their quality on a basis of their installation.  So how easy it was to build both on this laptop and on our supercomputer how usable it was.  So how easy it was for a user to get the simulations that they want.  Their documentation which is very important especially when you're trying to learn and  evaluate a finite element library in a week.  And we eventually came up with a few answers obviously the best library for what you get  depends how you weight these equations.  So if you look at pure performance for example if you're going to exascale performance really  matters MFEM would be the best one according to our results and when your simulations cost  about a thousand pounds a minute I think roughly at the exascale your developer time becomes  cheap again.  And having well documented easy to use portable build processes obviously very important.  User interaction was important and being able to support common open formats so those that  couldn't import the Gmesh one files were excluded and good tutorials examples and source comments.  I believe that's all I have to say.  Any questions.  So you discounted anything that's not compiled like from the start without I think much consideration  as far as I can tell.  And I wonder why that wasn't if you had actually done any benchmarking on which one anything  that had compiled that was using Python for instance like anything that was compiled.  So we did do some benchmarking.  So you asked we we discounted anything with an interpreted language like Python and asked  if did we do any benchmarking and we did.  So we did measure Fenix against Dolphin and in terms of scalability and performance it  was almost identical because it calls the C++ back end but there is this the memory overhead  is the main difference.  So this you used it.  This is like a first order mesh.  Yes.  Yeah.  Did you try anything with I don't know how many of the solvers support higher order but  did you try any higher order solves because that's really where MFM was designed to shine  is for like curved meshes and stuff that would use your GPU better.  We didn't try higher order meshes in this because implementing it in the various different  libraries becomes a lot more a lot more of an involved process which is also why we didn't  look at more involved equations for every single library.  But yes I agree MFM does shine there.  It's within the same line.  I mean the thing you want to simulate is from a physics point of view extremely challenging  as you said and as a test case you take two very very very simple things.  Do you think that actually is even representative for the thing you actually want to simulate  because they might shine very well in these simple problems and completely mislead we  fail in actually want to solve.  That is correct and it is a potential flaw.  However without implementing a full multi physics solver in every single back end we  chose a standard benchmark that we could definitely implement in every library.  Do you actually even think we have codes at this point that can actually do that at excess  scale level.  No because I would think you actually still people are developing these things and as  far as I know it doesn't get existed.  This is the start of a large package of work to develop the simulations we need.  So in terms of multi physics we sort of nailed by physics where you have two coupled physics  and we're starting on multi physics so three coupled physics domains are sort of coming  in there's nothing I'm aware of that does everything we need to.  And just as curiosity what exactly do you want to simulate of the whole interesting  you want to simulate the entire reactor you have specific research questions.  Ultimately yes we would like to simulate an entire reactor but on the way there there  are many different components such as the diverter and blankets where you have a lot  of physics interacting but you don't have to scale up to the entire reactor size and  so you can use a smaller computer.  Preferably before it does its first run.  So here we're dealing with multi physics why are we considering finite elements.  In terms of building a single solver that can solve everything I think it's probably  the best discretization.  I don't have any evidence to back that up necessarily it's conjecture I agree but it  seems the best way to be able to implement all the physics that you need.  Whereas I would agree if you were to do say a multiple coupled solvers then you would  look at other types of simulation.  Hi thank you for the very nice talk.  Is the problem you're trying to solve sensitive to numerical accuracy or instability and did  you try to assess those libraries on this problem?  We looked at so part of the reason for using the Poisson and time dependent equations is  that we could then have a known variable put in and so we have a definite solution so we  can compare the error there but in terms of further assessment that's as far as we went.  Thank you.  Since you're really valuing the long term sustainability of your project why only looking  into making everything in one solver and not go with the partitioned way for example?  Because the performance is not necessarily the answer here.  That seemed like two questions which would you like me to answer?  Let's start from the first one.  Can you repeat the first one?  Yes so why really trying to pick one of the, well you have so different applications why  really trying to pick one of the libraries for everything and not follow the partitioned  way that then you could have sustainability in the long term.  So the work I'm presenting here is based on looking at that one approach we are also  looking into the coupled approach so we are covering both bases there.  So for the solver part did you use just, so you used one solver for everything but I  mean like MFEM has lots of solver backends right?  You can actually plug in for all those different frameworks, different solvers that they are  really designed to work with.  So do you think your solver results would be better if you did that?  Potentially so this is one point that I'd possibly like to make in that we've tended  to follow what was recommended in tutorials and available documentation for each package  so we picked which solver is recommended for it which is more a question of does the library  set sensible defaults or advise sensible things for performance?  Well I was going to ask the same question but I have actually another one and I would  maybe just quickly say that like I think there are many problems where like you just cannot  like the library cannot set the defaults because the problem is very specific and it requires  still more knowledge but my actual question is like I want to suggest to you again to  maybe revaluate your position on non-compile languages because I, are you familiar with  FireDrake?  Sorry?  Are you familiar with FireDrake?  I've heard of FireDrake yes.  So I'm the original author of FireDrake and in my PhD I actually did, so it's an API compatible  implementation of Dolphin or Phoenix and we generate code for the entire solver at runtime  and I benchmarked up to 1536 cores in my PhD thesis which was like six years ago and I  was able to beat every other package.  That's about plain?  Well every other package, well I was sorry I was able to beat the Dolphin also the C++  version in terms of performance and like I wasn't, we weren't running into memory issues  so like just as another notch maybe.  The memory issues is a scaling thing so if you're only running single core with only  one interpreter it's a tiny amount of memory so it only really becomes a scaling problem  when you have multiple processes.  Again this comes back to we don't know what the extra scale looks like if it's fewer processes  with lots of GPU dispatch then perhaps the memory overhead of running an interpreter  isn't an issue as much as it would be if you run fully on multiple processes on CPUs.  Thanks.  Sorry we're out of time, we'll have to meet the speaker after.  Thank you very much, let's thank the speaker again.
exascale_fusion_sim_0_167.png exascale_fusion_sim_168_911.png exascale_fusion_sim_912_1079.png exascale_fusion_sim_1080_2339.png exascale_fusion_sim_2340_3395.png exascale_fusion_sim_3396_4691.png exascale_fusion_sim_4692_5975.png exascale_fusion_sim_5976_7535.png exascale_fusion_sim_7536_8267.png exascale_fusion_sim_8268_9083.png exascale_fusion_sim_9084_10475.png exascale_fusion_sim_10476_11375.png exascale_fusion_sim_11376_12083.png exascale_fusion_sim_12084_12707.png exascale_fusion_sim_12708_13691.png exascale_fusion_sim_13692_14975.png exascale_fusion_sim_14976_15671.png exascale_fusion_sim_15672_16367.png exascale_fusion_sim_16368_16871.png exascale_fusion_sim_16872_17231.png exascale_fusion_sim_17232_17363.png exascale_fusion_sim_17364_17423.png exascale_fusion_sim_17424_17471.png exascale_fusion_sim_17472_17591.png exascale_fusion_sim_17592_17663.png exascale_fusion_sim_17664_17795.png exascale_fusion_sim_17796_18467.png exascale_fusion_sim_18468_19331.png exascale_fusion_sim_19332_20015.png exascale_fusion_sim_20016_20711.png exascale_fusion_sim_20712_21347.png exascale_fusion_sim_21348_21731.png exascale_fusion_sim_21732_23219.png exascale_fusion_sim_23220_24599.png exascale_fusion_sim_24600_38159.png exascale_fusion_sim_38160_38281.png
Okay, we're ready to start our next talk from Alexander on finite element analysis. So please welcome Alexander. Good afternoon. My name is Alexander Dubas and I'm from the United Kingdom Atomic Energy Authority and I'm going to talk on selecting a finite element analysis back end for exascale fusion reactor simulations. First of all, if you're not familiar, I'm going to give you a brief introduction to fusion and the challenges behind it. Simply put, we take two isotopes of hydrogen, smash them together and we get some helium, a neutron and lots of energy out. So we take the two isotopes we use typically are deuterium and tritium. Deuterium is a naturally occurring isotope that's in everybody here in all the seawater and it's thoroughly abundant on earth. Tritium is less abundant but with a fusion reactor as well designed you can breed your very own fuel inside your reactor. The energy distribution mostly comes out in this neutron here and I'll come back to that later. It's about 80% of the energy. So how do you do fusion? Well, you need to generate a plasma first to get your nuclei free of their electron shielding and we do this by magnetically confining it in a set of magnetic fields which have a set of coils, an inner coil, poloidal and toroidal field coils which both confine and shape the plasma. In most fusion reactor designs these are superconductors. It's debatable as to whether the high temperature superconductors will work or low temperature superconductors. But currently in progress being built in the south of France is Eta. This one does use superconducting magnets and you can see the plasma chamber in the middle of this picture here. This is an international collaboration and its aim is to ultimately prove that we can get more energy out of fusion than we put in. It roughly costs and it's not finished yet so there are no actual figures but it's in the order of £20 billion. This is rather expensive so it would be nice if we could simulate this inside a computer and know it's going to work before we actually invest in a large building and spending a lot of money. So how do we simulate it? Well, it turns out it's actually quite challenging. First of all we need to capture the energy from these fast neutrons so we need neutron assimilations and because it's magnetically combined it's electromagnetic simulations. Obviously the magnets are going to maybe induce some structural deflections as well as the weight of the machine itself so that's going to deflect that and that's going to change your neutron assimilations. And you want generally to get the heat out of your system so we need thermal simulations and then we don't want all the heat to stay in one place so we use fluid simulations to extract it and so this is a highly connected multi-physics problem. So there are two ways of approaching the multi-physics issue. One is a single tightly coupled simulation so you have one single executable solving a large linear system with all the physics involved. Now this ensures that you're capturing all your strongly coupled phenomena but it's numerically challenging to build this and some say it's not even possible. The alternative is you take your best and class solvers for each of your independent physics and you couple them with a third party library. This is an easier method so you'll guarantee the best performance for each individual class of physics and it's easy to decouple the ones you don't need for your current simulation but their temporal accuracy may suffer and you might not capture strongly coupled physics if you get it wrong. So the approach we're focusing on in our current work is looking at a single tightly coupled simulation using finite elements and so we need to select a back end to build our finite elements simulation on. Now we need to select this finite element back end in an objective fashion rather than going oh I know how to use this one let's just use that because it's my favourite and my mate works on it so I'm presenting today our objective method which includes a lot of subjectivity but we'll get to that. So how do you define a set of objective criteria to benchmark your finite element libraries? Well for one to simulate a fusion reactor is going to require somewhere in the order of an exascale machine so it's an exascale simulation and it has to be designed as a parallel code from the outset. You're not going to reach the performance required without a parallel first approach and therefore parallel first is our first criteria. The second criterion so we need to be able to run on whatever computing resources we have available which could be anywhere in the world especially with the cloud including locations without internet access on your batch nodes could be any number of processes so you need to have one a license that will scale to any number of processes that you want to use and ideally you want to be able to permit extension and modification so naturally this would lead you to an open source license except not necessarily theoretically you could license software that is both permissive and proprietary but no one's that brave. So another problem there are no exascale machines yet so what does it look like? It could be a multitude of highly vectorized CPUs, it could be a mixture of open MP and MPI so shared and distributed memory paradigms, it could be solely GPU or heavily GPU so one of the criteria is to have a very portable library that supports a variety of architectures. One other criteria also is that we want to be able to add to our back end if we need to and so it has to be open to external contribution so that's permission and also we have to be able to add to it easily and not waste time so that generally requires that we follow good software engineering practices. Finally our main criterion is it has to be well supported including a good community around it, forums where questions are answered, mailing lists or IRC rooms, workshops and tutorials and good documentation both for developers and for users which are slightly different. Now there are also some implicit criteria, one is that it's a compiled language that we need, interpreted languages are getting very good but they still incur an overhead. We did benchmark Phoenix against its C++ back end Dolphin and performance was nearly identical apart from an extra memory overhead of the Python interpreter and when you scale it up to a million processes you're then using 80 terabytes of memory just in Python processes. The other implicit criteria is it's got to be production ready so to speak, it's got to have a stable API or at least the developers need to express it as confidence so we set a criteria of having a minimum version number of one which is somewhat arbitrary but it's a way of ruling out libraries that aren't ready for use yet. The other criteria to be actively supported has to be actively developed and so we excluded anything with no development in the past six months. You'd be surprised how many libraries there are that are effectively in the finite element graveyard. So our initial survey found 35 potential candidate libraries and we went through and eliminated those that weren't parallel first, the ones that were in early development or just as Alpha or Beta that were poorly supported, inextensible both from permission or from poor software practices and those that were abandoned. And we came up with a short list of deal two which you might have seen the talk yesterday, June Dolphin, LibMesh, MFEM, Moose and Necta++. Now obviously some subjective criteria were included and if you did this same process today you would get a slightly different set of results as more libraries have matured. So how have we measured the performance? Well I've levelled up from solving the Laplace equation on a unit square and I'm solving the Poisson equation on a unit cube and to also measure when there's a little bit more CPU work the time dependent equation was also measured. The mesh was a unit cube generated in G mesh as having an open source measure helps. But this led to one slight problem. We couldn't get the mesh into deal two which is rather unfortunate because it seems like a very good library and if you were in a talk yesterday the claims that it scales and performs better than commercial packages is corroborated by our results but we can't present them in comparison because the same mesh wasn't used. We also measured the memory usage because that's quite important when you have a limited amount of memory and at this stage we had to eliminate Necta++ which uses rather lots of memory partly because it didn't solve with piecewise linear elements so we had to use quadratic elements and thus we couldn't get the memory down to a reasonable level. Onto the scaling. We measured performance scaling up to a thousand cores. Obviously strong scaling it dropped off as our problem size here and this seems to suggest that Dolphin is the best library with Moose in second place and the rest below it. However looking into this a slightly more in depth it turns out a lot of time was spent using the mesh importer and we're not trying to measure how good the mesh importer is. We want to know how well it solves so we looked at the solver scaling instead and so this gave a different set of results with Dune appearing to scale very well, Mfem and Dolphin being roughly on a par and LibMesh not scaling so great. Moose isn't included in these results because we couldn't get the solver directly comparable solver timings out of Moose. However it's not just about scalability it's also about how quick it is so Dune seemed to scale very well but it took too long and it never reached the same speed level so Mfem and LibMesh were both really fast. Dolphin and Moose scaled well but were a bit slower maybe they overtake further down. And so that's the performance side of things. As Carol mentions I would say Mfem had a very very nice build process. It took literally half an hour to set it up on whichever machine I wanted to and Moose has a very nice multi-physics coupling designed to allow arbitrary physics coupling. So which one's the best? There wasn't a clear winner. The various libraries seem to sit on a parito front between performance and usability. So to pick a winner some advanced maths of essentially waiting their performance and their quality to try and give an objective ranking to their assessment and we scored them on their quality on a basis of their installation. So how easy it was to build both on this laptop and on our supercomputer how usable it was. So how easy it was for a user to get the simulations that they want. Their documentation which is very important especially when you're trying to learn and evaluate a finite element library in a week. And we eventually came up with a few answers obviously the best library for what you get depends how you weight these equations. So if you look at pure performance for example if you're going to exascale performance really matters MFEM would be the best one according to our results and when your simulations cost about a thousand pounds a minute I think roughly at the exascale your developer time becomes cheap again. And having well documented easy to use portable build processes obviously very important. User interaction was important and being able to support common open formats so those that couldn't import the Gmesh one files were excluded and good tutorials examples and source comments. I believe that's all I have to say. Any questions. So you discounted anything that's not compiled like from the start without I think much consideration as far as I can tell. And I wonder why that wasn't if you had actually done any benchmarking on which one anything that had compiled that was using Python for instance like anything that was compiled. So we did do some benchmarking. So you asked we we discounted anything with an interpreted language like Python and asked if did we do any benchmarking and we did. So we did measure Fenix against Dolphin and in terms of scalability and performance it was almost identical because it calls the C++ back end but there is this the memory overhead is the main difference. So this you used it. This is like a first order mesh. Yes. Yeah. Did you try anything with I don't know how many of the solvers support higher order but did you try any higher order solves because that's really where MFM was designed to shine is for like curved meshes and stuff that would use your GPU better. We didn't try higher order meshes in this because implementing it in the various different libraries becomes a lot more a lot more of an involved process which is also why we didn't look at more involved equations for every single library. But yes I agree MFM does shine there. It's within the same line. I mean the thing you want to simulate is from a physics point of view extremely challenging as you said and as a test case you take two very very very simple things. Do you think that actually is even representative for the thing you actually want to simulate because they might shine very well in these simple problems and completely mislead we fail in actually want to solve. That is correct and it is a potential flaw. However without implementing a full multi physics solver in every single back end we chose a standard benchmark that we could definitely implement in every library. Do you actually even think we have codes at this point that can actually do that at excess scale level. No because I would think you actually still people are developing these things and as far as I know it doesn't get existed. This is the start of a large package of work to develop the simulations we need. So in terms of multi physics we sort of nailed by physics where you have two coupled physics and we're starting on multi physics so three coupled physics domains are sort of coming in there's nothing I'm aware of that does everything we need to. And just as curiosity what exactly do you want to simulate of the whole interesting you want to simulate the entire reactor you have specific research questions. Ultimately yes we would like to simulate an entire reactor but on the way there there are many different components such as the diverter and blankets where you have a lot of physics interacting but you don't have to scale up to the entire reactor size and so you can use a smaller computer. Preferably before it does its first run. So here we're dealing with multi physics why are we considering finite elements. In terms of building a single solver that can solve everything I think it's probably the best discretization. I don't have any evidence to back that up necessarily it's conjecture I agree but it seems the best way to be able to implement all the physics that you need. Whereas I would agree if you were to do say a multiple coupled solvers then you would look at other types of simulation. Hi thank you for the very nice talk. Is the problem you're trying to solve sensitive to numerical accuracy or instability and did you try to assess those libraries on this problem? We looked at so part of the reason for using the Poisson and time dependent equations is that we could then have a known variable put in and so we have a definite solution so we can compare the error there but in terms of further assessment that's as far as we went. Thank you. Since you're really valuing the long term sustainability of your project why only looking into making everything in one solver and not go with the partitioned way for example? Because the performance is not necessarily the answer here. That seemed like two questions which would you like me to answer? Let's start from the first one. Can you repeat the first one? Yes so why really trying to pick one of the, well you have so different applications why really trying to pick one of the libraries for everything and not follow the partitioned way that then you could have sustainability in the long term. So the work I'm presenting here is based on looking at that one approach we are also looking into the coupled approach so we are covering both bases there. So for the solver part did you use just, so you used one solver for everything but I mean like MFEM has lots of solver backends right? You can actually plug in for all those different frameworks, different solvers that they are really designed to work with. So do you think your solver results would be better if you did that? Potentially so this is one point that I'd possibly like to make in that we've tended to follow what was recommended in tutorials and available documentation for each package so we picked which solver is recommended for it which is more a question of does the library set sensible defaults or advise sensible things for performance? Well I was going to ask the same question but I have actually another one and I would maybe just quickly say that like I think there are many problems where like you just cannot like the library cannot set the defaults because the problem is very specific and it requires still more knowledge but my actual question is like I want to suggest to you again to maybe revaluate your position on non-compile languages because I, are you familiar with FireDrake? Sorry? Are you familiar with FireDrake? I've heard of FireDrake yes. So I'm the original author of FireDrake and in my PhD I actually did, so it's an API compatible implementation of Dolphin or Phoenix and we generate code for the entire solver at runtime and I benchmarked up to 1536 cores in my PhD thesis which was like six years ago and I was able to beat every other package. That's about plain? Well every other package, well I was sorry I was able to beat the Dolphin also the C++ version in terms of performance and like I wasn't, we weren't running into memory issues so like just as another notch maybe. The memory issues is a scaling thing so if you're only running single core with only one interpreter it's a tiny amount of memory so it only really becomes a scaling problem when you have multiple processes. Again this comes back to we don't know what the extra scale looks like if it's fewer processes with lots of GPU dispatch then perhaps the memory overhead of running an interpreter isn't an issue as much as it would be if you run fully on multiple processes on CPUs. Thanks. Sorry we're out of time, we'll have to meet the speaker after. Thank you very much, let's thank the speaker again.
selecting a finite element analysis back end for you a brief introduction to fusion and the here in all the seawater and it's thoroughly the plasma. In most fusion reactor designs these of this picture here. This is an international that we can get more energy out of fusion than we expensive so it would be nice if we could finite elements and so we need to select a back end to build our finite elements simulation on. so I'm presenting today our objective method the order of an exascale machine so it's an first is our first criteria. The second criterion computing resources we have available which could will scale to any number of processes that you this would lead you to an open source license license software that is both permissive and heavily GPU so one of the criteria is to have a against its C++ back end Dolphin and performance there are that are effectively in the finite potential candidate libraries and we went through yesterday the claims that it scales and performs memory and at this stage we had to eliminate would be the best one according to our results processes obviously very important. User in terms of scalability and performance it was a potential flaw. However without implementing a of a large package of work to develop the simulations we need. So in terms of multi physics Hi thank you for the very nice talk. Is the performance is not necessarily the answer here. approach so we are covering both bases there. So it's a tiny amount of memory so it only really