The story of adding TPM support to oVirt
Zamazal, Milan
oVirt is an open source virtualization solution based on kvm, QEMU and libvirt. Trusted Platform Module (TPM) device support, which brings new security capabilities that modern operating systems utilize or even require, was added to oVirt recently. In theory, adding TPM support should be as easy as just adding a TPM device to the virtual machine libvirt XML. But features built on top of a lower-level virtualization platform are not always as easy to implement as they may initially seem to be. This talk will present the challenges experienced when adding TPM support to oVirt. The talk will explain that a supposedly complete feature support in libvirt/QEMU may still require challenging design considerations. What can be used easily in a simple virtual machine running on a desktop computer may not be enough to get the things working well and reliably in a virtual machine management running across many hosts. Some of the challenges experienced with TPM support have been sorted out while other ones still wait for a good solution. Although focusing on TPM, the lessons presented in this talk can apply to a wide range of features. Whatever we work on, we cannot be just passive consumers of features but we must look for the right ways of using them and be proactive in avoiding pitfalls.
 Hello, welcome in this presentation about adding TPM support to Overt. My name is Milan Zamasal and I work as a software developer at Red Hat. I participated on adding this feature and I'd like to share with you some experiences from its development. Red Platform Module or simply TPM is a piece of hardware provided some security related functions. It can be typically used for things like managing secret keys and data and storing them into secure memory. In virtualization it can be an emulated device serving the same purposes. Its presence is required by recent versions of Windows, but it can be also used for managing disk encryption keys or other purposes. Overt is an open source distributed virtualization solution based on QMU, which is an emulator of processors and other hardware, and Libvert, which manages virtual machines or simply VMs running using QMU or other hypervisors. There are many components involved, but the two most important ones from Overt are the Overt engine. The central virtualization manager and with the SM handling the hosts managed by the Overt engine. Why to talk about TPM in Overt? Because it is a good example of a feature that may look easier from outside than inside. You can look under the hood here and see how the feature is implemented and why some things work the way they do. I'm not going to talk about TPM itself or about how to use TPM in Overt. The letter has already been presented at the Overt conference last year and you can use the link here if you are interested in it. Adding TPM to a virtual machine is easy. It can be added to the Libvert XML description of the VM using this snippet. But is it really that easy? Unfortunately no. The nature is complex and we cannot fully understand even the purest science which is mathematics. At least in my region we talk about three big crises of mathematics. The first one was when Pythagoras and his companions had trouble to know what the length of the square diagonal is because they knew only integers and their fractions. And the contemporary world couldn't exist without calculus. And still calculus was used for about 150 years without good foundations and actually understanding what infinitely small is. And today we know that any theory including natural numbers cannot be both consistent and complete. Either there is a claim that is both true and false or there is a claim where we cannot decide whether it is true or false. And there may be future crises we don't know about yet. Computer practitioners work with numbers a lot but they often deal with problems such as off by one errors and they don't need advanced mathematics for this. So many problems can be resolved by just giving up on completeness and reducing our requirements. Does it mean we can handle software better than maths? We all know the answer. This is how one of my computers welcomed me this year. It stopped working properly and I found out that all my file systems are rate only, apparently due to some bug in a hardling handling in the butterf s. As in many other cases, reboot helped. Until the error has occurred again. But if you can still watch this stream it means that software at least partial works and it's not actually that bad. So will TPM work with just the simple device definition? In some sense yes, unless we want additional functionality such as to be sure that the VM actually starts retaining TPM data across VM restarts or having a proper user interface. What can prevent the VM from starting? The first thing is that TPM requires UEFI firmware, it doesn't work with BIOS. It is not a big problem but it must be addressed in the user interface including all the dependencies between different options and additional considerations such as different architectures. Even with UEFI firmware, the VM won't start if software TPM emulator is not present on the host. The emulator is needed only if the VM contains a TPM device. That means QM and LibVert don't depend on the corresponding package directly and we must add the given package dependency to Overt VDSM. And we must guarantee that the given package is present on hosts in the given cluster. For similar purposes we have the concept of cluster compatibility versions in Overt. TPM could be used on one's new cluster version with this feature was introduced and users upgraded their hosts to that cluster version. One more thing to the device definition. Here is the device definition we originally used. Can anybody spot what's wrong here? It is based on the implicit assumption that LibVert will select the best model for the given TPM device parameters. But this is not how LibVert works. LibVert is strictly backward compatible, which is good. In this case it selects the original alt model and we want to use the more moded model and we must specify it manually. Okay, the VM is running with the right device and the device is listed among other VM devices in the web user interface. The device is different from the other devices there and needs its own icon. What's the problem is adding an icon. It requires communication with the graphic designer, explaining what we need, discussing several proposed icon versions and so on. It's not something especially tricky, but it still requires a significant amount of work. And sometimes little isolated things to do can sit aside for a long time without getting attention. Now to the bigger problems. When a VM stops and is deleted from the host, its TPM data is deleted too. When the VM starts again, we don't have the secrets stored in TPM anymore. So we must find a mechanism for transferring the data between hosts and the central virtualization manager. It's not difficult to come with many ideas how to do it. We can use share storage for storing the data or we can use a central database or we can even attach the data to the XML definition of the VM. We can use events or we can change API calls to pass the data around. The problem is that each of these ideas has its tradeoffs and it's important to select a good approach from the beginning because once the selected solution is implemented, it's very difficult to change the overall mechanism to a different one. So this is a complicated decision. We know there are harder problems like making the code pretty and not ugly. The problem is we can have a piece of code that is both ugly and not so ugly, or code where we cannot agree whether it is ugly or not, all that without proper definition of ugliness. Developers can spend a lot of time discussing these philosophical issues and then be left with no time to implement features such as TPM. Fortunately, this didn't happen to TPM and we hopefully selected the right mechanism. That means the data is stored in the engine database and we pass it around using modified or newly introduced API calls. A temporary problem was that we had to require a new cluster compatibility version again because of the changed APIs. A good time to retrieve the TPM data is when the VM is no longer on the host but it is still present there. At the time, we can be sure that the data is final, it cannot be further modified and we can safely take it. Then we can call the VM destroy API call that deletes everything about the VM from the host. But what if something bad happens and the data retrieval API calls fail all the time? Then we would never call VM destroy and this can cause a lot of other problems. So we decided for a compromise solution. If the data retrieval calls fail several times, then we call VM destroy anyway. Then contingent TPM data modifications are lost. But this is not very different from real hardware. If hardware dies, we lose all the data stored within it. Proper backups are needed whether we use real or emulated hardware. The VM destroy call stops the VM if it is still running on the host. This is used for an implementation shortcut for power off operation in the OVET engine without proper shutdown. In such a case, there is only a single VM destroy API call and we have no space to retrieve the TPM data. Since it would be difficult to change the current powering of mechanism in the engine, we must retrieve the data before the destroy is called. That means the data may be inserimodified after. But again, it's not very different from real hardware where hard pressing a power off switch button is not a safe operation. Is it all what was needed? No, we mustn't forget about irregular flows. What if the VM crashes or the host disappears? When the VM crashes, it's no longer running on the host, but it's still there and this is the situation we already handle. A lost host is a different problem. We may lose all the TPM data modifications performed while the VM was running there. In order to prevent this, at least partially, we predictly retrieve TPM data while the VM is running. In order to avoid unnecessary API calls and data transfers, we attach a TPM data hash to other API calls and the engine retrieves the data only if the hash changes. Was it all what was needed? No, we read the TPM data from the software TPM data directory and if software TPM writes the data at the same moment we read it, then we can get corrupted data. Looking into software TPM sources, we could see that the problem is real and thought about ideas how to improve it. We send the idea to the software TPM developer on Friday afternoon and I was very pleased when FX was implemented and merged just a couple of hours later. Well, if you send a nice bug report, you increase your chances to get a nice response. Was it done? Unfortunately not, because there was still an old software TPM version in the host operating system. So we had to implement a workaround, watching follow-up data changes and reporting the changes only if they are stable. And there is still an additional risk that software TPM may change its data format in future versions. It would be nice if software TPM could still read the old data format in such a case, but there is no guarantee about it and we must check that in the future. Oh, and snapshots. Even the problem with my file system I mentioned before was related to snapshots. Is there anybody here who uses snapshots of any kind and has never had any problem with them? At least deleting a different snapshot than intended by mistake. How is it with snapshots and TPM? There is no problem with offline snapshots, the VM is not running and we can copy its data safely. The only question is where to put it and how. We decided to put it into snapshot OVF. But how about live snapshots? When we take the snapshot, the guest operating system is running and can in theory modify the data any time. And it's difficult to get TPM data exactly corresponding to the snapshot. So we attach the last data retrieved periodically to the snapshot and users must take this into account. Is it any different for snapshots with memory? There are some subtle but important differences. First, if the state of the TPM emulator doesn't correspond to the state of the VM memory, then the guest OS can fall into trouble, which is bad. And second, while the VM memory is being dumped to a snapshot, the guest OS is stopped, which creates a good opportunity to copy the TPM data safely. Unfortunately, LibVid doesn't have support for this. So we had to disable snapshots with memory for VMs with TPM until this is fixed, which is not very good. And here is a fairly complete picture, what all has to be considered regarding storing and transferring TPM data. So we are almost done with TPM data. Except for security, we may not expose the data. And one of the places where it can easily appear is logs. Not that we would like to log something like TPM data is this and this, but it can appear there as part of other data, for instance, API call arguments or responses. And this must be prevented. But not only that, let's not forget about irregular flows again. An unexpected error may occur and it may log trace back together with data to ease the debugging. And secret data must not appear there. It's advisable always check the logs. It happened to me once that I implemented a simple patch that worked, but it's periodically logged a harmless trace back by mistake. And what do people do when they experience some error? They look into the logs and catch on the first error they can see there. So I had to respond to several bug reports I had nothing to do with. The lesson learned. This advantage of an emulated device is that there is no real secure memory. Libvert provides at least a mechanism to protect the data with a passphrase. But with helpers like this or two-factor authentication or other security measures, it's always necessary to consider the context in order to prevent a false sense of increased security. In Overt, we would generally need the passphrase at the same places where we have the data. So we cannot use it meaningfully and use technical measures instead of trust in people with elevated system permissions. When we start a VM on host, we upload its TPM data to the local file system there, and it may be a danger for other data on the host. We must use sanity checks to prevent damages by mistake or by malicious requests. And also check the other components we use. We use tar format for software TPM data, and it may be tempting to use SHUTL Python library in Python code to unpack the data. Unfortunately, the library is insecure and we had to use GNU tar, which can provide the necessary guarantees. The virtualization part is almost done, but how about the guest operating systems? In order to avoid user confusion, we allow enabling TPM only for the guest operating systems we know about that they work well with TPM. Users can enable TPM support for other guest operating systems if they wish. There were many other things that had to be done in order to have a full TPM support in Overt, including TPM support in templates, in the REST API, and providing a comprehensive web user interface, including things like asking users before deleting TPM data. TPM is often good in software development, and we implemented TPM in such a way that it could be reused to support secure boot persistent data in Overt. Although the features are a bit different, it was relatively easy to add secure boot data persistence. As you could see, adding a proper TPM support to Overt was a complex task, despite it could be built on top of a big work already done by Libvert, QM, software TPM, and others. This wasn't unexpected, but there are always surprises in software development, and we had to deal with various challenges. I hope you enjoyed the story, and perhaps you can use some lessons from it to further enhance virtualization features in Overt or other products. The list here is nothing unknown to any experienced software developer, but it never hearts to repeat these basic rules, like paying attention to documentation rather than making implicit assumptions, always looking into the logs, paying attention to security, not forgetting about irregular flows, sharing where possible, and not making mistakes that will cause trouble later. But foremost, having the users on mind and cooperating efficiently towards reaching the final goal, although it will almost never be fully perfect. Thank you for watching this presentation, and if you have any questions, you can ask them here now or anytime later on the development and users over to mailing lists.
 Hello, welcome in this presentation about adding TPM support to Overt.  My name is Milan Zamasal and I work as a software developer at Red Hat.  I participated on adding this feature and I'd like to share with you some experiences  from its development.  Red Platform Module or simply TPM is a piece of hardware provided some security related  functions.  It can be typically used for things like managing secret keys and data and storing them into  secure memory.  In virtualization it can be an emulated device serving the same purposes.  Its presence is required by recent versions of Windows, but it can be also used for managing  disk encryption keys or other purposes.  Overt is an open source distributed virtualization solution based on QMU, which is an emulator  of processors and other hardware, and Libvert, which manages virtual machines or simply VMs  running using QMU or other hypervisors.  There are many components involved, but the two most important ones from Overt are the  Overt engine.  The central virtualization manager and with the SM handling the hosts managed by the Overt  engine.  Why to talk about TPM in Overt?  Because it is a good example of a feature that may look easier from outside than inside.  You can look under the hood here and see how the feature is implemented and why some things  work the way they do.  I'm not going to talk about TPM itself or about how to use TPM in Overt.  The letter has already been presented at the Overt conference last year and you can use  the link here if you are interested in it.  Adding TPM to a virtual machine is easy.  It can be added to the Libvert XML description of the VM using this snippet.  But is it really that easy?  Unfortunately no.  The nature is complex and we cannot fully understand even the purest science which is mathematics.  At least in my region we talk about three big crises of mathematics.  The first one was when Pythagoras and his companions had trouble to know what the length  of the square diagonal is because they knew only integers and their fractions.  And the contemporary world couldn't exist without calculus.  And still calculus was used for about 150 years without good foundations and actually  understanding what infinitely small is.  And today we know that any theory including natural numbers cannot be both consistent and  complete.  Either there is a claim that is both true and false or there is a claim where we cannot  decide whether it is true or false.  And there may be future crises we don't know about yet.  Computer practitioners work with numbers a lot but they often deal with problems such  as off by one errors and they don't need advanced mathematics for this.  So many problems can be resolved by just giving up on completeness and reducing our requirements.  Does it mean we can handle software better than maths?  We all know the answer.  This is how one of my computers welcomed me this year.  It stopped working properly and I found out that all my file systems are rate only, apparently  due to some bug in a hardling handling in the butterf s.  As in many other cases, reboot helped.  Until the error has occurred again.  But if you can still watch this stream it means that software at least partial works  and it's not actually that bad.  So will TPM work with just the simple device definition?  In some sense yes, unless we want additional functionality such as to be sure that the  VM actually starts retaining TPM data across VM restarts or having a proper user interface.  What can prevent the VM from starting?  The first thing is that TPM requires UEFI firmware, it doesn't work with BIOS.  It is not a big problem but it must be addressed in the user interface including all the dependencies  between different options and additional considerations such as different architectures.  Even with UEFI firmware, the VM won't start if software TPM emulator is not present on  the host.  The emulator is needed only if the VM contains a TPM device.  That means QM and LibVert don't depend on the corresponding package directly and we  must add the given package dependency to Overt VDSM.  And we must guarantee that the given package is present on hosts in the given cluster.  For similar purposes we have the concept of cluster compatibility versions in Overt.  TPM could be used on one's new cluster version with this feature was introduced and users  upgraded their hosts to that cluster version.  One more thing to the device definition.  Here is the device definition we originally used.  Can anybody spot what's wrong here?  It is based on the implicit assumption that LibVert will select the best model for the  given TPM device parameters.  But this is not how LibVert works.  LibVert is strictly backward compatible, which is good.  In this case it selects the original alt model and we want to use the more moded model and  we must specify it manually.  Okay, the VM is running with the right device and the device is listed among other VM devices  in the web user interface.  The device is different from the other devices there and needs its own icon.  What's the problem is adding an icon.  It requires communication with the graphic designer, explaining what we need, discussing  several proposed icon versions and so on.  It's not something especially tricky, but it still requires a significant amount of work.  And sometimes little isolated things to do can sit aside for a long time without getting  attention.  Now to the bigger problems.  When a VM stops and is deleted from the host, its TPM data is deleted too.  When the VM starts again, we don't have the secrets stored in TPM anymore.  So we must find a mechanism for transferring the data between hosts and the central virtualization  manager.  It's not difficult to come with many ideas how to do it.  We can use share storage for storing the data or we can use a central database or we can  even attach the data to the XML definition of the VM.  We can use events or we can change API calls to pass the data around.  The problem is that each of these ideas has its tradeoffs and it's important to select  a good approach from the beginning because once the selected solution is implemented,  it's very difficult to change the overall mechanism to a different one.  So this is a complicated decision.  We know there are harder problems like making the code pretty and not ugly.  The problem is we can have a piece of code that is both ugly and not so ugly, or code  where we cannot agree whether it is ugly or not, all that without proper definition of  ugliness.  Developers can spend a lot of time discussing these philosophical issues and then be left  with no time to implement features such as TPM.  Fortunately, this didn't happen to TPM and we hopefully selected the right mechanism.  That means the data is stored in the engine database and we pass it around using modified  or newly introduced API calls.  A temporary problem was that we had to require a new cluster compatibility version again  because of the changed APIs.  A good time to retrieve the TPM data is when the VM is no longer on the host but it is  still present there.  At the time, we can be sure that the data is final, it cannot be further modified and  we can safely take it.  Then we can call the VM destroy API call that deletes everything about the VM from the  host.  But what if something bad happens and the data retrieval API calls fail all the time?  Then we would never call VM destroy and this can cause a lot of other problems.  So we decided for a compromise solution.  If the data retrieval calls fail several times, then we call VM destroy anyway.  Then contingent TPM data modifications are lost.  But this is not very different from real hardware.  If hardware dies, we lose all the data stored within it.  Proper backups are needed whether we use real or emulated hardware.  The VM destroy call stops the VM if it is still running on the host.  This is used for an implementation shortcut for power off operation in the OVET engine  without proper shutdown.  In such a case, there is only a single VM destroy API call and we have no space to retrieve  the TPM data.  Since it would be difficult to change the current powering of mechanism in the engine,  we must retrieve the data before the destroy is called.  That means the data may be inserimodified after.  But again, it's not very different from real hardware where hard pressing a power off switch  button is not a safe operation.  Is it all what was needed?  No, we mustn't forget about irregular flows.  What if the VM crashes or the host disappears?  When the VM crashes, it's no longer running on the host, but it's still there and this  is the situation we already handle.  A lost host is a different problem.  We may lose all the TPM data modifications performed while the VM was running there.  In order to prevent this, at least partially, we predictly retrieve TPM data while the VM  is running.  In order to avoid unnecessary API calls and data transfers, we attach a TPM data hash  to other API calls and the engine retrieves the data only if the hash changes.  Was it all what was needed?  No, we read the TPM data from the software TPM data directory and if software TPM writes  the data at the same moment we read it, then we can get corrupted data.  Looking into software TPM sources, we could see that the problem is real and thought about  ideas how to improve it.  We send the idea to the software TPM developer on Friday afternoon and I was very pleased  when FX was implemented and merged just a couple of hours later.  Well, if you send a nice bug report, you increase your chances to get a nice response.  Was it done?  Unfortunately not, because there was still an old software TPM version in the host operating  system.  So we had to implement a workaround, watching follow-up data changes and reporting the changes  only if they are stable.  And there is still an additional risk that software TPM may change its data format in  future versions.  It would be nice if software TPM could still read the old data format in such a case, but  there is no guarantee about it and we must check that in the future.  Oh, and snapshots.  Even the problem with my file system I mentioned before was related to snapshots.  Is there anybody here who uses snapshots of any kind and has never had any problem with  them?  At least deleting a different snapshot than intended by mistake.  How is it with snapshots and TPM?  There is no problem with offline snapshots, the VM is not running and we can copy its  data safely.  The only question is where to put it and how.  We decided to put it into snapshot OVF.  But how about live snapshots?  When we take the snapshot, the guest operating system is running and can in theory modify  the data any time.  And it's difficult to get TPM data exactly corresponding to the snapshot.  So we attach the last data retrieved periodically to the snapshot and users must take this into  account.  Is it any different for snapshots with memory?  There are some subtle but important differences.  First, if the state of the TPM emulator doesn't correspond to the state of the VM memory,  then the guest OS can fall into trouble, which is bad.  And second, while the VM memory is being dumped to a snapshot, the guest OS is stopped,  which creates a good opportunity to copy the TPM data safely.  Unfortunately, LibVid doesn't have support for this.  So we had to disable snapshots with memory for VMs with TPM until this is fixed, which  is not very good.  And here is a fairly complete picture, what all has to be considered regarding storing  and transferring TPM data.  So we are almost done with TPM data.  Except for security, we may not expose the data.  And one of the places where it can easily appear is logs.  Not that we would like to log something like TPM data is this and this, but it can appear  there as part of other data, for instance, API call arguments or responses.  And this must be prevented.  But not only that, let's not forget about irregular flows again.  An unexpected error may occur and it may log trace back together with data to ease the  debugging.  And secret data must not appear there.  It's advisable always check the logs.  It happened to me once that I implemented a simple patch that worked, but it's periodically  logged a harmless trace back by mistake.  And what do people do when they experience some error?  They look into the logs and catch on the first error they can see there.  So I had to respond to several bug reports I had nothing to do with.  The lesson learned.  This advantage of an emulated device is that there is no real secure memory.  Libvert provides at least a mechanism to protect the data with a passphrase.  But with helpers like this or two-factor authentication or other security measures, it's always necessary  to consider the context in order to prevent a false sense of increased security.  In Overt, we would generally need the passphrase at the same places where we have the data.  So we cannot use it meaningfully and use technical measures instead of trust in people with elevated  system permissions.  When we start a VM on host, we upload its TPM data to the local file system there, and  it may be a danger for other data on the host.  We must use sanity checks to prevent damages by mistake or by malicious requests.  And also check the other components we use.  We use tar format for software TPM data, and it may be tempting to use SHUTL Python library  in Python code to unpack the data.  Unfortunately, the library is insecure and we had to use GNU tar, which can provide the  necessary guarantees.  The virtualization part is almost done, but how about the guest operating systems?  In order to avoid user confusion, we allow enabling TPM only for the guest operating  systems we know about that they work well with TPM.  Users can enable TPM support for other guest operating systems if they wish.  There were many other things that had to be done in order to have a full TPM support in  Overt, including TPM support in templates, in the REST API, and providing a comprehensive  web user interface, including things like asking users before deleting TPM data.  TPM is often good in software development, and we implemented TPM in such a way that  it could be reused to support secure boot persistent data in Overt.  Although the features are a bit different, it was relatively easy to add secure boot  data persistence.  As you could see, adding a proper TPM support to Overt was a complex task, despite it could  be built on top of a big work already done by Libvert, QM, software TPM, and others.  This wasn't unexpected, but there are always surprises in software development, and we  had to deal with various challenges.  I hope you enjoyed the story, and perhaps you can use some lessons from it to further  enhance virtualization features in Overt or other products.  The list here is nothing unknown to any experienced software developer, but it never hearts to  repeat these basic rules, like paying attention to documentation rather than making implicit  assumptions, always looking into the logs, paying attention to security, not forgetting  about irregular flows, sharing where possible, and not making mistakes that will cause trouble  later.  But foremost, having the users on mind and cooperating efficiently towards reaching the  final goal, although it will almost never be fully perfect.  Thank you for watching this presentation, and if you have any questions, you can ask them  here now or anytime later on the development and users over to mailing lists.
vai_story_tpm_0_167.png vai_story_tpm_168_647.png vai_story_tpm_648_1427.png vai_story_tpm_1428_2207.png vai_story_tpm_2208_2663.png vai_story_tpm_2664_3035.png vai_story_tpm_3036_3419.png vai_story_tpm_3420_5339.png vai_story_tpm_5340_6251.png vai_story_tpm_6252_6731.png vai_story_tpm_6732_7271.png vai_story_tpm_7272_8447.png vai_story_tpm_8448_9323.png vai_story_tpm_9324_10355.png vai_story_tpm_10356_10919.png vai_story_tpm_10920_12503.png vai_story_tpm_12504_13091.png vai_story_tpm_13092_14651.png vai_story_tpm_14652_15731.png vai_story_tpm_15732_16895.png vai_story_tpm_16896_18683.png vai_story_tpm_18684_20087.png vai_story_tpm_20088_21023.png vai_story_tpm_21024_21371.png vai_story_tpm_21372_22895.png vai_story_tpm_22896_23831.png vai_story_tpm_23832_24731.png vai_story_tpm_24732_25283.png vai_story_tpm_25284_25727.png vai_story_tpm_25728_26171.png vai_story_tpm_26172_26699.png vai_story_tpm_26700_27875.png vai_story_tpm_27876_28163.png vai_story_tpm_28164_28304.png
Hello, welcome in this presentation about adding TPM support to Overt. My name is Milan Zamasal and I work as a software developer at Red Hat. I participated on adding this feature and I'd like to share with you some experiences from its development. Red Platform Module or simply TPM is a piece of hardware provided some security related functions. It can be typically used for things like managing secret keys and data and storing them into secure memory. In virtualization it can be an emulated device serving the same purposes. Its presence is required by recent versions of Windows, but it can be also used for managing disk encryption keys or other purposes. Overt is an open source distributed virtualization solution based on QMU, which is an emulator of processors and other hardware, and Libvert, which manages virtual machines or simply VMs running using QMU or other hypervisors. There are many components involved, but the two most important ones from Overt are the Overt engine. The central virtualization manager and with the SM handling the hosts managed by the Overt engine. Why to talk about TPM in Overt? Because it is a good example of a feature that may look easier from outside than inside. You can look under the hood here and see how the feature is implemented and why some things work the way they do. I'm not going to talk about TPM itself or about how to use TPM in Overt. The letter has already been presented at the Overt conference last year and you can use the link here if you are interested in it. Adding TPM to a virtual machine is easy. It can be added to the Libvert XML description of the VM using this snippet. But is it really that easy? Unfortunately no. The nature is complex and we cannot fully understand even the purest science which is mathematics. At least in my region we talk about three big crises of mathematics. The first one was when Pythagoras and his companions had trouble to know what the length of the square diagonal is because they knew only integers and their fractions. And the contemporary world couldn't exist without calculus. And still calculus was used for about 150 years without good foundations and actually understanding what infinitely small is. And today we know that any theory including natural numbers cannot be both consistent and complete. Either there is a claim that is both true and false or there is a claim where we cannot decide whether it is true or false. And there may be future crises we don't know about yet. Computer practitioners work with numbers a lot but they often deal with problems such as off by one errors and they don't need advanced mathematics for this. So many problems can be resolved by just giving up on completeness and reducing our requirements. Does it mean we can handle software better than maths? We all know the answer. This is how one of my computers welcomed me this year. It stopped working properly and I found out that all my file systems are rate only, apparently due to some bug in a hardling handling in the butterf s. As in many other cases, reboot helped. Until the error has occurred again. But if you can still watch this stream it means that software at least partial works and it's not actually that bad. So will TPM work with just the simple device definition? In some sense yes, unless we want additional functionality such as to be sure that the VM actually starts retaining TPM data across VM restarts or having a proper user interface. What can prevent the VM from starting? The first thing is that TPM requires UEFI firmware, it doesn't work with BIOS. It is not a big problem but it must be addressed in the user interface including all the dependencies between different options and additional considerations such as different architectures. Even with UEFI firmware, the VM won't start if software TPM emulator is not present on the host. The emulator is needed only if the VM contains a TPM device. That means QM and LibVert don't depend on the corresponding package directly and we must add the given package dependency to Overt VDSM. And we must guarantee that the given package is present on hosts in the given cluster. For similar purposes we have the concept of cluster compatibility versions in Overt. TPM could be used on one's new cluster version with this feature was introduced and users upgraded their hosts to that cluster version. One more thing to the device definition. Here is the device definition we originally used. Can anybody spot what's wrong here? It is based on the implicit assumption that LibVert will select the best model for the given TPM device parameters. But this is not how LibVert works. LibVert is strictly backward compatible, which is good. In this case it selects the original alt model and we want to use the more moded model and we must specify it manually. Okay, the VM is running with the right device and the device is listed among other VM devices in the web user interface. The device is different from the other devices there and needs its own icon. What's the problem is adding an icon. It requires communication with the graphic designer, explaining what we need, discussing several proposed icon versions and so on. It's not something especially tricky, but it still requires a significant amount of work. And sometimes little isolated things to do can sit aside for a long time without getting attention. Now to the bigger problems. When a VM stops and is deleted from the host, its TPM data is deleted too. When the VM starts again, we don't have the secrets stored in TPM anymore. So we must find a mechanism for transferring the data between hosts and the central virtualization manager. It's not difficult to come with many ideas how to do it. We can use share storage for storing the data or we can use a central database or we can even attach the data to the XML definition of the VM. We can use events or we can change API calls to pass the data around. The problem is that each of these ideas has its tradeoffs and it's important to select a good approach from the beginning because once the selected solution is implemented, it's very difficult to change the overall mechanism to a different one. So this is a complicated decision. We know there are harder problems like making the code pretty and not ugly. The problem is we can have a piece of code that is both ugly and not so ugly, or code where we cannot agree whether it is ugly or not, all that without proper definition of ugliness. Developers can spend a lot of time discussing these philosophical issues and then be left with no time to implement features such as TPM. Fortunately, this didn't happen to TPM and we hopefully selected the right mechanism. That means the data is stored in the engine database and we pass it around using modified or newly introduced API calls. A temporary problem was that we had to require a new cluster compatibility version again because of the changed APIs. A good time to retrieve the TPM data is when the VM is no longer on the host but it is still present there. At the time, we can be sure that the data is final, it cannot be further modified and we can safely take it. Then we can call the VM destroy API call that deletes everything about the VM from the host. But what if something bad happens and the data retrieval API calls fail all the time? Then we would never call VM destroy and this can cause a lot of other problems. So we decided for a compromise solution. If the data retrieval calls fail several times, then we call VM destroy anyway. Then contingent TPM data modifications are lost. But this is not very different from real hardware. If hardware dies, we lose all the data stored within it. Proper backups are needed whether we use real or emulated hardware. The VM destroy call stops the VM if it is still running on the host. This is used for an implementation shortcut for power off operation in the OVET engine without proper shutdown. In such a case, there is only a single VM destroy API call and we have no space to retrieve the TPM data. Since it would be difficult to change the current powering of mechanism in the engine, we must retrieve the data before the destroy is called. That means the data may be inserimodified after. But again, it's not very different from real hardware where hard pressing a power off switch button is not a safe operation. Is it all what was needed? No, we mustn't forget about irregular flows. What if the VM crashes or the host disappears? When the VM crashes, it's no longer running on the host, but it's still there and this is the situation we already handle. A lost host is a different problem. We may lose all the TPM data modifications performed while the VM was running there. In order to prevent this, at least partially, we predictly retrieve TPM data while the VM is running. In order to avoid unnecessary API calls and data transfers, we attach a TPM data hash to other API calls and the engine retrieves the data only if the hash changes. Was it all what was needed? No, we read the TPM data from the software TPM data directory and if software TPM writes the data at the same moment we read it, then we can get corrupted data. Looking into software TPM sources, we could see that the problem is real and thought about ideas how to improve it. We send the idea to the software TPM developer on Friday afternoon and I was very pleased when FX was implemented and merged just a couple of hours later. Well, if you send a nice bug report, you increase your chances to get a nice response. Was it done? Unfortunately not, because there was still an old software TPM version in the host operating system. So we had to implement a workaround, watching follow-up data changes and reporting the changes only if they are stable. And there is still an additional risk that software TPM may change its data format in future versions. It would be nice if software TPM could still read the old data format in such a case, but there is no guarantee about it and we must check that in the future. Oh, and snapshots. Even the problem with my file system I mentioned before was related to snapshots. Is there anybody here who uses snapshots of any kind and has never had any problem with them? At least deleting a different snapshot than intended by mistake. How is it with snapshots and TPM? There is no problem with offline snapshots, the VM is not running and we can copy its data safely. The only question is where to put it and how. We decided to put it into snapshot OVF. But how about live snapshots? When we take the snapshot, the guest operating system is running and can in theory modify the data any time. And it's difficult to get TPM data exactly corresponding to the snapshot. So we attach the last data retrieved periodically to the snapshot and users must take this into account. Is it any different for snapshots with memory? There are some subtle but important differences. First, if the state of the TPM emulator doesn't correspond to the state of the VM memory, then the guest OS can fall into trouble, which is bad. And second, while the VM memory is being dumped to a snapshot, the guest OS is stopped, which creates a good opportunity to copy the TPM data safely. Unfortunately, LibVid doesn't have support for this. So we had to disable snapshots with memory for VMs with TPM until this is fixed, which is not very good. And here is a fairly complete picture, what all has to be considered regarding storing and transferring TPM data. So we are almost done with TPM data. Except for security, we may not expose the data. And one of the places where it can easily appear is logs. Not that we would like to log something like TPM data is this and this, but it can appear there as part of other data, for instance, API call arguments or responses. And this must be prevented. But not only that, let's not forget about irregular flows again. An unexpected error may occur and it may log trace back together with data to ease the debugging. And secret data must not appear there. It's advisable always check the logs. It happened to me once that I implemented a simple patch that worked, but it's periodically logged a harmless trace back by mistake. And what do people do when they experience some error? They look into the logs and catch on the first error they can see there. So I had to respond to several bug reports I had nothing to do with. The lesson learned. This advantage of an emulated device is that there is no real secure memory. Libvert provides at least a mechanism to protect the data with a passphrase. But with helpers like this or two-factor authentication or other security measures, it's always necessary to consider the context in order to prevent a false sense of increased security. In Overt, we would generally need the passphrase at the same places where we have the data. So we cannot use it meaningfully and use technical measures instead of trust in people with elevated system permissions. When we start a VM on host, we upload its TPM data to the local file system there, and it may be a danger for other data on the host. We must use sanity checks to prevent damages by mistake or by malicious requests. And also check the other components we use. We use tar format for software TPM data, and it may be tempting to use SHUTL Python library in Python code to unpack the data. Unfortunately, the library is insecure and we had to use GNU tar, which can provide the necessary guarantees. The virtualization part is almost done, but how about the guest operating systems? In order to avoid user confusion, we allow enabling TPM only for the guest operating systems we know about that they work well with TPM. Users can enable TPM support for other guest operating systems if they wish. There were many other things that had to be done in order to have a full TPM support in Overt, including TPM support in templates, in the REST API, and providing a comprehensive web user interface, including things like asking users before deleting TPM data. TPM is often good in software development, and we implemented TPM in such a way that it could be reused to support secure boot persistent data in Overt. Although the features are a bit different, it was relatively easy to add secure boot data persistence. As you could see, adding a proper TPM support to Overt was a complex task, despite it could be built on top of a big work already done by Libvert, QM, software TPM, and others. This wasn't unexpected, but there are always surprises in software development, and we had to deal with various challenges. I hope you enjoyed the story, and perhaps you can use some lessons from it to further enhance virtualization features in Overt or other products. The list here is nothing unknown to any experienced software developer, but it never hearts to repeat these basic rules, like paying attention to documentation rather than making implicit assumptions, always looking into the logs, paying attention to security, not forgetting about irregular flows, sharing where possible, and not making mistakes that will cause trouble later. But foremost, having the users on mind and cooperating efficiently towards reaching the final goal, although it will almost never be fully perfect. Thank you for watching this presentation, and if you have any questions, you can ask them here now or anytime later on the development and users over to mailing lists.
Hello, welcome in this presentation about adding development. Red Platform Module or simply TPM is keys or other purposes. Overt is an open source distributed virtualization solution based on QMU, Overt. The letter has already been presented at TPM to a virtual machine is easy. It can be added to the Libvert XML description of the VM using theory including natural numbers cannot be both work with just the simple device definition? In versions in Overt. TPM could be used on one's new running with the right device and the device is no time to implement features such as TPM. that we had to require a new cluster data is when the VM is no longer on the host but it is still present there. At the time, we can be on the host. This is used for an implementation to get a nice response. Was it done? OS is stopped, which creates a good opportunity TPM data. Except for security, we may not expose this must be prevented. But not only that, let's enabling TPM only for the guest operating systems we know about that they work well with TPM. Users things that had to be done in order to have a full TPM support in Overt, including TPM support data in Overt. Although the features are a bit proper TPM support to Overt was a complex task, despite it could be built on top of a big work