text: [SOLVED] Hyper-V avhdx files filling up host volume - Spiceworks
  by CCH87
  on Sep 27, 2021 at 12:14 UTC
  Next: Hyper V replication
  I have a single virtual Hyper-v host, running 3 Windows servers. My VM disk volume (V: on local host server, no shared storage for VMs) only had a few GB free, so I went looking for ways to save some space. I noticed I had a couple checkpoints for a couple of the servers from 2018, so I searched Google to make sure I could delete checkpoints on running VMs and all of the MS Technet articles I found said that I could, and even recommended removing checkpoints. So I deleted them. What none of the articles I read mentioned was that when you delete a checkpoint, Hyper-v creates a new .avhdx file, then merges, then deletes it. That becomes a problem when you are deleting checkpoints to try to save space, since you dont' have much.
  Anyway, that immediately filled up my VM disk volume and paused my servers. Luckily, I was able to move a "test" VM over to an external hard drive to save around 100GB and I was able to start all my VMs up again. That was a few days ago. Every night, there are a couple more .avhdx files created that are between .5 and 10 GB.
  My question is basically, what's the best way to fix this? Can I export that "problem" VM and then delete all of the .avhdx files and the VM itself in Hyper-v, then import it back? Can I manually delete the .avhdx files that aren't actively "growing" with the server running (I'm assuming they are basically abandoned and shouldn't affect the VM, since there is one .avhdx that is actively growing as the server is used?)? What's the safest option at this point? I have to do something pretty much immediately since the volume is filling up every night. I'm down to 17GB free at this point. Help? lol
  Here is a screenshot of the worst VM offender's disks:
  The highlighted avhdx file is the one that's actively growing.
  Edited Sep 27, 2021 at 12:43 UTC
  Robert5205 Sep 27, 2021 at 13:27 UTC
  Can I export that "problem" VM and then delete all of the .avhdx files and the VM itself in Hyper-v, then import it back?
  Yes. This is a fairly common way of solving the "no space" problem.
  Export the VM to a different media - external drive or network drive.
  Delete the VM on the host using hyper-v manager. This deletes the XML file that describes the VM.
  Remove all files associated with that VM such as the VHD files.
  Copy the exported VM back to the host.
  Import, using the same ID.
  Sep 27, 2021 at 12:52 UTC
  313 Helpful Votes
  You have a bit of a checkpoint issue¬ You need to remove probably ALL those checkpoints.
  What do you see if you do this:
  Get-VMCheckPoint -VMName WPLS1
  I assume you see a bunch of checkpoints listed. Start with the most recent and remove that one. Repeat until all the checkpoints are removed
  Sep 27, 2021 at 12:54 UTC
  If you do not have adequate space to do what I recommended above, then another solution is to turn the VM off, then use Export-VM to export it to another file location. Once you do this remove the original VM (and removing the checkpoints). Then either just import the exported VM and start the VM, OR move the exported files back to where the VM was previously, then re-import.
  CCH87
  Sep 27, 2021 at 13:04 UTC
  Thanks for the reply. No checkpoints are showing in the GUI (apparently I haven't installed the Hyper-v tools for powershell...it didnt' recognize the cmdlet). I only have 13GB free in the V: now, so I'm assuming even if I did have checkpoints left to remove, that would fail as well.
  I will see if the storage can hold out the rest of the business day and then shut down the VM and try the Export option. I'm pretty new to virtualization, and have never moved or exported/imported a production VM (I've done that in testing with throwaway servers), so I'm a little nervous, but I guess this ability is why I went virtual in the first place!
  Also, since you didn't mention it, I'm assuming deleting some of those old avhdx files is probably not a good idea?
  Sep 27, 2021 at 13:13 UTC
  What export-VM does is to export both the VM setup (essentially the XML describing the VM) and the VHDXs all compacted and without any checkpoints.
  To add the tools to the server:
  Add-WindowsFeature -Name Hyper-V -IncludeManagmentTools
  Sep 27, 2021 at 13:43 UTC
  86 Best Answers
  539 Helpful Votes
  The safest method to fix those bugged virtual machines would be either exporting them, removing the original, and importing them back using Hyper-V manager or just cloning the actual state using Windows Admin Center https://www.microsoft.com/en-us/windows-server/windows-admin-center or free V2V Converter https://www.starwindsoftware.com/starwind-v2v-converter. Backup and restore using Veeam B&R https://www.veeam.com/virtual-machine-backup-solution-free.html or your current backup software should do the job as well.
  Manually merging AVHDX files and reconfiguring the virtual machines to use original VHDX instead of VHDX requires lots of manual effort and leaves a lot of space for a mistake, so I recommend you avoid this path, especially given the alternatives mentioned above.
  Sep 27, 2021 at 14:39 UTC
  Thanks for the reassurance and step-by-step, Robert!
  I exported a Linux VM (Unifi controller) and deleted it to save me another 10GB so hopefully that'll hold me over til the end of the day! It is filling up pretty quick. In about an hour it went from 17 down to 8GB.
  Thanks again, to all of you!
  Sep 27, 2021 at 15:25 UTC
  Apparently, in Hyper-V 2012r2, the command is Get-VMSnapshot still. That worked (and showed no checkpoints). Just wanted to add for posterity. Also, it was my fault for not specifying server version in the OP.
  Sep 27, 2021 at 23:33 UTC
  So now I might be screwed... I shut the VM down and ran the export. The last step of the export was "Merging" which failed, because it started taking up space on the SOURCE (V:). How is that possible for an export to take up space in on the host's storage? The export files (on an external hard drive) now show a bunch of the .avhdx files as well. The event manager shows an error, but Hyper-v manager shows nothing in the "Status" section where it was displaying the export progress. I dont' feel like it's a good idea to delete everything on the host and try to import if the export had issues. Am I missing something? Is there no way to export a VM if you don't have like 2 or 3 times the storage space available already on the host?
  Here is the export files on the external hard drive:
  And the V: on the host:
  How comfortable would you be deleting this server and importing it back?
  Sep 27, 2021 at 23:40 UTC
  The event viewer had these two events:
  Sep 28, 2021 at 12:10 UTC
  I wanted to update anyone who might come across this with the same issue. So honestly, I'm not sure if the export was good or not. I do know that the export contained all of the .avhdx files that I was trying to get rid of to save space and the export was around 800GB for a 400GB server. After the export was "finished", the host immediately started filling up the local V: trying to merge and then ran out of space again and paused all my other VMs.
  After trying a few things that did not work, what I ended up doing was shutting down our SQL server (running Sage CRM) after verifying that it was running off of the .vhdx file as the disk and not a .avhdx. I then *cut and pasted* that .vhdx file to external media. This gave our host enough space to finish the merge on the problem VM and once it did that, all but about 4 of it's avhdx files got deleted. The 4 remaining avhdx files dont' appear to be in the "tree", because now in settings, that VM is showing that it's disk is the main .vhdx file. I'll work later on getting rid of the remaining avhdx files.
  Also, when I copied our SQL server's .vhdx file back into our host, it wouldn't start with a permissions error for the disk. After some looking online, I found that if you copy that file to a new folder within the same host, then copy it back, it works. I have no idea why that helped, but it fired right up after I did that.
  I can't recommend anyone do what I did, it just happened to work for me. I gambled that the 100GB SQL server would free up enough space for the merge to finish and it did..barely. It grew another 70 GB before it started deleting the old .avhdx files. I'm not sure what I would have done if it wasn't enough space. I wish Hyper-V would warn you about this space issue before creating or deleting checkpoints. I also can't believe you need 2x the space just to EXPORT a VM, but it's a lesson I won't forget anytime soon. Thanks again to everyone that helped and gave advice.
  Sep 30, 2021 at 10:33 UTC
  CCH87 wrote:
  I wish Hyper-V would warn you about this space issue before creating or deleting checkpoints.
  There is no space issue warning in regards to checkpoints because checkpoints are not backups and should not be used for longer than a couple of hours https://www.vmwareblog.org/snapshots-checkpoints-alone-arent-backups/. There is a good practice to schedule a small automated script that will check through all the virtual machines and automatically remove checkpoints that are older than a specified threshold (usually three days) https://community.spiceworks.com/topic/2216319-script-to-alert-to-old-checkpoints-in-hyper-v. However, you can schedule a script that will notify you about the free space left on VMs volume as well.
  I also can't believe you need 2x the space just to EXPORT a VM, but it's a lesson I won't forget anytime soon.
  This is why it is recommended to use 3rd party software where you can specify more advanced options like exporting the virtual machine to a different volume or network share right away.
  Sep 30, 2021 at 12:06 UTC
  I get that it's not a "backup" and shouldn't be kept long, but as I understand how checkpoints work and how they are deleted, it would make sense that I couldn't even take a checkpoint right now and remove it, regardless of how long I keep it. If Hyper-V creates a new .avhdx file the size of the parent .vhdx PLUS the checkpoint that's being deleted, then merges them and removes the checkpoint, it'll always fail if you don't have enough free space. So in my case, my VM is around 430 GB and I have 360 GB free. It would make sense for Hyper-V to warn me that no matter how long I keep a checkpoint, it WILL fail to delete, and if I do delete one, it'll cause the same problem I just got out of.
  I do agree with you that 3rd party software that does a V2V would have worked better (in this case where you lack enough free space for Hyper-V to do its thing), and is something I'll probably look into going forward.
  I still believe that there should be more in Microsoft's own literature on the process of merging when deleting a checkpoint. It seems like it'd be easy to get in my situation for someone who is deleting old checkpoints (whether they show in Hyper-V manager or not. I still have dozens of .avhdx files across 3 VMs that don't show as checkpoints in Hyper-V manager).
  As always, the Spiceworks community has been the BEST source of reliable information anywhere, and I really do appreciate the help I've gotten from everyone, on this particular issue and in general in the past. Ya'll are awesome!