More Hyper-V Troubleshooting Woes - but a script to help

In a previous post, I noted some challenges I faced with Hyper-V. The TL;DR is that Windows Update breaks Hyper-V, often as a result of a new WIndows Insider Build. In that post I explained the basic fix for this issue. 

Jeffrey Snover once pointed out that when Linux guys are faced with an issue for a second time - they create a script. Well - I am now facing this Hyper-V problem more frequently. The root issue is that an Update can break Hyper-V. The solution is to re-create the VMs. Which can be a lot of work.

As I recovered from the previous failure, I started to write a script to so many of the actions. This week, I had ANOTHER update (WSL) that broke Hyper-V. So to recover, I write the script below. I hope you find is useful.

# fixing hyperv

# a script in several parts

# part 1 - remove Hyper-V from Win 11

# run in elevated command prompt

# 1.1 View what is there

Get-windowsoptionalFeature -FeatureName Microsoft-hyper-V* -online |

  Format-Table -Property *Name,State

# 1.2 Remove the Hyper-V optionalfeature  

Get-windowsoptionalFeature -FeatureName Microsoft-hyper-V* -online |

  Disable-WindowsOptionalFeature -Online

# note this pops up a 'do you want to reboot' - say yes/.

#

# Note the reboot may fail (trying and not succeeding to stop the hyper-V service)

# Step 2. After the reboot from step 1

# 2.1 View Hyper-V Data Folder

$DataBase = "$env:ProgramData\Microsoft\Windows\Hyper-V"

Get-ChildItem -Path $DataBase

# 2.2 Remove the Hyper-V Data

Get-ChildItem -Path $DataBase -recurse |

  Remove-Item -Recurse -Force

# 2.3 Check the features

Get-windowsoptionalFeature -FeatureName Microsoft-hyper-V* -online |

Format-Table -Property *Name,State

# 2.4 Re-Add Hyper-V to Windows and Reboot

Get-WindowsOptionalFeature -FeatureName Microsoft-hyper-V* -online

| Enable-WindowsOptionalFeature -Online

# Reboot

# Step 3 - Creater New Switchs

New-VMSwitch -Name Internal -SwitchType Internal

$NIC = Get-NetAdapter | where {($_.status -eq 'up') -and ($_.Name -notmatch 'vEthernet')}

New-VMswitch -Name External -NetadapterName $Nic.Name -AllowManagementOS:$true

# Step 4 - recreate the VMs

# At this point, you may need to merge any difference disks. So far I have not automated this

# So use the GUI

# Cookham1

$vmbase = 'D:\VMs\Cookham1'

$Vhdx   = "$VMbase\Virtual Hard Disks\Cookham1.vhdx"

New-VM -VHDPath $vhdx -Generation 2 -MemoryStartupBytes 8gb -Name "Cookham1" -Path $vmbase

Start-VM -VMName 'Cookham1'

# Reskit VMs

# DC1

$vmbase = 'D:\v9\DC1'

$Vhdx   = "$VMbase\dc1.vhdx"

New-VM -VHDPath $vhdx -Generation 2 -MemoryStartupBytes 8gb -Name DC1 -Path $vmbase

Start-VM -VMName dc1

# DC2

$vmbase = 'D:\v9\DC2'

$Vhdx   = "$VMbase\dc2.vhdx"

New-VM -VHDPath $vhdx -Generation 2 -MemoryStartupBytes 8gb -Name DC2 -Path $vmbase

Start-VM -VMName DC2

# SRV1

$vmbase = 'D:\v9\SRV1'

$Vhdx   = "$VMbase\SRV1.vhdx"

New-VM -VHDPath $vhdx -Generation 2 -MemoryStartupBytes 6gb -Name SRV1 -Path $vmbase

Start-VM -VMName SRV1

# SRV2

$vmbase = 'D:\v9\SRV2'

$Vhdx   = "$VMbase\SRV2.vhdx"

New-VM -VHDPath $vhdx -Generation 2 -MemoryStartupBytes 6gb -Name SRV2 -Path $vmbase

Start-VM -VMName SRV2

# UKDC1

$vmbase = 'D:\v9\UKDC1'

$Vhdx   = "$VMbase\UKDC1.vhdx"

New-VM -VHDPath $vhdx -Generation 2 -MemoryStartupBytes 6gb -Name UKDC1 -Path $vmbase

Start-VM -VMName UKDC1

# Step 5. reset networking

# On DC1.DC2 - turn OFF the DHCP serice temporarily!

# Remove all Reservations and zones

# Then On a VM by VM basis:

#   Open the VM'S Hyperv- Settings and remove Nics

#   In the VM, use Device manager, view hidden devices and Remove ALL the Hyper-V NICS

#   In the VM's Settings - add two nics: Internal (Ethernet) and External (Ethernet #2)

#   Re-configure the Internal statisc NIC's IP address (see reskitnetowrk.ms)

# Recreate the DHCP Scopes and reservations and test

# Test internal communicationns (from VM to each DC and server)

Test-NetConnection DC1

Test-NetConnection DC2

Test-NetConnection UKDC1

Test-NetConnection SRV1

Test-NetConnection SRV2

Troubleshooting Hyper-V in Windows 11

 I love Hyper-V inside Windows 11 (and inside Windows 8.x and 10 in their day). An utterly simple hypervisor (to use) but one that provides a lot of benefits. I have written 4, and shortly 5, books that made use of this feature. I have created a nice server farm (2 forests, 3 domains, multiple servers, etc) which runs nicely on my boxes. The scripts to create the VMs, and configure them are all published to GitHub,. 

Now to run a VM at a reasonable speed, you need to give the VM resources. My old Precision T7500 did not and could not support the hardware necessary for Win 11. I now have a Precision T7920 with 2 16-core Xeon Gold processors, 128 GB of ram, 1 TB of Nvme SSD, 2x2TB plus 1x1TB SSDs. So a LOT of storage, a LOT of memory, and a lot OF CPU. My farm works great. Except when it doesn't.

I run this box as part of the Windows Insider's program. That means I get a more-or-less weekly upgrade that brings new features and bug fixes. It is a lot of fun - except sometimes the upgrade does NOT work. Over the years I have had numerous issues with an upgrade and have generally been able to back up to an earlier build or take a newer build (with a fix). For the most part this has been an inconvenience to me, but useful for the program for a while - I just love providing feedback.

One specific set of issues caused by an updated Insider build is that Hyper-V does not work. This has happened now around 5 times over the years. NOTE: This was originally written in 2002. Since this blog post was first published, this issue has affected nearly every Windows Insider update I have taken since.

Backing up to an earlier build sometimes cures the problems. But all to often, I just could not get things going again. This happend again a few days ago - and this blog post notes some of the issues and some troubleshooting tips (at that time).

So after an upgrade, starting the Hyper-V MMC showed something like this:

As I hope you can see, the MMC is showing two VMs trying to start. One is stuck at restoring (10%), the other at just restoring. Trying to stop a VM from PowerShell was also not successful. 

Troubleshooting tip #1

As long as you have the VHDX(s), you can always rebuild the VM. And if you created the VM with a PowerShell script, re-creating it is trivial.

So with that in mind, I did the obvious thing:  I uninstalled Hyper-V totally and rebooted. Then, I "hid" the folders containing my VMs and re-installed Hyper-V. My thinking was that this would clear everything away, and I could unhide the VM folders and re-import the VMs. Except it didn't work/

Having removed then re-added Hyper-V did NOT clear down the configuration. In fact, what I now see is:

So even though I removed and re-added Hyper-V (and hid all the folders containing the VMs/VHDXs) Hyper-V still remembered the old VMs. This seemed illogical until a nice MS person explained it as the de-installation does not take everything related to Hyper-V off your box - in case you want to reinstall it - your configuration magically reappears. 

Troubleshooting tip #2 and #3

A conversation on Twitter led to the discovery. The list of VMs that the MMC uses when it starts up is contained in a file: $env:ProgramData\Microsoft\Windows\Hyper-V\data.vmcx although it was different in earlier versions of Windows.

The second troubleshooting tip: is that it is in binary form, so hand editing is not really possible. I was also told that removing the file totally was a bad idea since it contains other valuable data, too.

So how to recover? 

Troubleshooting tip #4

As noted, the details of the VMs and Hyper-V itself are, by default, contained in a folder (with sub-folders) at $env:ProgramData\Microsoft\Windows\Hyper-V. So to get back to a pristine state, remove the Hyper-V feature from Windows 11. After the reboot, remove this folder totally then re-install Hyper-V and reboot. After the reboot, you should see a nice clean MMC console!

With a pristine Hyper-V environment, I have to re-create the virtual switches and then re-create the VMs. It appears that removing the Hyper-V feature deletes all the VM data except the actual VHDXs themselves. So recreating the VMs meant just building a new VM that contained the old VHDX. 

There was only one final problem - networking. If you remove a vNIC from a VM, and then add a new NIC, Windows sees the new NIC as, well, a NEW NIC and creates a new configuration for it. But it also keeps the old configuration. This can cause some issues, including not letting you rename the connection inside ncpa.cpl - Windows claims another connection already has that game.

If you open Device Manager, you see just the most recently added vNIC, something like this:

Troubleshooting tip #5

If you enable hidden devices inside Device Manager you can see the "removed" net adapters.

In Device Manager, click on View, then select Show Hidden Devices then you see:

You can then right-click each adapter and uninstall the device. If you then click Action, and then click Scan for hardware changes - you should see just the actual vNICs in your VM.

Troubleshooting tip #6

If you add a 'new' nic to a VM, Windows sees it as a brand new NIC, so sets the device to get its configuration from DHCP. 

This might be fine if the host was DHCP configured but not if you configured it with static IP addresses (eg for a DC or a DNS server). Fortunately, you have PowerShell and can easily script the NIC configuration. 

I hope this helps someone!

[note: this issue has been with me for a long while. This post was first written in May 2002, but the underlying issue has been consistent since them. I have made some updates (and fixed typos) to this post in May 2023.

Configuring PowerShell 7 With Group Policy

Introduction

Group policy is a feature of Windows Server Active Directory which automagically deploys groups of policies to users and computers. A policy is some computer setting you wish to enforce, such as which screen saver to use, what desktop background to use, or what the default execution policy should be. 

Windows PowerShell has for a while allowed you to set certain group policies to control how PowerShell works. Windows PowerShell 5.1 provides five specific policy settings. PowerShell 7 provides all the Windows PowerShell policies, plus one more. I describe each policy below. 

One neat feature of PowerShell 7 is that you can enable independent policy values for PowerShell 7 and Windows PowerShell. Or you can enable a  PowerShell 7 policy and take any values, such as the Execution Policy from Windows PowerShell policies.

You can set policies for a computer or a user. I base the Group Policy Editor and the PowerShell cmdlets make use of administrative templates stored C:\Windows\PolicyDefinitions folder in your DC (or to a central policy store shared on all DCs). The templates include an XML file that defines a policy or set of policy, which has the extension ADMX. Each template definition a localised set of strings stored in an ADML file. The ADMX file contains pointers to strings defined in the ADML file. Having both files enables the GP Editor to use localised language.

After you apply a policy, the group policy agent on the computer creates entries in the user or computer's registry policy area. You can see the policy if you use registry editor or PowerShell and navigate to HKCU:\Software\Policies\Microsoft\PowerShellCore  for user settings, or you can navigate to HKLM:\Software\Policies\Microsoft\PowerShellCore for computer settings.

The group policy agent runs each time the computer starts, and each time a user logs on. The agent also runs at intervals of 2 hours (less a random time up to 30 minutes). To immediately invoke the agent, you can use the gpupdate.exe console application. 

NOTE: This article started out simple, but as it grew, I've had to split it into two. In this article, I look at the GPO settings you can specify and which registry key(s) and value entries they use. Armed with this information, the next article looks at how you set each policy using PowerShell 7. 

PowerShell Group Policy settings

There are six PowerShell 7 related group policies you can deploy

• Execution Policy
• Console Session Configuration (new with PowerShell 7)
• Module Logging
• Script block logging
• Transcription
• Updatable Help.

Setting Policies

Before setting any policy, note that each policy sets one or more keys and value entries in the user or computer section of the registry. Most policies are a single registry key with one or more value entries. Let's look at the individual policies, and the registry keys and value entries deployed.

Since these policies are registry settings, you can deploy them using group policy, or you can set the registry key manually. You could use a .REG file containing the settings you want to apply to a host, or use PowerShell to set the keys and value entries needed for the policy. 

In the following section, I show how you can create the key and value entries needed for user settings of a policy. In the next article, you can see how to create and deploy the group policy settings using PowerShell cmdlets. 

Execution Policy

The Execution Policy policy defines a particular execution policy or whether to disallow all script execution. I would expect you to know that PowerShell's Execution Policy is not a security barrier. More a speed bump to stop a very in-experienced user to do something silly. I would hope you know it is pretty easy to work around a restricted execution policy. 

For most organisations, an Execution Policy of RemoteSigned is probably sufficient. Your IT Professionals might want to log in and have full access without having to work around a more restricted policy, so setting the policy to Unrestricted for members of the IT department might be useful. Mileage varies. 

The registry key set by the policy is
HKCU:\Software\Policies\Microsoft\PowerShellCore

This is for the user policy setting. If you are setting computer policies, you can use HKLM: instead, which is the case with all the other policy keys used.

There are two value entries for this policy:

• EnableScripts - This value entry enables this policy. It is of type dword and has a value of 1 if the policy is enabled 
• ExecutionPolicy - This is the execution policy to be applied. It is a string and can contain any valid PowerShell execution policy 

Here is how to set this policy using PowerShell, in HKCU:

# 1. Set Execution Policy

# Create Key

$Key = 'HKCU:\Software\Policies\Microsoft\PowerShellCore'

if (Test-Path $Key) {

  Write-Verbose "Registry Key exists [$key]"

}

Else {

  Write-Verbose "Creating registry key [$key]"

  New-Item -Path $Key

}

# Set value for execution policy 

Write-Verbose 'Setting Execution Policy On'

$CVHT1   = @{ Path  = $Key

              Name  = 'EnableScripts'

              Type  = 'Dword'

              Value = 1 }

Set-ItemProperty @CVHT1

Write-Verbose 'Setting Execution Policy to Unrestricted'

$CVHT2   = @{ Path  = $Key

              Name  = 'ExecutionPolicy'

              Type  = 'String'

              Value = 'Unrestricted'}

Set-ItemProperty @CVHT2

Console Session Configuration 

The Console Session Configuration specifies a remoting configuration endpoint to use. Remote sessions then run against that endpoint. You can specify any endpoint, include a JEA endpoint.

The  registry key set by this policy is:

HKCU:Software\Policies\Microsoft\PowerShellCore\ConsoleSessionConfiguration

There are two value entries for this key:

• EnableConsoleSessionConfiguration - this entry enables the policy. It is of type dword and has a value of 1 to indicate that the policy is applied.
• ConsoleSessionConfigurationName - this entry is the remoting endpoint that is used. It is a string, such as "PowerShell.7". 

Module Logging

Module Logging requires PowerShell to log module usage and for which modules. If you set this policy, PowerShell logs pipeline execution events for the specified modules to the PowerShell Core event log. 

Be careful with this setting as you could generate a lot of log entries. If you are going to log module events, have a strategy for reviewing the events generate

This policy requires the use of two registry keys. The first is:

HKCU:\Software\Policies\Microsoft\PowerShellCore\ModuleLogging 

There is one value entry for this key:

• EnableModuleLogging - This entry enables the policy. It is of type dword and has a value of 1 to indicate that the policy is applied.

The second key used by this policy is:

HKCU:\Software\Policies\Microsoft\PowerShellCore\ModuleLogging\ModuleNames

There are multiple value entries you can specify for this key, each one a module for logging. Each value entry has a name and a value of the module name. So if you wanted to log events for the FOO42 module, you would have a registry value name FOO42 with an associated value of FOO42. You can specify as many modules as you like.

Script block logging

You use the Script block logging policy to turn on PowerShell's logging of any important script blocks.  PowerShell does not log all script blocks only those that change something. Nevertheless, this policy can generate a lot of logging. In Windows PowerShell, setting this policy could result in performance degradation although has been much improved in PowerShell 7.

The  registry key used by this policy is:

HKCU:Software\Policies\Microsoft\PowerShellCore\ScriptBlockLogging

There is one value entry under this policy.

• EnableScriptBlockLogging - This entry enables the policy. It is of type dword and has a value of 1 to indicate that the policy is applied.

Transcription

The Transcription policy allows you to automatically create a transcript of every PowerShell session. 

The  registry key set by this policy is:

HKCU:\Software\Policies\Microsoft\PowerShellCore\Transcription

There are two value entries used by this policy:

• EnableTranscripting - This entry enables the policy. It is of type dword and has a value of 1 to indicate that the policy is applied.
• OutputDirectory - this is a string and specifies the folder in which PowerShell writes session transcripts. 

Updatable Help

The Updatable Help policy allows you to configure a default value fore Update-Help's parameter SourcePath. If you enable the policy, Update-Help uses the setting as a default location for new help information. You can always override this by specifying a different value for the source path when you run Update-Help.

The  registry key set by this policy is:

HKCU:\Software\Policies\Microsoft\PowerShellCore\UpdatableHelp

There are two value entries under this policy.

• EnableUpdateHelpDefaultSourcePath - This entry enables the policy. It is of type dword and has a value of 1 to indicate that the policy is applied.
• DefaultSourcePath -This parameter defines the location to be used as the default source path and is of type String. You would probably use a network share. Irrespective,  you should note that in this string you must escape any back slash character with aqn additional backslash. If the default source path is \\dc1\help, you see it as "\\\\dc1\\help".

Using Windows PowerShell Settings

As I mentioned above, you can set a policy for PowerShell 7 use but use whatever values you set for Windows PowerShell policy. In most or at least many cases, if you have Windows PowerShell policies set, you probably want the same policies set for PowerShell usage as well.

To set any of the 6 polices to apply to PowerShell 7, but using the Windows PowerShell session you set two value entries in the six polices review above. The first is the Enable value entry, which enables that policy. Then you add another value entry:

• UseWindowsPowerShellPolicySetting - This value entry indicates that the policy details should come from Windows PowerShell policies. It is of type dword and has a value of 1.

Summary

We looked at the six policies you can set and which registry key and value entries used by each policy. In the next article, I show you how to use the Group Policy cmdlets and PowerShell 7 to create and deploy these policies.