Previous Post in Series: Part 10: Deploy and Configure SDN Software Load Balancer Service
Welcome to part 11 of my Server 2016 Features Series, I thought it was worth posting on this one because I managed to brick my SDN environment (in my lab) because I just went for it without following any real process…COME ON, we’ve all done it
There won’t be much new in here that Greg Cusanza at MS hasn’t covered in the great documentation he uploaded recently, so if nothing else, you’ll get a similar experience to that but with a slight Scottish flavour
At the time of writing, backup and restore of SDN was introduced in the most recent Cumulative Update, and as I alluded to earlier, getting this piece wrong can break your SDN environment, so it’s important that you get it right.
OK, that’s enough scaremongering…let’s get to it.
You’re probably looking at the title of this post and thinking, “Shouldn’t it be called Backup, Update and Restore?” Normally you’d be right too but as I’ve had SDN deployed for a while now, the version I’m running doesn’t have the backup and restore capabilities. I therefore have to update the environment before I can back it up.
Backing up your SDN infrastructure isn’t just a case of backing up the Virtual Machines as this wouldn’t ensure that quorum is maintained across your Network Controller nodes, this will be explained in more detail later in the guide.
Updating Your SDN Infrastructure
All servers that make up your SDN infrastructure should be kept at the same patch level, these roles should be as follows:
- Management Console Servers – Including SCVMM
- Network Controller VMs
- Software Load Balancer VMs
- SDN enabled Hyper-V hosts
- RAS Gateway VMs
Management Console Servers
The first thing we’ll want to update is the server you manage your Network Controllers from, i.e. the server where you installed the RSAT-NetworkController feature. For my lab, this is my SCVMM server which is fortuitous as this server should be kept at the same patch level (as mentioned above) as the rest of the servers that make up your SDN infrastructure.
Just to be on the safe side, the first thing I did after updating SCVMM was to carry out a few ping tests from within one of my SDN enabled tenant VMs. The two you can see in the screenshot below are as follows:
On the left: An internal ping between two VMs on the same VM Network subnet
On the right: An external ping to a website on port 80. I was forced to use TCP ping as outbound ping (UDP) is locked down on my lab firewall and I haven’t gotten round to fixing it
Everything looks good so far…so that’s good!
Network Controller VMs
If there are any mistakes to be made during the SDN update process, it’s here
We want to work through patching each of our Network Controller nodes in turn, however after each node we want to make sure it’s fully online within the cluster before continuing onto the second node. You can check the status of all nodes within the cluster by running the following PowerShell command (elevated) from an NC VM:
Get-NetworkControllerNode
You can also run it from a remote server that has the RSAT-NetworkController tools installed by modifying the command slightly:
$Credential = Get-Credential #Enter the domain\username and password of an account in your NC management group in AD Get-NetworkControllerNode -ComputerName MultiNodeNC.domain.com -Credential $Credential
The output below shows the status of my NCs just after node 1 has been patched and rebooted, note that it’s currently showing as “Down”.
We’ll need to wait a few minutes until the status changes to “Up” before continuing on to node 2.
The same process should be repeated after patching nodes 2 and 3.
NOTE (Reiteration): CHECK THE STATUS OF EACH NODE AFTER PATCHING USING THE INSTRUCTIONS ABOVE
Once you have run through the above steps for each Network Controller node and all are back up and in a healthy state, one last step is required to update the microservices running within the Network Controller cluster. This job is kicked off by the system within roughly an hour, however this task can be initiated by ourselves to force the update to take place immediately.
Run the following within an elevated PowerShell console on one of your Network Controller nodes:
NOTE: This step can take a while to complete.
Update-NetworkController
Now that our Network Controller cluster has been updated it’s a good time to redo our Tenant VM pings tests (from earlier in this guide) to make sure everything is still working as expected…assuming we’re all good, let’s move on to our Software Load Balancers.
Software Load Balancer VMs
Nothing of note to be aware of here guys, just update each SLB VM in turn to maintain availability of the service throughout.
The only thing I noticed during the SLB update cycle was a single slow ping response when one of the SLB VMs was rebooting…not too bad
Your mileage here will vary of course depending on your networking and the responsiveness or the external source you’re pinging.
Hyper-V Hosts and RAS Gateway VMs
As mentioned in an earlier guide, I’ve not yet done much with the RAS Gateway role but have included it here as there are some best practice recommendations when updating the VMs. This section assumes a 3 node Hyper-V cluster each hosting a RAS Gateway VM.
Although the RAS Gateway VMs CAN be Live Migrated, doing so will drop any tenant connections currently using that VM as they’re failed over to a surviving Gateway. It’s for this reason that I’ve set my RAS Gateways to NOT live migrate, they will instead go down with their host (salute).
We can limit tenant downtime by patching a Hyper-V host and any RAS Gateway VMs it contains at the same time. Keep in mind that if you have 3 RAS Gateways, 1 will always be in standby and the other two active. Following the instructions below will result in the least tenant impact (in this scenario):
NOTE: Before this piece of work is implemented, I would recommend making your tenants aware of it and scheduling it for a time where it would have the least impact.
- Pick the Hyper-V host that’s running your standby RAS Gateway (We’ll call this RASGW1). Evacuate it of Live Migration capable VMs – NO TENANT IMPACT.
- Install updates on the RAS Gateway VM and reboot if required – NO TENANT IMPACT.
- Install updates on the host and reboot as required – NO TENANT IMPACT.
- Repeat the process for your second Hyper-V host and the RAS Gateway (We’ll call this RASGW2) hosted on it – TENANT CONNECTIONS FAILED OVER TO STANDY GATEWAY (RASGW1).
- Repeat the process for the 3rd Hyper-V host and its RAS Gateway (We’ll call this RASGW3) – TENANT CONNECTIONS FAILED OVER TO STANDBY GATEWAY (RASGW2)
If you have a cluster comprised of more than 3 hosts and/or more RAS Gateway VMs, continue the above process patching Hyper-V hosts running Gateway VMs first, and lastly, hosts without Gateway VMs.
NOTE: Remember to reimport the NetworkController PowerShell module on any server you plan to target your NCs from, otherwise you’re NC backup commands will fail.
Import-Module NetworkController
…and that’s it, your SDN infrastructure should now be fully up to date and was hopefully done with the absolute minimum of downtime for your tenant workloads. The above process should be followed any time you’re updating this environment.
Backup your SDN Infrastructure
As mentioned earlier in the guide, in addition to backing up the Network Controller VMs, we’ll also want to back up the NC database periodically. This is for much the same reason that we waited till all NC nodes were showing as “Up” when working through the VM update cycle. By taking a backup of this database, we can ensure that quorum between the NC nodes is maintained and can be restored to a healthy state in the event of a disaster, or for example not following the update process for the NC nodes themselves
Before continuing, you’ll need access to a SMB share as a user with Read/Write permissions to both that share and the file system. I’ll be using my lab SCVMM library share as it’s already set up with the required access…just make sure that you back up the contents of this share on a schedule. An idea would be to create a SMB share on a VM that you’re already backing up…but I’ll leave that up to you guys
NOTE: If you installed your Network Controller using a Group Managed Service Account (GMSA), you can use that here too.
Let’s step through it:
- Backup the Network Controller VMs using the backup tool of your choice. This gives you the ability to do a full SDN infrastructure restore if required and also grabs the required certificates for decrypting the database…so that’s nice.
- Stop the SCVMM service and take a backup of the SCVMM SQL database. Stopping the SCVMM service will ensure that no changes are made to the SCVMM database while the Network Controller database is being backed up, therefore making them inconsistent.
NOTE: Do NOT restart the SCVMM service until the Network Controller DB backup has been completed.
- Take a backup for the Network Controller database
- Start the SCVMM service
I’ve put together little PowerShell script for running through the entire process, here are a few notes on it:
- The credentials you use for the SCVMM SQL DB backup and for targeting your Network Controller service will be stored securely and can only be unencrypted (used by the script) if the script is launched by the user that stored them securely in the first place.
- You will be asked to enter a password for both of the above accounts on the first run of the script only.
- A user is created on the Network Controller for accessing the SMB share where the NC backup will be stored. On the first run of the script uncomment the following section “UNCOMMENT AND POPULATE USERNAME AND PASSWORD…” and enter a username and password that has access to the SMB share. This again only needs to be done once and the commented out section can be removed or commented out again on subsequent runs of the script.
- The script will keep checking the status of the backup job and will only continue when it has completed successfully (which in my experience doesn’t take long).
Things you’ll need to modify before running:
- $URI variable should be set to the REST URL for your Network controller e.g. “https://MultiNodeNC.domain.com”
- $NCUsername should be a user within your NC Management Group in AD e.g. “DOMAIN\NCAdmin”
- $SQLBKPUsername should be a user that has permissions to back up your SCVMM SQL DB
- $Server.Databases (within the $SQLScriptBlock) should be the name of your SCVMM SQL DB e.g. [‘SCVMMDB’]
- Invoke-Command -ComputerName should be the FQDN of the SQL server that hosts your SCVMM DB
- $BackupProperties.BackupPath should be the UNC path you’ve created for storing the Network Controller back files e.g. “\\MyShare.domain.local\Backups\NetworkController\$BackupTime“
Clear-Host
# A function for checking the status of the Network controller backup job and advising the user when complete
Function CheckBackupProgress
{
$BackupProgress = Get-NetworkControllerBackup -ConnectionUri $URI -Credential $Credential -ResourceId $Backup.ResourceId
if ($BackupProgress.Properties.ProvisioningState -eq "Succeeded")
{
Write-Host "Network Controller Backup Job Complete" -ForegroundColor Green
}
else
{
Write-Host "Backup job still running - waiting 15 seconds before continuing" -ForegroundColor Yellow
Start-Sleep -Seconds 15
CheckBackupProgress
}
}
# Network Controller URI - the "serverurl" from your SCVMM Network Controller network service
$URI = "<a href="https://MultiNodeNC.DOMAIN.COM">https://MultiNodeNC.DOMAIN.COM</a>"
# Create or Get Credential object for NCAdmin user
$NCUsername = "DOMAIN\NCAdmin"
$NCPasswordFile = "C:\Temp\NCAdminPWFile.txt"
$Credential = Get-ChildItem -Path $NCPasswordFile -ErrorAction SilentlyContinue
If ($Credential -eq $null)
{
(Get-Credential -UserName $NCUsername -Message "Please enter the password for the NCAdmin user").Password | ConvertFrom-SecureString | Out-File "C:\Temp\NCAdminPWFile.txt"
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $NCUsername, (Get-Content $NCPasswordFile | ConvertTo-SecureString)
else
}
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $NCUsername, (Get-Content $NCPasswordFile | ConvertTo-SecureString)
# Create of Get Credential object for SQL Backup user
$SQLBKPUsername = "DOMAIN\SQLBackupUser"
$SQLBKPPasswordFile = "C:\Temp\SQLBKPPWFile.txt"
$SQLBKPCredential = Get-ChildItem -Path $SQLBKPPasswordFile -ErrorAction SilentlyContinue
If ( $SQLBKPCredential -eq $null)
{
(Get-Credential -UserName $SQLBKPUsername -Message "Please enter the password for the SQL Backup user").Password | ConvertFrom-SecureString | Out-File "C:\Temp\SQLBKPPWFile.txt"
$SQLBKPCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $SQLBKPUsername, (Get-Content $SQLBKPPasswordFile | ConvertTo-SecureString)
}
$SQLBKPCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $SQLBKPUsername, (Get-Content $SQLBKPPasswordFile | ConvertTo-SecureString)
# Get or Create Credential object for File share user
$ShareUserResourceId = "BackupUser"
$ShareCredential = Get-NetworkControllerCredential -ConnectionURI $URI -Credential $Credential | Where {$_.ResourceId -eq $ShareUserResourceId } -Verbose
# UNCOMMENT AND POPULATE USERNAME AND PASSWORD DETAILS ON FIRST RUN OF THE SCRIPT ONLY
<# If ($ShareCredential -eq $null) { $CredentialProperties = New-Object Microsoft.Windows.NetworkController.CredentialProperties $CredentialProperties.Type = "usernamePassword" $CredentialProperties.UserName = "Username" $CredentialProperties.Value = "Password" $ShareCredential = New-NetworkControllerCredential -ConnectionURI $URI -Credential $Credential -Properties $CredentialProperties -ResourceId $ShareUserResourceId -Force } #>
# Stop SCVMM Service
Stop-Service SCVMMService -Verbose
# Create SCVMM SQL DB Backup
$SQLScriptBlock = {
Import-Module SQLServer
$Server = New-Object 'Microsoft.SqlServer.Management.SMO.Server' $inst
$BackupDir = $Server.Settings.BackupDirectory
$ServerName = $Server.Name
$DB = $Server.Databases['YOURDBNAME']
$DBName = $DB.Name
$SQLBackupTime = (get-date).ToString("s").Replace(":", "_")
$BackupFile = "$BackupDir\$($DBName)_db_$($SQLBackupTime).bak"
Backup-SqlDatabase -ServerInstance $ServerName -Database $DBName -BackupFile $BackupFile
}
Write-Host "Backing up SCVMM SQL Databse" -ForegroundColor Yellow
Invoke-Command -ComputerName "SQL DB FQDN" -ScriptBlock $SQLScriptBlock -Credential $SQLBKPCredential -Verbose
# Create Network Controller backup
$BackupTime = (get-date).ToString("s").Replace(":", "_")
$BackupProperties = New-Object Microsoft.Windows.NetworkController.NetworkControllerBackupProperties
$BackupProperties.BackupPath = "NC BACKUP SHARE LOCATION\$BackupTime"
$BackupProperties.Credential = $ShareCredential
Write-Host "Backing up Network Controller" -ForegroundColor Yellow
$Backup = New-NetworkControllerBackup -ConnectionURI $URI -Credential $Credential -Properties $BackupProperties -ResourceId $BackupTime -Force
# Check Status of Network Controller backup
CheckBackupProgress
# Start SCVMM Service
Start-Service SCVMMService -Verbose
Write-Host "SCVMM SQL DB and Network Controller Backups Successful" -ForegroundColor Green
Well, that should take care of the backup piece, let’s move on to the often more stressful restore piece.
Restore your SDN Infrastructure
For the purposes of this guide, I’ll only be restoring the Network Controller backup we took earlier. Depending on the situation you find yourself in though, here is the order you would follow assuming the worst had happened and you needed to do a full rebuild of your environment. These are only the steps, I’ll go through a subset of them in detail that suit the restore process we’ll follow in this guide.
Step 1: Redeploy your Hyper-V hosts/cluster and any required storage.
Step 2: Restore your Network Controller VMs, your Software Load Balancer VMs and your RAS Gateway VMs from backup.
Step 3: Stop the SLB Host agent and NC Host Agent services on all Hyper-V hosts.
Step 4: Stop your RAS Gateway VMs.
Step 5: Stop your Software Load Balancer VMs.
Step 6: Restore the Network Controller e.g. from a backup like the one we took earlier in this guide.
Step 7: Check the Network Controller restore Provisioning State to make sure the restore completed successfully.
Step 8: Restore the SCVMM VM (if required )and database from backup…remember to stop the SCVMM service or cluster role if it’s Highly Available.
Step 9: Restore workload VMs as required.
Step 10: Start the NC Host Agent and SLB Host agent services on all Hyper-V hosts.
Step 11: Start your Software Load Balancer VMs
Step 12: Start your RAS Gateway VMs
Step 13: Check the health on your SDN infrastructure.
Now hopefully you’ll never find yourself in a situation where all those steps are required, but hey, it happens right? At least now you know what to do
Let’s step through the restore process if we were ONLY to restore the SCVMM DB and Network Controller backup:
Step 1: Stop SLB Host Agent and NC Host Agent services on all Hyper-V hosts (this code was written to run from your SCVMM server). The SLB Agent is stopped first as it’s a dependant of the NC Host Agent service.
$HyperVHosts = Get-SCVMHost | ? HostCluster -match "Hyper-V Cluster FQDN"
Invoke-Command -ComputerName $HyperVHosts -ScriptBlock {
Stop-Service SLBHostAgent -Verbose
Start-Sleep -Seconds 5
Stop-Service NCHostAgent -Verbose
}
…and if you feel the need to confirm
Invoke-Command -ComputerName $HyperVHosts -ScriptBlock {Get-Service NCHostAgent,SLBHostAgent -Verbose}
Step 2: Stop your RAS Gateway VMs
The easiest way to do this is from within SCVMM as we can shut down the entire service in one go.
Navigate to “VMs and Services”, your Hyper-V cluster object and select “Services” from the ribbon
Right-click your RAS Service and select “Shut Down”, all little worryingly though is that it doesn’t prompt for confirmation…it just goes ahead and does it.
Expand the service and wait till the status of all nodes shows as “Stopped” before continuing.
Step 3: Repeat step 2 to shut down the Software Load Balancer service.
Step 4: Restore the Network Controller from the backup we took earlier in the guide.
Step 5: Restore the SCVMM SQL DB from the backup we took earlier
Step 6: Start the SCVMM Service
The following script takes care of steps 4, 5 and 6
This script was written to be run from your SCVMM server.
Open an elevated PowerShell ISE console, paste in the following code, modify it for your environment and run it:
Script Notes:
- The credentials you use for the SCVMM SQL DB backup and for targeting your Network Controller service will be stored securely and can only be unencrypted (used by the script) if the script is launched by the user that stored them securely in the first place.
- You will be asked to enter a password for both of the above account on the first run of the script only.
- A user is created on the Network Controller for accessing the SMB share where the NC backup will be stored. On the first run of the script uncomment the following section “UNCOMMENT AND POPULATE USERNAME AND PASSWORD…” and enter a username and password that has access to the SMB share. This again only needs to be done once and the commented section can be removed or commented out again on subsequent runs of the script.
NOTE: I’ve had to put in an “ipconfig /flushdns” into my “CheckRestoreProgress” function as it seems SCVMM will intermittently lose connection to the NC connection URI (MultiNodeNC.domain.com) while it’s being restored. This will also cause the odd connectivity error to be thrown while stepping through that loop, but that will stop about 30 seconds into the Network Controller restore job and the script will continue without issue. It’s therefore OK to ignore the red if it’s during that loop. I’m speaking to Microsoft to ask if that’s expected behaviour or not (I assume it is though as a connection issue doesn’t seem unreasonable during a service restore from backup). I’ve also not suppressed the connection errors as it’s possible we’d also suppress an error we want to see.
Things you’ll need to modify before running:
- $URI variable should be set to the REST URL for your Network controller e.g. “https://MultiNodeNC.domain.com“
- $NCUsername should be a user within your NC Management Group in AD e.g. “DOMAIN\NCAdmin”
- $SQLBKPUsername should be a user that has permissions to back up your SCVMM SQL DB
- $RestoreProperties.RestorePath should be the UNC path you’ve created for storing the Network Controller back files e.g. “\\MyShare.domain.local\Backups\NetworkController\2017-05-12T16_25_13 “
- $Server.Databases (within the $SQLScriptBlock) should be the name of your SCVMM SQL DB e.g. [‘SCVMMDB’]
- $Backupfile should be the name of your SCVMM SQL backup file. Just replace the text following _db_with the date of the backup you want to restore (it’s the end of your SQL backup filename)
- Invoke-Command -ComputerName should be the FQDN of the SQL server that hosts your SCVMM DB
Clear-Host
# A function for checking the status of the Network controller restore from backup job and advising the user when complete
Function CheckRestoreProgress
{
$RestoreProgress = Get-NetworkControllerRestore -ConnectionUri $URI -Credential $Credential -ResourceId $RestoreTime
if ($RestoreProgress.Properties.ProvisioningState -eq "Succeeded")
{
Write-Host "Network Controller Restore Job Complete" -ForegroundColor Green
}
else
{
Write-Host "Restore job still running - waiting 15 seconds before continuing" -ForegroundColor Yellow
ipconfig /flushdns
Start-Sleep -Seconds 15
CheckRestoreProgress
}
}
# Network Controller URI - the "serverurl" from your SCVMM Network Controller network service
$URI = "<a href="https://MultiNodeNC.DOMAIN.com">https://MultiNodeNC.DOMAIN.com</a>"
# Create or Get Credential object for NCAdmin user
$NCUsername = "DOMAIN\NCAdmin"
$NCPasswordFile = "C:\Temp\NCAdminPWFile.txt"
$Credential = Get-ChildItem -Path $NCPasswordFile -ErrorAction SilentlyContinue
If ($Credential -eq $null)
{
(Get-Credential -UserName $NCUsername -Message "Please enter the password for the NCAdmin user").Password | ConvertFrom-SecureString | Out-File "C:\Temp\NCAdminPWFile.txt"
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $NCUsername, (Get-Content $NCPasswordFile | ConvertTo-SecureString)
else
}
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $NCUsername, (Get-Content $NCPasswordFile | ConvertTo-SecureString)
# Create of Get Credential object for SQL Backup user
$SQLBKPUsername = "DOMAIN\SQLUser"
$SQLBKPPasswordFile = "C:\Temp\SQLBKPPWFile.txt"
$SQLBKPCredential = Get-ChildItem -Path $SQLBKPPasswordFile -ErrorAction SilentlyContinue
If ( $SQLBKPCredential -eq $null)
{
(Get-Credential -UserName $SQLBKPUsername -Message "Please enter the password for the SQL Backup user").Password | ConvertFrom-SecureString | Out-File "C:\Temp\SQLBKPPWFile.txt"
$SQLBKPCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $SQLBKPUsername, (Get-Content $SQLBKPPasswordFile | ConvertTo-SecureString)
}
$SQLBKPCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $SQLBKPUsername, (Get-Content $SQLBKPPasswordFile | ConvertTo-SecureString)
# Get SMB User credentials from the Network Controller and restore NC from backup
$ShareUserResourceId = "BackupUser"
$ShareCredential = Get-NetworkControllerCredential -ConnectionURI $URI -Credential $Credential | Where {$_.ResourceId -eq $ShareUserResourceId }
$RestoreProperties = New-Object Microsoft.Windows.NetworkController.NetworkControllerRestoreProperties
$RestoreProperties.RestorePath = "UNC Path to SMB share\Folder Name of NC Backup"
$RestoreProperties.Credential = $ShareCredential
$RestoreTime = (Get-Date).ToString("s").Replace(":", "_")
Write-Host "Restoring Network Controller from Backup" -ForegroundColor Yellow
New-NetworkControllerRestore -ConnectionURI $URI -Credential $Credential -Properties $RestoreProperties -ResourceId $RestoreTime -Force
# Check status of Network Controller Restore
CheckRestoreProgress
# Stop SCVMM Service
Stop-Service SCVMMService -Verbose
# Restore SCVMM SQL DB from backup
$SQLScriptBlock = {
Import-Module SQLServer
$Server = New-Object 'Microsoft.SqlServer.Management.SMO.Server' $inst
$BackupDir = $Server.Settings.BackupDirectory
$ServerName = $Server.Name
$DB = $Server.Databases['SCVMM DATABASE NAME']
$DBName = $DB.Name
$SQLBackupTime = (get-date).ToString("s").Replace(":", "_")
$BackupFile = "$BackupDir\$($DBName)_db_BACKUP DATE.bak"
Restore-SqlDatabase -ServerInstance $ServerName -Database $DBName -BackupFile $BackupFile
}
Invoke-Command -ComputerName "VMM SQL SERVER FQDN" -ScriptBlock $SQLScriptBlock -Credential $SQLBKPCredential
# Start SCVMM Service
Start-Service SCVMMService -Verbose
Write-Host "SCVMM SQL DB and Network Controller Restore Successful" -ForegroundColor Green
Step 7: Start the NC Host Agent and SLB Host agent services on all Hyper-V hosts.
$HyperVHosts = Get-SCVMHost | ? HostCluster -match "HYPER-V CLUSTER FQDN"
Invoke-Command -ComputerName $HyperVHosts -ScriptBlock {
Start-Service NCHostAgent -Verbose
Start-Sleep -Seconds 5
Start-Service SLBHostAgent -Verbose
}
Step 8: Confirm services started as expected.
Invoke-Command -ComputerName $HyperVHosts -ScriptBlock {Get-Service NCHostAgent,SLBHostAgent -Verbose}
Step 9: Start your Software Load Balancer and RAS Gateway services (See Step 2 substituting “Showdown” for “Start”)
Step 10: Leave things for a couple of minutes and then confirm you can ping both internally and externally from an SDN tenant connected VM.
Step 11: Run the Debug-NetworkControllerConfigurationState cmdlet to check the status of your SDN infrastructure.
$Credential = Get-Credential (Enter the credentials for a user within your NC Management AD group e.g. domain\NCAdmin) Debug-NetworkControllerConfigurationState -NetworkController "MultiNodeNC.domain.com" -Credential $Credential
If you see the following output, wait a few minutes and run the command again.
It should eventually look like this:
All going well, you have now successfully walked through a backup and restore of your Network Controller and SCVMM databases…nicely done!
I’m off to enjoy a long overdue beer, see you in the next post.