Virtual Apps and Desktops – VDAs will remain turned off after Restart Schedule

Reading Time: 3 minutes

Have you ever experienced that some of the Virtual Desktop Agents remain turned off after a scheduled reboot via the Delivery Group? This is something I could notice several times of the last years in various environments. Most of the time this only affects a few machines and it is not a big deal. During a current customer engagement I could see the same issues again but sometimes over 40% of the VDAs have been affected (not every day). This can happen if there is a boot storm occurring (PVS servers are exhausted) or the VDA registration process is not happening after 20 minutes (CTX223434). That was not the case in this specific environment. The interesting thing is that we never saw a power on action for the problem VDAs in VMware vCenter. You will find a lot of people in Citrix Discussions or Reddit who are struggling with the same problem but seems like there is not a real solution for this behavior.

That is why I came up with the idea to write a PowerShell script which needs to be run after the restart schedule which is doing the following things:

Checking for all faulty VDAs for the given Machine Catalog (PowerState: Off, RegistrationState: Unregistered, MaintenanceMode: False)
Powering on each VDA with a delay of 15 seconds (New-BrokerHostingPowerAction)
Sends an email notification (if enabled)

The great thing is we do not need to work with VMware PowerCLI, because we can start the VDAs with the integrated Citrix PowerShell SDK. This makes it work for all the environments out there, doesn’t matter if you are running Citrix Hypervisor, VMware, AHV etc.

Hope this is helping some people and I appreciate some feedback 🙂

Citrix_VDA_BootFix_V1.2.ps1

#==========================================================================
#
# Check for Unregistered VDAs after Scheduled Reboot
#
# AUTHOR: Julian Mooren
# DATE  : 13.08.2020
#
# VERSION: 1.2 
# - Added Mail Notifications
# - Improved Error Handling + Logging
# - Excluding VDAs in Maintenance Mode
#        
#==========================================================================

# Custom variables [edit]
$BaseLogDir = "C:\Logs"                              
$PackageName = "Citrix_VDA_Startup"    
$Catalog = "FARMP10"
$SendMail = $true
$SMTPSERVER = "EX-01.flashmob-saulgau.de"
$SendMailFrom = "citrix@flashmob-saulgau.de"
$SendMailTo= "julian.mooren@flashmob-saulgau.de"


# Global variables
$date = Get-Date -Format "dd-MM-yy_hh-mm"
$LogDir = (Join-Path $BaseLogDir $PackageName).Replace(" ","_")
$LogFileName = "$($PackageName)_$($Catalog)_$date.log"
$LogFile = Join-path $LogDir $LogFileName
[int]$BrokenVDA = "0"

# Create the log directory if it does not exist
if (!(Test-Path $LogDir)) { New-Item -Path $LogDir -ItemType directory | Out-Null }

Start-Transcript $LogFile | Out-Null

# Import the Citrix Powershell Snapins
Write-Host "I" "Import the Citrix Powershell Snapins"
try {
    Add-PSSnapin Citrix*
    Write-Host "S" "The Citrix Powershell Snapins was imported successfully"
} catch {
    Write-Host "E" "An error occurred trying to import the Citrix Powershell Snapins (error: $($Error[0]))"
    Exit 1
}


# Check for Machine Catalog
Write-Host "I" "Checking for Existence of Machine Catalog: $($Catalog)"
if((Get-BrokerDesktop -CatalogName $Catalog) -eq $null ) 
{
Write-Host "E" "Machine Catalog: $($Catalog) is not available" 
Write-Host "I" "Exiting Script....."
exit 1
}

else {Write-Host "S" "Machine Catalog: $($Catalog) is available"}


$VDAs = Get-BrokerDesktop -CatalogName $Catalog -RegistrationState Unregistered -PowerState Off -InMaintenanceMode $false  | Select HostedMachineName

if($VDAs -ne $null)
{
foreach($VDA in $VDAs)
{

Write-Host "I" "$($VDA.HostedMachineName) was powered off. Booting up the machine...."
try
{
New-BrokerHostingPowerAction -MachineName $VDA.HostedMachineName -Action TurnOn | Out-Null
$BrokenVDA++
}
catch 
{
Write-Host "E" "Problem with turning on the VDA: $($VDA.HostedMachineName)" 
Write-Host "I" "Exiting Script....."
exit 1
}


Write-Host "I" "Waiting for 15 seconds before starting the next VDA...."
Sleep 15 

}
}

else {Write-Host "I" "All VDAs are registered. Nothing to do....."}


# Send E-Mail Notification

if($SendMail -eq $true)
{

Write-Host "I" "Preparing E-Mail Notification."

$NewMsg = new-object Net.Mail.MailMessage
$SMTP = new-object Net.Mail.SmtpClient($SMTPSERVER)

$NewMsg.IsBodyHTML = $true
$NewMsg.From = $SendMailFrom
$NewMsg.To.Add($SendMailTo)
$NewMsg.Subject = "Citrix VDA Health Status - Catalog $($Catalog)"
$NewMsg.Body = @"
<p>The Machine Catalog $Catalog had $BrokenVDA unhealthy VDAs.</p>
"@

try
{
$SMTP.Send($NewMsg)
Write-Host "S" "E-Mail Notification was send successfully"
}
catch 
{
Write-Host "E" "(error: $($Error[0]))"
exit 1
}


}


Stop-Transcript

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

#==========================================================================

# Check for Unregistered VDAs after Scheduled Reboot

# AUTHOR: Julian Mooren

# DATE : 13.08.2020

# VERSION: 1.2

# - Added Mail Notifications

# - Improved Error Handling + Logging

# - Excluding VDAs in Maintenance Mode

#==========================================================================

# Custom variables [edit]

$BaseLogDir = "C:\Logs"

$PackageName = "Citrix_VDA_Startup"

$Catalog = "FARMP10"

$SendMail = $true

$SMTPSERVER = "EX-01.flashmob-saulgau.de"

$SendMailFrom = "citrix@flashmob-saulgau.de"

$SendMailTo= "julian.mooren@flashmob-saulgau.de"

# Global variables

$date = Get-Date -Format "dd-MM-yy_hh-mm"

$LogDir = (Join-Path $BaseLogDir $PackageName).Replace(" ","_")

$LogFileName = "$($PackageName)_$($Catalog)_$date.log"

$LogFile = Join-path $LogDir $LogFileName

[int]$BrokenVDA = "0"

# Create the log directory if it does not exist

if (!(Test-Path $LogDir)) { New-Item -Path $LogDir -ItemType directory | Out-Null }

Start-Transcript $LogFile | Out-Null

# Import the Citrix Powershell Snapins

Write-Host "I" "Import the Citrix Powershell Snapins"

try {

Add-PSSnapin Citrix*

Write-Host "S" "The Citrix Powershell Snapins was imported successfully"

} catch {

Write-Host "E" "An error occurred trying to import the Citrix Powershell Snapins (error: $($Error[0]))"

Exit 1

}

# Check for Machine Catalog

Write-Host "I" "Checking for Existence of Machine Catalog: $($Catalog)"

if((Get-BrokerDesktop -CatalogName $Catalog) -eq $null )

{

Write-Host "E" "Machine Catalog: $($Catalog) is not available"

Write-Host "I" "Exiting Script....."

exit 1

}

else {Write-Host "S" "Machine Catalog: $($Catalog) is available"}

$VDAs = Get-BrokerDesktop -CatalogName $Catalog -RegistrationState Unregistered -PowerState Off -InMaintenanceMode $false | Select HostedMachineName

if($VDAs -ne $null)

{

foreach($VDA in $VDAs)

{

Write-Host "I" "$($VDA.HostedMachineName) was powered off. Booting up the machine...."

try

{

New-BrokerHostingPowerAction -MachineName $VDA.HostedMachineName -Action TurnOn | Out-Null

$BrokenVDA++

}

catch

{

Write-Host "E" "Problem with turning on the VDA: $($VDA.HostedMachineName)"

Write-Host "I" "Exiting Script....."

exit 1

}

Write-Host "I" "Waiting for 15 seconds before starting the next VDA...."

Sleep 15

}

else {Write-Host "I" "All VDAs are registered. Nothing to do....."}

# Send E-Mail Notification

if($SendMail -eq $true)

{

Write-Host "I" "Preparing E-Mail Notification."

$NewMsg = new-object Net.Mail.MailMessage

$SMTP = new-object Net.Mail.SmtpClient($SMTPSERVER)

$NewMsg.IsBodyHTML = $true

$NewMsg.From = $SendMailFrom

$NewMsg.To.Add($SendMailTo)

$NewMsg.Subject = "Citrix VDA Health Status - Catalog $($Catalog)"

$NewMsg.Body = @"

<p>The Machine Catalog $Catalog had $BrokenVDA unhealthy VDAs.</p>

try

{

$SMTP.Send($NewMsg)

Write-Host "S" "E-Mail Notification was send successfully"

}

catch

{

Write-Host "E" "(error: $($Error[0]))"

exit 1

}

Stop-Transcript

Mike,
One possible solution would be to create a function that perform those tasks. Then call the function passing, in my case, the Desktop Group name as parameter . I this way you can perform these operation against the Desktop Groups that you interested in monitoring. You should explore the various Get-BrokerDesktop options to see if you can gather a collection of VDAs that make sense to you, then put that collection on the foreach cycle.

Here I created the function
Function CheckingVDAPoweredOff ($DesktopGroupName)
{Write-Host “Checking VDAs DesktopGroupName $DesktopGroupName”
$VDAs = Get-BrokerDesktop -DesktopGroupName $DesktopGroupName -RegistrationState Unregistered -PowerState Off -InMaintenanceMode $false | Select HostedMachineName

if($VDAs -ne $null)
{
foreach($VDA in $VDAs)
{Write-Host “$($VDA.HostedMachineName) was powered off. Booting up the machine….”
try
{
New-BrokerHostingPowerAction -MachineName $VDA.HostedMachineName -Action TurnOn | Out-Null
$BrokenVDA++
}
catch
{
Write-Host “E” “Problem with turning on the VDA: $($VDA.HostedMachineName)”
Write-Host “I” “Exiting Script…..”
exit 1
}

Write-Host “Waiting for 15 seconds before starting the next VDA….”
Sleep 15
}
}

else {Write-Host “All VDAs are registered. Nothing to do…..”}
}

#here I am setting the desktop Group name that I want to check
$DesktopGroupName = “APPS01”

#Here I am calling the function passing the Desktop Group Name
CheckingVDAPoweredOff $DesktopGroupName

$DesktopGroupName = “APPS02”
CheckingVDAPoweredOff $DesktopGroupName

11 comments

Roy Davis says:

13 August 2020 at 13 01 03 08038

Nice job

Frode Eggen says:

18 August 2020 at 9 09 06 08068

The script is working fine when I starting it in Powershell. But when I set it up as an Task Schedule is not working. Hav you tried to run the script from Task Schedule?

1. citrixguyblog says:
  
  18 August 2020 at 9 09 11 08118
  
  Did you configure the “log on as a batch job” setting?
  Screenshot: https://imgur.com/a/irreaNj
  
Frode Eggen says:

18 August 2020 at 10 10 25 08258

Thanks.
I had to set the Action in Task Schedule to:
Action: Start a Program
Program/Script: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Add arguments: -Executionpolicy bypass -file C:\thescript.ps1

That did the trick.

But it seems to run the script only work with Domain Admins role.

Mike Trajano says:

8 April 2021 at 1 01 16 04164

I have multiple Machine Catalogs so when I add them to the script $Catalog = “FARMP10″,”FARMP11” it errors out on the script with this error:
Stop-Transcript : An error occurred stopping transcription: The host is not currently transcribing.
At line:103 char:1
+ Stop-Transcript
+ ~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Stop-Transcript], PSInvalidOperationException
+ FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.StopTranscriptCommand

Any suggestions on how to add multiple Machine Catalogs to the script?

1. citrixguyblog says:
  
  8 April 2021 at 11 11 31 04314
  
  Hello Mike. I would suggest you to use an input paramater. Then you can use the same script and create different schedulded tasks for specifing the needed catalogs.
  Param ([parameter(Mandatory)] [String]$Catalog)
  
Mike says:

8 April 2021 at 15 03 36 04364

Im kind a new to PS. Where in your script would I enter the parameter so that I can enter multiple Machine Catalogs?

Thank you.

1. citrixguyblog says:
  
  8 April 2021 at 15 03 52 04524
  
  Paste the param to line 14 in the script.
  Run the script with the following command: Citrix_VDA_BootFix_V1.2.ps1 -Catalog ‘Name”
  
mike says:

9 April 2021 at 15 03 45 04454

Hello its me again. I did add the command on line 14 and it worked but the problem still exist which it only checks one Machine Catalog and not multiple. What I wanted to do with the Script is to check all the Machine catalogs that has unregistered machines and if unregistered to power them all up. Thank you for helping me out. I hate it when a scheduled reboot powers off a few servers from different Machine Catalogs and I have to power them on manually.

1. Marco Cortigiani says:
  
  2 December 2021 at 18 06 11 121112
  
  Mike,
  One possible solution would be to create a function that perform those tasks. Then call the function passing, in my case, the Desktop Group name as parameter . I this way you can perform these operation against the Desktop Groups that you interested in monitoring. You should explore the various Get-BrokerDesktop options to see if you can gather a collection of VDAs that make sense to you, then put that collection on the foreach cycle.
  
  Here I created the function
  Function CheckingVDAPoweredOff ($DesktopGroupName)
  {Write-Host “Checking VDAs DesktopGroupName $DesktopGroupName”
  $VDAs = Get-BrokerDesktop -DesktopGroupName $DesktopGroupName -RegistrationState Unregistered -PowerState Off -InMaintenanceMode $false | Select HostedMachineName
  
  if($VDAs -ne $null)
  {
  foreach($VDA in $VDAs)
  {Write-Host “$($VDA.HostedMachineName) was powered off. Booting up the machine….”
  try
  {
  New-BrokerHostingPowerAction -MachineName $VDA.HostedMachineName -Action TurnOn | Out-Null
  $BrokenVDA++
  }
  catch
  {
  Write-Host “E” “Problem with turning on the VDA: $($VDA.HostedMachineName)”
  Write-Host “I” “Exiting Script…..”
  exit 1
  }
  
  Write-Host “Waiting for 15 seconds before starting the next VDA….”
  Sleep 15
  }
  }
  
  else {Write-Host “All VDAs are registered. Nothing to do…..”}
  }
  
  #here I am setting the desktop Group name that I want to check
  $DesktopGroupName = “APPS01”
  
  #Here I am calling the function passing the Desktop Group Name
  CheckingVDAPoweredOff $DesktopGroupName
  
  $DesktopGroupName = “APPS02”
  CheckingVDAPoweredOff $DesktopGroupName
  
Jason says:

6 September 2022 at 14 02 26 09269

Has anyone had this same issue but it completely breaks the entire Delivery Group? We are seeing where a VDA stays in a powered off state and when this happens the DG no longer takes any sessions until the VDA is put in MM or powered on.

Leave a Reply Cancel reply