Saturday, January 6, 2018

Patching a VMware ESXi 5.5 host offline via CLI

One of the more common questions I’m asked every year is how to patch an ESXi host without using VMware Update Manager because it is quite common for ESXi hosts as well as vCenter to not have internet access. What I typically do when asked is point them to one of my previous posts where I demonstrated the process for a Nutanix infrastructure:

Upgrading Nutanix blades from ESXi 5.1 to 5.5
http://terenceluk.blogspot.com/2014/12/upgrading-nutanix-blades-from-esxi-51.html

… because the post includes the command for non-Nutanix deployments at the end but since I’ve received follow up questions about the process, I thought it would be a good idea to write a post that specifically demonstrates this for any ESXi deployments.

Step #1 – Identify and download desired ESXi build level

Begin by identifying the which build you’d like to patch the ESXi to:

https://kb.vmware.com/s/article/2143832

image

With the build number identified, proceed to the following URL to locate and download the patch:

https://my.vmware.com/group/vmware/patch#search

image

image

Download the patch:

image

The package you download should be a ZIP file containing files similar to the following screenshot:

vib20

index.xml

metadata.zip

vendor-index.xml

image

Step #2 – Upload patch to datastore

Upload the package to a datastore that the host you’re patching has access to by using a utility such as WinSCP:

image

Step #3 – Patch ESXi host via CLI

Proceed to either access the console directly on the server or SSH to it.

image

The command we’ll be using to install the patch will look as such:

esxcli software vib install -d <full path to patch file>

Note that it is important to specify the full path to the patch file because if you don’t then the follow error will be thrown:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

-d ESXi550-201612001.zip

[MetadataDownloadError]

Could not download from depot at zip:/var/log/vmware/ESXi550-201612001.zip?index.xml, skipping (('zip:/var/log/vmware/ESXi550-201612001.zip?index.xml', '', "Error extracting index.xml from /var/log/vmware/ESXi550-201612001.zip: [Errno 2] No such file or directory: '/var/log/vmware/ESXi550-201612001.zip'"))

url = zip:/var/log/vmware/ESXi550-201612001.zip?index.xml

Please refer to the log file for more details.

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #

image

Forgetting to include the -d will throw the following error:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

/vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

Error: Unknown command or namespace software vib install /vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

image

The patch will successfully install if the full path is specified and the output would look as such:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

-d /vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.3.95.4345813, VMware_bootbank_esx-base_5.5.0-3.100.4722766, VMware_bootbank_esx-tboot_5.5.0-3.100.4722766, VMware_bootbank_esx-ui_1.12.0-4722375, VMware_bootbank_lsi-msgpt3_00.255.03.03-2vmw.550.3.78.3248547, VMware_bootbank_misc-drivers_5.5.0-3.95.4345813, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.78.3248547, VMware_bootbank_net-igb_5.0.5.1.1-1vmw.550.2.54.2403361, VMware_bootbank_net-ixgbe_3.7.13.7.14iov-12vmw.550.2.62.2718055, VMware_bootbank_sata-ahci_3.0-22vmw.550.3.89.4179633, VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.95.4345813, VMware_locker_tools-light_5.5.0-3.92.4345810

VIBs Removed: Intel_bootbank_net-igb_5.2.7-1OEM.550.0.0.1331820, Intel_bootbank_net-ixgbe_3.21.4-1OEM.550.0.0.1331820, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.0.0.1331820, VMware_bootbank_esx-base_5.5.0-3.71.3116895, VMware_bootbank_esx-tboot_5.5.0-2.33.2068190, VMware_bootbank_lsi-msgpt3_00.255.03.03-1vmw.550.1.15.1623387, VMware_bootbank_misc-drivers_5.5.0-3.68.3029944, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.68.3029944, VMware_bootbank_sata-ahci_3.0-22vmw.550.3.68.3029944, VMware_bootbank_xhci-xhci_1.0-2vmw.550.3.68.3029944, VMware_locker_tools-light_5.5.0-3.68.3029944

VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-atiixp_0.4.6-4vmw.550.0.0.1331820, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-via_0.3.3-2vmw.550.0.0.1331820, VMware_bootbank_block-cciss_3.6.14-10vmw.550.0.0.1331820, VMware_bootbank_elxnet_10.2.309.6v-1vmw.550.3.68.3029944, VMware_bootbank_esx-dvfilter-generic-fastpath_5.5.0-0.0.1331820, VMware_bootbank_esx-xlibs_5.5.0-0.0.1331820, VMware_bootbank_esx-xserver_5.5.0-0.0.1331820, VMware_bootbank_ima-qla4xxx_2.01.31-1vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.550.0.0.1331820, VMware_bootbank_lpfc_10.0.100.1-1vmw.550.0.0.1331820, VMware_bootbank_lsi-mr3_0.255.03.01-2vmw.550.3.68.3029944, VMware_bootbank_misc-cnic-register_1.72.1.v50.1i-1vmw.550.0.0.1331820, VMware_bootbank_mtip32xx-native_3.3.4-1vmw.550.1.15.1623387, VMware_bootbank_net-be2net_4.6.100.0v-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2_2.2.3d.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2x_1.72.56.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-cnic_1.72.52.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_net-e1000_8.0.3.1-3vmw.550.0.0.1331820, VMware_bootbank_net-enic_1.4.2.15a-1vmw.550.0.0.1331820, VMware_bootbank_net-forcedeth_0.61-2vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-nx-nic_5.0.621-1vmw.550.0.0.1331820, VMware_bootbank_net-tg3_3.123c.v55.5-1vmw.550.2.33.2068190, VMware_bootbank_net-vmxnet3_1.1.3.0-3vmw.550.2.39.2143827, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.550.0.0.1331820, VMware_bootbank_qlnativefc_1.0.12.0-1vmw.550.0.0.1331820, VMware_bootbank_rste_2.0.2.0088-4vmw.550.1.15.1623387, VMware_bootbank_sata-ata-piix_2.12-10vmw.550.2.33.2068190, VMware_bootbank_sata-sata-nv_3.5-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-promise_2.12-3vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil24_1.1-1vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil_2.3-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-svw_2.3-3vmw.550.0.0.1331820, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.550.0.0.1331820, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.550.0.0.1331820, VMware_bootbank_scsi-aic79xx_3.1-5vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2fc_1.72.53.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2i_2.72.11.v55.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-fnic_1.5.0.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-hpsa_5.5.0-44vmw.550.0.0.1331820, VMware_bootbank_scsi-ips_7.12.05-4vmw.550.0.0.1331820, VMware_bootbank_scsi-lpfc820_8.2.3.1-129vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-sas_5.34-9vmw.550.3.68.3029944, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.550.0.0.1331820, VMware_bootbank_scsi-mpt2sas_14.00.00.00-3vmw.550.3.68.3029944, VMware_bootbank_scsi-mptsas_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-qla2xxx_902.k1.1-12vmw.550.3.68.3029944, VMware_bootbank_scsi-qla4xxx_5.01.03.2-6vmw.550.0.0.1331820, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.550.0.0.1331820

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #

imageimage

Proceed to vMotion the VMs on this host to another host or shut them down if they don’t need to be up and restart the host:

image

The new build number should be displayed once the host has successfully restarted:

image

Wednesday, December 20, 2017

Unable to expand virtual machine hard disk that is backed up by VMware VDP

Problem

You have a virtual machine in your vSphere environment that you would like to expand one of the hard disks but as you browse to the hard disk that needs to get expanded, you see that it is locked:

image

You’ve checked to confirm that there are no snapshots on the virtual machine:

image

Reviewing the files for the virtual machine show that there are CTK files for the disks that are locked and unable to be expanded in capacity:

image

Reviewing the disk for the VM in the vSphere Web Client displays the same behavior:

image

Shutting down and reviewing the VM’s summary tab indicate that the disks need to be consolidated:

Configuration Issues

Virtual machine disks consolidation is needed.

image

Attempting to consolidate the disks throws the following error:

Consolidate virtual machine disk files

Unable to access file since it is locked

See the error stack for details on the cause of this problem.

Error Stack

An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED.

image

image

Solution

One of the reasons why this error would be thrown is if you have this virtual machine configured to be backed up by a VMware VDP appliance and the appliance hasn’t released the disk at the end of a backup job or perhaps it did not complete properly.  To determine whether this is the case, open the settings of the VDP appliance and review whether this virtual machine’s disk is attached to it as such:

image

You can verify the disks that belong to the VDR appliance by reviewing the administration console as such:

image

To correct the problem, simply remove the disk from the VDP appliance and select the Remove from virtual machine when prompted with the Removal Options (this does not delete the disk):

image

Note that the disk will still be locked at this point:

image

Proceed to consolidate the disks:

image

The disk should be unlocked once the consolidation task completes.

Wednesday, December 13, 2017

Attempting to launch Citrix XenApp / XenDesktop 7.x desktop or application published by NetScaler fails with the error: “The connection to “XenApp Desktop” failed with status (Unknown client error 1110).”

Problem

You attempt to launch a XenApp or XenDesktop published application or desktop published by NetScaler but receive the following error message:

The connection to “XenApp Desktop” failed with status (Unknown client error 1110).

image

Solution

One of the possible causes of this error message is if you have incorrect or have not configured any STA (Secure Ticket Authority) on your NetScalers.  Navigate to NetScaler Gateway > Virtual Servers, open the properties of the virtual server, scroll down to Published Applications and ensure STA Servers are configured:

image

Monday, December 4, 2017

Attempting to run Windows update on a Windows 7 desktop fails with the error code 80248015

Problem

You attempt to run Windows updates on a Windows 7 desktop but notice that it fails with the following message:

Windows could not search for new updates

An error occurred while checking for new updates for your computer.

Error(s) found:

Code 80248015

Windows Update encountered an unknown error.

clip_image002

Solution

One of the ways to correct this issue is to navigate to the Windows directory and rename the SoftwareDistribution folder so that it would get recreated. 

C:\Windows\SoftwareDistribution

clip_image002[4]

Restart the desktop once the folder has been renamed and Windows update should now work as expected:

clip_image002[6]

Thursday, November 30, 2017

Unable to move Exchange 2016 resource mailboxes to another database from Exchange admin center

Problem

You would like to migrate Exchange 2016 resource mailboxes from one database to another but noticed that the Move Mailbox To another database is not available when you select the resource mailbox:

image

Shared mailboxes does not have this issue:

image

Solution

I’m not sure if this is by design but to move the resource mailboxes to another database via the Exchange admin center, use the migration option as such:

image

Using this wizard will allow you to select the resource mailboxes:

image

image

Friday, October 13, 2017

VMware VDP 6.1.2.19 backups fail with the error message: “An attempt made to backup a client failed because no data was found that matches the type of data that the job was configured for.”

Problem

You’ve noticed that backup jobs within VDP have been failing with the following errors:

VDP: An attempt made to backup a client failed because no data was found that matches the type of data that the job was configured for.

VDP: An unexpected error occurred with the following error code: 30983. More information may be available in the client logs which can be downloaded from the configuration application (https://<VDP hostname>:8543/VDP-configure).

image

Viewing the task failure report displays the following:

Reason: VDP: An attempt made to backup a client failed because no data was found that matches the type of data that the job was configured for.

Log file retrieved is empty.

image

Reason: VDP: An unexpected error occurred with the following error code: 30983. More information may be available in the client logs which can be downloaded from the configuration application (https://<VDP hostname>:8543/VDP-configure).

Failed to retrieve log file. This can happen if:

* Management Services were recently restarted.

* Regular log maintenance has removed old log files.

* Logs may be empty or non existent.

*An error may have occurred.

image

Solution

One of the reasons why the VDP backup jobs have failed and will continue to fail is if you have a mismatch between supported versions of VDP with the vCenter version that you are running.  In the example above, the failures started when the vCenter server was upgraded to:

Version: vCenter Server 5.5 Update 3e

Release Date: 2016-08-04

Build Number: 4180647

Installer Build Number: 4180646

image

The VDP appliance in the environment was at version 6.1.2.19:

image

The supported version for vCenter Server 5.5. Update 3e was 6.1.3 so updating the VDP appliance to 6.1.3.70 corrected the problem:

image

Thursday, October 12, 2017

ThinPrint redirected printer printing issue with VMware Horizon View Client 4.6.0

I recently ran into a strange issue with a client that had a satellite office with 4 users who use a Lenovo Windows 10 PC and VMware Horizon View client to connect back to their desktops in the main datacenter.  The satellite office is relatively new so the VMware Horizon View client installed onto the PCs were version 4.5.0 but two of the desktops recently had to be swapped out due to hardware failures and since a newer 4.6.0 client was out, I decided to go with the latest version.  Shortly after the new Lenovo PCs with the newer client were deployed, I received reports that:

  1. Attempting to print a multiple paged email from their Outlook 2010 client would insert blank pages in between pages with content and/or skip some pages all together
  2. Attempting to print attachments (e.g. PDF) on emails would send the print job but nothing prints
  3. Word documents print properly

I originally did not suspect this was related to VMware Horizon View so I sent my colleague out to have her check the drivers and any other Windows components worth reviewing.  A day goes by without a resolution so the issue was escalated to me and that was when I realized the only difference between the two non-working PCs and the other ones that were working was the newer VMware Horizon Client 4.6.0.9732:

image

Having exhausted most options that could be tried on the Windows 10 operating system, I quickly downgraded the client to 4.5.0.8090 and printing immediately functioning as expected:


image

I’m not completely sure there is a known bug for this client as the release notes here does not mention such an issue:

VMware Horizon Client 4.6 for Windows Release Notes
https://docs.vmware.com/en/VMware-Horizon-Client-for-Windows/4.6/rn/horizon-client-windows-46-release-notes.html

I hope this blog post will be able to help anyone who may come across this issue and in case anyone wants to compare operating version versions, below is the Windows OS on the Lenovo details:

image