This vCenter 6.0 error is rather common but generates an error that isn’t too helpful. It normally states that the web client service is unavailable and the error is apparent when accessing the URL. The other vCenter services are operating correctly and is running so it does not appear to have any impact on VMs.
Normally restarting the vCenter appliance (which would have to be done by SSHing into the vCenter or connecting to the ESXi host that the appliance is running on) will resolve the issue, but that is a very temporary error.
The most common reason for this type of error is that the service stops because the VM has run out of space on one of the volumes. This can be validated by running:
shell.set –enabled true
df -h
After this runs check if any volumes are at 100%, specifically: /dev/mapper/seat_vg-seat 100G 100G 0G 100% /storage/seat
What can be done at this point is you could modify the appliance and extend the partition of this disk from 100G to something higher. Once done you can run the following command to automatically expand the logical volume where the physical volume was increased:
/usr/lib/applmgmt/support/scripts/autogrow.sh
df -h
Running the df-h command again a few minutes after the autogrow runs will show that there is now sufficient disk space.
You should now be able to execute the following command to restart any stopped services:
service-control –start –all
Although this resolves the issue for now, it is not a permanent solution as the database will continue to grow until the real issue is addressed. After examining the database in my case, I found that the database was filling up constantly due to logons from some monitoring related accounts that were in place to monitor vCenter and underlying VMs.
The next step that should be done to ensure this does not reoccur is to access the settings of this vCenter server. Navigate to the vCenter Inventory List and select your vCenter. Access the database settings by following these steps:
- Select the ‘Manage’ tab
- Select ‘General’ pane and then click ‘Edit’ on the right of the settings screen
- Select the Database setting
- The default task retention and event retention days are 30 days. I reduced mine to 14 days for each to help reduce the amount of disk space used.
The final step was to reclaim the space in use by this database. Please keep in mind that I do forward all logs to LogInsight so there was no good reason for me to keep all this data within vCenter. I did the following to prune this data:
- SSH into vCenter appliance if you are not still connected
- Enable the shell
- ‘shell.set –enabled true’
- Stop the vCenter service
- ‘service vmware-vpxd stop’
- You will need to grep the password for the vCenter database to be used in a later step
- ‘cat /etc/vmware-vpx/embedded_db.cfg | grep PASSWORD
- Copy the above password at the end of this command to connect to the DB
- ‘/opt/vmware/vpostgres/1.0/bin/psql -d VCDB postgres
- Truncate the tables with the following command:
- ‘TRUNCATE TABLE vpx_event CASCADE;’
- Wait a few minutes for this to complete and check the disks now:
- ‘df -h’
- You can now restart the vCenter service
- ‘service vmware-vpxd start’
The vCenter database will now be pruned, log less data, and have a larger disk if that was a requirement.
Leave a Reply