The most recent checkpoint for the VBA appliance is outdated

Recently found one of our VBA appliances was producing the following error in vSphere

error> The most recent checkpoint for the VBA appliance is outdated

cp.20150714161043 Tue Jul 14 10:10:43 2015 valid hfs — nodes 1/1 stripes 344
cp.20150714172020 Tue Jul 14 11:20:20 2015 valid hfs — nodes 1/1 stripes 344

Opened a ticket with EMC and the following procedure was provided.

dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.

A status.dpn gives us a little more information on some of the maintenance services

Tue Jul 21 09:35:44 MDT 2015 [VBA01] Tue Jul 21 15:35:44 201 5 UTC (Initialized Tue Nov 4 20:34:05 2014 UTC)
Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Loa d UsedMB Errlen %Full Percent Full and Stripe Status by Disk
0.0 7.0.62-10 ONLINE fullaccess mhpu+0hpu+0hpu 4 false 0.5 8 7586 15696646 6.1% 6%(onl:116) 6%(onl:116) 6%(onl:115)
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwrit eable
System ID: 1415133245@00:50:56:88:1A:CC
All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full
Last checkpoint: cp.20150714172020 finished Tue Jul 14 11:20:40 2015 after 00m 2 0s (OK)
No GC yet
Last hfscheck: finished Tue Jul 14 11:28:07 2015 after 07m 19s >> checked 343 of 343 stripes (OK)


Although maintenance is running cp, gc and the hfscheck were suspended?


Maintenance windows scheduler capacity profile is active.
WARNING: cp is suspended permanently.
WARNING: gc is suspended permanently.
WARNING: hfscheck is suspended permanently.
The maintenance window is currently running.
Next backup window start time: Tue Jul 21 20:00:00 2015 MDT
Next maintenance window start time: Wed Jul 22 08:00:00 2015 MDT

The following commands will set hfscheck, gc and cp to on permanently.

avmaint sched resume hfscheck –permanent –ava
avmaint sched resume gc–permanent –ava
avmaint sched resume cp–permanent –ava

You may want to create a checkpoint manually
dpnctl stop maint
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Suspending maintenance windows scheduler…

avmaint checkpoint –ava
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>

dpnctl start maint
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Resuming maintenance windows scheduler…
dpnctl: INFO: maintenance windows scheduler resumed.

no comments


Cannot access NetWorker VBA GUI

Had an issue recently where the VBA config and FLR GUI’s were inaccessible.  It was easy enough to stop and start tomcat with the emwebapp script, but it didn’t work. EMC provided this process to re-register the certificate also.


1. Stop emwebapp
emwebapp.sh – -stop

2. Back up existing keystore
cp /root/.keystore /root/.keystore.sav

3.  List tomcat certificate – should see 1 certificate
/usr/java/latest/bin/keytool -list -keystore /root/.keystore -storepass changeit -alias tomcat

4. Delete tomcat certificate from keystore
/usr/java/latest/bin/keytool -delete -alias tomcat -storepass changeit

5. List tomcat certificate again – should return empty
/usr/java/latest/bin/keytool -list -keystore /root/.keystore -storepass changeit -alias tomcat

6. Regenerate certificate using SHA256
/usr/java/latest/bin/keytool -genkeypair -v -alias tomcat -keyalg RSA -sigalg SHA256withRSA -keystore /root/.keystore -storepass changeit -keypass changeit -validity 3650 -dname “CN=localhost.localdom, OU=Avamar, O=EMC, L=Irvine, S=California, C=US”

7. List tomcat certificate again – should see 1 certificate

8. Start emwebapp.sh
emwebapp.sh – -start

no comments


Unexpected Connection error with NetWorker and VBA

I was really looking forward to another idyllic day as a Backup Administrator.

Those days usually begin troubleshooting a few backup failures, drinking a lot of coffee and planning world domination. Sadly, I actually had to do some work today instead.

I found all my VMware protection policies had failed the previous evening.

error: Unable to connect to VBA, error Cannot establish session to VBA.
Logged onto vSphere and attempted to browse the backup recover area. Where we found the following error:

error: An unexpected connection error occurred and the cause could not be determined, Please check your EBR configuration screen to troubleshoot, or contact an administrator.

We rebooted the appliance and and it seemed to fix the issue for one. Ended up delving into the nuts and bolts of the various resources required to create a VBA backup. Let’s review these components.

EBR Config GUI

You can access the configuration interface by browsing to:


Login via the root credentials of the appliance. The default password is 8RttoTriz

Note: I don’t know about you, but try as I might I could not access this via chrome.


In the GUI you can view running service and restart if required. You can also configure the connection to NetWorker. To do this you will need to know the password for the vmuser account. Near as I can tell the is an ID internal to NetWorker that is used to establish communication. The default password is “changeme”


We hadn’t change this password. If you would like to change the password this can be done on the NetWorker server properties page under the misc tab. We also were not sure what the password was? Yeah, I know.

Re-establish communication between the appliance and NetWorker

Go to the NetWorker Config tab. Here you can enter the password and save to re-establish (and confirm the password) the interface with NetWorker. You will need to reboot the appliance after this.

6-24-2015 2-30-16 PM



This did resolve our issue and we can now browse the Backup Recovery interface in vSphere. With this re-established backups should run tonight. Fingers crossed.


no comments


Protecting the vCenter server database


Had a few VBA backup failures this morning. I had isolated the failures to my clients VBLOCK and was just about to start digging into the problem, when the VMware admin contracted me and advised he had found the issue. Apparently we had configured a VBA backup of the vCenter server.  I’m surprised we had not seen this issue before. While taking a quiesced snapshot or while deleting the snapshot of the database virtual machine the vCenter server can loose connectivity to the database. What is actually happening beneath the hood is interesting and pretty 

“The vCenter database layer (Vdb) replays the failed SQL statement requests to continue the vCenter operation. During the replay process, if it turns out that the previously failed SQL statement has been committed to the database, and if there is a unique constraint definition on the specific table, the ODBC driver reports the unique constraint violated error to the VMware VirtualCenter Server service and the service shuts down to prevent corruption of the vCenter Server database.”

Vmware reconfirms what I have always considered a best practice for protecting any application.

Currently, VMware does not support quiesced snapshots of virtual machines running the vCenter Server database. 

To work around this issue, use one of these options:
  • If quiesced snapshots are created by backup software to back up the virtual machine data, either use:
    • A backup solution that provides application-level quiescing.
    • Backup Agents in the guest operating system.Note: Any backup agent that quiesces the file system causes the issue described in this article.

  • If snapshots are created manually during virtual machine maintenance (for example, guest os patching, configuration changes), deselect the Quiesce guest file system option while taking the virtual machine snapshot.

See: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003674


no comments