22
July

The most recent checkpoint for the VBA appliance is outdated

Recently found one of our VBA appliances was producing the following error in vSphere

error> The most recent checkpoint for the VBA appliance is outdated

cplist
cp.20150714161043 Tue Jul 14 10:10:43 2015 valid hfs — nodes 1/1 stripes 344
cp.20150714172020 Tue Jul 14 11:20:20 2015 valid hfs — nodes 1/1 stripes 344

Opened a ticket with EMC and the following procedure was provided.

dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.

A status.dpn gives us a little more information on some of the maintenance services

status.dpn
Tue Jul 21 09:35:44 MDT 2015 [VBA01] Tue Jul 21 15:35:44 201 5 UTC (Initialized Tue Nov 4 20:34:05 2014 UTC)
Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Loa d UsedMB Errlen %Full Percent Full and Stripe Status by Disk
0.0 10.0.80.7 7.0.62-10 ONLINE fullaccess mhpu+0hpu+0hpu 4 false 0.5 8 7586 15696646 6.1% 6%(onl:116) 6%(onl:116) 6%(onl:115)
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwrit eable
System ID: 1415133245@00:50:56:88:1A:CC
All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full
Last checkpoint: cp.20150714172020 finished Tue Jul 14 11:20:40 2015 after 00m 2 0s (OK)
No GC yet
Last hfscheck: finished Tue Jul 14 11:28:07 2015 after 07m 19s >> checked 343 of 343 stripes (OK)

 

Although maintenance is running cp, gc and the hfscheck were suspended?

contd..

Maintenance windows scheduler capacity profile is active.
WARNING: cp is suspended permanently.
WARNING: gc is suspended permanently.
WARNING: hfscheck is suspended permanently.
The maintenance window is currently running.
Next backup window start time: Tue Jul 21 20:00:00 2015 MDT
Next maintenance window start time: Wed Jul 22 08:00:00 2015 MDT

The following commands will set hfscheck, gc and cp to on permanently.

avmaint sched resume hfscheck –permanent –ava
avmaint sched resume gc–permanent –ava
avmaint sched resume cp–permanent –ava

You may want to create a checkpoint manually
dpnctl stop maint
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Suspending maintenance windows scheduler…

avmaint checkpoint –ava
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>
<checkpoint
tag=”cp.20150721154550″
isvalid=”false”/>

dpnctl start maint
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Resuming maintenance windows scheduler…
dpnctl: INFO: maintenance windows scheduler resumed.

no comments

8
July

Cannot access NetWorker VBA GUI

Had an issue recently where the VBA config and FLR GUI’s were inaccessible.  It was easy enough to stop and start tomcat with the emwebapp script, but it didn’t work. EMC provided this process to re-register the certificate also.

 

1. Stop emwebapp
emwebapp.sh – -stop

2. Back up existing keystore
cp /root/.keystore /root/.keystore.sav

3.  List tomcat certificate – should see 1 certificate
/usr/java/latest/bin/keytool -list -keystore /root/.keystore -storepass changeit -alias tomcat

4. Delete tomcat certificate from keystore
/usr/java/latest/bin/keytool -delete -alias tomcat -storepass changeit

5. List tomcat certificate again – should return empty
/usr/java/latest/bin/keytool -list -keystore /root/.keystore -storepass changeit -alias tomcat

6. Regenerate certificate using SHA256
/usr/java/latest/bin/keytool -genkeypair -v -alias tomcat -keyalg RSA -sigalg SHA256withRSA -keystore /root/.keystore -storepass changeit -keypass changeit -validity 3650 -dname “CN=localhost.localdom, OU=Avamar, O=EMC, L=Irvine, S=California, C=US”

7. List tomcat certificate again – should see 1 certificate

8. Start emwebapp.sh
emwebapp.sh – -start

no comments

26
June

Unexpected Connection error with NetWorker and VBA

I was really looking forward to another idyllic day as a Backup Administrator.

Those days usually begin troubleshooting a few backup failures, drinking a lot of coffee and planning world domination. Sadly, I actually had to do some work today instead.

I found all my VMware protection policies had failed the previous evening.

error: Unable to connect to VBA, error Cannot establish session to VBA.
Logged onto vSphere and attempted to browse the backup recover area. Where we found the following error:

error: An unexpected connection error occurred and the cause could not be determined, Please check your EBR configuration screen to troubleshoot, or contact an administrator.

We rebooted the appliance and and it seemed to fix the issue for one. Ended up delving into the nuts and bolts of the various resources required to create a VBA backup. Let’s review these components.

EBR Config GUI

You can access the configuration interface by browsing to:

https://VBA:8543/ebr-configure/

Login via the root credentials of the appliance. The default password is 8RttoTriz

Note: I don’t know about you, but try as I might I could not access this via chrome.

VMUSER

In the GUI you can view running service and restart if required. You can also configure the connection to NetWorker. To do this you will need to know the password for the vmuser account. Near as I can tell the is an ID internal to NetWorker that is used to establish communication. The default password is “changeme”

CHANGING THE VMUSER PASSWORD

We hadn’t change this password. If you would like to change the password this can be done on the NetWorker server properties page under the misc tab. We also were not sure what the password was? Yeah, I know.

Re-establish communication between the appliance and NetWorker

Go to the NetWorker Config tab. Here you can enter the password and save to re-establish (and confirm the password) the interface with NetWorker. You will need to reboot the appliance after this.

6-24-2015 2-30-16 PM

 

 

This did resolve our issue and we can now browse the Backup Recovery interface in vSphere. With this re-established backups should run tonight. Fingers crossed.

 

no comments

19
March

EBR not sending summary report

This was a new one. My VMware admin dropped me a note wondering why he had not been receiving a backup summary report from NetWorker? I didn’t even know this was a configurable option? Sure enough after poking around the Web client interface I found it.

3-19-2015 4-11-53 PM

 

 

Then we found this very informative message in the log.

3-19-2015 4-15-19 PM

 

Some quick research found this is a known issue. It typically occurs around daylight saving time and is caused by a mismatch in time value between the summary report timer and the database. Some success had reported with rebooting the appliance.  There was some indication that editing the email option and saving without changes would rectify the issue.

 

 

no comments