13
June

A new start

I started this blog in 2012 when I was assigned to a new client and had to get up to speed with NetWorker. The purpose was to capture any and all learning’s here. Four years and close to 80 posts later, it has been an invaluable tool to capture and share knowledge. It’s not unusual for me to look up past issues here or to google them, only to be redirected to my own blog. I never considered an issue resolved, until it was captured here. Not unusual for others to find their way here also. Stats for the last month show 656 hits, mostly from India and the U.S. Its good to know I’m not here alone screaming in the dark.

 

 

Today I find myself at the start of a new opportunity with many great challenges and things to learn. So, expect this blog to not only be a great place for capturing knowledge for NetWorker and Avamar. It will now be a repository for new learning’s related to my old friend NetBackup, as well as storage and virtualization.

no comments

8
July

Cannot access NetWorker VBA GUI

Had an issue recently where the VBA config and FLR GUI’s were inaccessible.  It was easy enough to stop and start tomcat with the emwebapp script, but it didn’t work. EMC provided this process to re-register the certificate also.

 

1. Stop emwebapp
emwebapp.sh – -stop

2. Back up existing keystore
cp /root/.keystore /root/.keystore.sav

3.  List tomcat certificate – should see 1 certificate
/usr/java/latest/bin/keytool -list -keystore /root/.keystore -storepass changeit -alias tomcat

4. Delete tomcat certificate from keystore
/usr/java/latest/bin/keytool -delete -alias tomcat -storepass changeit

5. List tomcat certificate again – should return empty
/usr/java/latest/bin/keytool -list -keystore /root/.keystore -storepass changeit -alias tomcat

6. Regenerate certificate using SHA256
/usr/java/latest/bin/keytool -genkeypair -v -alias tomcat -keyalg RSA -sigalg SHA256withRSA -keystore /root/.keystore -storepass changeit -keypass changeit -validity 3650 -dname “CN=localhost.localdom, OU=Avamar, O=EMC, L=Irvine, S=California, C=US”

7. List tomcat certificate again – should see 1 certificate

8. Start emwebapp.sh
emwebapp.sh – -start

no comments

26
June

Unexpected Connection error with NetWorker and VBA

I was really looking forward to another idyllic day as a Backup Administrator.

Those days usually begin troubleshooting a few backup failures, drinking a lot of coffee and planning world domination. Sadly, I actually had to do some work today instead.

I found all my VMware protection policies had failed the previous evening.

error: Unable to connect to VBA, error Cannot establish session to VBA.
Logged onto vSphere and attempted to browse the backup recover area. Where we found the following error:

error: An unexpected connection error occurred and the cause could not be determined, Please check your EBR configuration screen to troubleshoot, or contact an administrator.

We rebooted the appliance and and it seemed to fix the issue for one. Ended up delving into the nuts and bolts of the various resources required to create a VBA backup. Let’s review these components.

EBR Config GUI

You can access the configuration interface by browsing to:

https://VBA:8543/ebr-configure/

Login via the root credentials of the appliance. The default password is 8RttoTriz

Note: I don’t know about you, but try as I might I could not access this via chrome.

VMUSER

In the GUI you can view running service and restart if required. You can also configure the connection to NetWorker. To do this you will need to know the password for the vmuser account. Near as I can tell the is an ID internal to NetWorker that is used to establish communication. The default password is “changeme”

CHANGING THE VMUSER PASSWORD

We hadn’t change this password. If you would like to change the password this can be done on the NetWorker server properties page under the misc tab. We also were not sure what the password was? Yeah, I know.

Re-establish communication between the appliance and NetWorker

Go to the NetWorker Config tab. Here you can enter the password and save to re-establish (and confirm the password) the interface with NetWorker. You will need to reboot the appliance after this.

6-24-2015 2-30-16 PM

 

 

This did resolve our issue and we can now browse the Backup Recovery interface in vSphere. With this re-established backups should run tonight. Fingers crossed.

 

no comments

19
March

EBR not sending summary report

This was a new one. My VMware admin dropped me a note wondering why he had not been receiving a backup summary report from NetWorker? I didn’t even know this was a configurable option? Sure enough after poking around the Web client interface I found it.

3-19-2015 4-11-53 PM

 

 

Then we found this very informative message in the log.

3-19-2015 4-15-19 PM

 

Some quick research found this is a known issue. It typically occurs around daylight saving time and is caused by a mismatch in time value between the summary report timer and the database. Some success had reported with rebooting the appliance.  There was some indication that editing the email option and saving without changes would rectify the issue.

 

 

no comments

25
February

Protecting the vCenter server database

 

Had a few VBA backup failures this morning. I had isolated the failures to my clients VBLOCK and was just about to start digging into the problem, when the VMware admin contracted me and advised he had found the issue. Apparently we had configured a VBA backup of the vCenter server.  I’m surprised we had not seen this issue before. While taking a quiesced snapshot or while deleting the snapshot of the database virtual machine the vCenter server can loose connectivity to the database. What is actually happening beneath the hood is interesting and pretty 


“The vCenter database layer (Vdb) replays the failed SQL statement requests to continue the vCenter operation. During the replay process, if it turns out that the previously failed SQL statement has been committed to the database, and if there is a unique constraint definition on the specific table, the ODBC driver reports the unique constraint violated error to the VMware VirtualCenter Server service and the service shuts down to prevent corruption of the vCenter Server database.”

Vmware reconfirms what I have always considered a best practice for protecting any application.

Currently, VMware does not support quiesced snapshots of virtual machines running the vCenter Server database. 

To work around this issue, use one of these options:
  • If quiesced snapshots are created by backup software to back up the virtual machine data, either use:
    • A backup solution that provides application-level quiescing.
    • Backup Agents in the guest operating system.Note: Any backup agent that quiesces the file system causes the issue described in this article.

  • If snapshots are created manually during virtual machine maintenance (for example, guest os patching, configuration changes), deselect the Quiesce guest file system option while taking the virtual machine snapshot.

See: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003674

 

no comments