NetWorker Build 774 has been released.

NetWorker Build 774 has been released.

It can be downloaded from ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/8.2/
This package contains the following cumulative fixes:

ID Details
(NW161917) ESC	NetWorker Escalation 22509: NSM DB2 PIT restore doesn't restore connecting directories' ACL on AIX
NW161624) ESC	NetWorker Escalation 22307: Device discovery raises alerts if udev-named library handle already configured: 14249:dvdetect: 'skipped as requested'
ESC	NetWorker Escalation 22567: Error counts not correctly handled with nsrsnmd & cdi changes
(NW159916) ESC 	NetWorker Escalation 21778: Disable "label" operation in AMM functionality: DataDomain devices unmount with "RPC severe Lost connection to media database"

no comments


Troubleshooting Cloning – Part 2 – Clone Wars

The clone wars continue. As you may recall from the Part 1, my NetWorker optimized clones have been hanging with this misleading error:

Waiting for 1 Writable volume on backup pool ‘Device’ disk(s) on nsr_server

To further complicate things, control over the clone from the console can be limited. I had been using the jobkill utility. Preston has a great write up on it here. The issue is that after killing the clone, the NMC console shows it as running still? Attempts to restart the clone via NMC resulted in the following error:

1 1429638078 event task manager Task aborted: task ‘clone.name Clone’ is already running

So is the job kill, really murdering all the required processes? Lets take a look.

My new env is Windows. So lets visit our old friend the task manager, here we want to look for nsrtask and nsrclone processes. First nsrclone. I’ve had to obfuscate the output, but what I found were the appropriate running clones. The specific job I needed to restart was not listed.



The same cannot be said for the clone jobs associated nsrtask process. There was indeed a process still hanging around.


After killing it, the state in NMC changed from Running to Interrupted. I could then restart the job.




All this just to get my clone going again. This is some progress as I had previously been restarting NetWorker, interrupting the service and other running clones.

no comments


Troubleshooting Cloning – Part 1

I’ve had some issues with cloning recently. It’s been an interesting issue that will require some more analysis, but for now I have a work around that has helped narrow the issue down.  It all started with this error on a running clone:

Waiting for 1 Writable volume on backup pool ‘Device’ disk(s) on nsr_server

After checking the device is mounted I started to search for other issues. As usual I found a great post over at Preston’s blog. There he explains:

“A core component in NetWorker’s media database design is that a saveset can only ever have one instance on a piece of media. This applies as equally to failed as complete saveset instances.The net result is that this error/situation will occur because it’s meant to – NetWorker doesn’t permit more than one instance of a saveset to appear on the same piece of physical media.”

So what I surmise happens is there is a failure during the clone operation. The aborted saveset on the destination needs to be removed or excluded. I have not been able to formulate an mminfo command to find aborted clone savesets yet? See part 2 when published. For now I would be satisfied to exclude the savesets. This specific clone job in question provides DR protection for one of my NetWorker servers. That is, it clones index, bootstrap and filesystem saves to an offsite DD. For reasons I cannot explain here at this time this clone was configured to go back 6 months. I know.



So after the clone had run for a few minutes I opened the clone properties and hit the “Preview saveset” button. What I found was there was a particular group of savesets from a specific date in the past identified and that date alone. My assumption is those savesets already exist on the destination and the clone job is not intelligent enough to identify and skip. The error “waiting for volume” is really misleading, at least to me. After greatly narrowing to number of days to look back and rerunning the clone job completed successfully.

Stay tuned for Part 2 -Identifying aborted clone savesets. Do you know how? Comment below!

no comments


This is embarrassing

That said, I’m never afraid too admit what I don’t know or when I’ve made a mistake. In hindsight, I can’t believe it took me this long to find this. In my defense, this has never been a requirement in to many of the NetWorker environments I’ve managed, until recently.

My client has some remote sites with some with limited bandwidth that we are attempting to backup over the wire to a DD at the home office. The issue is the clients at the site gets a little cranky when the backup hijacks their bandwidth during core business hours. We had been manually killing the backup job upon arriving in the office. The other day I was looking at the client properties and realized there were some variables that I had no idea what function they performed. One was hard limit. There you can set the runtime of the client backup in minutes. So yeah, I never realized there was a way to set a defined backup window for NetWorker clients. There I said it! In my defense, other backup products I had worked with have defined backup window resources.


no comments



Do you have multiple Avamar grids? You probably do. You may have even more with multiple versions. Check out the AVOpener tool


Drop this  on your desktop and use it to open the MCS Console GUI. You can also create favorites and customize it for your environment.Not officially supported by EMC.

no comments


EMC NetWorker Technical Advisory – Failed recovery may result in data loss

EMC published an advisory today specific to the following NetWorker versions. Data loss may be experienced when performing command line recoveries on the client and when performing recoveries from the Recover Wizard on the NetWorker Management Console (NMC).

EMC Software: NetWorker Server: NetWorker 8.0 through
EMC Software: NetWorker Server: NetWorker 8.0 SP1 through
EMC Software: NetWorker Server: NetWorker 8.0 SP2 through
EMC Software: NetWorker Server: NetWorker 8.0 SP3 through
EMC Software: NetWorker Server: NetWorker 8.0 SP4 through
EMC Software: NetWorker Server: NetWorker 8.1 through
EMC Software: NetWorker Server: NetWorker 8.1 SP1 through
EMC Software: NetWorker Server: NetWorker 8.1 SP2 through
EMC Software: NetWorker Server: NetWorker 8.2 through

Files on the local file system may be deleted after a failed recovery to the original file location.This issue does not occur if the recovery is directed to a location other than the original data location.  This is caused If NetWorker is unable to read the header of the data source that is used to recover, it processes the error and removes the contents of the file on the target system. However, at this point in the recover process nothing has been written to the target and the existing (original) data on the target system is removed.

EMC addressed this issue in the following NetWorker Client versions, and strongly recommends that impacted customers install the release when possible.

  • NetWorker Client and later
  • NetWorker Client and later
  • NetWorker Client and later

no comments


NetWorker released

NetWorker Build 753 has been released.
It can be downloaded from ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/8.2/
This package contains the following cumulative fixes:
ID Details
226673 (NW162239) ESC  Escalation 22704:Browseable recover: expand_check() function experiences massive delays after upgrade to 8.1.x from 7.x on AIX clients
225294 ESC  Escalation 22923:savepnpc commands with level skip hangs backup
225029 BUG  MMDB latency to respond to VBA savesets query, causes Restore Tab malfunction to list available backups
223984 ESC  Escalation 22756:Unable to set extended attribute ‘security.selinux’ Operation not permitted
223513 ESC  Escalation 22121:jobquery core during DPA 6.1 data collection with NetWortker
223175 (NW162157) ESC  Escalation 22649:nsrd/nsrmmdbd dead-lock during relabel
222890 (NW162215) ESC  Escalation 22688:NW:Avamar: Avamar does not delete all savesets even though nsravamar.raw shows them deleted
222833 (NW162150) ESC  Escalation 22643:[BZ: 232867] 8dot3name setting is lost after BMR using BMR_8.1.0.199 or above
206724 (NW161663) ESC  Expired cleaning tape (0 uses left) is being used for cleaning
206664 (NW162021) ESC  [MIGRATED TO BZ]Everytime a backup starts for a UNIX/Linux client, NetWorker queries LDAP for root account
204302 (NW161964) ESC  Failed recover deletes existing original file or folder on file system
199061 (NW161544) ESC  NMC is not displaying the ‘enabled/disabled’ field correctly after upgrade to
198242 (NW161619) ESC  Incosintency on used space reported on NMC, mminfo and Disk manager for AFTD device
190825 (NW154749) ESC  NetWorker server client parallelism silently changed to 12 after a restart when parallelism configured to <1
190583 ESC  snmd will not come up on slow systems owing to too short snmd poll timeout

no comments


EBR not sending summary report

This was a new one. My VMware admin dropped me a note wondering why he had not been receiving a backup summary report from NetWorker? I didn’t even know this was a configurable option? Sure enough after poking around the Web client interface I found it.

3-19-2015 4-11-53 PM



Then we found this very informative message in the log.

3-19-2015 4-15-19 PM


Some quick research found this is a known issue. It typically occurs around daylight saving time and is caused by a mismatch in time value between the summary report timer and the database. Some success had reported with rebooting the appliance.  There was some indication that editing the email option and saving without changes would rectify the issue.



no comments


New to NetWorker?


Congratulations! You have inherited some NetWorker infrastructure, either by your defined career path or by misfortune. You have a rudimentary knowledge of backup technologies, but you’re not sure where to start. This post is for you. First, head over to support.emc.com and sign up for an account. You may need to contact site support to associate your new user ID with your site id. What is your site ID? Glad you asked.

Things you should have when opening a support call

  • You should know your site ID
  • Host ID
  • NetWorker Version
  • Platform

Hopefully, somebody you work with has the site ID, if not your local sales rep might. If not contact support and have your ID associated. When associated you can open support requests online and select your site from a drop down list.

The host id is a unique ID NetWorker assigns when the software is installed. It is not to be confused with the host id of your NetWorker server or data zone.

  1. Open the NetWorker server’s NetWorker Management Console (Console) interface.
    2. Select NetWorker Administration.
    3. In the Administration interface, click the Configuration button.
    4. Right-click Registrations in the navigation tree, then right-click the NetWorker evaluation license (or any NetWorker license) in the Registrations area of the screen. The Properties window appears.
    5. In the Configuration area of the Properties window, the Host ID is the last of the parameters displayed.
    6. Click OK or Cancel to leave the Properties window.

Your version is easy enough to find. In the NMC select help and about.

With your new EMC ID you should also be able to access https://community.emc.com/. The ECN is your portal for all things EMC. Great forums, sometimes you might actually find an answer there. If you post be sure to give as much info about you infrastructure as possible, Version, platform, etc. There are some great, smart NetWorker guys that hang out there and not giving this context to your issue is a massive pet peeve.

Going back to support.emc.com. There you can find the NetWorker support portal. It’s pretty sexy. Here you can see other recommended resources. Remember, NetWorker modules have separate support pages. You can subscribe to the page and it will be saved in your product list for the future. Also I love the service life by version section as well as the shortcuts to open support tickets or live chat with EMC support.

3-18-2015 1-22-05 PM


Next check out the documentation section. Find the documentation portfolio for your NetWorker version. In there you will find the admin guide as well as a wealth of other information. One document that is worth a look if you’re new to NetWorker is “Theory of Core Operations” It may not be bundled in the portfolio, but is there if you search. The information in this guide is primarily intended to familiarize new EMC developers, test engineers, technical support engineers, product specialists, instructors, course developers, and information developers with NetWorker concepts.
This document addresses the EMC concept of “core engineering” function for the NetWorker product for Windows and UNIX operating systems.

Last but not least is http://nsrd.info/. This site is owned and administered by Preston de Guise. I’m not sure when he sleeps. He frequently post in depth content and his blog is a fantastic library of all things NetWorker. Be sure to check out his micro manuals. He has a new one called Turbo Charged NetWorker, which he intends to update regularly. He also published yearly NetWorker administrator survey results which he uses to capture trends in NetWorker uses and functions. Wow.

Also, you may pick up his book, Enterprise Systems Backup and Recovery: A Corporate Insurance Policy. A great read, I’m sure. Bought it a while ago and promise to read it one day.

So there you go. Did I miss anything? Comment below!

no comments


Backups are collaborative

This is an open letter to Windows server admins.

I have had the luxury of managing large backup environments at a few different organizations over the years. This is a great way for me to engage my skills in a very niche area as well as for my colleagues. Why for them? Well then they don’t have to manage backups as part of their daily operations. If I have learned anything since focusing in this area, it’s that nobody thinks about backups. Nobody wants to. Whether it’s in the early stages of a large system build, where we really should be brought in to consult and ensure that our backup application can ingest this new load or in daily ops. It is somehow expected that we will be there ready to drink from the veritable data fire hose and provide near zero downtime recovery.

The point I wanted to make was since the first tape drive was connected to the first mainframe, demands have required tighter integration between the backup application and your data. Whether it is a backup agent protecting your database, the vstorage API to protect your VM or VSS to capture a backup of your Windows server to ensure recoverability. They all require collaboration with backup administrators and systems administrators. Are you a Windows administrator? Are you familiar with the VSS process?  If not, you should be. Like the vstorage API and RMAN, VSS is just a mechanism that allows a backup application to capture the data. I don’t own the VSS component although I’m probably more familiar with it and better at resolving VSS issue than any Windows administrator.

If there is one thing I can assure you is this. VSS issues are not unique to any one backup product. Google “VSS backup issue” and you will find a multitude of forums from an array of products filled with angry, annoyed backup system administrators fuming over VSS.   I can generally and do fix most VSS issues without having to bother an admin. Sometimes I can’t and the only thing I can offer is to reboot the system. Is there a better solution? Yes, call Microsoft! Open a ticket, run some VSS traces. In short, work with me. Don’t treat me as a nuisance who is again requesting a reboot.

Collaborate with me because one day, you will need me by far more than I need you.

no comments

Back to top