New to NetWorker?


Congratulations! You have inherited some NetWorker infrastructure, either by your defined career path or by misfortune. You have a rudimentary knowledge of backup technologies, but you’re not sure where to start. This post is for you. First, head over to support.emc.com and sign up for an account. You may need to contact site support to associate your new user ID with your site id. What is your site ID? Glad you asked.

Things you should have when opening a support call

  • You should know your site ID
  • Host ID
  • NetWorker Version
  • Platform

Hopefully, somebody you work with has the site ID, if not your local sales rep might. If not contact support and have your ID associated. When associated you can open support requests online and select your site from a drop down list.

The host id is a unique ID NetWorker assigns when the software is installed. It is not to be confused with the host id of your NetWorker server or data zone.

  1. Open the NetWorker server’s NetWorker Management Console (Console) interface.
    2. Select NetWorker Administration.
    3. In the Administration interface, click the Configuration button.
    4. Right-click Registrations in the navigation tree, then right-click the NetWorker evaluation license (or any NetWorker license) in the Registrations area of the screen. The Properties window appears.
    5. In the Configuration area of the Properties window, the Host ID is the last of the parameters displayed.
    6. Click OK or Cancel to leave the Properties window.

Your version is easy enough to find. In the NMC select help and about.

With your new EMC ID you should also be able to access https://community.emc.com/. The ECN is your portal for all things EMC. Great forums, sometimes you might actually find an answer there. If you post be sure to give as much info about you infrastructure as possible, Version, platform, etc. There are some great, smart NetWorker guys that hang out there and not giving this context to your issue is a massive pet peeve.

Going back to support.emc.com. There you can find the NetWorker support portal. It’s pretty sexy. Here you can see other recommended resources. Remember, NetWorker modules have separate support pages. You can subscribe to the page and it will be saved in your product list for the future. Also I love the service life by version section as well as the shortcuts to open support tickets or live chat with EMC support.

3-18-2015 1-22-05 PM


Next check out the documentation section. Find the documentation portfolio for your NetWorker version. In there you will find the admin guide as well as a wealth of other information. One document that is worth a look if you’re new to NetWorker is “Theory of Core Operations” It may not be bundled in the portfolio, but is there if you search. The information in this guide is primarily intended to familiarize new EMC developers, test engineers, technical support engineers, product specialists, instructors, course developers, and information developers with NetWorker concepts.
This document addresses the EMC concept of “core engineering” function for the NetWorker product for Windows and UNIX operating systems.

Last but not least is http://nsrd.info/. This site is owned and administered by Preston de Guise. I’m not sure when he sleeps. He frequently post in depth content and his blog is a fantastic library of all things NetWorker. Be sure to check out his micro manuals. He has a new one called Turbo Charged NetWorker, which he intends to update regularly. He also published yearly NetWorker administrator survey results which he uses to capture trends in NetWorker uses and functions. Wow.

Also, you may pick up his book, Enterprise Systems Backup and Recovery: A Corporate Insurance Policy. A great read, I’m sure. Bought it a while ago and promise to read it one day.

So there you go. Did I miss anything? Comment below!

no comments


Backups are collaborative

This is an open letter to Windows server admins.

I have had the luxury of managing large backup environments at a few different organizations over the years. This is a great way for me to engage my skills in a very niche area as well as for my colleagues. Why for them? Well then they don’t have to manage backups as part of their daily operations. If I have learned anything since focusing in this area, it’s that nobody thinks about backups. Nobody wants to. Whether it’s in the early stages of a large system build, where we really should be brought in to consult and ensure that our backup application can ingest this new load or in daily ops. It is somehow expected that we will be there ready to drink from the veritable data fire hose and provide near zero downtime recovery.

The point I wanted to make was since the first tape drive was connected to the first mainframe, demands have required tighter integration between the backup application and your data. Whether it is a backup agent protecting your database, the vstorage API to protect your VM or VSS to capture a backup of your Windows server to ensure recoverability. They all require collaboration with backup administrators and systems administrators. Are you a Windows administrator? Are you familiar with the VSS process?  If not, you should be. Like the vstorage API and RMAN, VSS is just a mechanism that allows a backup application to capture the data. I don’t own the VSS component although I’m probably more familiar with it and better at resolving VSS issue than any Windows administrator.

If there is one thing I can assure you is this. VSS issues are not unique to any one backup product. Google “VSS backup issue” and you will find a multitude of forums from an array of products filled with angry, annoyed backup system administrators fuming over VSS.   I can generally and do fix most VSS issues without having to bother an admin. Sometimes I can’t and the only thing I can offer is to reboot the system. Is there a better solution? Yes, call Microsoft! Open a ticket, run some VSS traces. In short, work with me. Don’t treat me as a nuisance who is again requesting a reboot.

Collaborate with me because one day, you will need me by far more than I need you.

no comments


Purging Data with NetWorker and Data Domain

Purging data is sometimes required. There could be some legal requirement or more likely you are out of space on disk storage. Here is what I recently had to do.

Some data was identified to be purged. Specifically three clients.

The following tasks need to be completed:

  • Identify the save sets
  • Build the batch file
  • run nsrim -X
  • Run a clean on the DD

Identifying the save sets

There is a ton of info out there on mminfo. This is the command we will use to identify which save sets to remove. The command has two parts. A query portion where we feed in the required variables and a report portion where we can narrow the specific data we need. My command resembled the following.

mminfo -avot -q “client-clientname, volume=volumename.001” -r client,volume,ssid,cloneid

I ran the command and outputted to text. The output resembled the following.

client    volume         ssid          clone id

XXXXX volume.001 4065016450 1397439105

XXXXX volume.001 3997907656 1397439175

XXXXX  volume.001 3981130503 1397439239

XXXXX  volume.001 3947576123 1397439291

XXXXX  volume.001 3930798958 1397439341

XXXXX  volume.001 3914021948 1397439548

I’m really only concerned with the last two columns, as these are required to use with the nsrmm command to delete the data from the NetWorker databases.

The command we will use will look like this.

nsrmm-dy -S SSID/CloneID

Now that we have this we can build the batch file, as this is a windows system.

Building the batch

Just a lot of excelFU here. I saved the output from the mminfo command to a text file, then imported into using Data, From text. I selected Delimited and then selected “space” as the delimiter. This inserted my data into the columns nicely.

1 2



Next, I’ll delete everything in in columns A and B. In column A I will enter our nsrmm command as above. Then I’ll select the cell and drag it down to auto fill in the cells below.


Next I formatted all the cells as number and then entered the following formula into cell E2.

CONCATENATE(A3,” “,D3,”/”E3)

This will merge our command in column A, add a space after then our SSID and insert a / to seperate. Finally it will tag on the cloneid and output to one cell. Select the cell and drag down to auto populate. You will then have a column filled with individual commands. That entire column can be pasted into a batch file.


When complete run the batch, I have over 2000 rows so this will take while.

Run nsrim -X

nsrim -X will synchronize the media DB and wraps up the purging of this data from NetWorker

Run a clean on the DD

Start the DD clean. It can take some time, best to run when things are quiet. Here we can see we did not win a lot in the way of cleanable data?




Why is that? Reducing data retention has limited effect. When data is expired the pointers can be removed, but unique data is still needed to be retained for recovery.  Reducing retention is not always a positive thing as it can lead to a reduced pool of data to deduplicate  against. However, here we removed clients in their entirety? I can only assume that the data on this client was already highly deduplicated and there was actual precious little unique data identified. So are results are what they are. I hope yours are better. Let me know. Comment below.



no comments


Protecting the vCenter server database


Had a few VBA backup failures this morning. I had isolated the failures to my clients VBLOCK and was just about to start digging into the problem, when the VMware admin contracted me and advised he had found the issue. Apparently we had configured a VBA backup of the vCenter server.  I’m surprised we had not seen this issue before. While taking a quiesced snapshot or while deleting the snapshot of the database virtual machine the vCenter server can loose connectivity to the database. What is actually happening beneath the hood is interesting and pretty 

“The vCenter database layer (Vdb) replays the failed SQL statement requests to continue the vCenter operation. During the replay process, if it turns out that the previously failed SQL statement has been committed to the database, and if there is a unique constraint definition on the specific table, the ODBC driver reports the unique constraint violated error to the VMware VirtualCenter Server service and the service shuts down to prevent corruption of the vCenter Server database.”

Vmware reconfirms what I have always considered a best practice for protecting any application.

Currently, VMware does not support quiesced snapshots of virtual machines running the vCenter Server database. 

To work around this issue, use one of these options:
  • If quiesced snapshots are created by backup software to back up the virtual machine data, either use:
    • A backup solution that provides application-level quiescing.
    • Backup Agents in the guest operating system.Note: Any backup agent that quiesces the file system causes the issue described in this article.

  • If snapshots are created manually during virtual machine maintenance (for example, guest os patching, configuration changes), deselect the Quiesce guest file system option while taking the virtual machine snapshot.

See: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003674


no comments


What Does Total Source Capacity Mean?

Recently, when reviewing a report on a clients monthly backup statistics, an issue was highlighted with NetWorkers native licenses conformance output.




As we can see the output is not super informative. I have to admit, I was completely ignorant of exactly what Total Source Capacity meant? Now it seems self explanatory, but it took a few attempts from my SE of explaining before it sunk through the first few layers of my skull. I found this description from an EMC website that sums it up far better than I ever could.

“Data is measured as the largest aggregate full backup or synthetic full backup which is the combination of full backups plus incremental backups that are performed for all protected data by the NetWorker software over a two-month period (60 days). This is irrespective of where the data is backed up, for example, from tape, disk, VTL, Avamar ® Data Store, or Data Domain. The quantity of pre-deduplicated data is included in the calculation.”

So the measurement of total source capacity is not really directly tied to data moved or data stored, but rather the aggregate of the largest Full backup of all your protected clients over a 60 day period. My next challenge was how to determine this number? This particular client does use DPA, and I was getting ready to attempt to create a report that may pull out this information. Some googlefu found a recent question posted in the EMC community forum from a user looking for the exact same thing. After bumping some dude named Gareth came through with a deeply buried report template in DPA that would address this.



The report is called “Estimated Protected Capacity”. You can find this report under Status / Backup in the report menu. Just set you time period for 60 days and viola!


no comments


Configuring NMM Exchange Backups


First I know what you are all wondering, Dan how do you feel about Stephen Colbert being named as David Letterman’s replacement in 2015? I find Stephen Colbert both charming and handsome and look forward to meeting the real Colbert when he debuts. It’s a good thing Mr. Colbert can relay on his many talents to help fulfill a role as a lat night talk show host. Without those skills he may have ended up a lowly Data recovery specialist in Calgary faced with an NMM configuration.

That my friends, is what we call a segue in the biz.



Recently I rejoined the professional services team at my company. I had been embedded for sometime, but lets not talk about that. Suffice to say I’m excited to return to PS as a trusted adviser on all things NetWorker. Those words are a complete fabrication of reality and were proven when I attempted to configure NMM backups for my clients Exchange systems.

Given that this was my first project upon returning to PS I was eager to put a “W” up on the board. I had thought I was prepared having reviewed the NMM documentation. I found it slightly convoluted and confusing in some parts, but this is not an unusual circumstance for my brain. Perhaps I was delusional in assuming that the documentation contained all the knowledge needed to be successful? Yes, yes I was.

I don’t have the time to re-write the NMM documentation here. What I can do is fill you in on some glaring omissions form the documentation that EMC support was able to inform me of.

After adding the client check the client resources.

Not sure how critical this was but EMC support advised that a pool should not be specified.

4-11-2014 12-43-22 PM



Ensure the DAG entry is correct

4-11-2014 12-43-35 PM






Ensure all DNS aliases are entered

4-11-2014 12-43-51 PM

The remote access are should contain entries for all the exchange nodes.

4-11-2014 12-44-09 PM



The administration field should contain the following. An entry for the NMMBackupUser, this user should have been created with the required permissions and a mailbox assigned. There should also be entries for the exchange nodes and the Exchange DAG name.

4-11-2014 12-55-31 PM



Again, not sure how important, but the EMC support zeroed out these entries on the group properties


4-11-2014 12-44-49 PM



The snapshot policy and pool should be identified under the group properties.

4-11-2014 12-44-32 PM


Finally, ensure you have the following entries in administration  under setup in the NetWorker Server properties.

4-11-2014 12-55-31 PM



EMC documentation is pretty good, but I don’t know how these very critical items could be missed. I searched the NMM documentation portfolio for the following items.

I hope this post helps other users, VAR’s and EMC staff in any future NMM implementations.  All said, it seems to work well when you get it going.

no comments


Ask the expert?

I’ll be hosting a discussion over at the ECN network on NetWorker day to day operations.



I know, expert? I’ll try my best to impart something that may or may not resemble wisdom. I’m not making any promises.


no comments


Creating NetWorker Client Repositories

Creating client repositories is pretty easy. All you need is the package and the command line. It does get a little tricky when you need to create a cross platform repository. That is create Windows client repository on a Linux NetWorker server.

For same OS, it’s easy enough.  Move the package over to the NetWorker server. You may want to look at the LGTO meta file. It will break down the variables for how they should be entered into the command line.

The switches below denotate the product, platform and path to the client files.

nsrpush -a -p NetWorker -v -P linux_x86 -m /tmp/linux_x86 -U

Success adding product from:
Add to repository status: succeeded

Cross platform is a little more of a pain. You need to place the Windows client files on a like Windows client as well as on the NetWorker server. Then we will specify the Windows client in the command along with the path.


Below we add the C for the client name. This is the name of an existing networker client where we have placed the new client files as well as the path to the same files on our NIX NetWorker server.

nsrpush -a -p NetWorker -P win_x64 -v -m /tmp/win_x64 -c cwf161 -C ‘D:\\nw80sp1_win_x64\win_x64’ -W
Hostname and mount point recorded.
Success adding product from:



no comments


Troubleshooting NDMP backup issues

This morning I came in to find the NDMP backups of the Celerra had been hung overnight. The other filer was fine as were all the client backups.

So lets go through the list of things to check.

1. Is the filer up?

I know this sounds dumb, but really start with the simplest solution. Yes, the filer is online.

2.Is the NDMP service running on the filer?

To confirm run the following command from your NetWorker server.

inquire -N filer_name.

You will be prompted to enter the associated credentials. It will then return a list of devices attached to the filer. Compare this against what is in NetWorker. Yup! It all looks good.

3. Is there an issue with the media database?

I’ve seen this before. Where there is media available to use but networker is not using it. This should be associated with some alerts indicating “unable to allocate media” I wasnt seeing any such alert. At the time I had only scratch tapes, that is returned previously used media. I added some brand new never used tapes to see if that would work. I also ran some of the database checks. nsrim -X. No joy, if the problem is related in some way to this I’ll need some help from the vendor.

4. Are there any processes running for these jobs?

# ps -ef | grep FILER101
root 1512 3029 0 May01 ? 00:00:04 /usr/sbin/nsrndmp_save -T dump -F :sapvol01:uxvol01:filenetvol01 -s cls213.company.ca -c FILER101.company.ca -g FILER101_04 -LL -t 1367328945 -l 7 -q -W 78 -N /epvol08 /epvol08
root 1942 3047 0 02:14 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 SQL Prod SQL_DB_Prod
root 4735 3047 0 May01 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 Day 7 -N 1 FILER101_07
root 10453 3047 0 May01 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 Day 5 -N 1 FILER101_05
root 11767 16287 0 11:47 pts/3 00:00:00 grep FILER101
root 13881 3047 0 May01 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 Day 3 -N 1 FILER101_03
root 14461 3029 0 May01 ? 00:00:02 /usr/sbin/nsrndmp_save -T dump -s cls213.company.ca -c FILER101.company.ca -g FILER101_03 -LL -t 1366767465 -l 2 -q -W 78 -N /epvol07 /epvol07
root 14914 3047 0 May01 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 Day 1 -N 1 FILER101_01
root 15502 3029 0 May01 ? 00:00:02 /usr/sbin/nsrndmp_save -T dump -F :epvol11 -s cls213.company.ca -c FILER101.company.ca -g FILER101_01 -LL -t 1367367626 -l 4 -q -W 78 -N /epvol10 /epvol10
root 21381 3047 0 May01 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 Day 6 -N 1 FILER101_06
root 26306 3047 0 May01 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 Day 2 -N 1 FILER101_02
root 26418 3029 0 May01 ? 00:00:02 /usr/sbin/nsrndmp_save -T dump -F :SEIS:epvol01:epvol02:epvol03 -s cls213.company.ca -c FILER101.company.ca -g FILER101_02 -LL -l full -q -W 78 -N /EP /EP
root 30062 3047 0 Apr30 ? 00:00:00 /usr/sbin/savegrp -I -C FILER101 Day 4 -N 1 FILER101_04

5. Finally, check the logs.
Check /nsr/cores and the daemon log.

In this instance I found this message scrolling through.
83446 05/02/2013 02:13:28 PM nsrmmd NSR warning Ignoring shutdown request for nsrmmd 3 with pid 6870 because it’s currently busy.

Lets look at this PID…

root 6870 1 0 Apr16 ? 00:01:07 /usr/sbin/nsrmmd -b 2 -N 364266497 -n 3 -s cls213.company.ca -r FILER101 -t cls213.company.ca

My plan was to kill and restart, but before I could restart the jobs took off. I guess the process restarted on its own.
Really, this process is not specific to NDMP. Just wanted to map out my process when troubleshooting.

no comments


More fun with NSRADMIN


If you are responsible for maintenance and operations of some NetWorker infrastructure, you are doing yourself a huge disservice not getting to know NSRADMIN.

There is a some documentation out there, this micromanual  written by Preston over at nsrd.info is invaluable. So, lets talk about a recent use case that was of great help to me recently.

I have a large client that was migrating their VM’s into a new vCenter. Task 1. was to update the application information variable on all the client resources with the name of the new vCenter.  My friend and co-worker informed me that you can now edit multiple clients simultaneously with the NMC on NetWorker 8! I was pretty happy with this news for two reasons.

1. As previously stated, I’m not very smart. While I was sure there was going to be a way to do this with NSRADMIN I wasn’t sure how?

2. I’m also lazy and had no desire or time to figure this out.

I attempted to do this via the NMC initially. I didnt take the time to note the error, but it was something to the effect of  “Could not edit 6 of 18 selected clients…”. I kept trying with different blocks of clients and consistently had this error returned. Eventually I gave up and had more success with NSRADMIN. Here is what I did.

nsradmin> show name:; application information:
nsradmin> print type: NSR client; group: DEV4_VADP

The above returns a list of all clients in the group and the associated application information.

name: server_nameDPU101;
application information: VADP_HYPERVISOR=vdc01vmvc01.domain.com;

name: server_nameDXU101;
application information: “VADP_HYPERVISOR=vdc01vmvc03.domain.com”;


To update this variable for all the clients in the group, run the following.

nsradmin> update application information: VADP_HYPERVISOR=vsphere4.domain.com

You will be prompted to confirm for each client. I have not been able to find a way to force this. This could be scripted to make it even easier, but I was hammering this on the fly in the middle of the night and had to get this done as I wanted to go back to bed.

Lazy? Remember?



no comments

Back to top