16
March

Log File Size Management

networker is down

 

I got to thinking about log files size mgmt after I had an issue this week.

I came in the office to find NetWorker a smoking wreck, as pictured above. Unresponsive and our /nsr directory rapidly running out of space. We had just added 200 GB a few weeks ago. It now was 99% full with only 16 GB left. With a couple of hours that space would be filled also.

I had assumed that the index was the culprit. Perhaps there was huge new data load on one of the clients or the filer?

I was working with the storage team to get some space added, when they informed me it was the daemon.raw and .log files that were over 160 GB in size a growing rapidly. Still not 100% sure what the cause was? We had recently upgraded from 7.6sp3 to 8.0 sp1. There had been some issues with NDMP backups and I had requested the storage team increase the debug level of the NDMP daemon on the filer. I dont think this would inform the daemon files?

Anyway, we also found there were some runaway ndmp process on the filers and killed them.
With the crisis averted it was time to do some analysis on log file mgmt.

I know what you are thinking. Great way to spend a Saturday afternoon šŸ™‚

Again, nsradmin is you’re friend!

To trim the log files I restarted networker. This rolled them over and I then deleted.
Here are our current log files.

-bash-3.2$ ls -lh daemon*
-rw-r–r– 1 root root 36M Feb 19 12:26 daemon_130219_122717.log
-rw-r–r– 1 root root 56M Feb 19 12:26 daemon_130219_122717.raw
-rw-r–r– 1 root root 4.1M Feb 21 10:01 daemon_130221_104654.log
-rw-r–r– 1 root root 6.8M Feb 21 10:01 daemon_130221_104654.raw
-rw-r–r– 1 root root 19M Feb 28 10:29 daemon_130228_115135.raw
-rw-r–r– 1 root root 11M Feb 28 10:29 daemon_130228_115136.log
-rw-r–r– 1 root root 59M Mar 11 09:20 daemon_130311_092040.raw
-rw-r–r– 1 root root 47M Mar 11 09:20 daemon_130311_092041.log
-rw-r–r– 1 root root 13M Mar 16 12:29 daemon.log
-rw-r–r– 1 root root 25M Mar 8 11:37 daemon.log2
-rw-r–r– 1 root root 14M Mar 16 12:29 daemon.raw
So we can see our files aren’t out of control. Lets look at our log mgmt settings.

[root@cls###~]# nsradmin -p nsrexec
NetWorker administration program.
Use the “help” command for help, “visual” for full-screen mode.
nsradmin> . type:NSR log
Current query set

Here we see a few directives

nsradmin> print
type: NSR log;
administrator: root, “user=root,host=cls###.WDenergy.ca”;
owner: NetWorker;
maximum size MB: 2;
maximum versions: 10;
runtime rendered log: /nsr/logs/daemon.log;
runtime rollover by size: Disabled;
runtime rollover by time: ;
name: daemon.raw;
log path: /nsr/logs/daemon.raw;

So here we can see the max file size is set for 2 MB, but the associated directive is disabled. So, our log files can and will grow unabated.

Lets, enable it. NetWorker will then do a hourly check on the file. It will then be rolled over. A maximum of 10 log archives will be kept.

nsradmin> . type:NSR log;name:daemon.raw
Current query set
nsradmin> print
type: NSR log;
administrator: root, “user=root,host=cls###.WDenergy.ca”;
owner: NetWorker;
maximum size MB: 2;
maximum versions: 10;
runtime rendered log: /nsr/logs/daemon.log;
runtime rollover by size: Disabled;
runtime rollover by time: ;
name: daemon.raw;
log path: /nsr/logs/daemon.raw;
nsradmin> update runtime rollover by size: Enabled
runtime rollover by size: Enabled;
Update? y
updated resource id 12.0.230.115.0.0.0.0.82.37.185.74.0.0.0.0.10.204.4.77(3)

Lets view and confirm the change.
nsradmin> print
type: NSR log;
administrator: root, “user=root,host=cls###.WDenergy.ca”;
owner: NetWorker;
maximum size MB: 2;
maximum versions: 10;
runtime rendered log: /nsr/logs/daemon.log;
runtime rollover by size: Enabled;
runtime rollover by time: ;
name: daemon.raw;
log path: /nsr/logs/daemon.raw;
nsradmin> . type:NSR log;name:daemon.log

Had to wait a little bit, but less than an our later the rollover happened!

 

[root@cls### logs]# ls -lh daemon*
-rw-r–r– 1 root root 36M Feb 19 12:26 daemon_130219_122717.log
-rw-r–r– 1 root root 56M Feb 19 12:26 daemon_130219_122717.raw
-rw-r–r– 1 root root 4.1M Feb 21 10:01 daemon_130221_104654.log
-rw-r–r– 1 root root 6.8M Feb 21 10:01 daemon_130221_104654.raw
-rw-r–r– 1 root root 19M Feb 28 10:29 daemon_130228_115135.raw
-rw-r–r– 1 root root 11M Feb 28 10:29 daemon_130228_115136.log
-rw-r–r– 1 root root 59M Mar 11 09:20 daemon_130311_092040.raw
-rw-r–r– 1 root root 47M Mar 11 09:20 daemon_130311_092041.log
-rw-r–r– 1 root root 14M Mar 16 13:59 daemon_130316_135910.raw
-rw-r–r– 1 root root 13M Mar 16 13:59 daemon_130316_135911.log
-rw-r–r– 1 root root 75K Mar 16 14:09 daemon.log
-rw-r–r– 1 root root 25M Mar 8 11:37 daemon.log2
-rw-r–r– 1 root root 24K Mar 16 14:09 daemon.raw

 

 

no comments

7
March

The Transformation Continuesā€¦Backup Game Day Is Back

The Transformation Continues…Backup Game Day Is Back How do you turn backup and recovery into an offensive strategy that delivers game-changing business results? More than 4,000 of you tuned in last fall to hear the first part of the story, and on Monday, March 11, we’re back with the […]

no comments

6
March

NetWorker vStorage API missing from client

Wow, this was infuriating.
After upgrading to NetWorker 8 I noticed some VADP backups I had been testing started failing.

I found this message in the output

c:\mnt\client\client\tmp.
Temporary Directory for VADP created.
Temporary vmMntLoc Directory for VADP created.
*** vStorage API driver (\??\C:\Program Files\Legato\nsr\plugins\VDDK\bin\AMD64\vstor2-mntapi10-shared.sys) not installed by NetWorker. This may cause potential compatibility issues. ***

Yeah!? NetWorker can’t find the vStorage API?! Really?!

Then I noticed NetWorkers new default install path is C:\Program Files\EMC NetWorker. Thank you EMC for updating that. You only bought Legato 9 years ago.

The former install path was C:\Program Files\Legato

I searched the registry on my proxy and found this key

vstor2-mntapi10-shared

Then updated the path in the registry to C:\Program Files\EMC NetWorker\

Also of note, if you like to create clients by right clicking and selecting copy.
For VADP clients you need to ensure the field in application information on the Apps & Moduiles VADP_VM_NAME is updated.

no comments

6
March

NetWorker 8 Update

Completed the NetWorker server update. Here are a couple of things to look out for.

The lgtonmc db needs to double in size. So ensure you have enough disk space. I didn’t so I had to scramble to free up some space.

There have been changes made to the authentication between NMC and NetWorker. In 8 and higher the NMC console uses nsrauth authentication to communicate with the NetWorker server. By default all NetWorker hosts use nsrauth and fail over to oldauth if it doesn’t work. However your nsrexed service may not be configured to use nsrauth.

This will result in the following error when attempting to browse devices…
error: Unable to connect to server. Unable to authenticate with server authentication error. ; why = client credential too weak

Mine wasn’t. šŸ™ To change do the following on your NetWorker server.

nsradmin> . type: NSRLA
Current query set
nsradmin> print

nsradmin> print
type: NSRLA;
name: networker.server;
nsrmmd version: 8.0.1.1.Build.132;
nsrsnmd version: 8.0.1.1.Build.132;
NW instance info operations: ;
NW instance info file: ;
installed products: ;
servers: ;
auth methods: “0.0.0.0/0,oldauth”;
max auth attempts: 8;
administrator: root,
“user=administrator,host=networker.server”,
“user=root,host=networker.server”;
kernel arch: x86_64;
CPU type: x86_64;
machine type: desktop;
OS: Linux 2.6.18-194.el5;
NetWorker version: 8.0.1.1.Build.132;
client OS type: Linux;
CPUs: 8;
MB used: 969534;
IP address: 10.204.4.77, “fe80::223:7dff:fedc:568”;
environment variable names: ;

nsradmin> update auth methods: “0.0.0.0/0,nsrauth/oldauth”
auth methods: “0.0.0.0/0,nsrauth/oldauth”;
Update? y
updated resource id 4.0.230.115.0.0.0.0.82.37.185.74.0.0.0.0.10.204.4.77(225)
nsradmin> print
type: NSRLA;
name: networker.server;
nsrmmd version: 8.0.1.1.Build.132;
nsrsnmd version: 8.0.1.1.Build.132;
NW instance info operations: ;
NW instance info file: ;
installed products: ;
servers: ;
auth methods: “0.0.0.0/0,nsrauth/oldauth”;
max auth attempts: 8;
administrator: root,
“user=administrator,host=networker.server”,
“user=root,host=networker.server”;
kernel arch: x86_64;
CPU type: x86_64;
machine type: desktop;
OS: Linux 2.6.18-194.el5;
NetWorker version: 8.0.1.1.Build.132;
client OS type: Linux;
CPUs: 8;
MB used: 969534;
IP address: 10.204.4.77, “fe80::223:7dff:fedc:568”;
environment variable names: ;
nsradmin>

Also remember to email licensing@emc.com to get a new enabler code.
Provide the following.

host id
update enabler
server enabler

All the above can be found under registrations > Update for NetWorker 8

no comments

1
October

Taking VMware vSphere Storage APIs for Data Protection to the Limit: Pushing the Backup Performance Envelope

  VMworld 2012 in San Francisco has come and gone. VMworld EMEA in Barcelona is next week. It is a great event for technical audience; filled with numerous educational sessions, the largest solution pavilion and a lot of super awesome parties! VMworld issues call for papers each year. These […]

Click here to view original web page at www.symantec.com

no comments

1
October

Startup Nakivo Protects VMware Virtual Machines With Amazon Cloud Storage

Page 1 of 2 Nakivo, a startup developer of cloud storage technology, has entered the market with a cloud storage technology that takes advantage of Amazon Web Services to provide the capacity to protect VMware virtual machines and their data. Nakivo Backup & Replication provides customers and their channel […]

Click here to view original web page at www.pheedcontent.com

no comments

1
October

How Much Is Your Time Worth?

How Much Is Your Time Worth? If you’re like me, no matter what you do, your to-do list doesn’t seem to get any shorter. Laundry piles seem to reappear overnight; there’s always lawn to be cut and gardens to be weeded (does it seem like your yard gets bigger […]

Click here to view original web page at thebackupwindow.emc.com

no comments

1
October

How to make money from data protection

By Tom Giuliano, Senior Product Marketing Manager, EMC Backup Recovery Systems Division Data protection is like insurance – throw money at the chance that something will go wrong and hope that it doesn’t. But, with cloud computing models and the growing relevance of IT-as-a-service, data protection can make money for […]

Click here to view original web page at thebackupwindow.emc.com

no comments

14
September

Pogoplug Builds SOHO Cloud Storage On Amazon Glacier

Page 1 of 2 Cloud storage vendor Pogoplug has unveiled a new cloud storage service that marries a local hard drive to Amazon’s new low-cost Glacier archiving service to provide a cloud-based disaster recovery service for consumers and small businesses. The combination of the Pogoplug device and the Amazon […]

Click here to view original web page at www.pheedcontent.com

no comments

13
September

Acronis buys GroupLogic, targets mobile device backup

Backup software vendor Acronis Inc. today acquired file-sharing software company GroupLogic Inc. with the goal of extending its data protection capabilities to mobile device backup and file collaboration . GroupLogic will become a division of Acronis that will be headed by GroupLogic’s former CEO, Chris Broderick. Acronis did not […]

Click here to view original web page at www.pheedcontent.com

no comments

Back to top