Log File Size Management

networker is down

 

I got to thinking about log files size mgmt after I had an issue this week.

I came in the office to find NetWorker a smoking wreck, as pictured above. Unresponsive and our /nsr directory rapidly running out of space. We had just added 200 GB a few weeks ago. It now was 99% full with only 16 GB left. With a couple of hours that space would be filled also.

I had assumed that the index was the culprit. Perhaps there was huge new data load on one of the clients or the filer?

I was working with the storage team to get some space added, when they informed me it was the daemon.raw and .log files that were over 160 GB in size a growing rapidly. Still not 100% sure what the cause was? We had recently upgraded from 7.6sp3 to 8.0 sp1. There had been some issues with NDMP backups and I had requested the storage team increase the debug level of the NDMP daemon on the filer. I dont think this would inform the daemon files?

Anyway, we also found there were some runaway ndmp process on the filers and killed them.
With the crisis averted it was time to do some analysis on log file mgmt.

I know what you are thinking. Great way to spend a Saturday afternoon 🙂

Again, nsradmin is you’re friend!

To trim the log files I restarted networker. This rolled them over and I then deleted.
Here are our current log files.

-bash-3.2$ ls -lh daemon*
-rw-r–r– 1 root root 36M Feb 19 12:26 daemon_130219_122717.log
-rw-r–r– 1 root root 56M Feb 19 12:26 daemon_130219_122717.raw
-rw-r–r– 1 root root 4.1M Feb 21 10:01 daemon_130221_104654.log
-rw-r–r– 1 root root 6.8M Feb 21 10:01 daemon_130221_104654.raw
-rw-r–r– 1 root root 19M Feb 28 10:29 daemon_130228_115135.raw
-rw-r–r– 1 root root 11M Feb 28 10:29 daemon_130228_115136.log
-rw-r–r– 1 root root 59M Mar 11 09:20 daemon_130311_092040.raw
-rw-r–r– 1 root root 47M Mar 11 09:20 daemon_130311_092041.log
-rw-r–r– 1 root root 13M Mar 16 12:29 daemon.log
-rw-r–r– 1 root root 25M Mar 8 11:37 daemon.log2
-rw-r–r– 1 root root 14M Mar 16 12:29 daemon.raw
So we can see our files aren’t out of control. Lets look at our log mgmt settings.

[root@cls###~]# nsradmin -p nsrexec
NetWorker administration program.
Use the “help” command for help, “visual” for full-screen mode.
nsradmin> . type:NSR log
Current query set

Here we see a few directives

nsradmin> print
type: NSR log;
administrator: root, “user=root,host=cls###.WDenergy.ca”;
owner: NetWorker;
maximum size MB: 2;
maximum versions: 10;
runtime rendered log: /nsr/logs/daemon.log;
runtime rollover by size: Disabled;
runtime rollover by time: ;
name: daemon.raw;
log path: /nsr/logs/daemon.raw;

So here we can see the max file size is set for 2 MB, but the associated directive is disabled. So, our log files can and will grow unabated.

Lets, enable it. NetWorker will then do a hourly check on the file. It will then be rolled over. A maximum of 10 log archives will be kept.

nsradmin> . type:NSR log;name:daemon.raw
Current query set
nsradmin> print
type: NSR log;
administrator: root, “user=root,host=cls###.WDenergy.ca”;
owner: NetWorker;
maximum size MB: 2;
maximum versions: 10;
runtime rendered log: /nsr/logs/daemon.log;
runtime rollover by size: Disabled;
runtime rollover by time: ;
name: daemon.raw;
log path: /nsr/logs/daemon.raw;
nsradmin> update runtime rollover by size: Enabled
runtime rollover by size: Enabled;
Update? y
updated resource id 12.0.230.115.0.0.0.0.82.37.185.74.0.0.0.0.10.204.4.77(3)

Lets view and confirm the change.
nsradmin> print
type: NSR log;
administrator: root, “user=root,host=cls###.WDenergy.ca”;
owner: NetWorker;
maximum size MB: 2;
maximum versions: 10;
runtime rendered log: /nsr/logs/daemon.log;
runtime rollover by size: Enabled;
runtime rollover by time: ;
name: daemon.raw;
log path: /nsr/logs/daemon.raw;
nsradmin> . type:NSR log;name:daemon.log

Had to wait a little bit, but less than an our later the rollover happened!

 

[root@cls### logs]# ls -lh daemon*
-rw-r–r– 1 root root 36M Feb 19 12:26 daemon_130219_122717.log
-rw-r–r– 1 root root 56M Feb 19 12:26 daemon_130219_122717.raw
-rw-r–r– 1 root root 4.1M Feb 21 10:01 daemon_130221_104654.log
-rw-r–r– 1 root root 6.8M Feb 21 10:01 daemon_130221_104654.raw
-rw-r–r– 1 root root 19M Feb 28 10:29 daemon_130228_115135.raw
-rw-r–r– 1 root root 11M Feb 28 10:29 daemon_130228_115136.log
-rw-r–r– 1 root root 59M Mar 11 09:20 daemon_130311_092040.raw
-rw-r–r– 1 root root 47M Mar 11 09:20 daemon_130311_092041.log
-rw-r–r– 1 root root 14M Mar 16 13:59 daemon_130316_135910.raw
-rw-r–r– 1 root root 13M Mar 16 13:59 daemon_130316_135911.log
-rw-r–r– 1 root root 75K Mar 16 14:09 daemon.log
-rw-r–r– 1 root root 25M Mar 8 11:37 daemon.log2
-rw-r–r– 1 root root 24K Mar 16 14:09 daemon.raw

 

 

Category: NetWorker |

no comments

Comments are closed.