[comp.unix.questions] Ever Growing Log Files

tmcclory@odin.wright.edu (Tom McClory) (05/07/89)

As a fledgling system administrator, I'm discovering a number
of files that keep growing over time.  Some examples are:
	/usr/adm/wtmp
	/usr/adm/sulog
	various uucp log files
	etc.

My question is, what are the "standard" system files that
forever grow like this unless I do something explicit.
Either a direct answer, or a reference to the manual would
be appreciated.

We're running UNIX V.2 on an NCR TOWER 32-400.  In the not to distant
future, we will be upgrading to UNIX V.3.  Any hints on what changes
between V.2 and V.3 would be welcomed.

I would also appreciate any hints from veteran system administrators
for managing file systems including things that should be included 
into shell scripts run from cron, regular practice, and things not to do.
Disk space on our systems is tight, production systems run on
machines with only 85mb disks, including the O/S.

thanks in advance

tom

madd@bu-cs.BU.EDU (Jim Frost) (05/15/89)

In article <506@thor.wright.EDU> tmcclory@odin.wright.edu (Tom McClory) writes:
|As a fledgling system administrator, I'm discovering a number
|of files that keep growing over time.  Some examples are:
|	/usr/adm/wtmp
|	/usr/adm/sulog
|	various uucp log files
|	etc.
|
|My question is, what are the "standard" system files that
|forever grow like this unless I do something explicit.

Which files exist depend on your particular version of UNIX and which
daemons you are running.  You got most of the common ones,
/usr/adm/messages is another one (and generally the biggest excepting
heavily used programs such as UUCP).

What I do to help manage this is to add something like:

0 1 * * * tail -500 /usr/adm/messages > /usr/adm/messages.tmp \
          mv /usr/adm/messages.tmp /usr/adm/messages ; \
          date +"%h %d %H:%M %y: file truncated to 500 lines" >> \
          /usr/adm/messages

to crontab, which causes the messages file to be truncated to 500
lines (and the truncation logged) every night at 1am.  Note that the
lines should be joined at the backslashes (I don't believe cron would
like the above verbatim).  If you want to do this to a lot of files
you should make a more general-purpose shell script.

This technique has drawbacks; you'll loose older information and in
the worst case (some error happening repeatedly enough to push an
important message beyond the truncation, such as network errors) you
can loose something important.  Generally you know when you're having
a bad time, though, so this technique really helps management.

I really recommend adding this kind of automatic management to
machines which are seldom administered (eg workstations).

jim frost
madd@bu-it.bu.edu

jws@hpcljws.HP.COM (John Stafford) (05/18/89)

Beware of using tail to get the end of a log file and then mv to move
the end back to the original.  Some log files are never closed by their
owner, /usr/lib/cron/log being the most notorious.  Hence

   tail -100 /usr/lib/cron/log >/usr/lib/cron/log.temp
   mv /usr/lib/cron/log.temp /usr/lib/cron/log

is a disaster in the making.  The mv dutifully unlinks the old
/usr/lib/cron/log, but it is still open, so the directory entry
disappears, but the file doesn't.  The mv then creates a new
/usr/lib/cron/log that cron knows nothing about (until a reboot/cron
restart).  So after a while you notice that the tail end that you
"saved" is the only content of /usr/lib/cron/log and that it isn't
growing, but your file system is mysteriously getting full (as cron
dutifully continues to write to the unlinked original /usr/lib/cron/log).

If you want to keep a chunk, I prefer to keep it in a separate file,
something like:

   tail -100 /usr/lib/cron/log >/usr/lib/cron/log.old
   cp /dev/null /usr/lib/cron/log

If you really want to keep a chunk in the old file, it gets tricky

   tail -100 /usr/lib/cron/log >/usr/lib/cron/log.temp
   cp /usr/lib/cron/log.temp /usr/lib/cron/log
   rm /usr/lib/cron/log.temp

may or may not work (I've never tried it, so I don't know).  If cron
opens /usr/lib/cron/log with O_APPEND, it will work, if it doesn't, the
results will likely be other than expected (and are beyond the scope of
this course).