vancleef@nas.nasa.gov (Robert E. Van Cleef) (04/02/91)
Thanks to those who replied... To supply context, here was the original question: -------------------------------------------------------------------------- > From: vancleef@nas.nasa.gov (Robert E. Van Cleef) > Subject: log file and mail message filtering programs > Date: Fri, 22 Mar 91 15:12:40 GMT > Organization: NASA/Ames Research Center > Keywords: message filter logfiles errors system administration > > One of the major problems with the administration of a large number > of systems is the large volume of information that is generated everyday > by the systems. > > There is a massive amount of information that is available in the system > log files or system mail messages that the system administrator is forced > to ignore, or may not even be aware of, because of the large amount of > information and the enormous amount of noise. > > (It is almost as bad a trying to keep up with a USEnet newsgroup:) > > Has anyone done any work on developing intelligent filters that can monitor > the information generated by a couple of hundred workstations, filter out the > noise, and summarize the results? > > Any pointers would be welcome, and I will summarize any results that I receive. > > Bob > -- > Bob Van Cleef vancleef@nas.nasa.gov > NASA Ames Research Center (415) 604-4366 > --- > Perception is reality... -------------------------------------------------------------------------- Here is a summary of the replies. Apparently there is only one tool "watcher" freely available and one commercial product "XRSA" ... Look for a new book - "UNIX Tool Building" by Kenneth Ingham. It includes a description a tool called "watcher", also by Kenneth Ingham, which was also described in the paper: > "Keeping Watch Over the Flocks by Night (and Day)" > by Kenneth Ingham > Proceedings of the Summer 1987 USENIX Technical Conference and Exhibition, > Summer 1987, pp. 105-110. Thanks to: > From: smfedor@solar.lerc.nasa.gov (Gregory Fedor) > From: Fuat C. Baran <fuat@cunixf.cc.columbia.edu> > From: Scott Gasparian <gaspar@inf.ethz.ch> I was also sent some small script examples (included below) Thanks to: > From: Dan Chaney <chaney@ms.uky.edu> There is also a complany called XRSA, that provides a consulting/ system monitoring servic: (more below) > From: dick@ccnext.ucsf.edu (Dick Karpinski) > From: eci386!jmm@zoo.toronto.edu (John Macdonald) -------------------------------------------------------------------------- Here is the full collection of replies: hope it helps... Bob ------------------------- full text follows ------------------------------ > From smfedor@solar.lerc.nasa.gov Fri Mar 22 09:44:15 1991 > Date: Fri, 22 Mar 91 12:43:56 EST > From: smfedor@solar.lerc.nasa.gov (Gregory Fedor) > Message-Id: <9103221743.AA00298@solar.lerc.nasa.gov> > To: vancleef%nas.nasa.gov@amelia.nas.nasa.gov > Subject: Re: log file and mail message filtering programs > Newsgroups: comp.unix.admin > In-Reply-To: <1991Mar22.151240.6626@nas.nasa.gov> > Organization: NASA/Lewis Research Center, Cleveland > Status: RO > > In article <1991Mar22.151240.6626@nas.nasa.gov> you write: > >Has anyone done any work on developing intelligent filters that can > >monitor the information generated by a couple of hundred workstations, > >filter out the noise, and summarize the results? > > > >Any pointers would be welcome, and I will summarize any results that I receive. > > Bob, > > I am currently reading a book title _UNIX Tool Building_ by Kenneth Ingham. > In it he is walking the reader through the building of a utility called > "watcher" that he helped create at the University of New Mexico for monitoring > the status of many system. From what I've read so far, it sounds like this > is what you are looking for. > > Also, from what I gather it's available in comp.source.unix on uunet.uu.net. > I haven't had a chance yet to go check this out (I'm only on page 61 :). I > plan on retrieving it though and trying it out here at Lewis as well as taking > some concepts for a project I'm working on. > > I hope this helps. If you need any further information, drop me a line. > I look forward to hearing what other answers you get. > > -- > =============================================================================== > Gregory A. Fedor | Far from day, far from light \ > Sverdrup Technology Inc. | Out of time, out of sight \ > NASA Lewis Research Center | To a world, young and free \\-^-/___ > Cleveland, Ohio 44135-3191 | Weep no more, follow me |===[o]/ #o > (216) 433-8468 | /VVV > smfedor@lerc01.lerc.nasa.gov | Forever...Forever...Forever / > (128.156.10.14) | Voyagers 1 & 2 > =============================================================================== > > From fuat@cunixf.cc.columbia.edu Fri Mar 22 17:05:08 1991 > Received: by cunixf.cc.columbia.edu (5.59/FCB) > id AA10339; Fri, 22 Mar 91 20:04:57 EST > Date: Fri, 22 Mar 91 20:04:57 EST > From: Fuat C. Baran <fuat@cunixf.cc.columbia.edu> > Message-Id: <9103230104.AA10339@cunixf.cc.columbia.edu> > To: vancleef@nas.nasa.gov > Cc: fuat@cunixf.cc.columbia.edu > Subject: Re: log file and mail message filtering programs > Newsgroups: comp.unix.admin > In-Reply-To: <1991Mar22.151240.6626@nas.nasa.gov> > Organization: Columbia University Center for Computing Activities > Status: RO > > In article <1991Mar22.151240.6626@nas.nasa.gov> you write: > >Has anyone done any work on developing intelligent filters that can > >monitor the information generated by a couple of hundred workstations, > >filter out the noise, and summarize the results? > > Take a look at: > > "Keeping Watch Over the Flocks by Night (and Day)" > by Kenneth Ingham > Proceedings of the Summer 1987 USENIX Technical Conference and Exhibition, > Summer 1987, pp. 105-110. > > Kenneth Ingham has also written a book based on this paper (I'm not > sure of the name but it was something like "UNIX Tool Building." I > have the book at home, and can get you the details if you want). > --Fuat > -- > Internet: fuat@columbia.edu U.S. MAIL: Columbia University > BITNET: fuat@cunixc Center for Computing Activities > UUCP: ...!rutgers!columbia!cunixf!fuat 712 Watson Labs, 612 W115th St. > Phone: (212) 854-5128 Fax: (212) 662-6442 New York, NY 10025 > --------------------------------------------------------------- > From @s.ms.uky.edu:chaney@ms.uky.edu Fri Mar 22 20:34:38 1991 > From: Dan Chaney <chaney@ms.uky.edu> > Date: Fri, 22 Mar 1991 23:33:48 EST > X-Mailer: Mail User's Shell (7.2.0 10/31/90) > To: vancleef@nas.nasa.gov > Subject: Re: log file and mail message filtering programs > Message-Id: <9103222333.aa01669@s.s.ms.uky.edu> > Status: RO > > Newsgroups: comp.unix.admin > References: <1991Mar22.151240.6626@nas.nasa.gov> > > A lot of it depends on what sorts of things you want to keep up with, of > course. I keep track of mail daemons and queues through scripts that > know what 'normal' is and send mail when things don't quite match. That > is helpful to maintain 'running' programs. Checking for the presence > of TCP daemons is fairly simple if you assume the existence is proof > enough of a daemon's state. > echo "quit" | telnet mozart.ms.uky.edu 25 > if [ $? != 0 ]; then > echo "Problem with the daemon" > fi > That tells me if the smtp daemon is running. Along with sendmail -bp's on > other machines, I can usually catch a clogged mailer within an hour or > so (these scripts run every 4 hours, but that is just because I like > diligence in a major way) We also run MMDF on two machines and that > makes for lots of log files. Clever greps and diffs on 'ok' log files > brings my over all system mail down to a reasonable level. One helpful > trick I use is running scripts that write a lot of info to a specific > log - and overwrite the old data. This allows the full data to be at > least accessible, without getting in your way under 'normal' circumstances. > > I guess the theme is to train scripts what is normal or just do diffs > on a 'normal' output. I can provide you the scripts if you want. If > you want some ugly scripts, I'll show you the archive-maintaining scripts > that just tell me how things are and send nagging notes to all my archivers. > A truly obnoxious piece of scripting :-) > > -dan > ------------------------------------------------ > From gaspar@inf.ethz.ch Sat Mar 23 03:44:37 1991 > From: Scott Gasparian <gaspar@inf.ethz.ch> > Message-Id: <9103231146.AA06449@orion.inf.ethz.ch> > Cc: gaspar@orville.nas.nasa.gov > Subject: Re: log file and mail message filtering programs > Status: RO > > Have you heard of the program called "watcher" ? It takes input > from cron outputs, syslogs, msgs, etc, and compares them. If > something changes past a certain parameter (say load goes over > 20 or disk free goes over 90%), it mails a msg to set people. I > will try and remmeber where we got ours and send you more info. > I think it was U of New Mexico or something like that. > > very usefull little utility. Might be in *.sources.something. > > --gaspo. > > /----------------------------------------------------------------------------\ > | Scott "gaspo" Gasparian -- System Administrator | _>________ _<________ | > | Dept. Informatik, Eidg. Techn. Hochschule, Zurich |/[][][][][]\/[][][][][]\| > | ETH-Zentrum, CH-8092 Zurich. T# 01-01-254-7205 |`oo------oo'`oo------oo'| > | gaspar@inf.ethz.ch | "Good friends we've had, or good friends we've lost, | > | ..!ethz-inf!gaspar | along the way.In this proud land,you can't forget your| > | gaspo@scri.fsu.edu | past,so dry your tears I say. No woman, No cry." -BMW | > \----------------------------------------------------------------------------/ > > From dick@ccnext.ucsf.EDU Mon Mar 25 17:16:07 1991 > From: dick@ccnext.ucsf.edu (Dick Karpinski) > Message-Id: <9103260115.AA17508@ ccnext.ucsf.edu > > To: vancleef@nas.nasa.gov > Subject: XRSA does just that > Status: RO > > There is a commercial product from a software house in Canada which > does just that sort of thing. It's called eXpert Remote System > Administrator and uses possibly some AIish software in the central > host to reduce the data coming in to just the part that's most > interesting to the human attendants. They seem to want $20k/yr to > get into the game, so I'm interested in cheap clones. Many of us > human administrators ought to be willing to collaborate on a public > access package like that. PERL pops to mind as a useful tool for > many of these tasks. I have lotsa stuff from the xrsa folks if > that would interest you further. I'd like to pursue this matter > to the point of having some tools and a continuing sysadmin mailing > list for enhancements etc.... > > Dick > > Dick Karpinski Minicomputer Manager, UCSF Information Technology Services > Domain: dick@cca.ucsf.edu FAX: (415) 476-9537 (415) 476-4529 (11-7) > BITNET: dick@ucsfcca or dick@ucsfvm (415) 658-6803 (Home) > USPS: U-76 UCSF, San Francisco, CA 94143-0704 (415) 658-3797 (ans) > -------------------------------------------------------------------------- > From eci386!jmm@zoo.toronto.edu Tue Mar 26 09:16:12 1991 > From: eci386!jmm@zoo.toronto.edu (John Macdonald) > Date: Tue, 26 Mar 1991 11:53:38 EST > Newsgroups: comp.unix.admin > In-Reply-To: <1991Mar22.151240.6626@nas.nasa.gov> > Organization: Elegant Communications Inc. > X-Mailer: Mail User's Shell (7.1.2 7/11/90) > To: vancleef@nas.nasa.gov > Subject: Re: log file and mail message filtering programs > Message-Id: <9103261153.AA12599@eci386.UUCP> > Status: RO > > In article <1991Mar22.151240.6626@nas.nasa.gov> you write: > |One of the major problems with the administration of a large number > |of systems is the large volume of information that is generated > |everyday by the systems. > | > |There is a massive amount of information that is available in the system > |log files or system mail messages that the system administrator is forced > |to ignore, or may not even be aware of, because of the large amount of > |information and the enormous amount of noise. > | > |(It is almost as bad a trying to keep up with a USEnet newsgroup:) > | > |Has anyone done any work on developing intelligent filters that can monitor > |the information generated by a couple of hundred workstations, filter out > |the noise, and summarize the results? > | > |Any pointers would be welcome, and I will summarize any results that I receive. > > Well, we have had some previous email discussions about > XRSA - it can do much of this, and can be extended by us > to add the rest as a consulting project to any degree of > detail that you are willing to have us address. > > XRSA does a great deal of reduction and analysis of many > log files already. The reports that it generates are of > two major categories - daily and urgent. Daily reports > show interesting details about the systems. Urgent > reports only show indications of upcoming and current > problems. We typically expect that sys admins will normally > read urgents, and will read dailies only on a casual basis > or to obtain detailed background info for an unusually puzzling > urgent problem. > > There is a (very brief) summary provided for a group of > systems (currently it just states whether logs were > received, and whether there was an urgent condition, for > each monitored system in the group) which could be easily > extended to summarize any particular condition that you > wished to oversee. > > Feel free to request additional info from me. > > -- > Cure the common code... | John Macdonald > ...Ban Basic - Christine Linge | jmm@eci386 > --------------------------------------------------------------------------- > From eci386!jmm@zoo.toronto.edu Wed Mar 27 06:05:13 1991 > From: eci386!jmm@zoo.toronto.edu (John Macdonald) > Date: Wed, 27 Mar 1991 08:47:24 EST > In-Reply-To: Message dated Tue Mar 26 10:19 from vancleef@garg.nas.nasa.gov (Robert E. Van Cleef) Re: "Re: log file and mail message filtering programs" > X-Mailer: Mail User's Shell (7.1.2 7/11/90) > To: vancleef@garg.nas.nasa.gov (Robert E. Van Cleef) > Subject: Re: log file and mail message filtering programs > Message-Id: <9103270847.AA26712@eci386.UUCP> > Status: RO > > /===== Re: log file and mail message filtering programs ===== > || Quoting Robert E. Van Cleef, message dated Mar 26, 10:19 > |+----- > || John; > || > || Unfortunately, when I read the stuff you sent previously I interpreted > || it as a consulting service setup. I will see if I can dig out the oldd > || mail messages and re-read them... > || > || Bob > \========================= > > Hmm, our usual (almost constant) problem is that people > try and treat XRSA as strictly a product. In fact, it > is closer to being a consulting service than a product, > but a major portion of the consulting activity is carried > out automatically by software. > > Essentially, we provide consulting and licensed software > to a service provider, who can then use this to provide > sys admin service as a product to their customers. The > service provider can be either an separate company that > is providing service as a marketed product to its customers > (currently we have marketing agreements of this sort with > IBM and Bull and others of their ilk are close to signing), > or it can be a central support department within a large > organization that provides service to the rest of the > organization. > > -- > Cure the common code... | John Macdonald > ...Ban Basic - Christine Linge | jmm@eci386 > --------------------- end of forwarded material ------------------- Bob Van Cleef - vancleef@nas.nasa.gov RNS Distributed Systems NASA Ames Research Center (415) 604-4366 Mail Stop 258-6 FTS 464-4366 Moffet Field, CA 94035-1000 FAX (415) 604-4377 __ "If you're not a liberal at 20, you have no heart, and if you're not a conservative at 40, you have no head." Winston Churchill
jmm@eci386.uucp (John Macdonald) (04/08/91)
In article <1991Apr1.160108.12136@nas.nasa.gov> vancleef@nas.nasa.gov (Robert E. Van Cleef) writes: [ a summary of responses to his request for methods for automation of log analysis ] |Here is a summary of the replies. Apparently there is only one tool "watcher" |freely available and one commercial product "XRSA" ... [ ... ] |-------------------------------------------------------------------------- |Here is the full collection of replies: hope it helps... Bob [ ... ] |> From: dick@ccnext.ucsf.edu (Dick Karpinski) |> Subject: XRSA does just that |> |> There is a commercial product from a software house in Canada which |> does just that sort of thing. It's called eXpert Remote System |> Administrator and uses possibly some AIish software in the central |> host to reduce the data coming in to just the part that's most |> interesting to the human attendants. They seem to want $20k/yr to |> get into the game, so I'm interested in cheap clones. Many of us |> human administrators ought to be willing to collaborate on a public |> access package like that. PERL pops to mind as a useful tool for |> many of these tasks. I have lotsa stuff from the xrsa folks if |> that would interest you further. I'd like to pursue this matter |> to the point of having some tools and a continuing sysadmin mailing |> list for enhancements etc.... Umm, I hate to look like I'm doing marketing on the net, but Dick's figure is wrong except maybe in a specific sort of context. The base price for XRSA is about $2k/yr per system monitored. There are additional considerations possible (like if you want to license the entire suite of software and not use an external server it does get up to a starting price of $20k/yr, but that includes a minimum of 5 systems being supported). Robert's summary of replies included mine, so I won't repeat that info here, but anyone interested can send me email with any specific questions or for general info. We agree with Dick that Perl is a useful tool for doing many of the tasks - we use it in the central analysis portion of XRSA. -- sendmail - as easy to operate and as painless as using | John Macdonald manually powered dental tools on yourself - John R. MacMillan | jmm@eci386
rodgers@clausius.mmwb.ucsf.edu (04/14/91)
In <1991Apr8.145915.6596@eci386.uucp> jmm@eci386.uucp (John Macdonald) writes: >|Here is a summary of the replies. Apparently there is only one tool "watcher" >|freely available and one commercial product "XRSA" ... No, there is also the System Manager's Toolkit, from the Office of Technology Licensing at Berkeley. Contact otl@violet.berkeley.edu for information... Cheerio, Rick Rodgers R. P. C. Rodgers, M.D. (415)476-2957 (work) 664-0560 (home) UCSF Laurel Heights Campus UUCP: ...ucbvax.berkeley.edu!cca.ucsf.edu!rodgers 3333 California St., Suite 102 Internet: rodgers@maxwell.mmwb.ucsf.edu San Francisco CA 94118 USA BITNET: rodgers@ucsfcca
ingham@triton.unm.edu (Kenneth Ingham) (04/15/91)
>In article <1991Apr1.160108.12136@nas.nasa.gov> vancleef@nas.nasa.gov (Robert E. Van Cleef) writes: >Here is a summary of the replies. Apparently there is only one tool "watcher" >freely available and one commercial product "XRSA" ... The version of watcher available from the comp.sources.unix archives is a bit old. A newer version can be gotten via anonymous ftp from ariel.unm.edu. -- Kenneth Ingham ingham@ariel.unm.edu Hummin' lil Grumman N9646L