zeeff@b-tech.ann-arbor.mi.us (Jon Zeeff) (11/07/90)
>I have often wondered if there was a nice easy way to expire news until >a specified quantity of disk space if freed. This would help to maintain >free disk space which is currently consumed when news breaks through a >bottleneck in the network. Under this circumstance, an expire which What you need is my rnews.c package. It's faster, more secure, and does smart expires (you never overflow your news partition). Let me know if you want a copy. -- Jon Zeeff (NIC handle JZ) zeeff@b-tech.ann-arbor.mi.us
henry@zoo.toronto.edu (Henry Spencer) (11/07/90)
In article <1990Nov6.181747.21191@bhpcpd.kembla.oz.au> bernd@bhpcpd.kembla.oz.au (Bernd Wechner) writes: >"expire as usual, but keep going if less than X MB of free space exists" To do this well, especially in the presence of selective expiry as in C News, means you have to know how big each article is. We seriously looked at providing C News expire with this information, but the more we thought about the policy side, the more complicated it looked, so in the end we didn't do it. The only *simple* way to do it is repetitive expiry. >Although I haven't tried it I expect I can whip something up using the -n >option of Cnew's expire incrementing the date until the space is >acceptable... Please don't use -n for this without thinking about it real hard; -n is really meant only for testing. The preferred way to do it would be to dynamically generate the explist file with successively tighter numbers. That would also permit being selective about policy rather than applying the same to everything. -- "I don't *want* to be normal!" | Henry Spencer at U of Toronto Zoology "Not to worry." | henry@zoo.toronto.edu utzoo!henry
bernd@bhpcpd.kembla.oz.au (Bernd Wechner) (11/07/90)
I have often wondered if there was a nice easy way to expire news until a specified quantity of disk space if freed. This would help to maintain free disk space which is currently consumed when news breaks through a bottleneck in the network. Under this circumstance, an expire which keeps news for a specified period of time will let the disk overfill. In essence it would be nice to say: "expire as usual, but keep going if less than X MB of free space exists" Has this been done? Have I not RTFM closely enough? Although I haven't tried it I expect I can whip something up using the -n option of Cnew's expire incrementing the date until the space is acceptable. This would be slow though as it might possibly involve several expire runs. -- Bernd Wechner, Research Officer (bernd@bhpcpd.kembla.oz.au) BHP Coated Products Division, Research and Technology Centre Port Kembla, New South Wales, Australia.
mjr@hussar.dco.dec.com (Marcus J. Ranum) (11/07/90)
In article <1990Nov6.181747.21191@bhpcpd.kembla.oz.au> bernd@bhpcpd.kembla.oz.au (Bernd Wechner) writes: >I have often wondered if there was a nice easy way to expire news until >a specified quantity of disk space if freed. Nifty idea. Hmmm.... I wonder if a lazy expire could be developed. IE: each time expire is run, articles are placed on an "available to trash" list. "spacefor" would then delete as many articles off the trash list as needed to make the space available. That way you could tune your expiry so that when lots of useful discussions about Re^15 "should I check the return value of close(2)?" come across unix wizards, er, internals, it would grab the needed space out of a less important group, like news.admin. :) mjr.
oc@vmp.com (Orlan Cannon) (11/08/90)
In article <!PD_V*A@b-tech.uucp> zeeff@b-tech.ann-arbor.mi.us (Jon Zeeff) writes: >>I have often wondered if there was a nice easy way to expire news until >>a specified quantity of disk space if freed. > >What you need is my rnews.c package. It's faster, more secure, and >does smart expires (you never overflow your news partition). Said package was just recently posted in "alt.sources" (it reached here today). -- Orlan Cannon oc@vmp.com Video Marketing & Publications, Inc. (800) 627-4551 Oradell, NJ 07649
mrm@sceard.Sceard.COM (M.R.Murphy) (11/08/90)
In article <1990Nov6.181747.21191@bhpcpd.kembla.oz.au> bernd@bhpcpd.kembla.oz.au (Bernd Wechner) writes: >I have often wondered if there was a nice easy way to expire news until >a specified quantity of disk space if freed. This would help to maintain >free disk space which is currently consumed when news breaks through a >bottleneck in the network. Under this circumstance, an expire which >keeps news for a specified period of time will let the disk overfill. > >In essence it would be nice to say: > >"expire as usual, but keep going if less than X MB of free space exists" > >Has this been done? Have I not RTFM closely enough? >[...] I'm sorry this is so long, it started out to be a short reply. I've been running a non-traditional expire under C News for a while. I figured that expiration should be a multi-part thing: 1) keep history of stuff that has come by so that duplicates are not saved or propagated (it is important to keep history of articles that are no longer stored in the news spool), 2) keep enough space in the news spool so that incoming articles like the important things in news.groups can be accepted, 3) get rid of old articles so that new articles like the important things in alt.flame can be accepted, 1) above is the responsiblility of expire. 2) and 3) above are the responsiblity of a "trasher". The expire that comes with C News doesn't work too well in small model :-) However, getdate, spacefor, awk, and find work pretty slick, so... For that reason and for understandability, I replaced expire with a script that goes through the history file, and if the date and time is older than some desired date, don't write the line to a new history file. That is, the script sifts out the old stuff from history. It doesn't bother about deleting any of the files in a history line, it just worries about history. When the new file is complete, then new dbm (or in my case, dbz) files are created. Save the old, copy the new to the old, delete the saved old if you are short on space, and history is ok. The important part of the expire script is: ----- cd $NEWSCTL [ $makehistory = "yes" ] && { rm -f history.n history.n.pag history.n.dir now=`getdate now` ago=`awk "/^\/expired\// {print ($now-(86400*\$(3)))} {next}" explist` awk "{split(\$2,dates,\"~\");if(dates[1]>$ago)print \$0}" history >history.n mkdbm history.n [ -s history.n ] && mv history history.o && # install new ASCII history file mv history.n history && rm -f history.pag && # and related dbm files rm -f history.dir && mv history.n.pag history.pag && mv history.n.dir history.dir ----- This makes the creation of a new history file pretty simple. That's all that this expire does. Note that it grabs the history time from explist, which is otherwise unused. The trasher gets rid of old articles. If there is enough space on the news spool, there is no need to trash, otherwise 1) trash the worthless stuff first, 2) trash the stuff that only deserves keeping for a while, 3) trash from history starting with the oldest first. Since history is automagically in order from oldest first, this is pretty easy. 4) trash everything based on age (including in.coming and out.going, why play favorites :-) To trash is to remove news articles until a high limit is reached. The high limit should be higher than the low limit :-) This prevents unnecssary thrashing. The important part of the trasher looks like this: ----- TRASH=${NEWSCTL}/trashlist # make HILIM - LOLIM about a day or so of news in blocks LOLIM=25000 HILIM=50000 # if below low limit, trash files until high limit reached LC=`spacefor 512 articles` if [ $LC -lt $LOLIM ] ; then cat $TRASH | while read GRP FLGS do LC=`spacefor 512 articles` if [ $LC -lt $HILIM ] ; then find $GRP -type f $FLGS -print 2>/dev/null|xargs rm -f else break fi done LC=`spacefor 512 articles` fi # if still below low limit, remove oldest files from history ( inefficient ) if [ $LC -lt $LOLIM ] ; then sed -e 's/.* //' -e '/^$/d' -e '/control/d' -e 's/\./\//g' $NEWSCTL/history | while read art arts do if [ $LC -lt $HILIM ] ; then [ -f "./$art" ] && { rm -f $art $arts LC=`spacefor 512 articles` } else break fi done else break fi ----- Our $NEWSCTL/trashlist looks like (order counts): ----- in.coming -mtime +5 out.going -mtime +5 psu alt.sex junk -mtime +1 control -mtime +3 talk -mtime +1 soc -mtime +1 alt.flame -mtime +1 alt -mtime +4 trial -mtime +5 news.groups -mtime +2 news.lists -mtime +3 comp.os.vms -mtime +2 comp.mail.maps -mtime +2 comp.org.fidonet comp.binaries.amiga comp.binaries.apple2 comp.binaries.atari comp.binaries.hypercard comp.binaries.mac comp.binaries.os2 comp.binaries -mtime +1 comp.sys.amiga comp.sys.apple2 comp.sys.atari to -mtime +10 sci -mtime +2 rec -mtime +2 ca -mtime +14 comp -mtime +5 gnu -mtime +5 junk la -mtime +5 misc -mtime +5 news -mtime +5 rec -mtime +5 sci -mtime +5 sdnet -mtime +5 soc -mtime +5 talk -mtime +5 to -mtime +5 unix-pc -mtime +5 ----- This is our trashlist. If your favorite group is slighted, please accept my apology. If you leave out an heirarchy, it doesn't get expired. It can grow big. If you leave out an mtime, then everything in the heirarchy goes. You can put in an heirarchy more than once with decreasing times. Lots of ways to do what you want to do and to fine tune. To say that this isn't as well checked out as the rest of C News is a mild understatement, be careful, the risk is yours. If anyone feels like putting in fancier flags in trashlist, say to provide archiving before removal, it should be pretty easy. What have I done wrong? Let me know. -- Mike Murphy mrm@Sceard.COM ucsd!sceard!mrm +1 619 598 5874