rd@tarpit.uucp (Bob Thrush) (07/17/89)
I am finally posting the fix to the dreaded "vanishing inode
problem" for V/AT. The problem, as many have already indicated, is
due to an incorrect implementation of the inode cacheing algorithm in
the kernel ialloc and ifree functions. Rather than trying to patch
the problem, the fix replaces alloc.o in lib1. I expected to
have posted this much earlier; however, it seems that many things
conspired to hold it up. Anyway, with the advent of C News, this fix
is even more appropriate. This posting is followed by the
instructions for installing the fix.
Several people have tested the attached patch and have reported
favorable success. I have been using it since March 28 and it has
been weathering a full incoming news feed plus a few full/partial
outgoing feeds. I have not had a problem related to suddenly
vanishing inodes. Thanks to Mike Murphy and John Limpert for
independently testing the fix. (There were others whose names I
have misplaced. Sigh ;-})
I had gotten a very good description of the problem courtesy of
wayne@teemc. During several long nights, I managed to decompile
the errant alloc.o kernel module, add the few changes that were
mentioned in the attached posting, and remake the system.
On Nov. 24, 1987, Mayer Ilovitz posted a complete description of
the problem and a simple test to illustrate it. This helped me
understand the problem, fix it and exercise it. I have attached a
shar of that posting for those that wish to delve deeper,
otherwise hit 'n' now.
#! /bin/sh
# This is a shell archive. Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file". To overwrite existing
# files, type "sh file -c". You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g.. If this archive is complete, you
# will see the following message at the end:
# "End of shell archive."
# Contents: inode.notes
# Wrapped by rd@support on Sun Jul 16 20:01:48 1989
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'inode.notes' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'inode.notes'\"
else
echo shar: Extracting \"'inode.notes'\" \(14555 characters\)
sed "s/^X//" >'inode.notes' <<'END_OF_FILE'
XReplied: Thu, 03 Mar 88 02:04:40 EST
XReplied: sharkey!mailrus!rutgers!ksuvax1.cis.ksu.edu!scott (Scott Hammond)
XForwarded: Thu, 03 Mar 88 02:02:17 EST
XForwarded: Jeff Lewis <pur-ee!lewie>
XReturn-Path: uucp
X>From mailrus!rutgers!ksuvax1.cis.ksu.edu!/dev/null Wed Mar 1 12:17:45 1989 remote from sharkey
XReceived: by mailgw.cc.umich.edu (5.59/1.0)
X id AA10725; Wed, 1 Mar 89 12:17:45 EST
XReceived: by mailrus.cc.umich.edu (5.59/1.0)
X id AA04178; Wed, 1 Mar 89 12:22:13 EST
XReceived: from ksuvax1.UUCP by rutgers.edu (5.59/SMI4.0/RU1.1/3.03) with UUCP
X id AA07118; Wed, 1 Mar 89 12:05:02 EST
XReceived: by ksuvax1.cis.ksu.edu
X (5.59++/CIS1.1) id AA05544; Wed, 1 Mar 89 11:04:53 CST
XDate: Wed, 1 Mar 89 09:02:24 PST
XFrom: sharkey!mailrus!rutgers!ksuvax1.cis.ksu.edu!scott (Scott Hammond)
XMessage-Id: <8903011702.AA07131@rdahp.UUCP>
XTo: mailrus!sharkey!teemc!wayne, mailrus!sharkey!teemc!wayne
XSubject: Re: Dreaded System V inode problem
XNewsgroups: comp.bugs.sys5
XIn-Reply-To: <39256@teemc.UUCP>
XOrganization: R & D Associates, Marina del Rey, CA
XCc:
X
XIn article <39256@teemc.UUCP> you write:
X>
X> Several months ago, there was a discussion here about the way
X>System V eats inodes under certain circumstances (notably on the news
X>partition). Could someone who has corrected this please mail me a complete
X>description that I could forward to my vendor? I saved what I believed to
X>be pertinent articles at the time but they appear insufficient for him to
X>locate and correct the problem.
X>
X
XThis is the article I saved describing the problem. I have System V
Xsource code, and this explanation was sufficient for me to be able to
Xfind the bug myself. Unfortunately I haven't been able to 'fix' the
Xproblem because our source does not correspond to the release running
Xon our news machine, nor does it match the hardware.
X
X[I've been having some trouble with my main mail relay, so I'm sending
X this two ways. My apologies if you get two copies. Also beware the
X reply address.]
X--
XScott Hammond
XR & D Associates, Marina del Rey, CA
XEmail: rdahp!scott, rdahp!scott@sm.unisys.com, scott@harris.cis.ksu.edu
X--
XArticle 1730 of news.admin:
XPath: ksuvax1!uxc!tank!oddjob!uwvax!rutgers!njin!princeton!njsmu!mccc!pjh
X>From: pjh@mccc.UUCP (Pete Holsberg)
XNewsgroups: news.admin
XSubject: Re: Alzheimer's Syndrome
XKeywords: LONG!
XMessage-ID: <176@mccc.UUCP>
XDate: 26 Sep 88 16:35:54 GMT
XReferences: <576@mbph.UUCP> <406@amyerg.UUCP> <10611@ulysses.homer.nj.att.com> <12608@ncoast.UUCP>
XReply-To: pjh@mccc.UUCP (Pete Holsberg)
XOrganization: The College On The Other Side of Route 1
XLines: 251
X
XHere's what I found in my archives about the inode problem:
X
XPath: mccc!princeton!rutgers!cmcl2!phri!cooper!mayer
X>From: mayer@cooper.cooper.EDU (Mayer Ilovitz )
XNewsgroups: comp.sys.att,comp.unix.wizards
XSubject: Analysis & test for 3b inode problem: applies to ALL users of SYSTEM V
XKeywords: 3b, SYSTEM V, inodes
XMessage-ID: <1133@cooper.cooper.EDU>
XDate: 24 Nov 87 20:06:46 GMT
XOrganization: The Cooper Union (NY, NY)
XLines: 233
X
X
X Since I haven't seen anyone post a full description of the problem or
Xa test for it, here is my contribution.
X
X This document contains what I believe is a complete analysis of
Xthe System V inode allocation system and the problem that everyone is having.
XI have included a test procedure which should detect the problem on a UNIX
Xsystem and included a program that will help you perform the tests. Also I
Xhave some suggestions on properly fixing the bug.
X
X To begin with, let me describe what I have available to me.
XWe have a number of pretty standard Unix-PCs running System V 3.5 and
XSystem V 3.0 . We have a pair of 3b2-400 s running System V 3.0 Version 2. These
Xmachines each have a floppy diskette system. We also have an OLD 3b5 running
XSystem V Release 2 Version 2. I have access to the source for the Unix on the
X3b5 and the 3b2 systems. Our 3b5 runs our newsfeed using the rnews package.
XThis system has suffered the inode problems that everyone has been mentioning
Xon the net for the last few weeks. Since this system has no expendable files
Xsystems, I ran the tests on the 3b2 and the Unix-PC . Both of these systems
Xshowed the same error. From this, I suspect that all versions of ATT System V
Xunix have the problem. Furthermore, this problem may very well be in any
XATT System-V compatible version of Unix and may well have been present in
XSystem-III Unix. I therefore suggest, just to be on the safe side, that you
Xrun the test described below.
X
X The analysis and test was based on the source from the 3b5. A cursory
Xexamination of the source to the 3b2 showed the code to be essentially the
Xsame in the critical area though there are what appear to be minor cosmetic
Xchanges. For those of you with access to the souces, The file that needs
Xchanging is called alloc.c or s5alloc.c . If you don't have a file by this
Xname, look for a file that closely matches one of these names. The function
Xthat is causing the problem is called s5ialloc() or ialloc() .
X
X As far as I can tell ialloc and ifree are the low level inode
Xallocation control system. When an inode is needed, a call to ialloc() is made.
XWhen a file/directory is deleted, ifree() is used to release the inode.
XThese 2 functions use certain parameters that are kept in the superblock of
Xevery file system. tinode is the total number of free inodes in a file system.
XTo speed up inode allocation and freeing, the superblock maintains a table of
Xfree inodes. This table is called inode[]. The size of this table is given
Xby the #defined value NICINOD and is usually 100. ninode specifies the number
Xof free inodes available in inode[].
X
X When ifree() releases an inode, it first checks to see
Xif the inode table is full. If it isn't, the inode is added to the top of the
Xtable and ninode is adjusted. If the table is full and the inode being released
Xis less than the inode stored in inode[0], the newly released inode is put into
Xinode[0]. In this way, the allocation system knows where in the i-list a group
Xof free inodes are likely to be.
X
X When ialloc() is called, it tries to give the requesting process an
Xinode from inode[]. If none are available, ialloc() searches the i-list for
Xmore free inodes to reload inode[]. ialloc() will start this search begining
Xat the location of the last allocated inode as indicated by inode[] and
Xninode. The search continues untill NICINOD inodes are located or the end of
Xthe i-list is reached. inode[] will be reloaded from the top of the table
Xworking down to inode[0]. A mark is put in inode[] if less than 100 nodes were
Xfound. The next time inode[] runs out of nodes, this mark tells it to search
Xthe i-list from the very begining. If NO inodes were found during the search,
Xninode is SET TO 0 and the out of inodes error is printed on the system console.
X
X The problem that everyone is having is caused by the following
Xsituation. At the last reloading of inode[] exactly NICINOD inodes were found.
XTherefore, the inode at inode[0] is where the next search for inodes will begin.
XAs the system runs, more inodes are allocated and freed. Eventually, the last
Xfree inode in inode[] is allocated. The system waits until the next call to
Xialloc to determine if it needs to reload inode[]. If a node is released before
Xthe inode table is reloaded, the freed inode will go into inode[0], replacing
Xthe old value which would be used for searching the i-list. If the freed inode
Xwas higher in the i-list than the one it replaced in the table, ialloc will no
Xlonger know that it should check the lower portion of the i-list for free
Xinodes. It will think that everything below inode[0] is allocated already.
XIf a significant number of lower valued inodes are not freed before ialloc
Xhas to reload the inode table, ialloc will fail to find any free inodes even
Xthough they exist. Furthermore, because of the coding of ialloc(), unless an
Xinode is freed at some point, every time it tries looking for more inodes, it
Xwill start at the same place. So until the file system is dismounted and fsck'd,
Xunless some inodes are freed, the system will be stuck repeating the same search
Xand reporting the same failure.
X
X The original intent of the ialloc() - ifree() system is to minimize
Xthe time to find more free nodes by remembering the best location to start
Xsearching for more free inodes. Therefore, the best fix to ialloc would be
Xto first try to give the requesting process a free node. ialloc() should
Xthen IMMEDIATELY check to see if that was the last free inode it had, and if
Xit was, try reloading the inode table right then. This will prevent the
Xpossibility of the system from forgetting about the best place to search for
Xinodes. A side result of this is that the out of inodes message will appear
Xwhen the last free inode is allocated and not when ialloc failed to give
Xan inode. An argument could be made either way as to wether this side effect
Xis good or not. The other fix is to put a kludge into ialloc that, in the
Xevent that NO free inodes were found, it would immediately recycle through
Xthe i-list from the very beginning looking for inodes before deciding that there
Xare no free inodes left. If the i-list is large, this can be somewhat
Xinefficient.
X
X
X PROCEDURE TO TEST FOR THE 3B INODE ALLOCATION BUG
X
X
X This test is intended to be run on a floppy-based file system or an
Xexpendable file system. It is assumed that NICINOD, the number of inodes that
Xare stored in the superblock inode table is 100. If not, the test will have
Xto be adjusted accordingly.
X
X 1. create a file system with ~ 280 inodes using mkfs
X fsck the disk and mount it /mnt
X
X 2. verify with a df -t as to the number of free inodes and the total
X number of inodes in this file system.
X
X 3. allocate all the inodes on this filesystem. You can use the program
X fillnode given at the bottom of this document to help you do
X the job. The final result is that there should be 0 inodes left.
X Each file that you made on this disk should be named after its
X respective inode.
X
X 4. unmount the filesystem, do an fsck of the disk, remount and
X verify with a df -t that there are no free inodes.
X
X 5. free up the files with inodes 3-202. This will give you 200 free
X inodes on the filesystem. Verify this using step 4.
X
X 6. at this point, the file system will be mounted and the superblock
X inode table will contain inodes 3-102 for immediate allocation.
X
X 7. use fillnode to reallocate inodes 3-102. at this point you will have
X 100 free inodes when you do a df. This is the correct number of
X free nodes. At this point the superblock inode table will be
X empty.
X
X 8. use fillnode to allocate 1 inode. the inode that will be allocated is
X inode # 103. At this point the superblock inode table will have
X been reloaded from the i-list. the 0 element in the table will
X be inode 202 and the 99th element will be inode 103, which you
X just allocated.
X
X 9. Delete in order the files with inodes 30-39. At this point, the 0
X element in the inode table will be inode 31 while the 99th
X element will be inode 30. When you released inode 30, the
X table was not full, so it was put onto the top of the table.
X When inode 31 was released, the table was full so ifree checked
X to see if the just freed inode was less than the inode in the
X 0th element of the table. Since the 0th element up to this time
X was 202, ifree replaced the 0th element with inode 31. Note,
X The inode table is now full, containing 100 free inodes, the
X lowest free inode in the entire i-list being in the 0th element
X of the table. As you release inodes 32-39, they will fail the
X test by ifree, the result being that these inodes ARE free but
X simply aren't in the inode table. This is alright since when
X ialloc must reload its inode table it will start looking with
X the inode referenced in the 0th element of the table.
X
X 10. allocate another 100 inodes. fillnode will allocate in order
X inodes 30,104-201 and inode 31. At this point the superblock
X inode table is empty again. However, as always, ialloc will
X leave the table empty until it must allocate an inode and finds
X no inodes in the table.
X
X 11. free inode 240. At this point you have sealed your doom ! .
X ifree will put this inode into the lowest available entry in the inode table, DESTROYING ANY MEMORY THAT THE LOWEST FREE INODE IS
X AROUND INODE 31.
X
X 12. Do a df -t to confirm that you still have ~ 10 free inodes.
X
X 13. allocate an inode. This inode will be inode 240.
X
X 14. Do a df -t to confirm that you now have ~ 9 free inodes.
X
X 15. Call fillnode again and say goodbye to your free inodes!
X At this point you will get an out of inodes error on your
X console and the allocation attempt will return failure. A df -t
X say that there are NO free inodes. What happened was that after
X step 13 there were no free nodes in the superblock inode table.
X At this point, ialloc went searching through the i-list for
X more free inodes starting at the inode specified in the 0th
X element of the inode table. BUT this no longer references inode
X 31, where we know there is more space, but inode 240. ialloc
X searches from inode 240 to the end of the i-list, but all those
X inodes are allocated, so ialloc decides that there are no more
X free inodes and reports the out of inodes error,EVEN though
X you still have free inodes!.
X
X 16. unmount the filesystem. Do an fsck. This will report a bad inode
X count in the Superblock ( Sound familiar ) which you must
X fix. Remount and do an df -t to confirm that you really do
X still have a number of free inodes.
X
X IF THE SITUATIONS DESCRIBED IN THIS TEST HAPPEN TO YOU
X
X AND YOU ARE HAVING PROBLEMS BECAUSE OF THIS BUG
X
X CONTACT YOUR ATT CUSTOMER/TECH SUPPORT REP AND REPORT THE PROBLEM
X
Xbelow is the code for fillnode.c . This program will create a file in /mnt.
XThe file created will be named after the inode to which it was allocated.
XThe file will have 0 blocks allocated to it.
X
X#include <fcntl.h>
X#include <sys/types.h>
X#include <sys/stat.h>
Xmain()
X{
X int link(),open(),close(),fstat();
X struct stat buf;
X int fd;
X char name[30];
X
X if( (fd = open("/mnt/XXX",O_CREAT | O_WRONLY,0666) ) < 0 )
X {
X printf("can't open file\n");
X exit(2);
X }
X if( fstat(fd,&buf) < 0 )
X {
X printf("error fstating file\n");
X exit(3);
X }
Xprintf("inode is %d\n",buf.st_ino);
X sprintf(name,"/mnt/%d",buf.st_ino);
X close(fd);
X if( link("/mnt/XXX",name) < 0 )
X {
X printf("can't link to new name\n");
X exit(3);
X }
X if( unlink("/mnt/XXX") < 0 )
X {
X printf("can't unlink old file /mnt/XXX\n");
X exit(3);
X }
X exit(0);
X}
END_OF_FILE
if test 14555 -ne `wc -c <'inode.notes'`; then
echo shar: \"'inode.notes'\" unpacked with wrong size!
fi
# end of 'inode.notes'
fi
echo shar: End of shell archive.
exit 0
--
Bob Thrush UUCP: {ucf-cs,rtmvax}!tarpit!rd
Automation Intelligence, 1200 W. Colonial Drive, Orlando, Florida 32804rd@tarpit.uucp (Bob Thrush) (07/17/89)
Here are the rather brief instructions for rebuilding the V/AT
kernel to fix the vanishing inode problem as mentioned in the
previous posting. (You must have access to the Software
Development System to do this.)
0. Become root.
1. cd to the cf directory of your linkkit (usually /usr/linkkit/cf).
2. uudecode the attached as alloc.o in this directory.
3. (optional) you may want to backup ../lib1 since it will be updated.
4. `ar rv ../lib1 alloc.o' # This will replace the existing alloc.o
5. `make',`make install' and then reboot with the new kernel.
The original alloc.c apparently was unchanged since V/AT
1.3.8, so it is likely that systems older than 2.4 would be able
to use the corrected module. Beware that it has not been tested
on anything other than V/AT 2.4.
You may see a new advisory console message after installing
this fix. It will occur whenever the vanishing inode problem
would have otherwise appeared and was originally part of my
debugging code. I decided to leave it in, since it generally
indicates that file system may need to be remade with more
inodes. The message to the console will look like the following:
The vanishing inode problem would have occurred on maj(%d),min(%d)
There actually were %d inodes available, trying again
This posting is hereby placed in the public domain. If you
find it useful, let me know. If you have problems with it,
I would be glad to help via email. No other implied guarantees. :-)
----------------- snip here for uuencoded alloc.o -----------------
begin 600 alloc.o
M4@$#`!J,+R3N$0``.```````A!`N=&5X=```````````````)`D``(P```"T
M"@```````+<````@````+F1A=&$````D"0``)`D```0!``"P"0``VA$`````
M```"````0````"YB<W,`````*`H``"@*``!<````````````````````````
M`(````#(#```_W8&FNH&8`!$1(E&^(E6^NL4:@J+1O@%F@'_=OI0F@````"#
MQ`;%=OCVA)H!_W7BQ7;X]T0&__]_`^GY`,5V^/],!HMT!L'F`@-V^(Y>^HM$
M"(M4"HE&_(E6_H/Z`'4)@W[\`'4#Z<T`_W8&_W;^_W;\_W;Z_W;XF@T#,`"#
MQ`J%P'6LQ7;X]T0&__]^`^F!`,5V^/Z$F@'_=O[_=OS_=@::`````(/$!HE&
M](E6]K@``([8]@95!/]U+\5V],5T%HL$Q7;XB40&:,@`BT;X!0@`_W;Z4,5V
M]/]T&(M$%D!`4)H`````@\0*_W;V_W;TF@````"#Q`3%=OC&A)H!`(M&^`6:
M`?]V^E":`````(/$!,5V^/=$!O__?@G%=OB#?`8R?EO_=@9HD`!H;`F:````
M`(/$!L5V^,=$!@``Q7;XQX2J`0``QX2L`0``N```CMAK!@``!5":`````$1$
M_W8&:)``:'L)F@````"#Q`:X``".V,8&500<,\`STLG+_W;^_W;\_W8&F@``
M``"#Q`:)1O2)5O;_=O;_=O2:`````(/$!,5V^/>$K`'__W4(]X2J`?__=`W%
M=OB#K*H!`8.<K`$`Q7;XQH2<`0&+1O2+5O;KJ<@(``#_=@::Z@9@`$1$B4;\
MB5;^Q7;\QH2<`0'K%&H*BT;\!9H!_W;^4)H`````@\0&Q7;\]H2:`?]UXO]V
M!O]V"O]V"/]V_O]V_)H-`S``@\0*A<!T`^G8`,5V_/=$!O__?Q7%=OS'1`8!
M`,5V_,=$"```QT0*``#%=OR#?`8R?'W%=OS^A)H!_W8*_W8(_W8&F@````"#
MQ`:)1OB)5OK%=OR+1`;%=OC%=!:)!&C(`,5V^/]T&(M$%D!`4(M&_`4(`/]V
M_E":`````(/$"L5V_,=$!@``_W;Z_W;XF@````"#Q`3%=OS&A)H!`(M&_`6:
M`?]V_E":`````(/$!,5V_(M<!O]$!L'C`@->_(Y>_HM&"(M6"HE'"(E7"L5V
M_(.$J@$!@Y2L`0#%=OS&A)P!`<G+58OLQ78&BP0STCM6#'<<<@4[1@IW%<5V
M!HM$`HM4!#M6#'\=?`4[1@IW%O]V#FB0`&B$"9H`````@\0&N`$`R<LSP.OZ
MR!@``/]V!IKJ!F``1$2)1OR)5O[I@0)J"HM&_`6;`?]V_E":`````(/$!NEJ
M`L5V_/>$T`#__W\#Z<(`Q7;\_XS0`(NTT`#1Y@-V_(Y>_HN$T@")1NR%P'4#
MZ:$`_W;L_W8&F@````"#Q`2)1OB)5OJ#^@!U"8-^^`!U`^E-`L5V^/=$&/__
M=4K%=OB+1@B)1!C%=OB+1@J)1!K%=OB`3!#&N```CMBA5@3%=OB)1!RX``".
MV*%8!,5V^(E$'L5V^,=$(```QT0B``#'1O8``.D8`F@``&@``&@``&@``/]V
M^O]V^)H`````@\0,_W;Z_W;XF@````"#Q`3IF@'%=OS^A)L!Q7;\QX30`&0`
MQ7;\BX32`"7P_T")1NP%'P#!Z`0STHE&Z(E6ZNFL`/]VZO]VZ/]V!IH`````
M@\0&B4;RB5;TN```CMCV!E4$_W04_W;T_W;RF@````"#Q`2#1NP0ZVW%=O*+
M5!B+1!:)1NZ)5O#'1O8``.LXQ7;\]X30`/__?C/%=N[W!/__=1K%=OS_C-``
MB[30`-'F`W;\CE[^BT;LB832`/]&]O]&[(-&[D"#?O80?,+_=O3_=O*:````
M`(/$!,5V_/>$T`#__WXA@T;H`8-6Z@#%=OR+!#/2.U;J<@UV`^E#_SM&Z'8#
MZ3O_Q7;\QH2;`0"+1OP%FP'_=OY0F@````"#Q`3%=OSWA-``__]^'\5V_(NT
MT`!.T>8#=OR.7O['A-(```#%=OS'A-(```#%=OR#O-``9'0+Q7;\QX30`&0`
MZTW%=OS'A-````#%=OSWA*X!__]T28I&!C+D4(M&!L'H"%!HD`!HC@F:````
M`(/$",5V_/^TK@%HD`!HT@F:`````(/$!L5V_,>$T@`"`,5V_/:$FP'_=0/I
MB?WI;_W_=@9HH`!H"0J:`````(/$!K@``([8Q@95!!S%=OS'A*X!```SP#/2
MR<N+=O;!Y@(#=OB.7OK'1"0``,=$)@``_T;V@W[V#7SAQ7;\]X2N`?__=`?%
M=OS_C*X!Q7;\QH2<`0%H``!H``!H``!H``#_=OK_=OB:`````(/$#(M&^(M6
M^NNCR`0``/]V!IKJ!F``1$2)1OR)5O[%=OS_A*X!Q7;\]H2;`?]U*,5V_,:$
MG`$!Q7;\@[S0`&1\&,5V_(N$T@`[1@AV"L5V_(M&"(F$T@#)R\5V_(N<T`#_
MA-``T>,#7OR.7OZ+1@B)A](`Z^+("```QT;\``#'1OX``.MFQ7;\@SP!=5K%
M=OR+1`([1@9U3\5V_,5T"(M4&(M$%HE&^(E6^L5V^(-\!C)_"L5V^(.\T`!D
M?B+_=@9HH`!H%PJ:`````(/$!L5V^,=$!@``Q7;XQX30````BT;XBU;ZR<N#
M1OP0N```CMBA$@"+%A0`.U;^<@=WASM&_'>":*``:"$*F@````"#Q`0SP#/2
MZ\[(#```N*``CMCV!C@*_W0#Z8,!_@8X"L<&0`H`8,=&^```QT;Z``#IP@#%
M=OB#/`%T`^FS`,5V^,5T"(M4&(M$%HE&](E6]L5V]/:$G`'_=0/IE`#%=O3V
MA)L!_W0#Z8<`Q7;T]H2:`?]U?<5V]/:$G0'_=7/%=O3&A)P!`+@``([8H0``
MBQ8"`,5V](F$G@&)E*`!Q7;XBT0"F;N@`([;HTP*B19."K@``([8Q@95!`#'
M!G`$``+'!G($``#'!FX$``*+1O2C:@2+1O:C;`3&!E0$`<<&=`02`&B@`&@H
M"IH`````@\0$@T;X$+@``([8H1(`BQ84`#M6^G(-=@/I*/\[1OAV`^D@_\=&
M_```QT;^``#K5<5V_/9$$`%U2,5V_/=$$O__=#[%=OSV1!!&=#7%=OR`3!`!
MQ7;\_T02:```:```:```:```_W;^_W;\F@````"#Q`S_=O[_=OR:`````(/$
M!(-&_%RX``".V*$&`(L6"``[5OYR!W>8.T;\=Y-J_YH`````1$2XH`".V,8&
M.`H`R<LH"9``0"@C*6%L;&]C+F,)36EC<F]P;W)T(%)E=B!)9"`Q+C,N."`@
M,3`O,C(O.#8@+2!R9'1H<G5S:"!F:7@@,R\R."\X.0!"860@9G)E92!C;W5N
M=`!N;R!S<&%C90!B860@8FQO8VL`5&AE('9A;FES:&EN9R!I;F]D92!P<F]B
M;&5M('=O=6QD(&AA=F4@;V-C=7)R960@;VX@;6%J*"5D*2QM:6XH)60I"@!4
M:&5R92!A8W1U86QL>2!W97)E("5D(&EN;V1E<R!A=F%I;&%B;&4L('1R>6EN
M9R!A9V%I;@H`3W5T(&]F(&EN;V1E<P!B860@8V]U;G0`;F\@9G,```@````0
M`````0`*````#`````D`(P```!@````!`"4````8````"0`_````$`````(`
M:P```!`````"`'T````0`````0!_````!@````D`DP```!`````"`*8````9
M`````0"H````&0````D`M````!H````)`+H````:`````0#G````&P````$`
MZ0```!L````)`/4````<`````0#W````'`````D`#P$``!T````!`!$!```=
M````"0`M`0``'@````D`,`$``!(````!`#,!```?`````0`U`0``'P````D`
M4@$``"`````)`%@!```@`````0!=`0``(0````$`7P$``"$````)`&<!```B
M````"0!J`0``$@````$`;0$``!\````!`&\!```?````"0!U`0``&@````D`
M>P$``!H````!`(X!```C`````0"0`0``(P````D`H@$``"0````!`*0!```D
M````"0#A`0``$`````$`XP$```P````)``0"```8`````0`&`@``&`````D`
M)0(``!`````!`"<"```&````"0`Q`@``$`````(`;`(``",````!`&X"```C
M````"0"A`@``&P````$`HP(``!L````)`+<"```E`````0"Y`@``)0````D`
MT0(``!T````!`-,"```=````"0`\`P``)@````D`/P,``!(````!`$(#```?
M`````0!$`P``'P````D`6@,``!`````!`%P#```,````"0!G`P``$`````(`
M=@,``!@````!`'@#```8````"0!^`P``$`````(`C`,``!`````"`*T#```0
M`````@"V`P``)P````$`N`,``"<````)`,\#```0`````@#U`P``&@````D`
M^@,``!H````!``,$```:````"0`(!```&@````$`(P0``!`````"`"8$```H
M````"0`I!```*`````$`+`0``"@````)`"\$```H`````0`X!```*0````$`
M.@0``"D````)`$8$```J`````0!(!```*@````D`3@0``!`````"`'T$```0
M`````@")!```&0````$`BP0``!D````)`)<$```:````"0"=!```&@````$`
MJ00``!P````!`*L$```<````"0`1!0``'`````$`$P4``!P````)`#H%```0
M`````@!"!0``$`````(`5P4``!T````!`%D%```=````"0"_!0``*P````D`
MP@4``!(````!`,4%```L`````0#'!0``+`````D`U`4``"T````)`-<%```2
M`````0#:!0``+`````$`W`4``"P````)`/4%```0`````@#X!0``$`````(`
M_@4``"X````)``$&```2`````0`$!@``'P````$`!@8``!\````)``P&```:
M````"0`2!@``&@````$`7@8``"@````)`&$&```H`````0!D!@``*`````D`
M9P8``"@````!`'`&```I`````0!R!@``*0````D`AP8``!`````!`(D&```,
M````"0#Q!@``+P````$`]@8``"\````)`#8'```P````"0`Y!P``$@````$`
M/`<``!\````!`#X'```?````"0!A!P``,0````D`9@<``#$````!`&H'```Q
M`````0!Y!P``,@````D`?`<``!(````!`'\'```S`````0"!!P``,P````D`
MD0<``#0````)`)<'```4`````0"=!P``$`````(`H0<``!0````!`*4'```4
M`````0"L!P``+P````$`L0<``"\````)`+0'```0`````@"_!P``$`````(`
MW@<``!`````"`.L'```0`````@`*"```*`````D`#P@``"@````!`!,(```H
M`````0`H"```-`````D`+0@``!0````!`#$(```4`````0`T"```&@````D`
M.@@``!H````!`#\(```:`````0!%"```&@````$`2P@``!H````!`%,(```:
M`````0!9"```&@````$`70@``!H````!`&((```:`````0!G"```-`````D`
M:@@``!0````!`&T(```U`````0!O"```-0````D`>0@``#$````)`'X(```Q
M`````0"""```,0````$`C`@``!`````"`)0(```0`````@"9"```-@````$`
MG@@``#8````)`,P(```H````"0#/"```*`````$`T@@``"@````)`-4(```H
M`````0#>"```*0````$`X`@``"D````)`.P(```J`````0#N"```*@````D`
M^`@``#$````)`/T(```Q`````0`!"0``,0````$`$@D``#<````!`!0)```W
M````"0`9"0``-`````D`'PD``!0````!`"0)```2`````0`F"0``%P````D`
M+F9I;&4`````````_O\``&<!86QL;V,N8P``````````````86QL;V,`````
M`````0!H``(!`````-D!````````!```````9G)E90````#9`0```0`D``(!
M`````#0!````````!@``````8F%D8FQO8VL-`P```0`D``(!`````$4`````
M````"```````:6%L;&]C``!2`P```0!H``(!`````"T#````````"@``````
M:69R964```!_!@```0`D``(!`````&L`````````#```````9V5T9G,```#J
M!@```0!H``(!`````*(`````````#@``````=7!D871E``",!P```0`D``(!
M`````)@!````````$```````+G1E>'0``````````0````,!)`D``+<`!P``
M````````````+F1A=&$````D"0```@````,!!`$```(`````````````````
M+F)S<P`````H"@```P````,!7```````````````````````=7!O<G1I9``D
M"0```@````,`+C$R```````H"0```@````,`<VQE97````````````````(`
M8G)E860```````````````(`=0````````````````````(`8F-O<'D`````
M``````````(`8G)E;'-E``````````````(`=V%K975P``````````````(`
M+C0S``````!L"0```@````,`<')D978```````````````(`:'H`````````
M``````````(`9&5L87D```````````````(`+C0V``````!["0```@````,`
M9V5T8FQK``````````````(`8VQR8G5F``````````````(`8G=R:71E````
M``````````(`+C4Y``````"$"0```@````,`:6=E=`````````````````(`
M=&EM90````````````````(`:75P9&%T``````````````(`:7!U=```````
M``````````(`+C@V``````"."0```@````,`<')I;G1F``````````````(`
M+C@W``````#2"0```@````,`+C@X```````)"@```@````,`;6]U;G0`````
M``````````(`+C$P-@`````7"@```@````,`=@````````````````````(`
M+C$P.``````A"@```@````,`<&%N:6,```````````````(`:6YO9&5X```H
M"@```P````,`=W)I=&5I``````````````(`:6YO9&4```````````````(`
28F9L=7-H``````````````(`
`
end
----------------------------- and here ----------------------------
--
Bob Thrush UUCP: {ucf-cs,rtmvax}!tarpit!rd
Automation Intelligence, 1200 W. Colonial Drive, Orlando, Florida 32804