del@fnx.UUCP (Dag Erik Lindberg) (07/03/90)
First, let me appologize for the multiple cross posting. I would not do this
were it not for the unusual situation I find myself in. If I hit a group
that is inappropriate, please ignore this message.
In a nutshell: When news is unbatching, at some point the number of free
inodes suddenly goes to zero, and the news starts trying to fill the bit
bucket on the floor. Killing rnews, unmounting the file system, fsck'ing,
and remounting brings back the free inodes. The problem occurs at a random
time, from a hundred articles to a couple of thousand articles. The problem
does not occur with any other software I have tried which uses up large
quantities of inodes. (For example "find comp -print | cpio -pdvm" into
another directory on the news file system will happily use all the inodes
without exhibiting the problem.) I am not aware of any way that a user
program such as rnews can corrupt the free inode table.
The details: The system is a Mylex 20 Mhz w/64k cache, 8 MB memory, ATI
VGA wonder. I have unbatched news with no serial ports installed in the
system, same problem. I have tried a DTC RLL card, WD 7000 FASST SCSI card,
Adaptec 1542 SCSI, and all have shown the same problem. I am currently
running ISC Unix. I have stripped down the system to just the video card
and the disk controller to try to find this problem.
Having heard some rumors of Mylex motherboards being unreliable running
Unix, I have come close to deciding that the M.B. is the problem, but
for the following reasons I am hesitant to shell out for a new M.B.:
- I have pounded the system hard for up to 2 weeks with *NO* other
problems. "Hard" means 3 virtual screens active, 1 running VPIX (I
can keep several logins pretty active), and typically 2 users (not
myself) logged in on serial lines, 1 of them running VPIX on a Wyse60,
and uucp traffic doing mail, news, etc. With this load swapping starts
occurring even with 8 MB of memory.
- Mylex claims they have fixed their early problems with running Unix, and
my M.B. should work.
- If I 'fix' the file system by saving all data, mkfs, then restore, I can
run rnews and it will chug along until it is done or *legitimately* runs
out of inodes. The problem does not show up until I have run expire on
the news file system!
- This one I have not tried but twice, so I am not sure it is repeatable,
but I forced a rebuild of the free list using fsck, and shutdown the
system. After booting I again ran rnews with no problems, until I
run rnews.
I am desperate at this point, having spent unbelievable numbers of hours
on this problem. I just want to get it fixed, but have exhausted all my
ideas. If anyone has ever seen anything like this before, *please* drop
me a line via E-mail. Use Email as my news system is totally unreliable
right now. I'm only one hop away from uunet if you use the path below.
Thanks in advance!
--
del AKA Erik Lindberg uunet!pilchuck!fnx!del
Who is John Galt?fkk@wynge.Central.Sun.COM (Frank Kaefer - Sun Germany CSD - Munich) (07/03/90)
del@fnx.UUCP (Dag Erik Lindberg) writes: | In a nutshell: When news is unbatching, at some point the number of free | inodes suddenly goes to zero, and the news starts trying to fill the bit | bucket on the floor. Killing rnews, unmounting the file system, fsck'ing, | and remounting brings back the free inodes. The problem occurs at a random | time, from a hundred articles to a couple of thousand articles. The problem I have EXACTLY the same problem ! And I am also extremly desperate. My machine is a AT386 running Interactive IX 2.0.2. If anyone has any idea, I would be very glad to get some help. Frank. -- ============================================================================= Frank Kaefer | SUN Microsystems GmbH | Phone: (+49) 89 46008-321 German Answer Center | Am Hochacker 3 | FAX: (+49) 89 46008-400 Datacomm | D-8011 Grasbrunn | ============================================================================= e-mail: fkk@Germany.Sun.COM (...!sun!sunuk!sungy!fkk) fkk@sunmuc.UUCP (...!unido!sunmuc!fkk) suninfo!fkk fkk@stasys.sta.sub.org =============================================================================
cpcahil@virtech.uucp (Conor P. Cahill) (07/03/90)
In article <586@fnx.UUCP> del@fnx.UUCP (Dag Erik Lindberg) writes: > > [story of running out of inodes deleted] > >The details: The system is a Mylex 20 Mhz w/64k cache, 8 MB memory, ATI >VGA wonder. I have unbatched news with no serial ports installed in the >system, same problem. I have tried a DTC RLL card, WD 7000 FASST SCSI card, >Adaptec 1542 SCSI, and all have shown the same problem. I am currently >running ISC Unix. I have stripped down the system to just the video card >and the disk controller to try to find this problem. The only pertinent portion of this is the Operating system and you don't specify the version. This inode thing is a known bug that as far as I know was fixed in version 2.0.2 of 386/ix (I have been running news on a partition without running out of inodes for a year). There is a binary patch for microport unix that is reputed to work correctly for 386/ix. If you want to try it, send me email & I will send it to you. -- Conor P. Cahill (703)430-9247 Virtual Technologies, Inc., uunet!virtech!cpcahil 46030 Manekin Plaza, Suite 160 Sterling, VA 22170
bgi@stiatl.UUCP (Brad Isley) (07/03/90)
In article <fkk.646998181@wynge> fkk@wynge.Central.Sun.COM (Frank Kaefer - Sun Germany CSD - Munich) writes: > >I have EXACTLY the same problem ! And I am also extremly desperate. >My machine is a AT386 running Interactive IX 2.0.2. If anyone has >any idea, I would be very glad to get some help. You're experiencing the old SYSV missing inodes bug. We have it here, too. We have developed work-arounds. If you'd like them, let me know. I'd be glad to send them along... -- -----------------------------\ / ..and Apple thought GUI was theirs!.. \ bgi@SalesTech.COM 841-4169 \---| Yer local zymurgist & Amiga hacker/user | OR \ Klein bottle for sale- Inquire within / brad@slammer.UUCP 925-9663 Brad Isley, Sales Technologies, Inc.
doug@letni.UUCP (Doug Davis) (07/03/90)
>fkk@wynge.Central.Sun.COM (Frank Kaefer - Sun Germany CSD - Munich) writes: >>del@fnx.UUCP (Dag Erik Lindberg) writes: > >| In a nutshell: When news is unbatching, at some point the number of free >| inodes suddenly goes to zero, and the news starts trying to fill the bit >| bucket on the floor. Killing rnews, unmounting the file system, fsck'ing, >| and remounting brings back the free inodes. The problem occurs at a random >| time, from a hundred articles to a couple of thousand articles. The problem > Both of these sound like the now famous system V inode problem, if you have a source license I can send you some patchs that will fix the problem. Below is a file from Mayer Ilovitz that describes the problem in some detail. I redid ialloc() if any source licensees want it. Otherwise if your system exhibits this problem I would contact your vendor. Send 'em this code, tell 'em where to get a fix etc, etc.. Kudos to mayer, his description of the problem was right on. doug ------------------------------------------------------------------------------- From: mayer@cooper.cooper.EDU (Mayer Ilovitz ) Subject: Analysis & test for 3b inode problem: applies to ALL users of SYSTEM V Since I haven't seen anyone post a full description of the problem or a test for it, here is my contribution. This document contains what I believe is a complete analysis of the System V inode allocation system and the problem that everyone is having. I have included a test procedure which should detect the problem on a UNIX system and included a program that will help you perform the tests. Also I have some suggestions on properly fixing the bug. To begin with, let me describe what I have available to me. We have a number of pretty standard Unix-PCs running System V 3.5 and System V 3.0 . We have a pair of 3b2-400 s running System V 3.0 Version 2. These machines each have a floppy diskette system. We also have an OLD 3b5 running System V Release 2 Version 2. I have access to the source for the Unix on the 3b5 and the 3b2 systems. Our 3b5 runs our newsfeed using the rnews package. This system has suffered the inode problems that everyone has been mentioning on the net for the last few weeks. Since this system has no expendable files systems, I ran the tests on the 3b2 and the Unix-PC . Both of these systems showed the same error. From this, I suspect that all versions of ATT System V unix have the problem. Furthermore, this problem may very well be in any ATT System-V compatible version of Unix and may well have been present in System-III Unix. I therefore suggest, just to be on the safe side, that you run the test described below. The analysis and test was based on the source from the 3b5. A cursory examination of the source to the 3b2 showed the code to be essentially the same in the critical area though there are what appear to be minor cosmetic changes. For those of you with access to the souces, The file that needs changing is called alloc.c or s5alloc.c . If you don't have a file by this name, look for a file that closely matches one of these names. The function that is causing the problem is called s5ialloc() or ialloc() . As far as I can tell ialloc and ifree are the low level inode allocation control system. When an inode is needed, a call to ialloc() is made. When a file/directory is deleted, ifree() is used to release the inode. These 2 functions use certain parameters that are kept in the superblock of every file system. tinode is the total number of free inodes in a file system. To speed up inode allocation and freeing, the superblock maintains a table of free inodes. This table is called inode[]. The size of this table is given by the #defined value NICINOD and is usually 100. ninode specifies the number of free inodes available in inode[]. When ifree() releases an inode, it first checks to see if the inode table is full. If it isn't, the inode is added to the top of the table and ninode is adjusted. If the table is full and the inode being released is less than the inode stored in inode[0], the newly released inode is put into inode[0]. In this way, the allocation system knows where in the i-list a group of free inodes are likely to be. When ialloc() is called, it tries to give the requesting process an inode from inode[]. If none are available, ialloc() searches the i-list for more free inodes to reload inode[]. ialloc() will start this search begining at the location of the last allocated inode as indicated by inode[] and ninode. The search continues untill NICINOD inodes are located or the end of the i-list is reached. inode[] will be reloaded from the top of the table working down to inode[0]. A mark is put in inode[] if less than 100 nodes were found. The next time inode[] runs out of nodes, this mark tells it to search the i-list from the very begining. If NO inodes were found during the search, ninode is SET TO 0 and the out of inodes error is printed on the system console. The problem that everyone is having is caused by the following situation. At the last reloading of inode[] exactly NICINOD inodes were found. Therefore, the inode at inode[0] is where the next search for inodes will begin. As the system runs, more inodes are allocated and freed. Eventually, the last free inode in inode[] is allocated. The system waits until the next call to ialloc to determine if it needs to reload inode[]. If a node is released before the inode table is reloaded, the freed inode will go into inode[0], replacing the old value which would be used for searching the i-list. If the freed inode was higher in the i-list than the one it replaced in the table, ialloc will no longer know that it should check the lower portion of the i-list for free inodes. It will think that everything below inode[0] is allocated already. If a significant number of lower valued inodes are not freed before ialloc has to reload the inode table, ialloc will fail to find any free inodes even though they exist. Furthermore, because of the coding of ialloc(), unless an inode is freed at some point, every time it tries looking for more inodes, it will start at the same place. So until the file system is dismounted and fsck'd, unless some inodes are freed, the system will be stuck repeating the same search and reporting the same failure. The original intent of the ialloc() - ifree() system is to minimize the time to find more free nodes by remembering the best location to start searching for more free inodes. Therefore, the best fix to ialloc would be to first try to give the requesting process a free node. ialloc() should then IMMEDIATELY check to see if that was the last free inode it had, and if it was, try reloading the inode table right then. This will prevent the possibility of the system from forgetting about the best place to search for inodes. A side result of this is that the out of inodes message will appear when the last free inode is allocated and not when ialloc failed to give an inode. An argument could be made either way as to wether this side effect is good or not. The other fix is to put a kludge into ialloc that, in the event that NO free inodes were found, it would immediately recycle through the i-list from the very beginning looking for inodes before deciding that there are no free inodes left. If the i-list is large, this can be somewhat inefficient. PROCEDURE TO TEST FOR THE 3B INODE ALLOCATION BUG This test is intended to be run on a floppy-based file system or an expendable file system. It is assumed that NICINOD, the number of inodes that are stored in the superblock inode table is 100. If not, the test will have to be adjusted accordingly. 1. create a file system with ~ 280 inodes using mkfs fsck the disk and mount it /mnt 2. verify with a df -t as to the number of free inodes and the total number of inodes in this file system. 3. allocate all the inodes on this filesystem. You can use the program fillnode given at the bottom of this document to help you do the job. The final result is that there should be 0 inodes left. Each file that you made on this disk should be named after its respective inode. 4. unmount the filesystem, do an fsck of the disk, remount and verify with a df -t that there are no free inodes. 5. free up the files with inodes 3-202. This will give you 200 free inodes on the filesystem. Verify this using step 4. 6. at this point, the file system will be mounted and the superblock inode table will contain inodes 3-102 for immediate allocation. 7. use fillnode to reallocate inodes 3-102. at this point you will have 100 free inodes when you do a df. This is the correct number of free nodes. At this point the superblock inode table will be empty. 8. use fillnode to allocate 1 inode. the inode that will be allocated is inode # 103. At this point the superblock inode table will have been reloaded from the i-list. the 0 element in the table will be inode 202 and the 99th element will be inode 103, which you just allocated. 9. Delete in order the files with inodes 30-39. At this point, the 0 element in the inode table will be inode 31 while the 99th element will be inode 30. When you released inode 30, the table was not full, so it was put onto the top of the table. When inode 31 was released, the table was full so ifree checked to see if the just freed inode was less than the inode in the 0th element of the table. Since the 0th element up to this time was 202, ifree replaced the 0th element with inode 31. Note, The inode table is now full, containing 100 free inodes, the lowest free inode in the entire i-list being in the 0th element of the table. As you release inodes 32-39, they will fail the test by ifree, the result being that these inodes ARE free but simply aren't in the inode table. This is alright since when ialloc must reload its inode table it will start looking with the inode referenced in the 0th element of the table. 10. allocate another 100 inodes. fillnode will allocate in order inodes 30,104-201 and inode 31. At this point the superblock inode table is empty again. However, as always, ialloc will leave the table empty until it must allocate an inode and finds no inodes in the table. 11. free inode 240. At this point you have sealed your doom ! . ifree will put this inode into the lowest available entry in the inode table, DESTROYING ANY MEMORY THAT THE LOWEST FREE INODE IS AROUND INODE 31. 12. Do a df -t to confirm that you still have ~ 10 free inodes. 13. allocate an inode. This inode will be inode 240. 14. Do a df -t to confirm that you now have ~ 9 free inodes. 15. Call fillnode again and say goodbye to your free inodes! At this point you will get an out of inodes error on your console and the allocation attempt will return failure. A df -t say that there are NO free inodes. What happened was that after step 13 there were no free nodes in the superblock inode table. At this point, ialloc went searching through the i-list for more free inodes starting at the inode specified in the 0th element of the inode table. BUT this no longer references inode 31, where we know there is more space, but inode 240. ialloc searches from inode 240 to the end of the i-list, but all those inodes are allocated, so ialloc decides that there are no more free inodes and reports the out of inodes error,EVEN though you still have free inodes!. 16. unmount the filesystem. Do an fsck. This will report a bad inode count in the Superblock ( Sound familiar ) which you must fix. Remount and do an df -t to confirm that you really do still have a number of free inodes. IF THE SITUATIONS DESCRIBED IN THIS TEST HAPPEN TO YOU AND YOU ARE HAVING PROBLEMS BECAUSE OF THIS BUG CONTACT YOUR ATT CUSTOMER/TECH SUPPORT REP AND REPORT THE PROBLEM below is the code for fillnode.c . This program will create a file in /mnt. The file created will be named after the inode to which it was allocated. The file will have 0 blocks allocated to it. #include <fcntl.h> #include <sys/types.h> #include <sys/stat.h> main() { int link(),open(),close(),fstat(); struct stat buf; int fd; char name[30]; if( (fd = open("/mnt/XXX",O_CREAT | O_WRONLY,0666) ) < 0 ) { printf("can't open file\n"); exit(2); } if( fstat(fd,&buf) < 0 ) { printf("error fstating file\n"); exit(3); } printf("inode is %d\n",buf.st_ino); sprintf(name,"/mnt/%d",buf.st_ino); close(fd); if( link("/mnt/XXX",name) < 0 ) { printf("can't link to new name\n"); exit(3); } if( unlink("/mnt/XXX") < 0 ) { printf("can't unlink old file /mnt/XXX\n"); exit(3); } exit(0); }
rli@buster.irby.com (Buster Irby) (07/03/90)
cpcahil@virtech.uucp (Conor P. Cahill) writes: >In article <586@fnx.UUCP> del@fnx.UUCP (Dag Erik Lindberg) writes: >> >> [story of running out of inodes deleted] >> >>The details: The system is a Mylex 20 Mhz w/64k cache, 8 MB memory, ATI >>VGA wonder. I have unbatched news with no serial ports installed in the >>system, same problem. I have tried a DTC RLL card, WD 7000 FASST SCSI card, >>Adaptec 1542 SCSI, and all have shown the same problem. I am currently >>running ISC Unix. I have stripped down the system to just the video card >>and the disk controller to try to find this problem. >The only pertinent portion of this is the Operating system and you don't >specify the version. >This inode thing is a known bug that as far as I know was fixed in version >2.0.2 of 386/ix (I have been running news on a partition without running >out of inodes for a year). No, it did not. I ran 2.0.2 for only a few days before the bug reared its ugly head. Included below is the patch I received from T. William Wells. I applied it and it fixed the problem (Thanks again Bill). The problem does however appear to have been fixed in version 2.2. ------------------------inode.fix---------------------- From: uunet!twwells.com!bill (T. William Wells) Date: 16 Oct 89 18:54:06 EDT (Mon) Subject: Inode bug fix for ISC 2.0.2 Since I've got so many requests for the fix for the inode bug for ISC 2.0.2, I decided to post it. Sorry I've been so slothful on this; various things popped up and I wasn't able to get it all together. In case you want to know what this is: there is a kernel bug that, on occasion, and particularly if you are running a newsfeed, will cause the file system to think that there are no more free inodes, thus preventing creation of new files. Running fsck fixes the inode count but doesn't prevent the problem from occuring again. The problem is this: there is a "free inode cache" stored in the superblock for each file system; this block is kept in memory when the file system is mounted, thus the inode cache permits rapid allocation of inodes. When the cache is emptied, the kernel tries to read more inodes from the disk to fill the cache and then retries the allocate. If the kernel is unable to read more inodes from the disk, it assumes that there are no more free inodes. There is an optimization in the allocation code, which depends on the condition that the lowest free inode is always in the inode cache. What it does is to start the disk read from that lowest inode, instead of the first inode. This means that the inode table doesn't have to always be fully read, for what could be a significant savings in allocating inodes. (Consider what might happen when almost all inodes are in use.) However, the kernel does not maintain that condition properly. It is possible for the kernel to forget the lowest inode, with the effect that the kernel tries to read from some place too far in the inode table, and maybe discovers that there are no free inodes. When that happens, the kernel clears the available inode count, and the file system is essentially kaput. The right fix for this would be to always maintain that condition. However, a binary patch for that would be tricky at best, and maybe impossible. A patch that is possible is to have the allocation routine try from the beginning of the inode table whenever it fails to read inodes from the disk, relying on the free inode count to tell when the table is empty. This changes the condition that must be maintained to: the free inode count must always be accurate. (Having the free inode count never be larger than the actual number of free inodes is sufficient for the patch to not cause problems.) I made a similar patch for Microport SysV/386 3.0e and have been running it for most of this year without problems. I was asked to solve this for Interactive 2.0.2 and did so. However, I did the work on my Microport system, and the enclosed shell script works on that; it ought to work on an Interactive system but I've not tried this. With that caveat, here is what you do to patch your kernel. First, run the shell script. Make sure that it behaved correctly. Then save a copy of your good kernel and /etc/conf/pack.d/s5/Driver.o. If you are really paranoid, back up your whole system, though this shouldn't be necessary. Replace the Driver.o file with the one on /tmp. Finally, rebuild your kernel. That particular bug should never bite you again. Further kernel builds will have the bug fixed as well. If you have any problems, send me e-mail. I'll try to get back to you quicker than I did with this! If the patch causes some nasty kind of crash, please post immediately in hopes that others will read your message before having tried the patch. in=/etc/conf/pack.d/s5/Driver.o out=/tmp/Driver.o # check that we have the right Driver.o file if [ x"`sum $in`" != x"50880 81 $in" ]; then echo "sum failed" exit 1 fi if [ x"`sum -r $in`" != x"33908 81 $in" ]; then echo "sum -r failed" exit 1 fi # copy the file and make an appropriate fix { dd ibs=1 obs=1k count=1977 dd bs=19 count=1 of=/dev/null echo '\074\0144\017\0204\0327\0376\0377\0377\0146\c' echo '\0307\0207\0324\00\00\00\0144\00\0353\0151\c' dd bs=16k } <$in >$out # compare the list of differences against the expected differences cat <<\+ >/tmp/fix$$ 1978 75 74 1980 0 17 1981 0 204 1982 0 327 1983 164 376 1984 14 377 1985 146 377 1986 307 146 1987 207 307 1988 324 207 1989 0 324 1992 144 0 1993 0 144 1994 353 0 1995 152 353 1996 220 151 + if cmp -l $in $out | cmp -s - /tmp/fix$$; then rm /tmp/fix$$ exit 0 else rm /tmp/fix$$ echo "patch failed" exit 1 fi --- Bill { uunet | novavax | ankh | sunvice } !twwells!bill bill@twwells.com -- Buster Irby buster!rli