ir@cel.co.uk (ian reid) (09/20/90)
Forgive if this has been discussed before, but we have recently suffered a long
break in news at this site. My question is very simple, the reliability of the
System V 1K file system when a machine is subjected to a power cycle without
going through a proper shutdown sequence. Unfortunately on our system an UPS
is not an option (too price sensitive).
Our system is Interactive 386/ix 2.0.1.
What we observe is that occassionaly in the circumstances mentioned above, on
rebooting the system some indeterminate files have been corrupted and our
system will not boot in the way we expect. We boot up at run level 2 and
put our own files in /etc/rc2.d to tailor the behaviour of the system. None
of the files in /etc/rc2.d seems to be incorrect, or indeed any of the files
we load on the system (we also have our own inittab). We boot a kernel
on a floppy then mount the hard disk to determine this. Obviously our start up
performs an fsck on the file system if needed. The problem will occur even if
the machine has been idle for hours, and we have no cron jobs other than the
standard root etc ones supplied by Interactive.
In other lives I have seen similar problems on other System V based kit, so I
believe this to be a generic problem.
I know that System V uses a cacheing system for writing to disks, and that
the contents of this cache are flushed to the disk at intervals controlled
by a kernel parameter (at least in the case of SysVR3.2). The crucial data
structure is I believe the super-block which describes the organisation of the
disk. If the copy of this held on the disk ever becomes out of step with the
disk itself this can I believe cause the symptons I have described. But as
explained above this problem has been observed at a time when the disk contents
and thus the superblock should not be changing.
We are using a 40mb disk with a single / file system built using the default
paramters.
So my questions are:-
1) Do other people see this problem.
2) Are my comments (sketchy though they are) on the workings of the
file system correct?
3) How can we minimise, or alleviate it happening.
4) I have heard of file system hardening, which as I understand it was
something AT&T put into the kernel around the early days of
System V I believe to go some way towards reducing this problem.
Is this correct, and if so what exactly did it involve?
RTFMs, flames, email etc all welcomed. I will summarise if there is sufficient
interest but please note there is a long delay at our site.
------------------------------------------------------------------------------
Disclaimer: The opinions expressed are entirely my own, and not necessarily
those of Crosfield Electronics.
Name: Ian Reid. S-Mail: Peripheral Products,
R & D Department,
E-Mail: ir@cel.uucp Crosfield Electronics Ltd.,
or Three Cherry Trees Lane,
...mcvax!ukc!uk.co.cel!ir Hemel Hempstead,
Hertfordshire.
V-Mail: +44 442 230000 extn 3759 HP2 7RH
England.
------------------------------------------------------------------------------larry@nstar.uucp (Larry Snyder) (09/21/90)
ir@cel.co.uk (ian reid) writes: >Forgive if this has been discussed before, but we have recently suffered a long >break in news at this site. My question is very simple, the reliability of the >System V 1K file system when a machine is subjected to a power cycle without >going through a proper shutdown sequence. Unfortunately on our system an UPS >is not an option (too price sensitive). NSTAR is here primarily for news distribution - and sometimes looses power twice a week anywhere from 1 to 5 minutes (that is normal here). The system comes right back up, rechecks the file system (that takes some time) then continues processing mail. We are running 2.2 386/ix with SCSI drives - and have the write behind in the kernel set to 180 seconds - so at the worst case we will loose some news - -- Larry Snyder, Northern Star Communications, Notre Dame, IN USA {larry@nstar, uunet!sco!romed!nstar!larry, nstar!larry@ndmath.math.nd.edu} usenet newsfeeds available Public Access Unix Site (219) 289-0282 (5 high speed lines)
cpcahil@virtech.uucp (Conor P. Cahill) (09/21/90)
In article <5869@suns302.cel.co.uk> ir@cel.co.uk (ian reid) writes: >We are using a 40mb disk with a single / file system built using the default >paramters. Your problems would probably diminish if you had a root file system that was fairly stable (i.e. no user files being created/deleted there). You can do this by creating additional partitions and moving your news stuff to the other partition. The big problem with power loss is that it may be doing things to your system that no OS could protect against. For example, at the point of power loss the disk controller may be dumping junk information to different parts of your disk. I'm not saying that this is happening, just saying that it could. The loss of power to a system (especially from a provider loss which can include all kinds of yucky power things like spikes) is a very bad thing for a system. > 1) Do other people see this problem. We originaly ran our 386/ix systems with no protection (other than the standard surge protection) and had several power losses (usually around two to three a month) without experiencing the problems that you describe. > 2) Are my comments (sketchy though they are) on the workings of the > file system correct? Split you filesystems so that root and /usr are both as small as they have to be and have little or no user data on them (i.e. create yet another partition for your user data (total of three)). > 3) How can we minimise, or alleviate it happening. Get a UPS. Ensure that you have a good surge protector. > 4) I have heard of file system hardening, which as I understand it was > something AT&T put into the kernel around the early days of > System V I believe to go some way towards reducing this problem. > Is this correct, and if so what exactly did it involve? File system hardening involves an ordered updating of the information related to file system changes which should make it harder for fs corruption to occur. -- Conor P. Cahill (703)430-9247 Virtual Technologies, Inc., uunet!virtech!cpcahil 46030 Manekin Plaza, Suite 160 Sterling, VA 22170
art@pilikia.pegasus.com (Art Neilson) (09/22/90)
In article <5869@suns302.cel.co.uk> ir@cel.co.uk (ian reid) writes: > > [ stuff about file system getting hosed when power is cycled > without performing graceful shutdown ... ] Every UNIX I have seen behaves in the manner you describe. If you hit the red switch or experience a power outage without performing a graceful shutdown, you deserve whatever you get. Most times fsck will fix whatever mess is created by this, however there are no guarantees that a critical data won't get hosed. If your UNIX box is used for real production work, you are quite foolish not to put it on an UPS. Just how valuable is your data to you ??? Certainly more than the price of an UPS, especially if your business depends on the data stored in your machine. -- Arthur W. Neilson III | ARPA: art@pilikia.pegasus.com Bank of Hawaii Tech Support | UUCP: uunet!ucsd!nosc!pegasus!pilikia!art
avg@hq.demos.su (Vadim G. Antonov) (09/22/90)
In article <1990Sep22.041723.1599@pilikia.pegasus.com> art@pilikia.pegasus.com (Art Neilson) writes: >Every UNIX I have seen behaves in the manner you describe. Hmmm... May be your path never crossed some UNIXes? For example Unix BSD 4.2 and later has a different file system structure which handles power fails more carefully. Another example is BSD 2.9 - it has file structure similar with v7's (but with 1Kb blocks). The main difference is in algorithms: the 2.9's kernel performed critical file system operations in the strict order - for exmaple it guarantees that if you delete a file inode will be cleared before releasing data blocks which in turn will be done before releasing this inode. Thus if power would fail during removing a file the worst thing which can happen is missed blocks - but generally file structure will remain correct. There are some similar tricks in BSD 2.9, but I do not want to describe it - get sources and read it yourself. Hmmm... +1 to BSD over SysV. (What's a pity there are no BSD on 386s!) \ |\/| / Vadim Antonov -_(____)_- The actual image DEMOS, Moscow, USSR ( ) of my cat. (It is NOT a joke!) -------||------- || \/
cpcahil@virtech.uucp (Conor P. Cahill) (09/23/90)
In article <1990Sep22.160132.24999@hq.demos.su> avg@hq.demos.su (Vadim G. Antonov) writes: > algorithms: the 2.9's kernel performed critical file system > operations in the strict order - for exmaple it guarantees > that if you delete a file inode will be cleared before releasing > data blocks which in turn will be done before releasing this inode. This is the same as (or very similar to) the algorithm used in "today's" system V implementations. > Thus if power would fail during removing a file the worst thing > which can happen is missed blocks - but generally file structure > will remain correct. There are some similar tricks in BSD 2.9, This is partially wrong. This software change only guards against a system stop where there are no hardware glitches. A power failure can wreak havoc all over the system if the *hardware* does not correctly handle the loss of power during an otherwise normal operation. From the original posting about this problem, I would sumise that his hardware has some problems with power loss (or that a simple power loss is not the only thing happening). > but I do not want to describe it - get sources and read it > yourself. Hmmm... +1 to BSD over SysV. No. In this simple test they are about the same (and they are growing closer every day). -- Conor P. Cahill (703)430-9247 Virtual Technologies, Inc., uunet!virtech!cpcahil 46030 Manekin Plaza, Suite 160 Sterling, VA 22170
avg@hq.demos.su (Vadim G. Antonov) (09/24/90)
In article <1990Sep22.215128.16693@virtech.uucp> cpcahil@virtech.UUCP (Conor P. Cahill) writes: >A power failure can wreak havoc all over the system if the *hardware* >does not correctly handle the loss of power during an otherwise normal >operation. ... Or if disk dirver does not handle drive's idle state properly. Some controllers require clearing registers when device is inactive. It usually is not documented - and requires very careful design and testing of drivers. (Have you ever heard about testing Unix disk drivers on power failures?) Practically all machines provide power fail interrupts - and I don't know why Unix device drivers have no "xxpwfail" entries. Anyway I'm quite sure *any* device can correctly handle power fails - if you handle device properly :-). It seems to me the best way to protect disks from accidental damaging by power fails is to start recalibrating or moving heads to landing zone - usually quite simple logic circuitry protects from writing while heads move. Hmmm... For example I've never seen any recommendations about such things in DEC's HW manuals, but I've seen that DEC's PDP-11 systems clear controller after completing disk i/o. Needless to say when I've made a similar trick in Unix I've never got powerfail problems again. My $0.02 is: the quailty of disk drivers determines the reliability of a whole system. It is possible to work even on Soviet hardware if you wrote good drives. :-) I think Western software manufacturers should buy Soviet computers to test their systems on :-). "One who can pilot Camel can fly on anything" [Back In The Future, if my memory is right]. Vadim Antonov DEMOS, Moscow, USSR The cat is sleeping under sofa. (It is NOT a joke!)
chris@mimsy.umd.edu (Chris Torek) (09/25/90)
In article <1990Sep23.184158.841@hq.demos.su> avg@hq.demos.su (Vadim G. Antonov) writes: >Practically all machines provide power fail interrupts - I know of a number that do not; but in any case: >and I don't know why Unix device drivers have no "xxpwfail" entries. power failure interrupts are not any good unless they are guaranteed to occur sufficiently early, and usually they are not. The power supply system on the main CPU (the thing that has a `power fail' interrupt) is quite often completely independent of the power supply for the disk drives. If the electronics on the drive are in an indeterminate state, nothing done at the CPU will guarantee anything. >Anyway I'm quite sure *any* device can correctly handle power fails - >if you handle device properly :-). Power fail handling is like lightning protection: you can only do so much; if Nature is out to get you, you are doomed. (Lightning strikes have been known to dance teasingly around all the grounding posts, giggle in circles round and round as your hair stands on end, then viciously zap straight into the heart of your computer. Well, maybe not quite. :-) ) Designing systems that act properly on power failure is, however, tricky: >It seems to me the best way to protect disks from accidental >damaging by power fails is to start recalibrating or moving >heads to landing zone - usually quite simple logic circuitry >protects from writing while heads move. Let me tell you about ... Century Data Systems T-300s. (To be fair, I am not sure where the problem was located. The T300s were merely the end of the chain.) We have a couple of Xerox file servers with big washtub drives. These drives have a power fail system that retracts the heads (quite reasonably) so that they will not land on the disk when it stops spinning. Apparently it turns off the write current at the same time, because a simple power failure does not damage anything. Unfortunately, there are not-so-simple power failures. Thunderstorms (those thing that Californians never see :-) ) often cause momentary power failures---anywhere from a fraction of a second to several seconds. (Power distribution systems have thing called `lightning arrestors' that temporarily open the circuit to prevent serious overvoltages. There are two major variants, air and oil. Lightning will jump a simple air gap so the air versions blow compressed gas across the gap. I know nothing about the oil versions, other than that they explode very prettily, like oil-filled transformers. :-) ) Anyway, as it happens, under certain conditions the T300s would detect a power failure and begin retracting the heads. Then the power would come back on, the electronics would think, `oh, everything is OK', and the write current would turn on---while the heads were still spiraling down the pack. The result was invariably a hopelessly damaged pack. Hundreds of `bad' sectors appeared in a spiral pattern, and the only means of recovery was to reformat (followed by a tedious restore from the backup server, all the while hoping desperately that another storm would not come up during the multi-hour restore). The problem has finally been fixed: the file servers are now on a ten-minute UPS, and only a long-term power failure---the kind the drives were engineered to handle---will get through. -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 405 2750) Domain: chris@cs.umd.edu Path: uunet!mimsy!chris
avg@hq.demos.su (Vadim G. Antonov) (09/25/90)
In article <26688@mimsy.umd.edu> chris@mimsy.umd.edu (Chris Torek) writes: >Unfortunately, there are not-so-simple power failures. Hmmm... Our backbone host at Kurchatov Institute of Atomic Energy has a funny surge protector - so called "motor-generator". This device includes synchronous AC motor paired with dinamo-machine producing AC of the same voltage. Protection is absolute. (Another interesting feature of our power supply at KIAE is that the secondary power supply goes directly from atomic generator - yes, there are some working nuclear reactors in the Moscow!) Vadim Antonov DEMOS, Moscow, USSR (It is NOT a joke!)
steve@altos86.Altos.COM (Steve Scherf) (09/26/90)
In article <1990Sep22.041723.1599@pilikia.pegasus.com> art@pilikia.pegasus.com (Art Neilson) writes: >In article <5869@suns302.cel.co.uk> ir@cel.co.uk (ian reid) writes: >> >> [ stuff about file system getting hosed when power is cycled >> without performing graceful shutdown ... ] > >Every UNIX I have seen behaves in the manner you describe. If you >hit the red switch or experience a power outage without performing a >graceful shutdown, you deserve whatever you get. The Pick engineers here at Altos tell us that if a Pick system goes down ungracefully, it's time to re-install... -- Steve Scherf steve@Altos.COM ...!{sun|sco|pyramid|amdahl|uunet}!altos!steve These opinions are solely mine, but others may share them if they like.
dhesi%cirrusl@oliveb.ATC.olivetti.com (Rahul Dhesi) (09/26/90)
In <1990Sep22.041723.1599@pilikia.pegasus.com> art@pilikia.pegasus.com
(Art Neilson) writes:
Just how valuable is your data to you ??? Certainly more than the
price of an UPS,...
Now, now, let's not jump to conclusions.
--
Rahul Dhesi <dhesi%cirrusl@oliveb.ATC.olivetti.com>
UUCP: oliveb!cirrusl!dhesi