[comp.unix.sysv386] Reliability of System V 1K file system

ir@cel.co.uk (ian reid) (09/20/90)

Forgive if this has been discussed before, but we have recently suffered a long
break in news at this site.  My question is very simple, the reliability of the
System V 1K file system when a machine is subjected to a power cycle without
going through a proper shutdown sequence.  Unfortunately on our system an UPS
is not an option (too price sensitive).

Our system is Interactive 386/ix 2.0.1.

What we observe is that occassionaly in the circumstances mentioned above, on
rebooting the system some indeterminate files have been corrupted and our
system will not boot in the way we expect.  We boot up at run level 2 and
put our own files in /etc/rc2.d to tailor the behaviour of the system.  None
of the files in /etc/rc2.d seems to be incorrect, or indeed any of the files
we load on the system (we also have our own inittab).  We boot a kernel 
on a floppy then mount the hard disk to determine this.  Obviously our start up 
performs an fsck on the file system if needed.  The problem will occur even if
the machine has been idle for hours, and we have no cron jobs other than the
standard root etc ones supplied by Interactive.

In other lives I have seen similar problems on other System V based kit, so I
believe this to be a generic problem.

I know that System V uses a cacheing system for writing to disks, and that
the contents of this cache are flushed to the disk at intervals controlled
by a kernel parameter (at least in the case of SysVR3.2).  The crucial data
structure is I believe the super-block which describes the organisation of the
disk.  If the copy of this held on the disk ever becomes out of step with the
disk itself this can I believe cause the symptons I have described.  But as 
explained above this problem has been observed at a time when the disk contents
and thus the superblock should not be changing.

We are using a 40mb disk with a single / file system built using the default
paramters.

So my questions are:-

	1) Do other people see this problem.
	2) Are my comments (sketchy though they are) on the workings of the
	   file system correct?
	3) How can we minimise, or alleviate it happening.
	4) I have heard of file system hardening, which as I understand it was
	   something AT&T put into the kernel around the early days of 
	   System V I believe to go some way towards reducing this problem.
	   Is this correct, and if so what exactly did it involve?

RTFMs, flames, email etc all welcomed.  I will summarise if there is sufficient
interest but please note there is a long delay at our site.

------------------------------------------------------------------------------
Disclaimer:  The opinions expressed are entirely my own, and not necessarily 
             those of Crosfield Electronics.

Name:    Ian Reid.                      S-Mail:  Peripheral Products,
                                                 R & D Department,
E-Mail:  ir@cel.uucp                             Crosfield Electronics Ltd.,
                or                                 Three Cherry Trees Lane,
         ...mcvax!ukc!uk.co.cel!ir               Hemel Hempstead,
                                                 Hertfordshire.
V-Mail:  +44 442 230000 extn 3759                HP2  7RH
                                                 England.
------------------------------------------------------------------------------

larry@nstar.uucp (Larry Snyder) (09/21/90)

ir@cel.co.uk (ian reid) writes:

>Forgive if this has been discussed before, but we have recently suffered a long
>break in news at this site.  My question is very simple, the reliability of the
>System V 1K file system when a machine is subjected to a power cycle without
>going through a proper shutdown sequence.  Unfortunately on our system an UPS
>is not an option (too price sensitive).

NSTAR is here primarily for news distribution - and sometimes looses power
twice a week anywhere from 1 to 5 minutes (that is normal here).  The system
comes right back up, rechecks the file system (that takes some time) then
continues processing mail.  We are running 2.2 386/ix with SCSI drives - and
have the write behind in the kernel set to 180 seconds - so at the worst case
we will loose some news -
-- 
       Larry Snyder, Northern Star Communications, Notre Dame, IN USA 
 {larry@nstar, uunet!sco!romed!nstar!larry, nstar!larry@ndmath.math.nd.edu}
                          usenet newsfeeds available
         Public Access Unix Site (219) 289-0282 (5 high speed lines)

cpcahil@virtech.uucp (Conor P. Cahill) (09/21/90)

In article <5869@suns302.cel.co.uk> ir@cel.co.uk (ian reid) writes:
>We are using a 40mb disk with a single / file system built using the default
>paramters.

Your problems would probably diminish if you had a root file system that
was fairly stable (i.e. no user files being created/deleted there).  You 
can do this by creating additional partitions and moving your news stuff 
to the other partition.

The big problem with power loss is that it may be doing things to your
system that no OS could protect against.  For example, at the point of
power loss the disk controller may be dumping junk information to different
parts of your disk.  I'm not saying that this is happening, just saying
that it could.  The loss of power to a system (especially from a provider
loss which can include all kinds of yucky power things like spikes) is
a very bad thing for a system.

>	1) Do other people see this problem.

We originaly ran our 386/ix systems with no protection (other than the  
standard surge protection) and had several power losses (usually around
two to three a month) without experiencing the problems that you describe.

>	2) Are my comments (sketchy though they are) on the workings of the
>	   file system correct?

Split you filesystems so that root and /usr are both as small as they
have to be and have little or no user data on them (i.e. create yet another
partition for your user data (total of three)).

>	3) How can we minimise, or alleviate it happening.

Get a UPS.  Ensure that you have a good surge protector.

>	4) I have heard of file system hardening, which as I understand it was
>	   something AT&T put into the kernel around the early days of 
>	   System V I believe to go some way towards reducing this problem.
>	   Is this correct, and if so what exactly did it involve?

File system hardening involves an ordered updating of the information 
related to file system changes which should make it harder for fs corruption
to occur. 


-- 
Conor P. Cahill            (703)430-9247        Virtual Technologies, Inc.,
uunet!virtech!cpcahil                           46030 Manekin Plaza, Suite 160
                                                Sterling, VA 22170 

art@pilikia.pegasus.com (Art Neilson) (09/22/90)

In article <5869@suns302.cel.co.uk> ir@cel.co.uk (ian reid) writes:
>
> [ stuff about file system getting hosed when power is cycled
>   without performing graceful shutdown ... ]

Every UNIX I have seen behaves in the manner you describe.  If you
hit the red switch or experience a power outage without performing a
graceful shutdown, you deserve whatever you get.  Most times fsck will
fix whatever mess is created by this, however there are no guarantees
that a critical data won't get hosed.  If your UNIX box is used for real
production work, you are quite foolish not to put it on an UPS.  Just how
valuable is your data to you ???  Certainly more than the price of an UPS,
especially if your business depends on the data stored in your machine.
-- 
Arthur W. Neilson III		| ARPA: art@pilikia.pegasus.com
Bank of Hawaii Tech Support	| UUCP: uunet!ucsd!nosc!pegasus!pilikia!art

avg@hq.demos.su (Vadim G. Antonov) (09/22/90)

In article <1990Sep22.041723.1599@pilikia.pegasus.com> art@pilikia.pegasus.com (Art Neilson) writes:
>Every UNIX I have seen behaves in the manner you describe.

	Hmmm... May be your path never crossed some UNIXes?
	For example Unix BSD 4.2 and later has a different file system
	structure which handles power fails more carefully.
	Another example is BSD 2.9 - it has file structure similar
	with v7's (but with 1Kb blocks). The main difference is in
	algorithms: the 2.9's kernel performed critical file system
	operations in the strict order - for exmaple it guarantees
	that if you delete a file inode will be cleared before releasing
	data blocks which in turn will be done before releasing this inode.
	Thus if power would fail during removing a file the worst thing
	which can happen is missed blocks - but generally file structure
	will remain correct. There are some similar tricks in BSD 2.9,
	but I do not want to describe it - get sources and read it
	yourself. Hmmm... +1 to BSD over SysV. (What's a pity there are
	no BSD on 386s!)
					 \ |\/| /
	Vadim Antonov                   -_(____)_-     The actual image
	DEMOS, Moscow, USSR             (        )     of my cat.
	(It is NOT a joke!)          -------||-------
					    ||
					    \/

cpcahil@virtech.uucp (Conor P. Cahill) (09/23/90)

In article <1990Sep22.160132.24999@hq.demos.su> avg@hq.demos.su (Vadim G. Antonov) writes:
>	algorithms: the 2.9's kernel performed critical file system
>	operations in the strict order - for exmaple it guarantees
>	that if you delete a file inode will be cleared before releasing
>	data blocks which in turn will be done before releasing this inode.

This is the same as (or very similar to) the algorithm used in "today's"
system V implementations.

>	Thus if power would fail during removing a file the worst thing
>	which can happen is missed blocks - but generally file structure
>	will remain correct. There are some similar tricks in BSD 2.9,

This is partially wrong.  This software change only guards against a system
stop where there are no hardware glitches.  A power failure can wreak havoc
all over the system if the *hardware* does not correctly handle the loss
of power during an otherwise normal operation.  From the original posting
about this problem, I would sumise that his hardware has some problems
with power loss (or that a simple power loss is not the only thing
happening).


>	but I do not want to describe it - get sources and read it
>	yourself. Hmmm... +1 to BSD over SysV.

No.  In this simple test they are about the same (and they are growing
closer every day).



-- 
Conor P. Cahill            (703)430-9247        Virtual Technologies, Inc.,
uunet!virtech!cpcahil                           46030 Manekin Plaza, Suite 160
                                                Sterling, VA 22170 

avg@hq.demos.su (Vadim G. Antonov) (09/24/90)

In article <1990Sep22.215128.16693@virtech.uucp> cpcahil@virtech.UUCP (Conor P. Cahill) writes:
>A power failure can wreak havoc all over the system if the *hardware*
>does not correctly handle the loss of power during an otherwise normal
>operation.

	... Or if disk dirver does not handle drive's idle state
	properly. Some controllers require clearing registers when
	device is inactive. It usually is not documented - and
	requires very careful design and testing of drivers.
	(Have you ever heard about testing Unix disk drivers on
	power failures?) Practically all machines provide power fail
	interrupts - and I don't know why Unix device drivers
	have no "xxpwfail" entries. Anyway I'm quite sure *any* device
	can correctly handle power fails - if you handle device
	properly :-).

	It seems to me the best way to protect disks from accidental
	damaging by power fails is to start recalibrating or moving
	heads to landing zone - usually quite simple logic circuitry
	protects from writing while heads move.

	Hmmm... For example I've never seen any recommendations about
	such things in DEC's HW manuals, but I've seen that DEC's PDP-11
	systems clear controller after completing disk i/o. Needless to
	say when I've made a similar trick in Unix I've never got
	powerfail problems again.

	My $0.02 is: the quailty of disk drivers determines the
	reliability of a whole system. It is possible to work even
	on Soviet hardware if you wrote good drives. :-) I think
	Western software manufacturers should buy Soviet computers
	to test their systems on :-). "One who can pilot Camel can
	fly on anything" [Back In The Future, if my memory is right].

	Vadim Antonov
	DEMOS, Moscow, USSR           The cat is sleeping under sofa.
	(It is NOT a joke!)

chris@mimsy.umd.edu (Chris Torek) (09/25/90)

In article <1990Sep23.184158.841@hq.demos.su>
avg@hq.demos.su (Vadim G. Antonov) writes:
>Practically all machines provide power fail interrupts -

I know of a number that do not; but in any case:

>and I don't know why Unix device drivers have no "xxpwfail" entries.

power failure interrupts are not any good unless they are guaranteed
to occur sufficiently early, and usually they are not.  The power supply
system on the main CPU (the thing that has a `power fail' interrupt)
is quite often completely independent of the power supply for the disk
drives.  If the electronics on the drive are in an indeterminate state,
nothing done at the CPU will guarantee anything.

>Anyway I'm quite sure *any* device can correctly handle power fails -
>if you handle device properly :-).

Power fail handling is like lightning protection: you can only do so much;
if Nature is out to get you, you are doomed.  (Lightning strikes have been
known to dance teasingly around all the grounding posts, giggle in circles
round and round as your hair stands on end, then viciously zap straight
into the heart of your computer.  Well, maybe not quite. :-) )

Designing systems that act properly on power failure is, however, tricky:

>It seems to me the best way to protect disks from accidental
>damaging by power fails is to start recalibrating or moving
>heads to landing zone - usually quite simple logic circuitry
>protects from writing while heads move.

Let me tell you about ... Century Data Systems T-300s.

(To be fair, I am not sure where the problem was located.  The T300s were
merely the end of the chain.)

We have a couple of Xerox file servers with big washtub drives.  These
drives have a power fail system that retracts the heads (quite reasonably)
so that they will not land on the disk when it stops spinning.  Apparently
it turns off the write current at the same time, because a simple power
failure does not damage anything.

Unfortunately, there are not-so-simple power failures.  Thunderstorms
(those thing that Californians never see :-) ) often cause momentary
power failures---anywhere from a fraction of a second to several
seconds.  (Power distribution systems have thing called `lightning
arrestors' that temporarily open the circuit to prevent serious
overvoltages.  There are two major variants, air and oil.  Lightning
will jump a simple air gap so the air versions blow compressed gas
across the gap.  I know nothing about the oil versions, other than that
they explode very prettily, like oil-filled transformers. :-) )

Anyway, as it happens, under certain conditions the T300s would detect
a power failure and begin retracting the heads.  Then the power would
come back on, the electronics would think, `oh, everything is OK', and
the write current would turn on---while the heads were still spiraling
down the pack.

The result was invariably a hopelessly damaged pack.  Hundreds of `bad'
sectors appeared in a spiral pattern, and the only means of recovery
was to reformat (followed by a tedious restore from the backup server,
all the while hoping desperately that another storm would not come up
during the multi-hour restore).  The problem has finally been fixed:
the file servers are now on a ten-minute UPS, and only a long-term power
failure---the kind the drives were engineered to handle---will get through.
-- 
In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 405 2750)
Domain:	chris@cs.umd.edu	Path:	uunet!mimsy!chris

avg@hq.demos.su (Vadim G. Antonov) (09/25/90)

In article <26688@mimsy.umd.edu> chris@mimsy.umd.edu (Chris Torek) writes:
>Unfortunately, there are not-so-simple power failures.

	Hmmm... Our backbone host at Kurchatov Institute of Atomic
	Energy has a funny surge protector - so called "motor-generator".
	This device includes synchronous AC motor paired with dinamo-machine
	producing AC of the same voltage. Protection is absolute.

	(Another interesting feature of our power supply at KIAE is that
	the secondary power supply goes directly from atomic generator -
	yes, there are some working nuclear reactors in the Moscow!)

	Vadim Antonov
	DEMOS, Moscow, USSR
	(It is NOT a joke!)

steve@altos86.Altos.COM (Steve Scherf) (09/26/90)

In article <1990Sep22.041723.1599@pilikia.pegasus.com> art@pilikia.pegasus.com (Art Neilson) writes:
>In article <5869@suns302.cel.co.uk> ir@cel.co.uk (ian reid) writes:
>>
>> [ stuff about file system getting hosed when power is cycled
>>   without performing graceful shutdown ... ]
>
>Every UNIX I have seen behaves in the manner you describe.  If you
>hit the red switch or experience a power outage without performing a
>graceful shutdown, you deserve whatever you get.

The Pick engineers here at Altos tell us that if a Pick system goes down
ungracefully, it's time to re-install...
-- 
Steve Scherf
steve@Altos.COM    ...!{sun|sco|pyramid|amdahl|uunet}!altos!steve

These opinions are solely mine, but others may share them if they like.

dhesi%cirrusl@oliveb.ATC.olivetti.com (Rahul Dhesi) (09/26/90)

In <1990Sep22.041723.1599@pilikia.pegasus.com> art@pilikia.pegasus.com
(Art Neilson) writes:

     Just how valuable is your data to you ???  Certainly more than the
     price of an UPS,...

Now, now, let's not jump to conclusions.
--
Rahul Dhesi <dhesi%cirrusl@oliveb.ATC.olivetti.com>
UUCP:  oliveb!cirrusl!dhesi