[comp.unix.xenix] An interesting property of 'dd' and virtual memory

frankb@usource.UUCP (Frank Bicknell) (05/27/89)

I was experimenting with using huge buffers to write/read data
to/from the tape device.  I decided to see just how large it
would let me make that buffer.

First, it's important to know that I'm running SCO Xenix 2.3.1
(386) on a machine with only 2.5M of RAM... 

I noticed that if I use a very large buffer on the 'tape' side
of the dd command, the tape makes longer runs with fewer
underrun errors.  I found that using a 1M buffer size (1024k)
allows tape runs of about 13 seconds or so.  The time it takes
to write that 1M to the hard disk is about 25s, so we know where
the bottleneck is, eh?

Anyway, a command to do the above:

dd if=/dev/erct0 of=tape.file ibs=1024k

Works very nicely.  I tried ibs=2048k and received quite a
surprise!  It seems that before starting the copy process, a
contiguous buffer must be found (my guess).  A short pause
with a bunch of disk activity (probably other pages being
swapped out of memory?) before the tape drive starts on the
1024k buffer seems to verify this.  If I increase the buffer
size to 2M (which I don't have in physical ram after taking
out OS space and video space, etc) the thing goes off
swapping forever! 

No kill command will kill it, no interrupt/quit buttons
allowed (as it's probably tied up in the kernel).  Other
processes do get an occasional slice of time thrown at them
so fortunately it is possible to shut the system down, but
that was the only way I could find to stop the runaway dd.  

This is all harmless enough if one remembers not to request more
memory than you actually have, but I thought this was one of the
wonders of virtual memory: one needn't worry if one asks for too
much, pages are moved around to allow pieces to be accessible to
the requesting process.  It seems as if 'dd' is making some sort
of special request that is bypassing this virtual feature, ie
locking itself in physical memory.  Is this in fact what 'dd' is
doing?
-- 
Frank Bicknell
UniSource; 1405 Main St, Ste 709; Sarasota, FL 34236
killer!usource!frankb || frankb@usource.UUCP