[comp.os.vms] Record-access libraries

eric@snark.UUCP (Eric S. Raymond) (09/23/88)

In article <207@cvbnet2.uucp>, aperez@cvbnet2.UUCP (Arturo Perez Ext.) writes:
> What I'm curious about is the fact that I've never heard of any record
> access libraries for Unix.  I know that I've written simpleminded record
> access applications.  I'm sure other people have as well.  Is there anyone
> actually selling record access libraries for the Unix community?  If not
> why isn't anyone doing it?

Shortest answer: because it's not worth doing.

Medium-length answer: because, for the most common case of sequentially-
accessed fixed-length records, 'record access' under UNIX is adequately
handled from C by fread(3)/fwrite(3) of a struct buffer. The less common cases
are traditionally handled by ASCII encoding with a \n-terminated line
structure, the one kind of variable-length 'record' for which there is strong
support under UNIX.

Long answer: well, what does a record-access library *do*, anyhow, that byte-
stream access doesn't give you? There are basically, two possible answers:

    1. Encourage code simplification by provision of 'higher-level' primitives.
    2. Access speed or space optimization.

Both of these turn out to be mostly illusion. UNIX's byte-stream primitives
are sort of ultimately simple, certainly a lot simpler than the thicket of RMS
entry points. The text-file support in stdio is pretty clean and basic too
(which is why it's become a de-facto standard for C implementations even on
non-UNIX systems).

A possibility of significant speed or space optimization over stdio or simple
fread/fwrite only enters if you're talking about access to variable-length
records that can't reasonably be represented in ASCII. Such applications tend
to a) require indexed access, in which case you're already talking database
and the 'record library' likely has to be custom to handle housekeeping info,
or b) be sufficiently speed or space-critical that no savvy programmer is
really going to like someone else's packaged library for access.

And don't forget that at some level the record-access has to be doing
byte-stream I/O *anyway*. The only way it can win is by buffering. And wouldn't
you rather tune your own buffer sizes?

Record-access libraries sound like a decent idea, but in practice they tend to
introduce a lot of interface complexity (which makes your code ugly) and
premature optimization (which actually hurts performance). VMS's RMS could
stand as a perfect example of both these lossages.
-- 
      Eric S. Raymond                     (the mad mastermind of TMN-Netnews)
      UUCP: ...!{uunet,att,rutgers}!snark!eric = eric@snark.UUCP
      Post: 22 S. Warren Avenue, Malvern, PA 19355      Phone: (215)-296-5718

bzs@xenna (Barry Shein) (09/25/88)

Eric, I disagree and think you dismiss the value of record access
libraries too glibly. There's more to it than fixed and blocked record
formats.

Consider various associative schemes like ISAM or B-TREE management of
records by keys (dbm is similar to this under Unix.) These can be very
critical to manageable performance in certain areas, particularly the
management of large numbers of records where lookup is only by a very
few keys and quick insertion of new key/record pairs is needed. Your
typical example is a customer data base where lookup/insertion keys
is based on some customer code.

Particular implementation strategies are a whole other kettle of fish.

	-Barry Shein, ||Encore||

eric@snark.UUCP (Eric S. Raymond) (09/27/88)

In article <3717@encore.uucp>, bzs@xenna (Barry Shein) writes:
> Eric, I disagree and think you dismiss the value of record access
> libraries too glibly. There's more to it than fixed and blocked record
> formats.
> 
> Consider various associative schemes like ISAM or B-TREE management...

I agree that libraries like UNIX dbm are a Good Thing -- but then ISAM, B-TREE
management, and other associative retrieval schemes are not within the scope
of the original question. Let's not muddy the waters by confusing 'record
access' in the RMS "fixed and blocked record" sense with more general database
access techniques.
-- 
      Eric S. Raymond                     (the mad mastermind of TMN-Netnews)
      UUCP: ...!{uunet,att,rutgers}!snark!eric = eric@snark.UUCP
      Post: 22 S. Warren Avenue, Malvern, PA 19355      Phone: (215)-296-5718

oz@yunexus.UUCP (Ozan Yigit) (09/30/88)

[disclaimer: I am an ex-VMS-hack, used RMS extensively in the past, like
its power, puke at the sight of its interface, and here, I am speaking of
facilities, rather than a messy filesystem format to support multiple
file formats]

In article <e1Lt8#2CJhqZ=eric@snark.UUCP> Eric S. Raymond writes:
> ...other associative retrieval schemes are not within the scope
>of the original question. Let's not muddy the waters by confusing 'record
>access' in the RMS "fixed and blocked record" sense with more general 
> database access techniques.

They happen to be an integral part of RMS, and extremely well done in
functionality, leaving the "goriness" of the interface aside.
[Reportedly, even the designer of RMS admitted to its ugliness :-)] 
If I had a "library" half as powerful as some of RMS's facilities, I would
be damn happy... It is a shame that best un*x can come with is ndbm (and
that is, if you are lucky :-) Why do you think andrew@alice (Bell Labs)
was asking about something that can handle gigantic databases ??
Aren't you curious why more and more programs across variants of un*xes 
rely on dbm/ndbm type databases ?? It must be RMSicide !!

RMS Record access you say ?? this concept is especially revolting for our
purist(!) byte-stream-with-a-nl (usually 512 bytes or shorter for idiotic
programs :-) perspective.  Alas, we merrily keep faking that our byte
stream is in fact a collection of average size lines (records for less
religious :-) and rush to our soapboxes at the suggestion that something
as ugly as RMS can actually be useful. (*because* of its support of "more
general database access tecniques".)

You know, there is no such thing as "a little bit of purity". On the other
hand, crap is crap.

[Sounds of shuffling a plane ticket to alaska, to escape flames...]

oz
-- 
Reflections are 	        	| Usenet: ...!utzoo!yunexus!oz
images of tarnished aspirations.	|   ...uunet!mnetor!yunexus!oz
			RACTER		| Bitnet: oz@[yulibra|yuyetti]
[an Artifically Insane program.]	| Phonet: +1 416 736-5257x3976

ccdn@levels.sait.edu.au (DAVID NEWALL) (10/05/88)

In article <864@yunexus.UUCP>, oz@yunexus.UUCP (Ozan Yigit) writes:
> RMS Record access you say ?? this concept is especially revolting for our
> purist(!) byte-stream-with-a-nl (usually 512 bytes or shorter for idiotic
> programs :-) perspective.  Alas, we merrily keep faking that our byte
> stream is in fact a collection of average size lines (records for less
> religious :-) and rush to our soapboxes at the suggestion that something
> as ugly as RMS can actually be useful. (*because* of its support of "more
> general database access tecniques".)

I know little about VMS; not nothing, because the Institute (where I work)
uses VMS as it's priniciple operating system.  On the other hand, before
I joined the Institute, `it' knew little about Unix; Unix is a new thing
here.

Something that has surprised me a few times, is how often RMS seems to get
in the way of what you are doing.

For example, there was a tar tape written on some Unix machine, from which
some files were to be extracted.  Easy to do if you have pd-tar running on
VMS, but we didn't.  (This actually happened before I started working for
the Institute, and so my understanding of the problem is vaugue at best).
The thought was that the tar file could be extracted from the tape and
kermitted to the Unix box.  (We had just got the Unix machines, you must
understand, and hadn't yet got sophisticated things like tape drives or
ethernet cards).  I don't quite understand why that didn't work, but I
recall people mumbling about "RMS" and "File Type" or "Record Format".

Or then there was the time I suggested backing up our Unix machines across
ethernet, onto the VAX tape drive.  The answer then was similarly something
about "FILES-11", "too difficult".

Please excuse my quite obvious ignorance of VMS; I use it very little.
This is not intended as a criticism of VMS.  I am just surprised at the
apparent stumbling blocks one can encounter using RMS.  The power provided
seems to have cost in terms of flexibility.

David Newall                     Phone:  +61 8 343 3160
Unix Systems Programmer          Fax:    +61 8 349 6939
Academic Computing Service       E-mail: ccdn@levels.sait.oz.au
SA Institute of Technology       Post:   The Levels, South Australia, 5095