howard@brl-tgr.ARPA (Howard Walter <howard>) (05/02/85)
At BRL, we have been running the two latest versions of commercial Ingres produced by Relational Technology. Version 2.0 required kernel modifications to implement a lock driver in order for RTIngres to run in multi-user mode. Version 2.1, the latest version for Unix, requires a binary-only module to be linked with the normal kernel routines. Mike Scott, R. T. Director for New Environments (415-769-1400), told me that sockets couldn't be used for the lock driver since that would only work on 4.2 systems and shared memory was out since that only worked on Sys5; kernel mods would work on any version of UNIX. We are concerned about the security implications of binary-only modules added to our kernel. We are also having problems trying to run RTIngres on newer kernels (pre-4.3). Relational Technology would be happy to sell us the source for the kernel mod (as well as the rest of RTIngres) for only a few hundreds of thousands of dollars! Is anyone else running RTIngres? How do you feel about using binary-only modules in your kernel? Howard Walter howard@brl ...seismo!brl-bmd!howard
jbn@wdl1.UUCP (05/11/85)
RTI needs to get their act together. Requiring kernel changes for an applications program is totally out of line in today's world. The ``Ingres lock device'' is a hack that dates back to at least PWB/Unix and maybe before. There is enough interprocess communication available now that it shouldn't be necessary. JN
kalash@ucbcad.UUCP (Joe Kalash) (05/13/85)
> The ``Ingres lock device'' is a hack that dates back to at least PWB/Unix > and maybe before. There is enough interprocess communication available > now that it shouldn't be necessary. In fact, it dates to V6. However, the obvious question is which version of IPC should they use? Currently Sys V ipc is not compatible with BSD ipc, which is not compatible with Version 7 ipc (the happily gone mpx files), etc. While I wrote the ipc version of the driver for 4.2 (back when I was chief programmer for the project), I was able to get away with it because the project ONLY supports Ingres on 4.(current release). RTI has to support their system on a multitude of systems. Joe Kalash kalash@berkeley ucbvax!kalash
eric@osiris.UUCP (Eric Bergan) (05/15/85)
> > The ``Ingres lock device'' is a hack that dates back to at least PWB/Unix > > and maybe before. There is enough interprocess communication available > > now that it shouldn't be necessary. > > In fact, it dates to V6. However, the obvious question is > which version of IPC should they use? Currently Sys V ipc is not > compatible with BSD ipc, which is not compatible with Version 7 > ipc (the happily gone mpx files), etc. > RTI has to support their system on a multitude of systems. Not only that, RTI offers Ingres under VMS, and on IBM mainframes. A consistent lock interface would be important in trying to bring out new versions in a timely manner on all the different machines. And we are talking about installing a driver here, not some change to the file system, or to the scheduler. If security is the concern, I would point out that the Ingres backend runs suid to "ingres", and all the databases are owned by this user, so you are relying on the trustworthiness of the backend protection code to keep users from reading potentially sensitive database information. I installed Ingres in the Pentagon for the Department of the Army, to be used by several thousand users, and no one there complained about any possible breaches (it was not, however, handling DOD classified material). As Thompson said in his Turing address - you can only completely trust software when you have written all of it yourself - including operating system, compilers, linkers, applications, etc. -- eric ...!seismo!umcp-cs!aplvax!osiris!eric
gwyn@brl-tgr.ARPA (Doug Gwyn <gwyn>) (05/15/85)
> ... RTI has to support their system on a multitude of systems.
Bogus! Other vendors seem to be able to design software to cope
with a variety of systems without asking one to modify his kernel.
For example, one can write per-system-type user-mode "lock"
interface modules that work acceptably well.
Not only that, but when RTI made our Ingres users install their
lock driver, it turned out that their software still didn't
support concurrent database updates! What was the point of the
lock driver??sigma@usl.UUCP (Spiros Triantafyllopoulos) (05/18/85)
In article <410@wdl1.UUCP> jbn@wdl1.UUCP writes: > > RTI needs to get their act together. Requiring kernel changes for >an applications program is totally out of line in today's world. The >``Ingres lock device'' is a hack that dates back to at least PWB/Unix >and maybe before. There is enough interprocess communication available >now that it shouldn't be necessary. > > JN You bet they need! I am writing a restricted natural language interface for it, storing the dictionary in Ingres itself. Not much of it, ~200 words so far, but it just takes FOR EVER!. The worst thing is that a lot of the time is spent for interprocess communication... pipe'em in, flush 'em out, etc! Anyone wants to buy a NL interface for RTI, anyway? :-) Oh, before I forget: has anyone done any serious development work on RTI using EQUEL? I want to see how other people think about it. (In terms of performance, ease of programming, cludge-ness, etc) As usual, please respond by e-mail... Spiros ut-sally!usl!sigma
kalash@ucbcad.UUCP (Joe Kalash) (05/18/85)
> > ... RTI has to support their system on a multitude of systems. > > Bogus! Other vendors seem to be able to design software to cope > with a variety of systems without asking one to modify his kernel. > For example, one can write per-system-type user-mode "lock" > interface modules that work acceptably well. Really, how does one write such wonderful beasties? Indeed, what do you mean? You have lots of buzz words, but you didn't say anything. > > Not only that, but when RTI made our Ingres users install their > lock driver, it turned out that their software still didn't > support concurrent database updates! What was the point of the > lock driver?? The point of the lock driver is to keep two people from updating the same relation at the same time. If I do a "give everyone a 10% raise", at the same time you are doing a "Give everyone in the shoe department a 5% raise", if these two updates do not start and finish independently some employees in the shoe dept. could get a 10% raise, some could get a 5% raise, and some could get a 15.5% raise. Not uselly considered useful. Joe Kalash kalash@berkeley ucbvax!kalash
jbn@wdl1.UUCP (05/22/85)
One other alternative would be to run the network version of
Ingres, where there is one server and N client modules. Unfortunately,
we tried to install network Ingres on our diskless SUN workstations, and
the installation procedure didn't really comprehend diskless stations
properly. So we can't say whether it really works or not. We're still
running Ingres on one station only.
How does Ingres work under NFS? Any reports yet?
John Naglegwyn@brl-tgr.ARPA (Doug Gwyn <gwyn>) (05/23/85)
> > For example, one can write per-system-type user-mode "lock" > > interface modules that work acceptably well. > > Really, how does one write such wonderful beasties? Indeed, what > do you mean? You have lots of buzz words, but you didn't say anything. This is such a standard technique for portable packages that I didn't think anyone would fail to understand (but then obviously RTI doesn't or we wouldn't be having this conversation). To make it clearer, consider that the application code is written once, in a system- independent manner, and that all access to any host features that are not fully standardized (e.g. beyond stdio, strcpy, etc.) is done by means of a separate support library. The interface to the routines in the support library is carefully designed to be achievable on a wide variety of systems. For each target system, there is an implementation of the support library whose internal details are system-specific, but whose interface used by the application is invariant across all systems. Now, I'm not going to design a portable interface to file/record locking facilities for you; that's the job of the software engineering (what's that?) department. Get to work! > > Not only that, but when RTI made our Ingres users install their > > lock driver, it turned out that their software still didn't > > support concurrent database updates! What was the point of the > > lock driver?? > > The point of the lock driver is to keep two people from updating > the same relation at the same time. ... Good grief, I just explained that RTI Ingres FAILED to do this right in spite of their "lock driver".
kalash@ucbcad.UUCP (Joe Kalash) (05/25/85)
]]] This is getting silly.... [[[ > > This is such a standard technique for portable packages that I didn't > think anyone would fail to understand... > the support library is carefully designed to be achievable on a wide > variety of systems. For each target system, there is an implementation > of the support library whose internal details are system-specific, but > whose interface used by the application is invariant across all systems. You are not listening very well. Let me say this slowly. Unix V6, V7, Sys III, Sys V, and 4.1 do not have and useful IPC system for writing a lock driver with any performance (I suppose you could write out info to a file, and read it in, but the peformance would be prohibitive). > > Now, I'm not going to design a portable interface to file/record > locking facilities for you; That I can certianly beleive. > ... Get to work! I do not now, nor have I ever worked for RTI. Joe Kalash kalash@berkeley ucbvax!kalash
gwyn@brl-tgr.ARPA (Doug Gwyn <gwyn>) (05/27/85)
There are really only two important UNIX variants for the near future, UNIX System V and 4.2BSD. Both have facilities for database locking, and UNIX System V now has the /usr/group lockf system call for record locking. (I have considered putting this into the 4.3BSD kernel, but there are several semantic clashes with the 4BSD flock facility and it is not obvious how to resolve them.) I even have a user-mode lockf emulation for 4.2BSD; although it isn't terrific, given the limited abilities of flock, it is adequate for use (if careful) in making a general-purpose DBMS that would port directly to UNIX System V. Commercial UNIX systems like Xenix have provided record-locking support for a long time now, generally along the lines of lockf. Even without lockf, UNIX System V IPC can be (and has been) used to implement reasonable database concurrent access. I repeat, there are numerous DBMSes commercially available for a wide variety of UNIX systems that work fine without kernel hacking. Why is RTIngres such a conspicuous exception? Sure, Ingres was developed in the dark ages, but hasn't RTI had plenty of opportunity to improve it? P.S. I am informed that we removed the RTIngres "lock driver" from our one system that had it installed. RTIngres was not correctly handling concurrent update of databases anyway, so the sole reason for the "lock driver" was invalidated. P.P.S. I'm not a database expert (yet), but I can certainly recognize poor software design when I see it (which is all too often, alas).
jas@rtech.UUCP (Jim Shankland) (06/05/85)
I have been following with interest the recent discussion on
concurrency managemenet in RTI's INGRES, particularly Doug Gwyn's
increasingly acerbic broadsides. As someone who works at RTI and is
familiar with the issues involved, I believe I am in a position to
correct some of the misperceptions Doug has been sharing with the net.
Briefly, Doug's arguments are: it is reprehensible of us to
require our users to install a pseudo-device driver for INGRES
concurrency control. We have no right to expect our users to make
kernel mods in order to run INGRES, a mere applications program.
Writing a user-level lock manager is straightforward, albeit different
for different versions of UNIX. Furthermore, there are always the
"flock" system call of 4.2bsd, and the "lockf" system call of the
/usr/group standard. Finally, Doug says that INGRES does not even make
use of the lock pseudo-device ("concurrent updates are not supported"),
so that making the kernel mod is a pointless exercise. Doug clearly
implies that we at RTI don't know what we're doing. His last word (so
far) is: "I'm not an expert on database systems (yet), but I recognize
poor software design when I see it."
I'll dispose of the easiest complaint first by noting that it
is simply incorrect that we do not make use of the lock pseudo-device.
We support concurrent updates, as well as multi-statement
transactions. We use the locking pseudo-device to ensure that all
concurrent transaction executions are serializable, while still
maximizing concurrent access to shared data -- precisely what
concurrency management in a DBMS is all about.
I'm certainly not wild about the idea of a lock device driver,
and I don't think anyone else at RTI is, either. We have been and will
continue to investigate alternatives. However, there are serious (not
necessarliy disqualifying) disadvantages to all alternatives we have
found so far.
The "flock" system call, which locks an entire file at a time,
is inadequate for our purposes; we need a finer granularity of locking
than that. The "lockf" system call, which permits locking of an
arbitrary, contiguous subsection of a file, is getting closer. But it
is currently available on few, if any, of the systems on which we offer
INGRES. Neither call provides more than two lock modes, does any
deadlock detection, or any reasonable cleanup after a process exits
abnormally (merely releasing the locks that process holds is NOT
enough!). These are all serious shortcomings. Granted, we could make
INGRES use these calls; but we would be offering a product with
significantly reduced functionality compared with what we now offer.
The sad fact is that UNIX offers grossly inadequate concurrency
control for a real DBMS. Anyone who doubts this might do well to look
at silly, old VAX/VMS; its lock manager puts UNIX to shame.
That leaves us with the possibility of a user-level lock
manager process. Such a process presumably receives lock request
messages on some sort of named message channel, and sends response
messages back. Named FIFO's and System V messages come to mind for
System V, and sockets for 4.2bsd. The three major problems with this
approach are: (1) how does the lock manager (asynchronously) find out
about abnormal termination of a client?; (2) if every client requires
its own communications channel, how does the lock manager support more
than 20 simultaneous clients, given UNIX's open-file limit? (Yes, this
IS a requirement!); and (3) a lock request will now take 4 system
calls' worth of overhead, rather than 1 (client writes request, server
reads, server writes response, client reads).
Spiros Triantafyllopoulos has contributed to the discussion by
saying that INGRES is slow, and spends too much time in inter-process
communication. Spiros appeared to be agreeing with Doug, yet Spiros'
goals are diametrically opposed to Doug's. Doug brushes aside the
performance issue by saying that a user-mode lock manager will perform
"acceptably;" but will it be acceptable to Spiros?
The point of all this is not to insist that the pseudo-device
driver is a good idea, or to say that all other solutions are
unworkable. Rather, I hope I have shown that the issues are far from
cut-and-dried: that every possible approach, including the
pseudo-device driver, has serious disadvantages. As I mentioned above,
we are actively investigating alternatives. We welcome constructive
suggestions as well as just expressions of preference, particularly
when they shed more light than heat on this complex subject.
Jim Shankland
..!ucbvax!mtxinu!rtech!jas
..!ihnp4!pegasus!rtech!jaschris@umcp-cs.UUCP (Chris Torek) (06/08/85)
By the way, 4.3 will probably come with a default of 40 simultaneous file opens. (They finally got rid of the vread and vwrite hack.) So under 4.3 you could do lock stuff with a socket and select (select for read is true if the other end of the socket has been closed). If anyone wants more than 40 simultaneous lock fd's, just increase NOFILE.... -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 4251) UUCP: seismo!umcp-cs!chris CSNet: chris@umcp-cs ARPA: chris@maryland