rcw@qetzal.UUCP (sysop) (07/01/87)
Hello Gurus, I have been trying to write some routines to read from mag tape using the read() system call, and have discovered an interesting thing. I guess it makes sense, but I am not sure how to get around it. If I have a tape where logical record size = 80 bytes, and physical record size = 8000 bytes, and I do two discrete read() calls, from /dev/rmt0, I get the first 80 bytes from each PHYSICAL record, not 160 bytes in succession. (Wicat 160 w/Uniplus V.2 + Cipher 1600bpi drive) I can understand how this might be a characteristic of the tape drive hardware. It looked like setvbuf might provide a way around this problem, but ours doesn't seem to work. I'd be interested in hearing how others have solved this problem in general terms. THe manual makes no mention of this sort of behavior in the description for read(). Please send mail, and if this turns out to be a general problem, I'll post a summary. If this turns out to be a site specific problem, I'll flame the Unix vendor. Gracias. -- Robert C. White, Jr. Graphics Information, Inc. **************** UUCP: ihnp4!upba!qetzal!rcw isis!qetzal!rcw * OIL/GAS/LAND * USPS: 3067 Robin Way, Denver, CO 80222 * CARTOGRAPHY * ATT : +1 303 759-3666 ****************
jerry@oliveb.UUCP (07/10/87)
In article <153@qetzal.UUCP> rcw@qetzal.UUCP (sysop) writes: >physical record size = 8000 bytes, and I do two discrete read() calls, >from /dev/rmt0, I get the first 80 bytes from each PHYSICAL record, >not 160 bytes in succession. (Wicat 160 w/Uniplus V.2 + Cipher >1600bpi drive) This is primarily a characteristic of the tape hardware. Once you start reading a record you can't just top in the middle and wait for the next read. The alternative is to buffer the data. This allows you to return the requested number of bytes and save the rest of the record for the next read. The problem with this is how big do you make the buffer? If the tape has a maximum record size and it is not unreasonable then the driver can include buffer(s) of that size. But some drives allow records approaching the size of physical memory! Unix provides a normal and a raw interface to the tape drive. The normal one will provide the buffering you desire providing the records are less than the buffer size. The buffer size varies depending on the version but is on the order of 512 or 1024 bytes. Not much help for the tape you have. The raw interface provides no buffering so you must read a record in using a single read large enough to hold the maximum expected record. If the record is smaller the value returned from read will tell you that. The utility "dd" includes a number of features usefull for dealing with tape records. It can read one input record size and write out a compleatly different one. If your program can accept input from a pipe then you can "dd if=/dev/tape_name ibs=8000 | yourprog" and the pipe will provide the buffering you need. The alternative is to dd the tape into a disk file and process it there. Jerry Aguirre
kvt@drutx.ATT.COM (TranKV) (03/25/88)
To all UNIX/C gurus out there: When using read(2)/write(2) to access a physical device in raw mode (9-track tape devices specifically), are there a standard errno's that read(2)/write(2) returns to signify end of physical tapes? Or do they vary among UNIX versions and tape drives? If they are indeed different, what one can do in application programs to make them portable when changing tape drives or machines? All suggestions are appreciated. Kim Tran AT&T EUO
kvt@drutx.ATT.COM (TranKV) (03/25/88)
In article <7043@drutx.ATT.COM>, kvt@drutx.ATT.COM (TranKV) writes: > To all UNIX/C gurus out there: > > When using read(2)/write(2) to access a physical device in raw mode > (9-track tape devices specifically), are there a standard errno's that > read(2)/write(2) returns to signify end of physical tapes? Or do they > vary among UNIX versions and tape drives? If they are indeed different, > what one can do in application programs to make them portable when > changing tape drives or machines? All suggestions are appreciated. > > Kim Tran > AT&T EUO > To add more spices to the discussion, I should have thrown in some evidences in the first posting. Here they are: - Using our 3B5 with UNIX V 2.0, we got errno=5 for both end of tape read(2) and write(2). - Using our 3B2/600 with UNIX V 3.1 with 9-track tape drive hooked up through SCSI, we got errno=28 for end of tape write(2) and errno=0 for end of tape read(2). - We wrote a small C program to write(2) 5120 bytes at a time to a 9-track tape on the 3B5 (using raw mode) until we got to the end of tape. Then we tried to read(2) the tape (5120 bytes at a time) on the 3B2. To our surprise, 3B2 can read more than the 3B5 wrote and the 3B2 cannot detect end of tape without giving error warning. Reverse the read/write order, the 3B5 seems read less than what was put on tape by the 3B2. So we're confused. And help is definitely needed. Kim Tran
kvt@drutx.ATT.COM (TranKV) (03/25/88)
When read(2)/write(2) are used to access a physical device in raw mode
(9-track tape devices specifically), are there a standard errno's that
read(2)/write(2) returns to signify end of physical tapes? Or do they
vary among UNIX versions and tape drives? If they are indeed different,
what one can do in application programs to make them portable when
changing tape drives or machines?
To add more spices to the discussion, here are some interesting
phenomenon:
- Using our 3B5 with UNIX V 2.0, we got errno=5 for both end of
tape read(2) and write(2).
- Using our 3B2/600 with UNIX V 3.1 with 9-track tape drive hooked
up through SCSI, we got errno=28 for end of tape write(2) and
errno=0 for end of tape read(2).
- We wrote a small C program to write(2) 5120 bytes at a time
to a 9-track tape on the 3B5 (using raw mode) until we got to
the end of tape. Then we tried to read(2) the tape (5120 bytes
at a time) on the 3B2. To our surprise, 3B2 can read more than
the 3B5 wrote and the 3B2 cannot detect end of tape without
giving error warning. Reverse the read/write order, the 3B5
seems read less than what was put on tape by the 3B2.
So we're confused. And help is definitely needed.
Kim Tran
AT&T EUOhenry@utzoo.uucp (Henry Spencer) (03/27/88)
> When read(2)/write(2) are used to access a physical device in raw mode > (9-track tape devices specifically), are there a standard errno's that > read(2)/write(2) returns to signify end of physical tapes? The magtape driver unfortunately is an area where many people have exercised their creativity, to the detriment of standardization and portability... -- "Noalias must go. This is | Henry Spencer @ U of Toronto Zoology non-negotiable." --DMR | {allegra,ihnp4,decvax,utai}!utzoo!henry
atbowler@watmath.waterloo.edu (Alan T. Bowler [SDG]) (04/08/88)
In article <7051@drutx.ATT.COM> kvt@drutx.ATT.COM (TranKV) writes: > >When read(2)/write(2) are used to access a physical device in raw mode >(9-track tape devices specifically), are there a standard errno's that >read(2)/write(2) returns to signify end of physical tapes? Or do they >vary among UNIX versions and tape drives? If they are indeed different, >what one can do in application programs to make them portable when >changing tape drives or machines? Henry Spencer has observed that the mag tape drivers and their status returns tend to be non-standard. Part of the reason of course is that the original Unix drivers ignored end of tape. That aside I will point out that most tape hardware has NO end-of-tape status on a read. On a write, the hardware usually returns "successful write, and by the way you just passed the foil marker". The software is expected to observe the end of tape indication and write something on the tape that the reader will recognize as an end of tape. On the older operating systems, this usually is an eof marker and a trailer label indicating the name of the reel that the data is continued on. The Unix convention of return statuses from write does not lend itself well to this concept of "the operation worked but there is something that requires your attention soon (i.e. before you run out of wrapup space)". I.e. if the write is successful, then it should be returning the number of bytes written, and so the programmer doesn't get the idea he should check errno. If the write returns -1 to tell the user he should check errno, the program has lost the "actual byte count written" data that the program needs to continue (assuming he is not using a raw device). The early Unix mag tape drivers solved the problem by ignoring the end of tape status, and assuming programs would precalculate a "guestimate" of how much the tape would hold. Since then various things have been done depending on the expected use pattern at individual sites.
jwp@larry.sal.wisc.edu (Jeffrey W Percival) (05/24/89)
I have a TK50 tape with 3 80-byte records followed by a tape file mark, which is in turn followed by many 8192-byte records. I wrote a little C program that does a read(2) in a tight loop, and reports the number of bytes read. We expect this: 80 bytes read 80 bytes read 80 bytes read 0 bytes read 8192 bytes read 8192 bytes read ... Right? We get this output on a VS2000 with Ultrix 2.2, and a VS2000 with Ultrix 3.0. However, on 2 of our MicroVAX II's running Ultrix 3.0, we see this: 80 bytes read 80 bytes read 80 bytes read 0 bytes read 0 bytes read 0 bytes read 0 bytes read 0 bytes read 0 bytes read 0 bytes read 0 bytes read ... It looks like the read(2) is not getting past the file mark. This could be a problem in the C routine read(2), the Ultrix system, or in the TK50 controller. Has anyone seen this, or know what's going on? -- Jeff Percival (jwp@larry.sal.wisc.edu)
john@polyof.UUCP ( John Buck ) (05/25/89)
In article <187@larry.sal.wisc.edu>, jwp@larry.sal.wisc.edu (Jeffrey W Percival) writes: > ... I wrote a little > C program that does a read(2) in a tight loop, and reports the number > of bytes read. We expect this: > 80 bytes read > 80 bytes read > 80 bytes read > 0 bytes read > 8192 bytes read > 8192 bytes read > ... > Right? We get this output on a VS2000 with Ultrix 2.2, and a VS2000 I would not expect it to do what you said. > with Ultrix 3.0. However, on 2 of our MicroVAX II's running Ultrix 3.0, > we see this: > 80 bytes read > 80 bytes read > 80 bytes read > 0 bytes read > 0 bytes read > 0 bytes read > ... > Jeff Percival (jwp@larry.sal.wisc.edu) Generally, most tape drivers I've dealt with, require you to close(2) the tape device, then re-open it. Closing the device (on the no-rewind device) causes the tape to advance past the next EOF mark. When you re-open the device, the next read you do will start at the next "file" on the tape. Attempting to read(2) past the EOF mark continues to generate "0" byte reads, as it should. What you probably want to do is: open(no-rewind-tape); read(); read(); read(); close(); open(no-rewind-tape); while(read() != 0){ ... } close(); Perhaps DEC changed the way the tape driver works for different type machines (tape drives?)? John Buck john@polyof.poly.edu
envbvs@epb2.lbl.gov (Brian V. Smith) (05/25/89)
According to the manual entry for mtio(4) on Ultrix 3.0
(quoted w/o permission):
...
done with a tape ioctl call. A zero byte count is returned
when a tape mark is read, but another read will fetch the
first record of the next tape file. When a file open for
writing is closed, two end-of-files (EOF) are written.
Therefore, you could expect to get 0 bytes read for two reads
(because of two possible EOF's on the tape), but after that it should
read the next file.
I haven't tried it on our machines, but it looks like what you have
run into is a real bug.
Brian Smith
Lawrence Berkeley Laboratory
bvsmith@lbl.govres@ihlpb.ATT.COM (Rich Strebendt) (05/25/89)
In article <187@larry.sal.wisc.edu>, jwp@larry.sal.wisc.edu (Jeffrey W Percival) writes: | I have a TK50 tape with 3 80-byte records followed by a tape file mark, | which is in turn followed by many 8192-byte records. I wrote a little | C program that does a read(2) in a tight loop, and reports the number | of bytes read. We expect this: | | 80 bytes read | 80 bytes read | 80 bytes read | 0 bytes read | 8192 bytes read | 8192 bytes read | ... | | Right? We get this output on a VS2000 with Ultrix 2.2, and a VS2000 | with Ultrix 3.0. However, on 2 of our MicroVAX II's running Ultrix 3.0, | we see this: | | 80 bytes read | 80 bytes read | 80 bytes read | 0 bytes read | 0 bytes read | 0 bytes read | 0 bytes read | 0 bytes read | 0 bytes read | 0 bytes read | 0 bytes read | ... | | It looks like the read(2) is not getting past the file mark. This | could be a problem in the C routine read(2), the Ultrix system, or | in the TK50 controller. Has anyone seen this, or know what's going on? In order to read two files off of a tape on any reasonable system, the scenario should be the following: Open the device w/o rewind on close Read the first file -- perhaps until EOF is encountered Close the device The tape unit should now position the tape with the head just past the EOF mark. Open the device (typically with rewind) Read the second file until EOF is encountered or until program is happy Close the device and watch the tape reel spin to BOT On all of the tape controllers with which I have worked, if EOF has been encountered the controller does not attempt to read the tape. It just sends back another "you already found EOF, dummy" indication. This is to protect against a program ignoring EOF and attempting to read a couple thousand feet of possibly blank tape -- one heck of an Inter-Block Gap!! Rich Strebendt ...!att!ihlpj!res
jwp@larry.sal.wisc.edu (Jeffrey W Percival) (05/25/89)
In article <466@polyof.UUCP> john@polyof.UUCP ( John Buck ) writes: >Attempting to read(2) past the EOF mark continues to generate "0" byte >reads, as it should. The mtio(4) section in the 4.3BSD manual says: "A zero byte count is returned when a tape mark is read, but another read will fetch the first record of the new tape file." so I guess "should" has to have more of a context. -- Jeff Percival (jwp@larry.sal.wisc.edu)
grr@cbmvax.UUCP (George Robbins) (05/25/89)
In article <2721@helios.ee.lbl.gov> envbvs@epb2.lbl.gov (Brian V. Smith) writes: > According to the manual entry for mtio(4) on Ultrix 3.0 > (quoted w/o permission): > > done with a tape ioctl call. A zero byte count is returned > when a tape mark is read, but another read will fetch the > first record of the next tape file. When a file open for > writing is closed, two end-of-files (EOF) are written. And the tape is repositioned after the first of the two tape marks... > Therefore, you could expect to get 0 bytes read for two reads > (because of two possible EOF's on the tape), but after that it should > read the next file. No, the bug is most likely that the Ultrix driver *is* playing the sticky EOF game. If the program closed/opened the input file it would probably work fine, though the man pages imply this is not necessary. -- George Robbins - now working for, uucp: {uunet|pyramid|rutgers}!cbmvax!grr but no way officially representing arpa: cbmvax!grr@uunet.uu.net Commodore, Engineering Department fone: 215-431-9255 (only by moonlite)
ggs@ulysses.homer.nj.att.com (Griff Smith) (05/25/89)
In article <10530@ihlpb.ATT.COM>, res@ihlpb.ATT.COM (Rich Strebendt) writes: [deleted description of standard System V tape device driver behavior] > On all of the tape controllers with which I have worked, if EOF has been > encountered the controller does not attempt to read the tape. It just sends > back another "you already found EOF, dummy" indication. Perhaps you haven't used DEC products? > This is to protect > against a program ignoring EOF and attempting to read a couple thousand feet of > possibly blank tape -- one heck of an Inter-Block Gap!! So what? A well designed controller will time out after about 20 feet. This is another round in the battle between the `let's protect people from their stupidity' school and the `give me simple tools that do what I say' school. If you have ever tried to debug a damaged tape, the gratuitous file positioning enforced by the System V drivers can be maddening. I have some software that implements IBM tape input access methods. The software is convenient to use on BSD systems where EOF isn't sticky; people can use the auto-rewind tape names reliably, even though each dataset is actually three files. For the System V version of the software I have to document the feature that the auto-rewind files must not be used; the @#$%@# operating system rewinds the tape after I `close' to get to the next file. If it is really necessary to have user friendly tape behavior, there should be some sort of `half-baked' tape device that implements it. The raw device should do nothing but basic I/O and control operations, and it should have no strange side-effects. I would even be happy to see auto-rewind disappear for the raw device. -- Griff Smith AT&T (Bell Laboratories), Murray Hill Phone: 1-201-582-7736 UUCP: {most AT&T sites}!ulysses!ggs Internet: ggs@ulysses.att.com
jwp@larry.sal.wisc.edu (Jeffrey W Percival) (05/25/89)
In article <6975@cbmvax.UUCP> grr@cbmvax.UUCP (George Robbins) writes: > No, the bug is most likely that the Ultrix driver *is* playing > the sticky EOF game. If the program closed/opened the input file > it would probably work fine, though the man pages imply this is > not necessary. Just a reminder: in my original posting I pointed out that we observe *two* behaviors here: the documented one AND the "sticky EOF" one. Ultrix 3.0 and 2.2 on a VS2000 worked as TFM says, but the Ultrix 3.0 on two MVII's had a sticky EOF. So the phrase "THE Ultrix driver" is not useful. 1. DEC has created both behaviors. Is one acknowledged by them to be an error? 2. If one is an error, which one? 3. Is a fix in the works? Enquiring minds want to know! By the way: I discovered this by trying to read a VMS saveset tape, which is a multi-file thing. The vmsbackup program in the uunet archives assumes the documented behavior of mtio(4). -- Jeff Percival (jwp@larry.sal.wisc.edu)
grr@cbmvax.UUCP (George Robbins) (05/25/89)
In article <190@larry.sal.wisc.edu> jwp@larry.sal.wisc.edu.UUCP (Jeffrey W Percival) writes: > In article <6975@cbmvax.UUCP> grr@cbmvax.UUCP (George Robbins) writes: > > No, the bug is most likely that the Ultrix driver *is* playing > > the sticky EOF game. If the program closed/opened the input file > > it would probably work fine, though the man pages imply this is > > not necessary. > > Just a reminder: in my original posting I pointed out that we observe > *two* behaviors here: the documented one AND the "sticky EOF" one. > Ultrix 3.0 and 2.2 on a VS2000 worked as TFM says, but the Ultrix 3.0 > on two MVII's had a sticky EOF. ... > So the phrase "THE Ultrix driver" is not useful. >1. DEC has created both behaviors. Is one acknowledged by them to be an error? >2. If one is an error, which one? >3. Is a fix in the works? You haven't provided enough information to determine how the problem factors across the different machines and operating systems. Also, without knowing the program, one can't tell if stdio is involved or just read/write and whether one of the systems might have had some "System V environment" conditions set either when the program was compile or run. Is the Ultrix workstation stuff still separate from the regular stuff? If so then it's 3.0 apples and 3.0 oranges anyway. Is the TK50 on one system attached thru a differnt controller (i.e. different driver) on the VS2000 than on the other systems? SCSI vs. Q-bus, no? If you want the program to work reliably across the universe of BSD and System V systems, then you have to do the close/reopen when you get an EOF indication, consequently requiring that you specify the no-rewind magtape name. The sticky EOF is certainly irriating, and in this case seems to disagree with what the documentation says, though it seems the documentation was incomplete with respect to writing EOF's. Right or wrong? Depends on which background you're coming from. I don't know about a fix, have you made an attempt to contact software suport? Inconsistant behaviour is as much as "bug" as right or wrong. -- George Robbins - now working for, uucp: {uunet|pyramid|rutgers}!cbmvax!grr but no way officially representing arpa: cbmvax!grr@uunet.uu.net Commodore, Engineering Department fone: 215-431-9255 (only by moonlite)
keith@sequoia.UUCP (Keith Pyle) (05/26/89)
The original post on this topic in message <187@larry.sal.wisc.edu> by jwp@larry.sal.wisc.edu (Jeffrey W Percival) noted a problem when reading a tape with multiple files under Ultrix 3.0. He found that read(2) returns 0 bytes whenever an EOF is encountered as expected but that it continued to return 0 on subsequent calls. This behaviour was questioned as it was expected that the subsequent reads would return data from the following file of the tape. In article <2721@helios.ee.lbl.gov>, envbvs@epb2.lbl.gov (Brian V. Smith) writes: >According to the manual entry for mtio(4) on Ultrix 3.0 >(quoted w/o permission): > > ... > done with a tape ioctl call. A zero byte count is returned > when a tape mark is read, but another read will fetch the > first record of the next tape file. When a file open for > writing is closed, two end-of-files (EOF) are written. > >Therefore, you could expect to get 0 bytes read for two reads >(because of two possible EOF's on the tape), but after that it should >read the next file. This is essentially the same statement found in the mtio(4) documentation for SunOS 4.0 and DYNIX 3.0.12, and in mt(7) for HPUX. More important to this discussion, this is the observed behaviour on these systems. There is one point Brian did not mention: when a close is issued, two EOFs are indeed written, but the tape will be positioned between them if the device is a no-rewind device. Thus, two successive files may have only one intervening EOF if the device was of the no-rewind type. Clearly, the documented behaviour is for read(2) to return 0 once for each EOF found on the tape. I recently wrote a set of programs for tape duplication that runs on most of the Unix systems here (Sun, Sequent, Ultrix, HP) and Ultrix was the only one that did not function in accordance with the documentation. As pointed out by some others responding to the original post, you can deal with this situation (can you spell B-U-G? I thought that you could! :-)) by (1) use a no-rewind device, (2) close and reopen the device on encountering an EOF. Another technique (kludge) is to perform an MTIOCTOP ioctl call specifying a forward space (MTFSF) of 0 files. The use of this ioctl call in this way may not work for all systems according to the person who described it to me. In either case, the handling of an EOF is substantially slower than on systems which do what they say they will do. -- ----------------------------------------------------------------------------- Keith Pyle UUCP: ...!cs.utexas.edu!execu!keith Execucom Systems Corp., Austin, Texas Internet: execu!keith@cs.utexas.edu keith%execu.uucp@cs.utexas.edu Disclaimer: What?? You actually believed me? -----------------------------------------------------------------------------
jwp@larry.sal.wisc.edu (Jeffrey W Percival) (05/27/89)
In article <6982@cbmvax.UUCP> grr@cbmvax.UUCP (George Robbins) writes: >You haven't provided enough information to determine how the problem factors >across the different machines and operating systems. Also, without knowing >the program, one can't tell if stdio is involved or just read/write and >whether one of the systems might have had some "System V environment" >conditions set either when the program was compile or run. Is the Ultrix >workstation stuff still separate from the regular stuff? If so then it's >3.0 apples and 3.0 oranges anyway. Is the TK50 on one system attached >thru a differnt controller (i.e. different driver) on the VS2000 than >on the other systems? SCSI vs. Q-bus, no? I should have also mentioned: the sticky EOF occurs on the MVII TK50's, but *not* on the 6250-BPI reel-to-reel drives on the very same machines. So it's not even a workstation/microvax difference, but a difference between units on the same machine! Nice portability, eh? "Sure we can restore your VMS backup tape. Oh wait, not on the TK50... do you happen to have your backup on a regular tape?" -- Jeff Percival (jwp@larry.sal.wisc.edu)
grr@cbmvax.UUCP (George Robbins) (05/27/89)
In article <191@larry.sal.wisc.edu> jwp@larry.sal.wisc.edu.UUCP (Jeffrey W Percival) writes: > > I should have also mentioned: the sticky EOF occurs on the MVII TK50's, > but *not* on the 6250-BPI reel-to-reel drives on the very same machines. > > So it's not even a workstation/microvax difference, but a difference > between units on the same machine! Nice portability, eh? Completely different controller/driver no? -- George Robbins - now working for, uucp: {uunet|pyramid|rutgers}!cbmvax!grr but no way officially representing arpa: cbmvax!grr@uunet.uu.net Commodore, Engineering Department fone: 215-431-9255 (only by moonlite)
envbvs@epb2.lbl.gov (Brian V. Smith) (05/27/89)
In article <6999@cbmvax.UUCP> grr@cbmvax.UUCP (George Robbins) writes: >In article <191@larry.sal.wisc.edu> jwp@larry.sal.wisc.edu.UUCP (Jeffrey W Percival) writes: >> >> I should have also mentioned: the sticky EOF occurs on the MVII TK50's, >> but *not* on the 6250-BPI reel-to-reel drives on the very same machines. >> >> So it's not even a workstation/microvax difference, but a difference >> between units on the same machine! Nice portability, eh? > >Completely different controller/driver no? But the same manual entry/system call. If there is supposed to be a difference in the different controllers/drivers/tape drives then it should be stated. It simply looks like a bug. _____________________________________ Brian V. Smith (bvsmith@lbl.gov) Lawrence Berkeley Laboratory We don't need no signatures!
grr@cbmvax.UUCP (George Robbins) (05/27/89)
In article <2738@helios.ee.lbl.gov> envbvs@epb2 (Brian V. Smith) writes: > In article <6999@cbmvax.UUCP> grr@cbmvax.UUCP (George Robbins) writes: > > > >Completely different controller/driver no? > > But the same manual entry/system call. If there is supposed to be a > difference in the different controllers/drivers/tape drives then it should > be stated. It simply looks like a bug. Yes, of course it's a bug. Inconsistancies are bugs. You now have an explanation, a work-around and a valid problem description. If you consider it a bug, the next step is to file an "SPR" with DEC or otherwise follow up the problem through DEC software support channels. -- George Robbins - now working for, uucp: {uunet|pyramid|rutgers}!cbmvax!grr but no way officially representing arpa: cbmvax!grr@uunet.uu.net Commodore, Engineering Department fone: 215-431-9255 (only by moonlite)