cmf851@anu.oz.au (Albert Langer) (06/15/91)
1. Now that DAT is available with fairly high speed random access, is there any software available to actually USE this? (Or for Exabyte which is only slightly slower, or indeed any older software to use seeks on a raw character device?) 2. Do the drivers normally provided on Unix systems (e.g. ISC 386/ix 2.2) allow for random seeks or would new drivers as well as application software be needed? (If so, are any such drivers available?). 3. Main application I have in mind is for offline archiving of large volumes of files to be available for mail requests. Tape could be written once, or with supplements added in batches after filemarks. Requests get batched or handled individually but either way spin the tape at high speed between locations where actual file data has to be extracted rather than scanning the data at ordinary speed (and tying up the CPU with scanning). 4. Is it practical to do this with filemarks instead of fully random seeking? (i.e. THOUSANDS of filemarks?) Seems undesirable to me as there would be a stop-start after each logical file was written, and also I imagine that scanning for a filemark is slower than scanning to a specified block count? Is that right? 5. A nice way to do it might be something that makes a directory showing the byte offset to each file as files are being streamed out through cpio or tar. Then to read you just sort the requests. Normal recovery through cpio or tar would be available, and faster direct access as well. 6. I understand that dataDAT allows random writing as well, but only DDS is popular which does not allow random writing. Is there any way with DDS to leave huge gaps and waste tape in order to allow a primitive form of random writing between file marks or pseudo end of media marks? Seems to me that should be PHYSICALLY possible regardless of the format, but I can't figure out how to do it. Alternatively, or perhaps it is the same question, is there some way to use numerous "partitions" with DDS instead of just two? 7. If such software isn't yet available how about some DAT vendors (or even SCSI adaptor vendors) developing some? Seems to me it would widen the DAT (and/or Exabyte) market beyond backup streaming (and incidentally provide for fast recovery of files from backups). Once a crude prototype was freely available from a DAT vendor, others would develop new and better applications. Would also widen appeal of SCSI (less substantially). 8. If this isn't the best newsgroup to ask in, which should I try? P.S. While I am at it, an entirely separate issue, which also happens to relate to Random Access DAT. I have read of VERY expensive "juke boxes" for tapes as well as for WORMs and CD ROM. 9. Is there any reason a crude version of such a juke box could not be constructed quite cheaply? (e.g. a rather slow one, using toy components?) It doesn't sound all that complex to me for a toy robot to pick tapes from a rack and drop them into a slot and vice versa. Anybody working on this? 10. Are audio juke boxes for DAT available or planned? If so is anybody going to convert them for computer use? (Just take the the electronics from a computer DAT and use the mechanical parts from an audio juke box). Hmm, that reminds me of another issue. 11. The Wangtek DAT I am using has a JVC drive and an entirely separate Wangtek SCSI controller which is merely screwed together to make it look like a single full height drive. Would the JVC drive be an ordinary audio DAT drive and would there be some simple way one could feed output from an ADC to it or input to a DAC from it (at least standalone audio DACs are readily available for use with high end CD players)? 12. Likewise, could one just add a SCSI controller board to a consumer audio DAT drive to get a computer DAT (as I believe Wangtek may have done)? I guess consumer audio DAT will soon be dramatically cheaper than before and it might be worth looking into. (Or are computer DAT drives expected to quickly follow consumer audio prices down?) Seems to me that with CD ROMS there may be some technical differences in seek-time requirements etc to explain a continuing large price differential, but for DAT the ONLY technical differences should be in controller electronics and price differentials may be purely a marketing decision that could be challenged by some small PCB producer buying audio drives and adding their PCBs. -- Opinions disclaimed (Authoritative answer from opinion server) Header reply address wrong. Use cmf851@csc2.anu.edu.au
neese@adaptx1.UUCP (06/16/91)
>/* ---------- "Random Access DAT" ---------- */ > >1. Now that DAT is available with fairly high speed random access, is >there any software available to actually USE this? (Or for Exabyte >which is only slightly slower, or indeed any older software to use >seeks on a raw character device?) BEWARE! The method youare refering to is called QFA (Quick File Access) and is implemented very differently from manufacturer to manufacturer. This was due to the lack of a defination in the SCSI spec when these devices were designed. The spec now contains the new commands for this, but it will be a while before everyone implments them. >2. Do the drivers normally provided on Unix systems (e.g. ISC 386/ix 2.2) >allow for random seeks or would new drivers as well as application >software be needed? (If so, are any such drivers available?). No, for the reason above. I wouldn't want to write a tape specific driver that would not work with a limited number of drives. I am sure none of the suppliers of UNIX's would either. >STUFF DELETED< Roy Neese Adaptec Senior SCSI Applications Engineer UUCP @ neese@adaptex uunet!cs.utexas.edu!utacfd!merch!adaptex!neese
kev@hpcpbla.HP.COM (Kevin Jones) (06/17/91)
> 1. Now that DAT is available with fairly high speed random access, is > there any software available to actually USE this? (Or for Exabyte > which is only slightly slower, or indeed any older software to use > seeks on a raw character device?) > > 2. Do the drivers normally provided on Unix systems (e.g. ISC 386/ix 2.2) > allow for random seeks or would new drivers as well as application > software be needed? (If so, are any such drivers available?). > The DDS format has counts of: #blocks, #filemarks, #setmarks written in each physical "frame" of data placed on tape. This enables a drive in "fast search mode" (typically 200x read/write speed) to skip directly to any specified block, filemark or setmark. (PS. Setmarks are "super filemarks" and tend to be unique to DDS. You don't have to use them if you don't want to. Most people don't know what they are, and are probably no worse off as a result :-) A drive will automatically use "fast search" mode when told to skip blocks/filemarks/setmarks (SCSI "SPACE" command), or to position to an absolute location on tape (SCSI "LOCATE" command). Most drivers out there should be able to handle block/mark skipping. The problem with fast searching is knowing where to go, thus either you have a smart backup application which keeps track of the order in which it puts files onto tape, or you have a dumb backup application and keep track of the order of files yourself. You could then use something like the 'mt' command to position the tape in front of a file you want to read. > 3. Main application I have in mind is for offline archiving of large > volumes of files to be available for mail requests. Tape could be > written once, or with supplements added in batches after filemarks. > Requests get batched or handled individually but either way > spin the tape at high speed between locations where actual file > data has to be extracted rather than scanning the data at ordinary > speed (and tying up the CPU with scanning). > > 4. Is it practical to do this with filemarks instead of fully > random seeking? (i.e. THOUSANDS of filemarks?) Seems undesirable > to me as there would be a stop-start after each logical file was With DDS, a filemark uses 4 bytes of space, and a drive can "write" one in approx 5 milli-seconds. (this assumes you write the filemark in "immediate report" mode to prevent a time/space wasting flush of the drive's data buffe to tape). Thus there is not neccesarily a stop-start after a filemark is written. (NB. This is not true for 8mm, where filemarks waste a lot of space and time). > written, and also I imagine that scanning for a filemark is slower > than scanning to a specified block count? Is that right? Makes no difference in DDS. > 5. A nice way to do it might be something that makes a directory > showing the byte offset to each file as files are being streamed > out through cpio or tar. Then to read you just sort the requests. > Normal recovery through cpio or tar would be available, > and faster direct access as well. You mean: have a number of cpio/tar archives on tape interspersed with filemarks, and skip to the archive you're interested in ??? I guess this might work, provided cpio/tar doesn't cause a rewind when it "opens" the device for read/write. (I don't know much about this sort of stuff, so I skip to the next question ...........) > 6. I understand that dataDAT allows random writing as well, but only > DDS is popular which does not allow random writing. Is there any way with DDS > to leave huge gaps and waste tape in order to allow a primitive form > of random writing between file marks or pseudo end of media marks? > Seems to me that should be PHYSICALLY possible regardless of the > format, but I can't figure out how to do it. Alternatively, or perhaps > it is the same question, is there some way to use numerous "partitions" > with DDS instead of just two? No and No :-( The way that most helical scan tape formats work means that you cannot equate a specific size of data to a specific length of tape, eg. "10 Meggabytes of data = 1 meter of tape" doesn't work. Mechanisms that undermine this sort of algebra: 1. Auto-rewrites of bad frames. To ensure data integrity, data is read by a "trailing read head" soon after it is written. If the read shows problems then the data is repeated (Rewritten). Rewrites occur until the data is known to be written OK. Rewrites can thus result in "more tape for the same amount of data". 2. Flushes of partial groups. A "group" in DDS is a 126Kbytes of data which can contain multiple blocks/filemarks packed and written in a "single indivisible operation". Groups are built in a drive's buffer and written once they are "full" of data/marks. For data integrity purposes it is possible to force the writing of a group before it is full to "ensure the stuff is on tape". It still uses 126Kbytes-worth of tape however. Reserving "spaces" into which you can fit "known amounts of data" therefore becomes very difficult. Leaving "huge gaps" can pose great difficulty, especially to the fast-search algorithms. The muck up the tape format badly. This is, in effect, what the DATA/DAT format did (along with more exotic things like spares tables on tape etc....). The DDS format thus restricts you to 2 partitions and you will find drives will not allow you to read "beyond the most recent end of data" (ie. you will not be able to get past the point at which you last stopped writing in order to read "old" data). > 7. If such software isn't yet available how about some DAT vendors > (or even SCSI adaptor vendors) developing some? Seems to me it would > widen the DAT (and/or Exabyte) market beyond backup streaming > (and incidentally provide for fast recovery of files from backups). > Once a crude prototype was freely available from a DAT vendor, others > would develop new and better applications. > Would also widen appeal of SCSI (less substantially). > Its a matter of convincing DAT vendors (people who make iron) that developing a bit of software for a change is in their own best interests. Why hasn't this happened ??? I don't know. > 8. If this isn't the best newsgroup to ask in, which should I try? > > P.S. While I am at it, an entirely separate issue, which also > happens to relate to Random Access DAT. > > I have read of VERY expensive "juke boxes" for tapes as well as for > WORMs and CD ROM. > > 9. Is there any reason a crude version of such a juke box could not > be constructed quite cheaply? (e.g. a rather slow one, using toy > components?) It doesn't sound all that complex to me for a toy robot > to pick tapes from a rack and drop them into a slot and vice versa. > Anybody working on this? > > 10. Are audio juke boxes for DAT available or planned? If so is > anybody going to convert them for computer use? (Just take the > the electronics from a computer DAT and use the mechanical parts > from an audio juke box). I havn't seen any coming from Japan. There are some "stacker" devices for DAT on the market. Typical capacity approx 12 cassettes. > > Hmm, that reminds me of another issue. > > 11. The Wangtek DAT I am using has a JVC drive and an entirely > separate Wangtek SCSI controller which is merely screwed together > to make it look like a single full height drive. Would the JVC > drive be an ordinary audio DAT drive and would there be some > simple way one could feed output from an ADC to it or input to > a DAC from it (at least standalone audio DACs are readily > available for use with high end CD players)? A lot of DAT drives use mechanisms developed for audio. The similarity ends at this point. Most of the electronics (doens to the read/write amplifiers in some cases) will have undergone radical change to support computer data. Bolting on ADC's/DAC's becomes a non-trivial operation. > > 12. Likewise, could one just add a SCSI controller board to > a consumer audio DAT drive to get a computer DAT (as I > believe Wangtek may have done)? I guess consumer audio DAT "SCSI Controllers" for DDS-DAT drives tend to be fairly complicated. In order to generate DDS format, handle the SCSI interface, maintain data integrity etc.... they usually end up with a component list that goes something like: 1 16 bit processor, A couple of 8 bit microcontrollers, 0.5 - 1 Meg of DRAM, 128K-256Kbytes of PROM/ROM/FLASH memory "containing" 30,000 lines-worth of source code (that count doesn't include comments). 1 SCSI protocol controller chip, 1 or more DAT formatting chips ........ Assuming that you could get a consumer DAT drive, a controller board "off the shelf" that implements SCSI/DDS, plug them together, perform a LOT of data integrity and reliability testing and modify the design as you go so that you end up with a quality product......... In other words, "designing and adding a SCSI controller" may be all that WangDAT or any other DAT vendor has done, but that's where the difficulty is, and where a lot of the development and product costs are incurred. That's what you're paying any perceived "extra" for. > will soon be dramatically cheaper than before and it > might be worth looking into. (Or are computer DAT drives > expected to quickly follow consumer audio prices down?) > Seems to me that with CD ROMS there may be some technical > differences in seek-time requirements etc to explain a > continuing large price differential, but for DAT the > ONLY technical differences should be in controller > electronics and price differentials may be purely a > marketing decision that could be challenged by some > small PCB producer buying audio drives and adding their > PCBs. Price "erosion" (I think that might be a marketing term) will occur. I wouldn't expect DAT to become as cheap/cheaper than CD rom. There's an awfull lot more mechanism and electronics in a DAT drive than in a CD ROM. > > > -- > Opinions disclaimed (Authoritative answer from opinion server) > Header reply address wrong. Use cmf851@csc2.anu.edu.au ----------------------------------------------------------------- Kevin Jones. | Hewlett Packard Ltd, | Computer Peripherals Bristol, kev%hpcpbla@hplb.hpl.hp.com | Filton Road, | Stoke Gifford, Tel: 011 44 272 799910 (ext 22351) | Bristol. BS12 6QZ. | ENGLAND. ----------------------------------------------------------------- This response does not represent the official position of, or statement by, the Hewlett-Packard Company. The above data is provided for informational purposes only. It is supplied without warranty of any kind.
cmf851@anu.oz.au (Albert Langer) (06/18/91)
Thanks for the detailed and helpful answers to my questions Kevin. Now here's some follow ups: In article <9850028@hpcpbla.HP.COM> kev@hpcpbla.HP.COM (Kevin Jones) writes: >> 1. Now that DAT is available with fairly high speed random access, is >> there any software available to actually USE this? [...] >> >> 2. Do the drivers normally provided on Unix systems (e.g. ISC 386/ix 2.2) >> allow for random seeks or would new drivers as well as application >> software be needed? (If so, are any such drivers available?). > [detailed explanation of DDS and SCSI operation omitted] >A drive will automatically use "fast search" mode when told to >skip blocks/filemarks/setmarks (SCSI "SPACE" command), or to >position to an absolute location on tape (SCSI "LOCATE" command). >Most drivers out there should be able to handle block/mark skipping. I take it that, and the detail I have omitted about how it works, means the answer to my Q2 is "Yes" i.e. a Unix device driver normally used just for backup streaming can be expected to properly implement commands to seek to any given location. One would not expect such a driver to be implemented like a tty or lineprinter that doesn't know what "seek" means (even though that would be sufficient for backup streaming). Please confirm. >The problem with fast searching is knowing where to go, thus either >you have a smart backup application which keeps track of the order in >which it puts files onto tape, or you have a dumb backup application >and keep track of the order of files yourself. You could then use >something like the 'mt' command to position the tape in front of >a file you want to read. I take it, from the lack of any other answer to my Q1, that you are unaware of any software that uses the DDS facilities available, although you are familiar with the technical details of what can be done. This seems strange. What's the point of providing this stuff if nobody uses it? This also relates to Q7 - you seem to know your stuff and work for HP on DDS DAT - why don't they produce some sample random access software instead of just selling it as a backup streamer? >> 3. Main application I have in mind is for offline archiving of large >> volumes of files to be available for mail requests. [...] >> >> 4. Is it practical to do this with filemarks instead of fully >> random seeking? (i.e. THOUSANDS of filemarks?) Seems undesirable >> to me as there would be a stop-start after each logical file was > >With DDS, a filemark uses 4 bytes of space, and a drive can "write" >one in approx 5 milli-seconds. (this assumes you write the filemark >in "immediate report" mode to prevent a time/space wasting flush of >the drive's data buffe to tape). >Thus there is not neccesarily a stop-start after a filemark is >written. I take it from this, and the reference to mt in your answer to my Q1 and Q2, that your answer to my Q4 is "Yes". i.e. that one could put literally thousands of files separated by filemarks on tape in the normal way using successive outputs to a non-rewind device driver and just use "mt" for skipping to the appropriate file for retrieval and still get the full benefits of fast tape movement without having to use any "seek" commands at block level. This implies that the application described in Q3 may as well use that simple filemark method (Q4) and there is no point bothering about seeks to specific block numbers for it. Please confirm. >> 5. A nice way to do it might be something that makes a directory >> showing the byte offset to each file as files are being streamed >> out through cpio or tar. Then to read you just sort the requests. >> Normal recovery through cpio or tar would be available, >> and faster direct access as well. > >You mean: have a number of cpio/tar archives on tape interspersed with >filemarks, and skip to the archive you're interested in ??? >I guess this might work, provided cpio/tar doesn't cause a rewind >when it "opens" the device for read/write. (I don't know much about >this sort of stuff, so I skip to the next question ...........) The way I understand it, cpio and tar just write to stdout. When the cpio or tar process finishes and closes stdout it is not cpio or tar but the device driver that causes a rewind unless it is a non-rewind device driver - usually both are available (using a 1 bit difference in minor device numbers). Thus storing a series of cpio or tar files separated by filemarks should be absolutely no different in principle from just storing any other files (e.g. using cat or cp). What I meant was using a SINGLE tape file produced by cpio or tar to hold thousands of individual files and EITHER recover individual files or sets of them using cpio or tar (which implies slowly reading through the whole tape file), OR seek directly to the tape blocks that hold the individual file required. The latter alternative would require building some sort of index to the cpio or tar file as it is being produced. Seems to me that since you say there is no performance penalty for using filemarks I may as well just use the "mt" approach. [Q6. Thanks for finally convincing me there is no way around the limitation of two append only partitions by providing full details.] >> [Q9 and Q10 re juke boxes] >I havn't seen any coming from Japan. There are some "stacker" devices >for DAT on the market. Typical capacity approx 12 cassettes. I take it the distinction between "stacker" and "juke box" is that with a "stacker" you cannot randomly select the next cassette but only get them one after the other in the order they were stacked (and you would have to reload manually after getting to number 12). Please confirm. If so, that would still be very useful for some computer purposes, though less useful than a fully random access juke box. It could even be used for the file archive application responding to mail requests that I mentioned in Q3 (as well as for ordinary very large backups etc.) There would just be a batch cycle proceeding through all 12 tapes to fill any mail requests (though having to restack after each batch is unfortunate, it could be done only once or twice per day). Can you please provide any details you have available on who is marketing "stackers" and at what price? Also, are they audio or computer devices and if audio, would there be any difficulty using the stacker mechanism with a computer DAT drive? >> 11. The Wangtek DAT I am using has a JVC drive and an entirely >> separate Wangtek SCSI controller which is merely screwed together >> to make it look like a single full height drive. Would the JVC >> drive be an ordinary audio DAT drive and would there be some >> simple way one could feed output from an ADC to it or input to >> a DAC from it (at least standalone audio DACs are readily >> available for use with high end CD players)? > >A lot of DAT drives use mechanisms developed for audio. The similarity >ends at this point. Most of the electronics (doens to the read/write >amplifiers in some cases) will have undergone radical change to >support computer data. Bolting on ADC's/DAC's becomes a non-trivial >operation. I don't understand this answer. My understanding was that the audio DAT format is fully digital and the DDS format is just a particular form of it that defines block numbers etc. If so, why should only the mechanism be the same and not also the digital electronics (including read/write amplifiers)? If there is a digital signal being recorded and played back, why should accessing that for audio be any more difficult than recording an audio CD (digitally) on a DAT? I can understand there might be extra stuff for fast searches etc but surely more than the mechanism is the same. >> 12. Likewise, could one just add a SCSI controller board to >> a consumer audio DAT drive to get a computer DAT (as I >> believe Wangtek may have done)? [...] >In other words, "designing and adding a SCSI controller" may be >all that WangDAT or any other DAT vendor has done, but that's where >the difficulty is, and where a lot of the development and product >costs are incurred. That's what you're paying any perceived "extra" >for. I just saw a note about SCSI DATs available for USD $1500, c.f. $2500 when the one I'm using was bought. I guess the current prices are closer to reflecting the design costs you describe. >Price "erosion" (I think that might be a marketing term) will occur. >I wouldn't expect DAT to become as cheap/cheaper than CD rom. >There's an awfull lot more mechanism and electronics in a DAT >drive than in a CD ROM. Hmm, well CD is becoming VERY cheap and I can see how the mechanism differences alone will always keep DAT more expensive. But electronics is becoming so cheap that I think one can hope for SCSI controllers for audio DAT drives that don't add much to the cost (with the code and integration development costs amortized over many thousands of users, what's 0.5 MB of RAM and a few VLSI chips these days :-) If audio DAT takes off as a mass market competitor to ordinary audio cassettes (and CDs) then prices would have to end up comparable to CDs even though a bit greater (and less than VCRs which have similar mechanisms and are now getting to be fairly cheap). It will be VERY interesting when a typical western style home with hi fi equipment has 1.2 Gb or 2.4 Gb of tape storage sitting there waiting to be used, along with the 550 MB CD audio/CD ROM and the workstation quality HDTV (and 2 x 64 kbs circuit switched + 1 x 16 kbs packet switched ISDN line) - adding a keyboard and CPU box to get a pretty powerful computing environment won't be such a big deal :-) Anyway, thanks again for the helpful answers. -- Opinions disclaimed (Authoritative answer from opinion server) Header reply address wrong. Use cmf851@csc2.anu.edu.au
kev@hpcpbla.HP.COM (Kevin Jones) (06/18/91)
All conclusions drawn in your response to my posting are correct. Some specific points: > that you are unaware of any software that uses the > DDS facilities available, (ie. w.r.t fast search etc....) On PC's, applications like SYTOS allow selective store/recovery. On workstations, I am not aware of any good backup (lets say "data archiving") applications (that's not saying none exist). > with a "stacker" you cannot randomly select the next cassette Correct. > Can you please provide any details you have available on who > is marketing "stackers" and at what price? I'm not sure about what stackers are currently on the market. I've seen lots of "demo" type stuff that "looses" the occasional cassette. Many stackers have SCSI and/or other "computer" interfaces and claim to be able to do random access. Unless you've got a device driver for such a machine then you still have no random access ability, ie. you've got a stacker that might be a juke box if someone wrote a driver for it. > I don't understand this answer. My understanding was that the audio DAT > format is fully digital and the DDS format is just a particular form of Correct. > it that defines block numbers etc. If so, why should only the mechanism DDS is "built on top" of the audio format. It defines a further level of blocking (23 audio "frames" form a single 126Kbyte DDS "group"). A DDS "group" contains blocks of user data (multiple blocks can be packed into a group, a block can span one or more groups, no space is wasted irrespective of block size) and a "block access table" that is, in effect, an INDEX for locations of blocks in the group and their sizes. The "block access table" also contains information on the prescence and quantity of filemarks/setmarks. The drives embedded software is responsible for the creation of block access tables on writes, and their use in "unpacking" groups on reads. > be the same and not also the digital electronics (including read/write > amplifiers)? If there is a digital signal being recorded and played > back, why should accessing that for audio be any more difficult than >recording an audio CD (digitally) on a DAT? I can understand there > might be extra stuff for fast searches etc but surely more than the > mechanism is the same. A lot of the DAT digital electronics can be "re used" in a computer product. Specifically, circuitry to do the 1st and 2nd levels of error correction, and servo circuitry. There are many functions not performed in audio dat though: Read After Write, Buffering of Data (typically 0.5 Mbytes), 3rd level error correction, SCSI (not an insignificant part!!!), DDS format (built on top of DAT audio format), extensive quantities of performance/data-integrity oriented embedded software. These functions contribute more to the cost of a product than the "audio" functions (including the mechanism). ----------------------------------------------------------------- Kevin Jones. | Hewlett Packard Ltd, | Computer Peripherals Bristol, kev%hpcpbla@hplb.hpl.hp.com | Filton Road, | Stoke Gifford, Tel: 011 44 272 799910 (ext 22351) | Bristol. BS12 6QZ. | ENGLAND. ----------------------------------------------------------------- This response does not represent the official position of, or statement by, the Hewlett-Packard Company. The above data is provided for informational purposes only. It is supplied without warranty of any kind.
stevej@hpcpbla.HP.COM (Steve Jerman) (06/18/91)
Hi Albert, I'll try and answer some of these applications questions as I'm more involved with the applications side than Kev. Bare in mind that my knowledge is HP focused. >I take it that, and the detail I have omitted about how it works, means >the answer to my Q2 is "Yes" i.e. a Unix device driver normally used >just for backup streaming can be expected to properly implement >commands to seek to any given location. One would not expect such a driver >to be implemented like a tty or lineprinter that doesn't know what "seek" >means (even though that would be sufficient for backup streaming). Please >confirm. Yes, most U**x sequential tape drivers have a 'mt' command and ioctl control for the drive. This means that space commands can be used from applications. Almost all the Unix systems I have knowledge of have this sort of capability although some systems implement it in different ways. >I take it, from the lack of any other answer to my Q1, that you are >unaware of any software that uses the DDS facilities available, >although you are familiar with the technical details of what can >be done. This seems strange. What's the point of providing this stuff >if nobody uses it? This also relates to Q7 - you seem to know your >stuff and work for HP on DDS DAT - why don't they produce some >sample random access software instead of just selling it as a >backup streamer? There are third party solutions for HP and other kit that provide fast file recovery facilities. As far as HP products are concerned. The following is available (not we are working on more than this): o Fbackup (supplied with HPUX) supports f.f.r from HP-UX 8.0 o Omniback (separate network backup product) on Domain and very soon on HP-UX. This is not an exhaustive list, just the main workstation products. Also see appended utility, should work on most U**x systems and makes small archives easily accessible. >I take it from this, and the reference to mt in your answer to my Q1 >and Q2, that your answer to my Q4 is "Yes". i.e. that one could >put literally thousands of files separated by filemarks on tape >in the normal way using successive outputs to a non-rewind device >driver and just use "mt" for skipping to the appropriate file for >retrieval and still get the full benefits of fast tape movement >without having to use any "seek" commands at block level. This >implies that the application described in Q3 may as well use >that simple filemark method (Q4) and there is no point bothering >about seeks to specific block numbers for it. Please confirm. Yup - see end of message. >The way I understand it, cpio and tar just write to stdout. When the >cpio or tar process finishes and closes stdout it is not cpio or tar but >the device driver that causes a rewind unless it is a non-rewind device driver >- usually both are available (using a 1 bit difference in minor >device numbers). Thus storing a series of cpio or tar files separated >by filemarks should be absolutely no different in principle from just >storing any other files (e.g. using cat or cp). >What I meant was using a SINGLE tape file produced by cpio or tar to >hold thousands of individual files and EITHER recover individual files >or sets of them using cpio or tar (which implies slowly reading through >the whole tape file), OR seek directly to the tape blocks that hold >the individual file required. The latter alternative would require >building some sort of index to the cpio or tar file as it is being >produced. Seems to me that since you say there is no performance >penalty for using filemarks I may as well just use the "mt" approach. see end again. >I take it the distinction between "stacker" and "juke box" is that >with a "stacker" you cannot randomly select the next cassette but >only get them one after the other in the order they were stacked >(and you would have to reload manually after getting to number 12). >Please confirm. correct - also implies cheaper and simpler. >If so, that would still be very useful for some computer purposes, >though less useful than a fully random access juke box. It could >even be used for the file archive application responding to mail >requests that I mentioned in Q3 (as well as for ordinary very >large backups etc.) There would just be a batch cycle proceeding >through all 12 tapes to fill any mail requests (though having >to restack after each batch is unfortunate, it could be done >only once or twice per day). Can you please provide any details >you have available on who is marketing "stackers" and >at what price? Also, are they audio or computer devices and if audio, >would there be any difficulty using the stacker mechanism with a >computer DAT drive? Sorry I don't have that information. >I don't understand this answer. My understanding was that the audio DAT >format is fully digital and the DDS format is just a particular form of >it that defines block numbers etc. If so, why should only the mechanism >be the same and not also the digital electronics (including read/write >amplifiers)? If there is a digital signal being recorded and played >back, why should accessing that for audio be any more difficult than >recording an audio CD (digitally) on a DAT? I can understand there >might be extra stuff for fast searches etc but surely more than the >mechanism is the same. The DDS format is an extra logical layer on top of the audio format. (like OSI layers). The mechanism could be the same between Audio and digital data but complication would be added to separate of the audio and digital data and also to route that data to a speaker jack - no impossible though. >I just saw a note about SCSI DATs available for USD $1500, c.f. >$2500 when the one I'm using was bought. I guess the current >prices are closer to reflecting the design costs you describe. Price erosion is a wonderful thing ... Hope this helps Steve Jerman Hewlett Packard Computer Peripherals Bristol Address: Email: Filton Road stevej@hpcpbla.bri.hp.com Stoke Gifford Bristol BS12 6QZ UK Phone: Fax: (44) 272 799910 x22424 (44) 272 236194 This response does not represent the official position of, or statement by, the Hewlett-Packard Company. The above data is provided for informational purposes only. It is supplied without warranty of any kind.
stevej@hpcpbla.HP.COM (Steve Jerman) (06/18/91)
Oops I forgot the script ... #!/bin/ksh # Demo to show use on mt and cpio on DDS drive. set -x TAPE=/dev/foxy_n # (major num 54, minor num 0x0e0503) DIR1=/bin DIR2=/etc DIR3=/usr/lib #backup /bin #rewind tape mt -t $TAPE rew #backup directories for dir in $DIR1 $DIR2 $DIR3 do find $dir -print | cpio -oB > $TAPE mt -t $TAPE eof mt -t $TAPE bsf 1 done #no rewind mt -t $TAPE rew # space out 2 filemarks - to the lib directory mt -t $TAPE fsf 2 # and read the directory cpio -iBt < $TAPE
kev@hpcpbla.HP.COM (Kevin Jones) (06/19/91)
> BEWARE! The method youare refering to is called QFA (Quick File Access) > and is implemented very differently from manufacturer to manufacturer. This > was due to the lack of a defination in the SCSI spec when these devices > were designed. The spec now contains the new commands for this, but it will > be a while before everyone implments them. There are no problems with DDS "Random Access". * Not with the tape format * Not with the search algorithms * Not with the SCSI interface spec * Not with any host driver that has correctly implemented the SCSI "Space" command. There are now 3 methods of achieving "Random Access" via SCSI commands that I'm aware of: 1. Using the SCSI "SPACE" command. This can instruct the drive to space FILEMARKS, SETMARKS, RECORDS and Sequential Filemarks/Setmarks (a "sequential" mark is a group of marks, eg. "Space 2 sequential Filemarks" = Space to the next pair of filemarks). All spacing is done relative to the current tape position. This is the command which would be used in response to an 'mt' on UNIX. Most host drivers implement it. Its trivial. There is no ambiguity or misunderstandings in the SCSI spec about this command. DDS will "fast search" in response to a SPACE. 8mm products will NOT since they cannot space to a "logical position". 2. Using the SCSI "READ POSITION" and "LOCATE" commands. I think these may be the commands Roy was warning you about. A "Read Position" causes the drive to return the "current tape address". You can subsequently send this address back to the drive to get it to position to the point where you issued the LOCATE. This seems popular with PC's where a backup application can do a "LOCATE" before writing a file. It can then create a tape index containing filenames and "Locate positions". There are some problems with using LOCATE/READ-POSITION: 1. Many drivers don't implement these commands. This is no big deal for DDS/UNIX since its better to use "SPACE" anyhow. 2. Until recently, the "current tape address" was a bit vendor specific. ie. you could be positioned before file "x" and LOCATE shows you are at tape address 1234, whereas in a different manufacturers drive, at the same tape position, LOCATE returns tape address 1235. Not all that important if you stick with the same drive. If however you do a LOCATE on one then feed that address into a READ POSITION on another then you end up in the wrong place. This problem has now been addressed by the ANSI comittee. Prior to that, a common understanding had developed amongst DDS manufacturers about the definition of "current tape address" such that interchange problems between different vendors will not be a problem. 8mm products cannot utilise the LOCATE/READ-POSITION commands to fast search. See below....... 3. The third method of "Random Access" is very much vendor specific. My information on this "Random Access" method is a bit sketchy, but it goes something like: A vendor unique SCSI command allows you to read the current "track address" from the drive. A second vendor unique SCSI command causes the drive to position to a specified track address. This all sounds like the LOCATE and READ POSITION commands. The main difference is that the address returned is the physical tape address. With LOCATE/READ-POSITION, logical addressing means that "incrementing by 1" causes you to move up 1 block or 1 mark (it could be argued that "READ-POSITION" is now redundant since block addressing was standardised, and that a smart enough driver could "mirror" what the drive would return in response to a READ-POSITION command), with these vendor unique commands, there is no such correlation. Since these commands are only a couple of months old, I would expect to see them supported only on drivers that have been written for 8mm products. They are not as useful as the LOCATE/READ-POSITION commands, and NOWHERE NEAR AS USEFUL as the SPACE command. ----------------------------------------------------------------- Kevin Jones. | Hewlett Packard Ltd, | Computer Peripherals Bristol, kev%hpcpbla@hplb.hpl.hp.com | Filton Road, | Stoke Gifford, Tel: 011 44 272 799910 (ext 22351) | Bristol. BS12 6QZ. | ENGLAND. ----------------------------------------------------------------- This response does not represent the official position of, or statement by, the Hewlett-Packard Company. The above data is provided for informational purposes only. It is supplied without warranty of any kind.
neese@adaptx1.UUCP (06/20/91)
>> BEWARE! The method youare refering to is called QFA (Quick File Access) >> and is implemented very differently from manufacturer to manufacturer. This >> was due to the lack of a defination in the SCSI spec when these devices >> were designed. The spec now contains the new commands for this, but it will >> be a while before everyone implments them. > >There are no problems with DDS "Random Access". > * Not with the tape format > * Not with the search algorithms > * Not with the SCSI interface spec > * Not with any host driver that has correctly implemented the > SCSI "Space" command. > >There are now 3 methods of achieving "Random Access" via SCSI commands >that I'm aware of: > >1. Using the SCSI "SPACE" command. This can instruct the drive to > space FILEMARKS, SETMARKS, RECORDS and Sequential Filemarks/Setmarks > (a "sequential" mark is a group of marks, eg. "Space 2 sequential > Filemarks" = Space to the next pair of filemarks). > All spacing is done relative to the current tape position. > This is the command which would be used in response to an 'mt' on > UNIX. Most host drivers implement it. Its trivial. There is no > ambiguity or misunderstandings in the SCSI spec about this command. > > DDS will "fast search" in response to a SPACE. > 8mm products will NOT since they cannot space to a "logical position". Even this has caveates. As long as you stick to using blocks or filemarks as the refernce to space by, you will be okay. Not all tape drives support seqiential filemarks, end-of-data, setmarks, or sequential setmarks. These are optional in the SCSI-2 spec and support will vary from vendor to vendor. >2. Using the SCSI "READ POSITION" and "LOCATE" commands. > I think these may be the commands Roy was warning you about. > A "Read Position" causes the drive to return the "current tape > address". You can subsequently send this address back to the > drive to get it to position to the point where you issued the > LOCATE. This seems popular with PC's where a backup application > can do a "LOCATE" before writing a file. It can then create > a tape index containing filenames and "Locate positions". > > There are some problems with using LOCATE/READ-POSITION: > 1. Many drivers don't implement these commands. > This is no big deal for DDS/UNIX since its better to use > "SPACE" anyhow. > 2. Until recently, the "current tape address" was a bit vendor > specific. ie. you could be positioned before file "x" and > LOCATE shows you are at tape address 1234, whereas in a > different manufacturers drive, at the same tape position, > LOCATE returns tape address 1235. Not all that important if > you stick with the same drive. If however you do a LOCATE > on one then feed that address into a READ POSITION on another > then you end up in the wrong place. > > This problem has now been addressed by the ANSI comittee. > Prior to that, a common understanding had developed amongst > DDS manufacturers about the definition of "current tape address" > such that interchange problems between different vendors will > not be a problem. > > 8mm products cannot utilise the LOCATE/READ-POSITION > commands to fast search. See below....... One thing you left out of this. This is what will kill you. The mode sense command for setting up the partitions is vendor specific. This was due to a lack of a standard format for this data. SCSI-2 set it right, but very few drives have moved to SCSI-2 yet. When they do, life will be easier. >3. The third method of "Random Access" is very much vendor specific. > My information on this "Random Access" method is a bit sketchy, > but it goes something like: > > A vendor unique SCSI command allows you to read the current > "track address" from the drive. A second vendor unique SCSI command > causes the drive to position to a specified track address. > > This all sounds like the LOCATE and READ POSITION commands. > The main difference is that the address returned is the physical > tape address. With LOCATE/READ-POSITION, logical addressing > means that "incrementing by 1" causes you to move up 1 block > or 1 mark (it could be argued that "READ-POSITION" is now redundant > since block addressing was standardised, and that a smart enough > driver could "mirror" what the drive would return in response > to a READ-POSITION command), with these vendor unique commands, > there is no such correlation. > > Since these commands are only a couple of months old, I would > expect to see them supported only on drivers that have been written > for 8mm products. They are not as useful as the LOCATE/READ-POSITION > commands, and NOWHERE NEAR AS USEFUL as the SPACE command. Bingo!. Roy Neese Adaptec Senior SCSI Applications Engineer UUCP @ neese@adaptex uunet!cs.utexas.edu!utacfd!merch!adaptex!neese
cmf851@anu.oz.au (Albert Langer) (06/22/91)
Thanks to Steve Jermin, Kevin Jones and Roy Neese for various answers. My summary would be as follows: 1. Effective use of the high speed random access tape drive mechanism can be achieved by just using mt and the non-rewind device. One can use mt to skip backwards or forwards by literally hundreds or thousands of filemarks and it will do so at full speed. 2. It is unlikely that the ordinary seek ioctl mechanism will be properly implemented on current unix drivers. 3. SCSI DAT drives could be used for audio input and output with a certain amount of difficulty. 4. It would be impractical to use audio DAT for SCSI computer purposes as that would amount to designing a SCSI DAT drive controller just as the existing suppliers do. A couple of supplementary questions: 5. If I use the script provided to put say 500 directories onto tape in one session and then for a year of 365 days I keep rewinding to the start and spacing out to the end and then appending another 10 directories to the tape (so I end up with about 4,150 cpio files on tape, each separated by a filemark), can I later decide to truncate to say 3000 and rewrite everything after that? i.e. I would rewind and space out to file 3000 and then just start appending files as usual - losing whatever was on the tape after file 3000 but retaining everything before then. 6. I don't understand why different interpretations of SCSI and DDS etc should prevent unix drivers from properly implementing the seek ioctl on the raw tape device. All I want is the unix semantics of a file of bytes which I can position to any arbitrary byte within (and if I try to seek past the end of file tape mark that is an error). If the implementations are different that might cause media exchange problems but I would still expect to be able to seek forward or backwards on the same machine at high speed. This should be a separate facility to spacing forward or backwards to an end of file tapemark. Anyway, thanks again. Albert -- Opinions disclaimed (Authoritative answer from opinion server) Header reply address wrong. Use cmf851@csc2.anu.edu.au
olson@anchor.esd.sgi.com (Dave Olson) (06/23/91)
In <1991Jun21.200338.23597@newshost.anu.edu.au> cmf851@anu.oz.au (Albert Langer) writes: | Thanks to Steve Jermin, Kevin Jones and Roy Neese for various answers. | | My summary would be as follows: | | 1. Effective use of the high speed random access tape drive mechanism | can be achieved by just using mt and the non-rewind device. One can | use mt to skip backwards or forwards by literally hundreds or thousands | of filemarks and it will do so at full speed. Yes, this works amazingly quickly. The first time I tried it, I thought the driver or firmware was broken, until I verified it on the SCSI analyzer. | 2. It is unlikely that the ordinary seek ioctl mechanism will be | properly implemented on current unix drivers. There is no 'ordinary' seek ioctl, that I know of. IRIX 4.0 on the SGI machines has added an MTSEEK ioctl that uses the SCSI 1 seekblock cmd on scsi 1 drives, and the locate command on SCSI 2 drives like the Python DAT. Perhaps from your discussion below, you were thinking of the lseek system call? | 3. SCSI DAT drives could be used for audio input and output with a | certain amount of difficulty. The main problem is to define a data format, and to modify the DAT firmware to use that format; most of the DDS drives seem to have all the hardware necessary to do this, unlike the CDROM drives. This is not trivial, but not technically impossible. | 5. If I use the script provided to put say 500 directories onto tape | in one session and then for a year of 365 days I keep rewinding to the | start and spacing out to the end and then appending another 10 directories | to the tape (so I end up with about 4,150 cpio files on tape, each | separated by a filemark), can I later decide to truncate to say 3000 and | rewrite everything after that? i.e. I would rewind and space out to file | 3000 and then just start appending files as usual - losing whatever was | on the tape after file 3000 but retaining everything before then. Yes, the DDS DAT format allows overwrite, unlike QIC drives; it just doesn't allow update in place. The new EOD will be wherever you stop writing. | 6. I don't understand why different interpretations of SCSI and DDS etc | should prevent unix drivers from properly implementing the seek ioctl | on the raw tape device. All I want is the unix semantics of a file of | bytes which I can position to any arbitrary byte within (and if I try | to seek past the end of file tape mark that is an error). If the | implementations are different that might cause media exchange problems | but I would still expect to be able to seek forward or backwards on | the same machine at high speed. This should be a separate facility to | spacing forward or backwards to an end of file tapemark. Arbitrary byte seeks won't work, unless you use a blocksize of 1 byte, which actually works, at least on IRIX 4.0; you could have problems with some drivers. All of the space block, seek block, and locate commands work in terms of blocks. Using very small block sizes will waste tape (not as much as 8mm, but still a fair amount), and reduce data rates considerably. If you are willing to live with arbitrary block offsets, then there should be no problem, and high speed seek (to block, fm, or setmark) works fine, and should work on any driver that implements the capability. Finer buffering can be done in the user program. Some vendors may implement 'block' (much like the block disk interface), rather than 'raw' tape, but not all (IRIX doesn't). As has been discussed in some of the other comp.unix* newsgroups, most tape drives make it impossible to imitate byte stream semantics without adding a buffering layer (and updates in place just won't work at all). You also need to give up variable blocksizes to make this work at all.
cmf851@anu.oz.au (Albert Langer) (06/24/91)
Dave, Thanks for following up and confirming most of my summary and answering the remaining questions. In article <1991Jun23.072002.25736@odin.corp.sgi.com> olson@anchor.esd.sgi.com (Dave Olson) writes: >| 2. It is unlikely that the ordinary seek ioctl mechanism will be >| properly implemented on current unix drivers. > >There is no 'ordinary' seek ioctl, that I know of. IRIX 4.0 >on the SGI machines has added an MTSEEK ioctl that uses the >SCSI 1 seekblock cmd on scsi 1 drives, and the locate command on >SCSI 2 drives like the Python DAT. Perhaps from your discussion >below, you were thinking of the lseek system call? Yes, that was what I should have said. >| 6. I don't understand why different interpretations of SCSI and DDS etc >| should prevent unix drivers from properly implementing the seek ioctl >| on the raw tape device. All I want is the unix semantics of a file of >| bytes which I can position to any arbitrary byte within (and if I try >| to seek past the end of file tape mark that is an error). If the >| implementations are different that might cause media exchange problems >| but I would still expect to be able to seek forward or backwards on >| the same machine at high speed. This should be a separate facility to >| spacing forward or backwards to an end of file tapemark. > >Arbitrary byte seeks won't work, unless you use a blocksize of 1 >byte, which actually works, at least on IRIX 4.0; you could have >problems with some drivers. All of the space block, seek block, >and locate commands work in terms of blocks. Using very small >block sizes will waste tape (not as much as 8mm, but still a fair >amount), and reduce data rates considerably. > >If you are willing to live with arbitrary block offsets, then >there should be no problem, and high speed seek (to block, fm, or >setmark) works fine, and should work on any driver that implements >the capability. Finer buffering can be done in the user >program. Some vendors may implement 'block' (much like the block >disk interface), rather than 'raw' tape, but not all (IRIX doesn't). > >As has been discussed in some of the other comp.unix* newsgroups, >most tape drives make it impossible to imitate byte stream semantics >without adding a buffering layer (and updates in place just won't work >at all). You also need to give up variable blocksizes to make this >work at all. Sorry, but I still don't get it. Why doesn't the kernel driver just add the buffering layer in order to provide the lseek call? The DAT drive I am using complains if you send it blocks that aren't a multiple of 1K anyway (and reads them back padded to 1K). Surely anything that has an lseek call of byte granuality is likely to use a buffer between itself and the raw blocks on the physical device? My understanding is that a "block" device adds the kernel cache buffers to a raw device but the lseek is already available in the raw character device. It's easy to have lseek on a disk device but not on a tty or lpt. Strikes me it should also be easy to have lseek on a tape? -- Opinions disclaimed (Authoritative answer from opinion server) Header reply address wrong. Use cmf851@csc2.anu.edu.au
olson@anchor.esd.sgi.com (Dave Olson) (06/25/91)
In <1991Jun24.155738.17279@newshost.anu.edu.au> cmf851@anu.oz.au (Albert Langer) writes: | Sorry, but I still don't get it. Why doesn't the kernel driver just | add the buffering layer in order to provide the lseek call? The DAT | drive I am using complains if you send it blocks that aren't a multiple | of 1K anyway (and reads them back padded to 1K). Surely anything that | has an lseek call of byte granuality is likely to use a buffer between | itself and the raw blocks on the physical device? My understanding is | that a "block" device adds the kernel cache buffers to a raw device | but the lseek is already available in the raw character device. It's | easy to have lseek on a disk device but not on a tty or lpt. Strikes | me it should also be easy to have lseek on a tape? My original point is that many tape drivers do NOT implement lseek, and thus the buffering layer doesn't exist. Confining lseek to non-variable blocked tapes, and requiring the offset on entry to the driver to be a multiple of the block size; then doing the required space block or locate command automatically isn't that hard, but doesn't meet your requirements. You would also run into a problem once you got past 2Gb in capacity, since on most 32 bit systems, off_t (lseek's 3rd arg type) is a signed 31 bit number... 'Just' adding the buffering layer is non-trivial, unless it exactly matches the semantics of the existing block buffering layer (disk drives). Unfortunately, tape does not. If you were willing to live with a readonly block interface, and write the tape some other way, it would be much easier. One of the side effects of the block interface is asynchronous writes; which make it rather hard to report the errors that would be likely to occur, since one can't update in place with DDS DAT. Writing a different buffering system, unique to the tape driver would itself be a significant amount of work, for very little gain, although it certainly isn't impossible. It still wouldn't completely match the byte stream model.
cmf851@anu.oz.au (Albert Langer) (06/25/91)
In article <1991Jun25.072303.27866@odin.corp.sgi.com> olson@anchor.esd.sgi.com (Dave Olson) writes: >[...] You would also run into a problem >once you got past 2Gb in capacity, since on most 32 bit systems, >off_t (lseek's 3rd arg type) is a signed 31 bit number... > >'Just' adding the buffering layer is non-trivial, unless >it exactly matches the semantics of the existing block buffering >layer (disk drives). Unfortunately, tape does not. If you >were willing to live with a readonly block interface, and >write the tape some other way, it would be much easier. I could certainly live with a readonly block interface between filemarks, especially since the tape has to be rewritten from a filemark anyway. I could also live with a 2GB file size limit (again between filemarks) since quite a few other pointers would break on anything larger anyway (apart from current tape capacity being smaller than 2GB for DAT). However I do now understand, from your explanation, that adding lseek is not as trivial as I thought, and since the filemark mechanism only occupies 4bytes and does not slow things down I guess it will do for most applications (including the one I had in mind - just storing thousands of files offline for retrieval in batches or on a delay of 12 seconds or so rewind time) so that explains why normal unix drivers may not include it. I still think that since it is relatively easy for just a readonly block interface, even though non-trivial, and since that is all that could ever be possible for DDS, the unix driver SHOULD implement lseek. One possible application would be for a huge hashed database, prepared online, to be dumped to tape and then be accessible with a 12 second or so delay, or in batches. But I guess anybody doing that can just go ahead and write the driver too. Anyway, ALL my questions are answered now. Thanks to all who gave advice... it's appreciated. -- Opinions disclaimed (Authoritative answer from opinion server) Header reply address wrong. Use cmf851@csc2.anu.edu.au