jonathan@cs.pitt.edu (Jonathan Eunice) (04/07/91)
A few questions about V.42 data compresssion (or is is V.42bis? I can't keep track): 1. If one has V.42 compression, with 4:1 compression ratios claimed (and, apparently, ratios rivaling compress/LWZ achieved), does it still make sense to compress data before sending it over the wire? I'd think not--just let the modem h/w do the work. 2. Would it be possible for the modem to hand the system data that is still V.42 compressed? No need to use compress, the program, when the h/w already puts it in a suitable form. Software for online V.42 compression/decompression shouldn't be any harder to develop than compress, right? And if there were lots of V.42 modems around, which seems to be increasingly the case, V.42 might become a second de facto standard in addition to today's compress. 3. If #2 is OK, could the system use a modem as a jury-rigged compression/decompression engine? While perhaps not as good as dedicated h/w, most systems don't use their modems 100% of the time, so it could be a cheap alternative that wouldn't load the system down. Of course, if the modem were busy when someone wanted to compress/decompress, software emulation could be transparently substituted. So, am I in deep left field, or what?
dcarr@hobbit.gandalf.ca (Dave Carr) (04/10/91)
In <10334@pitt.UUCP> jonathan@cs.pitt.edu (Jonathan Eunice) writes: >A few questions about V.42 data compresssion (or is is V.42bis? I can't >keep track): >1. If one has V.42 compression, with 4:1 compression ratios claimed > (and, apparently, ratios rivaling compress/LWZ achieved), does it One would hope the compression ratios are similar, since the basis for V.42 bis data compression is LZW. The main differences are: (1) When the compressor table becomes full, V.42 bis recovers nodes in the compression dictionary one at a time as needed. Compress waits for the compression ratio to drop. (2) V.42 bis can control EXPANSION. When the modem detects uncompressable data, it can switch into transparent mode, and then switch back when the data again compresses. Unix Compress has no such feature. (3) The size of the compression tables may be much smaller in modems. Typically, Unix compress uses up to 16-bit codes, while modems MAY suuport only 12-bit codes. > still make sense to compress data before sending it over the wire? > I'd think not--just let the modem h/w do the work. In fact, the modem may do a BETTER job. You may find that there is a quite a difference in the compression ratio between different V.42 bis modems. The algorithm for encoding and decoding are standardized, but the decision to escape to/from transparent mode is not. >2. Would it be possible for the modem to hand the system data that is > still V.42 compressed? Yes, it possible. Although it would be preferrable if the host-modem interface was bit-synchronous, since the data may not be an integral number of octets. > Software for online > V.42 compression/decompression shouldn't be any harder to develop > than compress, right? And if there were lots of V.42 modems around, > which seems to be increasingly the case, V.42 might become a second > de facto standard in addition to today's compress. Sure, just pay the ~$50,000 licensing fees, and away you go. >3. If #2 is OK, could the system use a modem as a jury-rigged > compression/decompression engine? While perhaps not as good as > dedicated h/w, most systems don't use their modems 100% of the time, > so it could be a cheap alternative that wouldn't load the system > down. Of course, if the modem were busy when someone wanted to > compress/decompress, software emulation could be transparently > substituted. Well, depending on the processor load to handle the I/O, it might be cheaper to just do the compression in the main system. LZW encoding is fairly fast after all, it was design for disk subsystem. >So, am I in deep left field, or what? I don't know. Is there a short-stop between you and the plate ?
tnixon@hayes.uucp (04/11/91)
In article <10334@pitt.UUCP>, jonathan@cs.pitt.edu (Jonathan Eunice) writes: > A few questions about V.42 data compresssion (or is is V.42bis? I can't > keep track): V.42bis is data compression which runs on top of V.42 error control. > 1. If one has V.42 compression, with 4:1 compression ratios claimed > (and, apparently, ratios rivaling compress/LWZ achieved), does it > still make sense to compress data before sending it over the wire? > I'd think not--just let the modem h/w do the work. I always advise people to compress before sending if possible. An offline compression program can make multiple passes over the data and really optimize the compression, while a modem only has one shot at it. Also, the compression acheivable in the modem is limited by the ratio between the DTE-DCE speed and the carrier speed. Modem-based compression is great for online text-based applications, but if you're sending files and have some control over the process, offline compression is better. > 2. Would it be possible for the modem to hand the system data that is > still V.42 compressed? There's no way for a _standard_ V.42 modem to pass the compressed data to the PC. You wouldn't really want to, anyway -- there's no way to delimit the file boundaries, etc. While I suppose some manufacturer could produce a proprietary feature to allow the compressed data to be passed to the software, I doubt it would ever become part of the standard. > 3. If #2 is OK, could the system use a modem as a jury-rigged > compression/decompression engine? Again, perhaps some manufacturer will develop this as a proprietary feature, but it won't become part of a standard. > So, am I in deep left field, or what? No, these things have been thought about before, but the standards community didn't think it was feasible or reasonable. -- Toby Nixon, Principal Engineer | Voice +1-404-840-9200 Telex 151243420 Hayes Microcomputer Products Inc. | Fax +1-404-447-0178 CIS 70271,404 P.O. Box 105203 | UUCP uunet!hayes!tnixon AT&T !tnixon Atlanta, Georgia 30348 USA | Internet hayes!tnixon@uunet.uu.net
dcarr@hobbit.gandalf.ca (Dave Carr) (04/12/91)
In <3908.280363d4@hayes.uucp> tnixon@hayes.uucp writes: >I always advise people to compress before sending if possible. An >offline compression program can make multiple passes over the data >and really optimize the compression, while a modem only has one shot >at it. CAN make multiple passes over the data. In reality, most commercial or shareware compression programs make only a single pass. I was suprised to find out my favorite (PKZIP) does only one pass. LZW based algorithm are usually one-pass. LHarc is one-pass. Unix Compress is one-pass. >Also, the compression acheivable in the modem is limited by >the ratio between the DTE-DCE speed .. I agree. So why do modems and PC get it together and get a decent interface worked out. Why not Ethernet ? >and the carrier speed. I don't see this at all. Perhaps the link level protocol. Perhaps the line error rate. But the compression should be independant on the carrier speed.
jonathan@cs.pitt.edu (Jonathan Eunice) (04/13/91)
It looks like my idea of using modems with V.42 compression as a general-purpose compression engine is going to achieve naught. Several people warned me that V.42, while a nice optimization, is little more. Most implementations will give compression ratios and speeds substantially inferior to that of host based compression. The speed of communications with the modem appears to be a major bottleneck, as the fastest the host and modem can typically communicate is far below the compression rate of a program like compress or PKZIP running on any reasonable platform. V.42 also lacks the handy mutli-file bundling and CRC error checking of some host-based archive/compress packages. Oh, well. It was just an idea. I expect to have more of them in the future. Perhaps some will pan out, making be rich and famous beyond my wildest imaginings. Just not this one. jse
marc@aria.ascend.com (Marco S Hyman) (04/14/91)
In article <1991Apr12.132116.11546@hobbit.gandalf.ca> dcarr@hobbit.gandalf.ca (Dave Carr) writes: > I don't see this at all. Perhaps the link level protocol. > Perhaps the line error rate. But the compression should be > independant on the carrier speed. Dave, how would you feel if you kept putting bytes into one end of a data link and nothing came out the other end? Why? Because the first end can still do some more compression on the data and doesn't have a new symbol to send. Compression efficiency is only half the problem -- you want to keep your bandwidth filled. What's the use of compressing more while the bandwidth sits idle. When the (very) efficient compression symbol is finally sent there is even more delay added to output the cleartext at DTE interface speeds. All The only result of making compression independant of the carrier speed is increased response time. Throughput is not the only factor in communcations links -- you can never forget about response time. -- // marc // work: marc@ascend.com uunet!aria!marc // home: marc@dumbcat.sf.ca.us {ames,sun}!pacbell!dumbcat!marc
tnixon@hayes.uucp (04/14/91)
In article <1991Apr12.132116.11546@hobbit.gandalf.ca>, dcarr@hobbit.gandalf.ca (Dave Carr) writes: >>Also, the compression acheivable in the modem is limited by >>the ratio between the DTE-DCE speed .. > > I agree. So why do modems and PC get it together and get a decent > interface worked out. Why not Ethernet ? Well, we do have EIA-530 now, which goes up to 2.1 Mbps, and is pin-compatible with EIA-232. And TIA TR-30.2 just started work on a 52 Mbps interface standard. SCSI is another option, as are various LAN methods. The easiest way to get high speed, of course, is to use an internal modem and have it right on the bus, but it seems that most people want external modems these days. >>and the carrier speed. > > I don't see this at all. Perhaps the link level protocol. > Perhaps the line error rate. But the compression should be > independant on the carrier speed. I said the _ratio_ between the DTE-DCE speed and the carrier speed. If the DTE-DCE speed is 38400 and the carrier speed is 9600, it's hard to get more than 4-to-1 compression, because you're underrunning the transmitter. -- Toby Nixon, Principal Engineer | Voice +1-404-840-9200 Telex 151243420 Hayes Microcomputer Products Inc. | Fax +1-404-447-0178 CIS 70271,404 P.O. Box 105203 | UUCP uunet!hayes!tnixon AT&T !tnixon Atlanta, Georgia 30348 USA | Internet hayes!tnixon@uunet.uu.net
dcarr@hobbit.gandalf.ca (Dave Carr) (04/15/91)
In <529@aria.ascend.com> marc@aria.ascend.com (Marco S Hyman) writes: >Dave, how would you feel if you kept putting bytes into one end of a data link >and nothing came out the other end? Why? Because the first end can still do >some more compression on the data and doesn't have a new symbol to send. >Compression efficiency is only half the problem -- you want to keep your >bandwidth filled. What's the use of compressing more while the bandwidth sits >idle. When the (very) efficient compression symbol is finally sent there is >even more delay added to output the cleartext at DTE interface speeds. All >The only result of making compression independant of the carrier speed is >increased response time. >Throughput is not the only factor in communcations links -- you can never >forget about response time. I agree. Assume the Forwarding Algorithm is send packet when the link is idle. Use the SAME algorithm in a NON-COMPRESSION link as well. Both compressed and uncompressed links will attain a steady state throughput. I think you will find that the compression ratio is still independent of the link speed.
pplacewa@bbn.com (Paul Placeway) (04/23/91)
dcarr@hobbit.gandalf.ca (Dave Carr) writes: < In <3908.280363d4@hayes.uucp> tnixon@hayes.uucp writes: < >I always advise people to compress before sending if possible. An < >offline compression program can make multiple passes over the data < >and really optimize the compression, while a modem only has one shot < >at it. < CAN make multiple passes over the data. In reality, most commercial < or shareware compression programs make only a single pass. I was < suprised to find out my favorite (PKZIP) does only one pass. < LZW based algorithm are usually one-pass. LHarc is one-pass. < Unix Compress is one-pass. Unix compress(1) is (sorta) two-pass: if the resulting compressed file would be bigger than the original, it doesn't compress it. Of course this is only true of comressing a *file*, compressing pipe input is, of course, one pass. Several of the Macintosh compressor/archiver programs are multi-pass. StuffIt is a notable example -- StuffIt 1.5.1 tries nothing, Huffman and LZW and picks whichever is smallest. It almost allways uses LZW, though, since (order-0) Huffman coding is pretty weak. < >Also, the compression acheivable in the modem is limited by < >the ratio between the DTE-DCE speed .. < I agree. So why do modems and PC get it together and get a decent < interface worked out. Why not Ethernet ? Probably because compression is not the job of the network. Ethernet itself is PACKET based. Even if you had the hardware to do the compression in real time, compressing 1K packets might not buy much. Compressing E-net speed would require special hardware anyway. You can spend your extra $$$ on getting ~30 Mbit/sec out of Ethernet, I'd rather spend my $$$ on a twisted-pair FDDI-oid interface at 100 Mbits/sec. -- Paul Placeway <pplaceway@bbn.com>
dcarr@hobbit.gandalf.ca (Dave Carr) (04/23/91)
In <63840@bbn.BBN.COM> pplacewa@bbn.com (Paul Placeway) writes: My original comment was: I agree. So why do modems and PC get it together and get a decent interface worked out. Why not Ethernet ? >Probably because compression is not the job of the network. Ethernet >itself is PACKET based. Even if you had the hardware to do the >compression in real time, compressing 1K packets might not buy much. Oh, I (and Telebit, Microcom,..) would say the opposite. It is the job of the network to appear seamless. Do you want the application layer to start saying "oh you want to go to that destination, then you must compress the data first !". Hardly. Compression after all just saves money. You could always buy a more expensive link. On a single 1K byte packet, you should get 2:1 assuming the tables are reset between packets. The learning time of LZW (for example Storers AP variant) is extremely fast. If you are working on a single connection (say a FTP transfer) the compression approaches the compression figures of compressing before transmission. Never as good as precompressing, but transparent to the application. >Compressing E-net speed would require special hardware anyway. You >can spend your extra $$$ on getting ~30 Mbit/sec out of Ethernet, I'd >rather spend my $$$ on a twisted-pair FDDI-oid interface at 100 Mbits/sec. I wasn't talking about compressing the Ethernet. Most PC's already have an Ethernet connection. What I meant was put an Ethernet connection on the modem to eliminate the 38.4Kbps restriction. The modem would compress over the Wide Area Link, an talk to the PC using IP over the Ethernet. And for the record, compressing at E-net speeds wouldn't require special hardware. Check out Ross Williams paper "The World's Fastest Adaptive Text Compression Algorithm". It can compress 250 Kbytes/sec on a 1 MIPS machine. Take an 80960CA processor (or your average 680x0) and you can get E-net speeds (1.25 MBytes/sec).