[comp.sources.d] Problems with Compress

peter@sugar.UUCP (Peter da Silva) (09/20/87)

Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
uses a huge amount of memory and may not even be available on small machines
like PDP-11s and Intels.
-- 
-- Peter da Silva `-_-' ...!hoptoad!academ!uhnix1!sugar!peter
--                 'U`  Have you hugged your wolf today?

brianc@cognos.uucp (Brian Campbell) (09/24/87)

In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
! uses a huge amount of memory and may not even be available on small machines
! like PDP-11s and Intels.

   Don't suck Intel machines into this -- 16-bit compress is available for
both DOS and Xenix.  It also saves a considerable (meaning not negligible)
amount of space.  Try as I might, I just can't feel sorry for you.  Surely
you have access to some machine that can do 16-bit uncompresses?
-- 
Brian Campbell        uucp: decvax!utzoo!dciem!nrcaer!cognos!brianc
Cognos Incorporated   mail: POB 9707, 3755 Riverside Drive, Ottawa, K1G 3Z4
(613) 738-1440        fido: sysop@163/8

satan@geac.UUCP (The Big One Himself) (09/27/87)

In article <1493@cognos.UUCP> brianc@cognos.UUCP (Brian Campbell) writes:
>In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
>! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
>! uses a huge amount of memory and may not even be available on small machines
>! like PDP-11s and Intels.
>
>   Don't suck Intel machines into this -- 16-bit compress is available for
>both DOS and Xenix.  It also saves a considerable (meaning not negligible)
>amount of space.  Try as I might, I just can't feel sorry for you.  Surely
>you have access to some machine that can do 16-bit uncompresses?
>-- 
>Brian Campbell        uucp: decvax!utzoo!dciem!nrcaer!cognos!brianc
>Cognos Incorporated   mail: POB 9707, 3755 Riverside Drive, Ottawa, K1G 3Z4
>(613) 738-1440        fido: sysop@163/8

Sorry Brian, not all of us are on the fed's gravy train with access to ATs
and RTs and other expensive new hardware, we must cope with DOS or Venix on 
PCs or XTs, or else old PDP-11s.  None of which support 16-bit compress

By the way, locating a friendly site with a set of tools can sometime be
a big problem when the overall gain of transferring a 12-bit compressed
image versus a 16-bit image is very small (not negigilbe but small).  Since
you indicate that the request is an imposition, please post your sources
for your wonderful compress, otherwise, you should, perhaps, switch feet 
before venting any frustration you might be experiencing.

-- 
(__|__)
   |
   |        What's a soul anyway --- you hardly know its there!
   |

johnm@auscso.UUCP (John B. Meaders) (09/28/87)

Some of us unfortunate souls are stuck running Xenix on 8086 based machines
(mine is Compaq Deskpro).  When I get out of school I will be able to afford
a real computer.  Until then, or until I can find a buyer for this machine,
I am stuck with maximum user processes of 399k.  So, yes, please stick to
12-bit compresses.
-- 
John B. Meaders, Jr.  1114 Camino La Costa #3083, Austin, TX  78752
ATT:  Voice:  +1 (512) 451-5038  Data:  +1 (512) 371-0550
UUCP:   ...!ut-sally!ut-ngp!auscso!jclyde!john
                                  \johnm

jfh@killer.UUCP (10/02/87)

In article <1493@cognos.UUCP>, brianc@cognos.uucp (Brian Campbell) writes:
> In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
> ! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
> ! uses a huge amount of memory and may not even be available on small machines
> ! like PDP-11s and Intels.
> 
>    Don't suck Intel machines into this -- 16-bit compress is available for
> both DOS and Xenix.  It also saves a considerable (meaning not negligible)
> amount of space.  Try as I might, I just can't feel sorry for you.  Surely
> you have access to some machine that can do 16-bit uncompresses?

Two years ago I munged on Compress 3.0 and produced a compress that produced
16 bit compression in less than 128K of total memory.  (256K if you count
a unix kernel running there also.)

This week I started working on Compress 4.0 so it would do likewise.  Most
of the time has been spent working on the macros that simulate VM, trying
to get them to run no more than 3 times slower ...

So, what I want is a few people with little tiny machines that need to undo
16 bit compression.  I ain't gonna post it until it works right.

- John.
-- 
John F. Haugh II		HECI Exploration Co. Inc.
UUCP:	...!ihnp4!killer!jfh	11910 Greenville Ave, Suite 600
"Don't Have an Oil Well?"	Dallas, TX. 75243
" ... Then Buy One!"		(214) 231-0993

allbery@ncoast.UUCP (Brandon Allbery) (10/02/87)

As quoted from <1488@geac.UUCP> by satan@geac.UUCP (The Big One Himself):
+---------------
| In article <1493@cognos.UUCP> brianc@cognos.UUCP (Brian Campbell) writes:
| >In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
| >! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
| >! uses a huge amount of memory and may not even be available on small machines
| >! like PDP-11s and Intels.
| >
| >   Don't suck Intel machines into this -- 16-bit compress is available for
| >both DOS and Xenix.  It also saves a considerable (meaning not negligible)
| >amount of space.  Try as I might, I just can't feel sorry for you.  Surely
| >you have access to some machine that can do 16-bit uncompresses?
| 
| Sorry Brian, not all of us are on the fed's gravy train with access to ATs
| and RTs and other expensive new hardware, we must cope with DOS or Venix on 
| PCs or XTs, or else old PDP-11s.  None of which support 16-bit compress
+---------------

Nor are small machines alone.  Ncoast has fried its cache chips a few times
as a result of 16-bit compresses... 68000, NOT an Intel or PDP-11!
-- 
	    Brandon S. Allbery, moderator of comp.sources.misc
  {{harvard,mit-eddie}!necntc,well!hoptoad,sun!mandrill!hal}!ncoast!allbery
ARPA: necntc!ncoast!allbery@harvard.harvard.edu  Fido: 157/502  MCI: BALLBERY
   <<ncoast Public Access UNIX: +1 216 781 6201 24hrs. 300/1200/2400 baud>>
"`You left off the thunderclap and the lightning flash.', I told him.
`Should I try again?'  `Never mind.'"     --Steven Brust, JHEREG

rpw3@amdcad.UUCP (10/03/87)

My Fortune System has only about 700K usable user memory, and running 16-bit
"compress" (triggered by unbatching news) made it swap so bad that UUCP would
time out! So I did some comparisons of what the various "-b" settings did to
the compression ratio, versus the memory size of "compress" compiled for various
maximum supported "-b" values. I'm sorry I don't have the table at hand, but
the following table of pseudo-values is roughly what I remember for compress
version 3.6 (I have *not* tried compress 4.0 yet, although I've heard it's
considerably smaller -- folded some tables or something) when run on several
large batches of news (*not* with any uuencoded binaries in them):

	WARNING: From vague recollection!

	MaxBits	MemSize	Compressed file (normalized to -b16 == "1")

	-b 16	640K	1
	-b 15	360K	1.02
	-b 14	220K	1.05
	-b 13	150K	1.15
	-b 12	105K	1.25

Note that the memory size goes up as a power of 2 of the MaxBits, plus some
base size for the program and misc variables. Note also that (as best I recall)
the real "knee" in compression was between "-b 14" and "-b 13", as was the
knee of the memory size. Since my problem was at a different hard limit than
a 64k+64k machine, "-b15" worked just fine for me, and given the figures, none
of my news feeds minded cranking down the compression by 1 (even if it meant,
as it did in one case, cranking *all* their outgoing compressed batch down to
"-b 15").

Given compress 4.0, somebody might want to repeat my experiment again, and see
if there is a MaxBits value which is above the compression "knee", but within
the memory size limits for these small machines. (Remember to include mainly
text groups in your test.) For example, if it turns out that "-b14" will fit,
and the penalty is small, your neighbors (indeed, the net) may be willing to
switch to "-b14" as the "standard".

By the way, you big guys would win from this, as well. "-b16" makes your systems
thrash around a lot more than they really need to...


Rob Warnock
Systems Architecture Consultant

UUCP:	  {amdcad,fortune,sun,attmail}!redwood!rpw3
ATTmail:  !rpw3
DDD:	  (415)572-2607
USPS:	  627 26th Ave, San Mateo, CA  94403

peter@sugar.UUCP (Peter da Silva) (10/04/87)

In article <1493@cognos.UUCP>, brianc@cognos.uucp (Brian Campbell) writes:
> In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
> ! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
> ! uses a huge amount of memory and may not even be available on small machines
> ! like PDP-11s and Intels.
>    Don't suck Intel machines into this -- 16-bit compress is available for
> both DOS and Xenix.

Xenix 286, yes. Not on the 8088, though. And it still takes up something
in the vicinity of half a meg to hold the tables for 16 bit compression.

> It also saves a considerable (meaning not negligible)
> amount of space.  Try as I might, I just can't feel sorry for you.  Surely
> you have access to some machine that can do 16-bit uncompresses?

I do now... a Xenix 286 machine. I can also run it on my Amiga if I throw
away ALL other programs and libraries, a rather distasteful action.

Since the posting I'm talking about was a bunch of compressed Amiga images
it's a bit strange. At the very least people should refrain from 16bit
compressing data intended for machines with under half a meg of free RAM
easily available.

A couple of jobs back I was restricted to machines with 128K maximum
code+data.
-- 
-- Peter da Silva  `-_-'  ...!hoptoad!academ!uhnix1!sugar!peter
-- Disclaimer: These U aren't mere opinions... these are *values*.

darrylo@hpsrlc.HP.COM (Darryl Okahata) (10/05/87)

In comp.sources.d, peter@sugar.UUCP (Peter da Silva) writes:

> In article <1493@cognos.UUCP>, brianc@cognos.uucp (Brian Campbell) writes:
> > In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
> > ! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
> > ! uses a huge amount of memory and may not even be available on small machines
> > ! like PDP-11s and Intels.
> >    Don't suck Intel machines into this -- 16-bit compress is available for
> > both DOS and Xenix.
> 
> Xenix 286, yes. Not on the 8088, though. And it still takes up something
> in the vicinity of half a meg to hold the tables for 16 bit compression.

     It is *possible* to get an 8088 to handle 16-bit compression; you just
need the right compiler.  Just for fun, I hacked at compress to run with 16
bits on an 8088 and managed to get it to work.  The only problems are (1) it
does use up gobs of memory (the program needed 450K just to load!) and (2)
it is slower than molasses in January due to the way the compiler handled
arrays spanning multiple segments.

     [ ... ]
> -- 
> -- Peter da Silva  `-_-'  ...!hoptoad!academ!uhnix1!sugar!peter
> -- Disclaimer: These U aren't mere opinions... these are *values*.
> ----------

     -- Darryl Okahata
	{hplabs!hpccc!, hpfcla!} hpsrla!darrylo
	CompuServe: 75206,3074

Disclaimer: the above is the author's personal opinion and is not the
opinion or policy of his employer or of the little green men that
have been following him all day.

mc68020@gilsys.UUCP (Thomas J Keller) (10/06/87)

In article <4786@ncoast.UUCP>, allbery@ncoast.UUCP (Brandon Allbery) writes:
> As quoted from <1488@geac.UUCP> by satan@geac.UUCP (The Big One Himself):
> +---------------
> | In article <1493@cognos.UUCP> brianc@cognos.UUCP (Brian Campbell) writes:
> | >In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
> | >! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
> | >! uses a huge amount of memory and may not even be available on small machines
> | >! like PDP-11s and Intels.
> Nor are small machines alone.  Ncoast has fried its cache chips a few times
> as a result of 16-bit compresses... 68000, NOT an Intel or PDP-11!

   As I stated in my summary, I cannot imagine a smaller 68000 based machine
than the Tandy 6000, and my Tandy handles 16 bit compress/uncompress just fine.
I have to admit that I couldn't handle more than 13bits until I learned how to
patch my kernal to permit >256K user process limits, but once I did that, 
ZIPPETY-POW!, off it went!  (the ZIPPETY-POW!, by the by, comes from one
Katy the Caterpillar (Disney CHannel), to give attribution where attribution
is due)

   Brandon, if you are SERIOUS about 16 bit compresses causing hardware
failures, I strongly suggest that there is a ***MAJOR*** flaw in the design
of your hardware.

-- 
Tom Keller 
VOICE  : + 1 707 575 9493
UUCP   : {ihnp4,ames,sun,amdahl,lll-crg,pyramid}!ptsfa!gilsys!mc68020

jfh@killer.UUCP (The Beach Bum) (10/10/87)

In article <858@sugar.UUCP>, peter@sugar.UUCP (Peter da Silva) writes:
> In article <1493@cognos.UUCP>, brianc@cognos.uucp (Brian Campbell) writes:
> > In article <780@sugar.UUCP> peter@sugar.UUCP (Peter da Silva) writes:
> > ! Will people who compress stuff for distribution PLEASE use 12-bit. 16-bit
> > ! uses a huge amount of memory and may not even be available on small machines
> > ! like PDP-11s and Intels.
> >    Don't suck Intel machines into this -- 16-bit compress is available for
> > both DOS and Xenix.
> 
> Xenix 286, yes. Not on the 8088, though. And it still takes up something
> in the vicinity of half a meg to hold the tables for 16 bit compression.

You go to a 8088 machine to try this out on?  I did learn today that there
is a decompress program out there that does't need the mega-tables like
compress.  Don't know what to say ...

OK - the response was good to `So you want to beta-test my latest compress
hack?'.  What just about everyone forgot was - what is your operating
system this week?  Seriously guys and dolls, I don't recall seeing one
single unix-clone in the bunch of letters I got.

Just for your info - I did manage to decompress a 16 bit compressed (actually
had only gotten to 15 bit compression) file in only  2 times the normal time
using right at 64K text+data on a 68020.  Unfortunately, 16-bit compression
in 64K or less is VERY slow.  Works real well in 256K, and almost like
a dream in 512K or more (wonder why ;-)

So - if you wrote before let me know what the OS of the week is.  I should
have my compress tested on a 3B2 tonight and ready for bizarreware
by tomorrow.

- John.
-- 
John F. Haugh II		HECI Exploration Co. Inc.
UUCP:	...!ihnp4!killer!jfh	11910 Greenville Ave, Suite 600
"Don't Have an Oil Well?"	Dallas, TX. 75243
" ... Then Buy One!"		(214) 231-0993