[comp.compression] gif encoding questions

rg@msel.unh.edu (Roger Gonzalez) (04/09/91)

Let c = (2**bitsize) - 1

When encoding in GIF format, do you increase the bitsize when you
_output_ code c, or when you _add_ an entry to the c-th entry in the 
string table?

What about when c = 4095?  Do you output the clear code and reinitialize
after 4095 is output, or after the 4095th entry in the table is used?

Why have I seen gray scale (256) GIFs that don't have simple color maps like
00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF

Some of them are really strange...


-Roger
-- 
"The question of whether a computer can think is no more interesting
 than the question of whether a submarine can swim" - Edsgar W. Dijkstra 
rg@[msel|unhd].unh.edu        |  UNH Marine Systems Engineering Laboratory
r_gonzalez@unhh.bitnet        |  Durham, NH  03824-3525


-- 
"The question of whether a computer can think is no more interesting
 than the question of whether a submarine can swim" - Edsgar W. Dijkstra 
rg@[msel|unhd].unh.edu        |  UNH Marine Systems Engineering Laboratory
r_gonzalez@unhh.bitnet        |  Durham, NH  03824-3525

toddpw@nntp-server.caltech.edu (Todd P. Whitesel) (04/10/91)

rg@msel.unh.edu (Roger Gonzalez) writes:

>Let c = (2**bitsize) - 1

>When encoding in GIF format, do you increase the bitsize when you
>_output_ code c, or when you _add_ an entry to the c-th entry in the 
>string table?

You increase the bitsize whenever the maximum code value you could output
can't be represented with the current bitsize. For encoders this means
you do it when you add the c-th entry.

>What about when c = 4095?  Do you output the clear code and reinitialize
>after 4095 is output, or after the 4095th entry in the table is used?

According to the 89a standard you are not required to output a clear code
at all. However I would encourage outputting a clear code INSTEAD of filling
the 4095th table entry; this way the decompressor sees the clear code before
there is any question as to its bitsize. When I was first teaching myself GIF
I forgot to max out the bitsize at 12 and would lose sync with the data on
only some of my GIFs when the table maxed out for the first time, because the
clear code was being read as 13 bits. This is not overridingly important but
it's a nice touch to know that your picture can be read in full by a decoder
that almost works.

>Why have I seen gray scale (256) GIFs that don't have simple color maps like
>00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF

Who knows? Maybe they have been sorted by popularity or something.

Todd Whitesel
toddpw @ tybalt.caltech.edu

marcos@netcom.COM (Marcos H. Woehrmann) (04/10/91)

In article <1991Apr9.142255.186@unhd.unh.edu>, rg@msel.unh.edu (Roger Gonzalez) writes:
> 
> When encoding in GIF format, do you increase the bitsize when you
> _output_ code c, or when you _add_ an entry to the c-th entry in the 
> string table?
> 
> What about when c = 4095?  Do you output the clear code and reinitialize
> after 4095 is output, or after the 4095th entry in the table is used?

  if (hashLocation>maxHashTable)
    panic("Internal error while writing Gif (1)\n");
  if (lastUsed==maxUsable) {
    if (codeSize==maxBits) { /* can't increase bit size, clear table! */
      resetTable(clearCode,out);
      return;
    } else {
      codeSize++;
      maxUsable=1 << codeSize;
    }
  }

However, according to the GIF89A spec (availble from CompuServe and from
Simtel): "There is not a requirement to send a clear code when the string
table is full".  (And they go on to say you could actually clear the table
before it is full, though how you decide the optimal time to do that
is left as an exercise to the interested reader :-) ).

As an aside does anyone know if any of the GIF writers make use of this yet?
The document I have points out that most (all?) gif readers will break
unless you only issue a clear code when the table is full, and suggests that
gif writers don't implement a early or late clear code until after Jan. 1991,
by which time all the gif readers will have been fixed. 


> 
> Why have I seen gray scale (256) GIFs that don't have simple color maps like
> 00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF
> 
> 

The GIF spec suggests that you sort the Global Color Table (aka palette)
by order of importance (which most people of interpreted to mean in order
of frequency).  "This assists a decoder, with fewer available colors, in
choosing the best subset of colors..."

marcos

-- 
Marcos H. Woehrmann    {claris|apple}!netcom!marcos  |  marcos@netcom.COM

 Oh, I'm sure you've heard it all before, but remember it's not what you
 say, it's how you say it, and how much you're paid to do so, and besides
 who's listening anyway.  No one, that's who, because it's all been said
 and done and done and done and done to death.  Let's talk about art,
 said the fool to the idiot.  --Lydia Lunch

alan@ukpoit.co.uk (Alan Barclay) (04/10/91)

In article <1991Apr9.142255.186@unhd.unh.edu> rg@msel.unh.edu (Roger Gonzalez) writes:
>Why have I seen gray scale (256) GIFs that don't have simple color maps like
>00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF
>
>Some of them are really strange...


Basically there are three bytes per colour in the colour maps, therefore
00 00 00 is the black, 01 01 01 is the first shade of grey, 02 ...
FF FF FF is white. If when the colour is used it looks up so 0 is 00 00 00,
1 is 01 01 01 etc. There is no reason why 125 cannot refer to 00 00 00 while
126 refers to 53 53 53. And some scanner software will allocate in this
fashion. In addition the difference between 53 53 53 and 52 53 54 is so
small that it will still look grey, so you might call it a grey scale when
it is really a colour GIF with different near grey colours, chosen for
asethetic reasons.
-- 
  Alan Barclay
  iT                                |        E-mail : alan@ukpoit.uucp
  Barker Lane                       |        BANG-STYLE : .....!ukc!ukpoit!alan
  CHESTERFIELD S40 1DY              |        VOICE : +44 246 214241

rg@msel.unh.edu (Roger Gonzalez) (04/12/91)

>In article <1991Apr9.142255.186@unhd.unh.edu> I write:
>>Why have I seen gray scale (256) GIFs that don't have simple color maps like
>>00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF
>>
>>Some of them are really strange...

But what I meant was: 
When looking at grayscale 256 GIFs, I would expect
>>00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF
but have seen (in ppmtogif results, for example)

000000   47-49-46-38-37-61-00-02   00-02-E7-00-00-00-00-00   GIF87a@B@B.@@@@@
000010   C9-C9-C9-8F-8F-8F-55-55   55-1B-1B-1B-E4-E4-E4-AA   ......UUU[[[....
000020   AA-AA-70-70-70-36-36-36   FF-FF-FF-C5-C5-C5-8B-8B   ..ppp666........
000030   8B-51-51-51-17-17-17-E0   E0-E0-A6-A6-A6-6C-6C-6C   .QQQWWW......lll
000040   32-32-32-FB-FB-FB-C1-C1   C1-87-87-87-4D-4D-4D-13   222.........MMMS

And I was wondering why the screwy ordering.  Sorry if I wasn't clear.

I've been trying to reverse engineer what my problems are with my
encoder by comparing my output with the results of rawtopgm | ppmtogif,
but my comparison can only go so far before I hit a root code that 
is different that mine because their greyscale map is different.

Why am I writing my own encoder when I could swipe code from everyone else?
'Cause I'm a masochist.  So what if its a memory pig, I don't use hashing 
yet, and its slower than shit?  I'm having fun!  

If you are equally masochistic, and would like to give me a hand, my
almost-but-not-quite encoder is on msel.unh.edu in /pub/gif for ftp-ing.

I'm really close.. I get one good scan line out (using VUIMAGE on a
PC), and then a shitty scan line, and then it stops. I think it barfs
because I don't change bit sizes where it expects me to.  Sigh.

-Roger
-- 
"The question of whether a computer can think is no more interesting
 than the question of whether a submarine can swim" - Edsgar W. Dijkstra 
rg@[msel|unhd].unh.edu        |  UNH Marine Systems Engineering Laboratory
r_gonzalez@unhh.bitnet        |  Durham, NH  03824-3525

jms@tardis.Tymnet.COM (Joe Smith) (04/12/91)

In article <1991Apr11.213950.16265@unhd.unh.edu> rg@msel.unh.edu (Roger Gonzalez) writes:
:When looking at grayscale 256 GIFs, I would expect
:>>00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF
:but have seen (in ppmtogif results, for example)
:000000   47-49-46-38-37-61-00-02   00-02-E7-00-00-00-00-00   GIF87a@B@B.@@@@@
:000010   C9-C9-C9-8F-8F-8F-55-55   55-1B-1B-1B-E4-E4-E4-AA   ......UUU[[[....
:000020   AA-AA-70-70-70-36-36-36   FF-FF-FF-C5-C5-C5-8B-8B   ..ppp666........
:000030   8B-51-51-51-17-17-17-E0   E0-E0-A6-A6-A6-6C-6C-6C   .QQQWWW......lll
:000040   32-32-32-FB-FB-FB-C1-C1   C1-87-87-87-4D-4D-4D-13   222.........MMMS
:
:And I was wondering why the screwy ordering.  Sorry if I wasn't clear.

Have you tried creating a histogram of those files?  Try counting how many
pixels use color 0, how many pixels use color 1, etc.  If it turns out
that most of the pixels in the decoded picture use color 0, then that
would explain the screwy ordering.

-- 
Joe Smith (408)922-6220 | SMTP: jms@tardis.tymnet.com or jms@gemini.tymnet.com
BT Tymnet Tech Services | UUCP: ...!{ames,pyramid}!oliveb!tymix!tardis!jms
PO Box 49019, MS-C51    | BIX: smithjoe | CA license plate: "POPJ P," (PDP-10)
San Jose, CA 95161-9019 | humorous dislaimer: "My Amiga 3000 speaks for me."

zap@lysator.liu.se (Zap Andersson) (04/12/91)

rg@msel.unh.edu (Roger Gonzalez) writes:

>>In article <1991Apr9.142255.186@unhd.unh.edu> I write:
>>>Why have I seen gray scale (256) GIFs that don't have simple color maps like
>>>00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF
>>>
>>>Some of them are really strange...

>But what I meant was: 
>When looking at grayscale 256 GIFs, I would expect
>>>00 00 00 01 01 01 02 02 02 03 03 03 ... FE FE FE FF FF FF
>but have seen (in ppmtogif results, for example)

>000000   47-49-46-38-37-61-00-02   00-02-E7-00-00-00-00-00   GIF87a@B@B.@@@@@
>000010   C9-C9-C9-8F-8F-8F-55-55   55-1B-1B-1B-E4-E4-E4-AA   ......UUU[[[....
>000020   AA-AA-70-70-70-36-36-36   FF-FF-FF-C5-C5-C5-8B-8B   ..ppp666........
>000030   8B-51-51-51-17-17-17-E0   E0-E0-A6-A6-A6-6C-6C-6C   .QQQWWW......lll
>000040   32-32-32-FB-FB-FB-C1-C1   C1-87-87-87-4D-4D-4D-13   222.........MMMS

Guess (from looking in my own code to a similar hack) is that they created
a palette entry when they needed one, i.e. the palette-builder was unaware
that the image ewwas goung to be a grayscale thing, but entered things
into the palette when they were needed, just like my program does. 
(However, my program KNOWS when it's suppose to use gfrayscale if
I tell it too). 


>And I was wondering why the screwy ordering.  Sorry if I wasn't clear.

>I've been trying to reverse engineer what my problems are with my
>encoder by comparing my output with the results of rawtopgm | ppmtogif,
>but my comparison can only go so far before I hit a root code that 
>is different that mine because their greyscale map is different.

Would you like a DEcoder? I could easily mail one to you and help you
sort this out. SOMWHERE I have an encoder too, but dunno where :-(



>Why am I writing my own encoder when I could swipe code from everyone else?
>'Cause I'm a masochist.  So what if its a memory pig, I don't use hashing 
>yet, and its slower than shit?  I'm having fun!  

SOunds like a sound mind in an unsound body :-)

>If you are equally masochistic, and would like to give me a hand, my
>almost-but-not-quite encoder is on msel.unh.edu in /pub/gif for ftp-ing.

I enjoy better being SADISTIC (so we form a matching pair :-) so I
will MAIL you de encoders as soon as I find it. Har har har.


I>I'm really close.. I get one good scan line ou

t (using VUIMAGE on a
>PC), and then a shitty scan line, and then it stops. I think it barfs
>because I don't change bit sizes where it expects me to.  Sigh.

>-Roger
>-- 
>"The question of whether a computer can think is no more interesting
> than the question of whether a submarine can swim" - Edsgar W. Dijkstra 
>rg@[msel|unhd].unh.edu        |  UNH Marine Systems Engineering Laboratory
>r_gonzalez@unhh.bitnet        |  Durham, NH  03824-3525

THAT was a true statement indeed!

-- 
* * * * * * * * * * * * * * * * *          (This rent for space)
* My signature is smaller than  * Be warned! The letter 'Z' is Copyright 1991
* yours!  - zap@lysator.liu.se  * by Zap Inc. So are the colors Red, Green and
* * * * * * * * * * * * * * * * * Greenish-yellow (Blue was taken by IBM) 
--
* * * * * * * * * * * * * * * * *          (This rent for space)
* My signature is smaller than  * Be warned! The letter 'Z' is Copyright 1991
* yours!  - zap@lysator.liu.se  * by Zap Inc. So are the colors Red, Green and
* * * * * * * * * * * * * * * * * Greenish-yellow (Blue was taken by IBM) 

gwyn@smoke.brl.mil (Doug Gwyn) (04/13/91)

In article <1991Apr10.034324.28971@netcom.COM> marcos@netcom.COM (Marcos H. Woehrmann) writes:
-The document I have points out that most (all?) gif readers will break
-unless you only issue a clear code when the table is full, ...

The BRL-CAD "gif-fb" utility implements the GIF87a spec exactly.

-The GIF spec suggests that you sort the Global Color Table (aka palette)
-by order of importance (which most people of interpreted to mean in order
-of frequency).  "This assists a decoder, with fewer available colors, in
-choosing the best subset of colors..."

Yeah, the GIF specs are pretty terrible.

marcos@netcom.COM (Marcos H. Woehrmann) (04/16/91)

In article <15810@smoke.brl.mil>, gwyn@smoke.brl.mil (Doug Gwyn) writes:
> In article <1991Apr10.034324.28971@netcom.COM> marcos@netcom.COM (Marcos H. Woehrmann) writes:
> -The document I have points out that most (all?) gif readers will break
> -unless you only issue a clear code when the table is full, ...
> 
> The BRL-CAD "gif-fb" utility implements the GIF87a spec exactly.
> 

Yes, but the document I was referring to was the GIF89A spec;  and it
actually implies that not issuing a clear code when the table is full
was allowed under the GIF89A spec (just no one did it).

marcos

-- 
Marcos H. Woehrmann    {claris|apple}!netcom!marcos  |  marcos@netcom.COM

 Oh, I'm sure you've heard it all before, but remember it's not what you
 say, it's how you say it, and how much you're paid to do so, and besides
 who's listening anyway.  No one, that's who, because it's all been said
 and done and done and done and done to death.  Let's talk about art,
 said the fool to the idiot.  --Lydia Lunch