[comp.lang.c] Fatal Bug in VAX C pointer indirection

corwin@bsu-cs.UUCP (Paul Frommeyer) (05/11/87)

(I mistakenly posted this to comp.lang.c++)

Several other students and us were working on a software engineering project
involving the display of 39 or more colors on a VT240. We were writing in
VAX C under VMS 4.5. Our program used several lists to keep track of all
the data it needed for window size, current display, etc. Our lists were
interconnected, and in setting up list elements for new windows we 
used extensive pointer indirection in calls to malloc(). 

What we found was, to say the least, interesting. Third-level pointer
indirection, such as list1->link2->link3->data worked fine in a call
such as list1->link2->link3 = (struct lista *) malloc (sizeof(struct lista));
Analysis in VAX DEBUG would reveal that link3 had a value, say, of 55080.

Now, if we attempted another call to malloc like
	pointer1->link2->link3 = (struct lista *) malloc... etc.
there was no problem. This would give link3 a value of, say, 55200 if
struct lista was 120 bytes long. But at one point we needed to
	pointer1->ptr2->ptr3->ptr4->ptr5->ptr6 = (struct lista *)... etc.

When we tried this, all hell broke loose! Malloc calls the system service
for allocating more virtual memory, in addition to performing some VAX C
housekeeping chores. Now, we would expect ptr6 to have a value of
55200, assuming malloc was giving contiguous memory, which it seems to do.
What we never expected was to get a value BETWEEN 55080 and 55200.

But that's exactly what happened! When malloc was called with more than
3 levels of pointer indirection, why, it just crashed! It would give a value
usually about 4 bytes more than the last byte for the structure pointed to by
link3, say about 55084. Malloc was overlapping virtual memory!! It took one 
of our team members a day to figure out what was going wrong! Nowhere in any of
the VAX C manuals that we read does it caution against this! Noncontiguous
allocation might be expected, but overlapping addresses was definitely dirty
pool on the part of the computer. After much discussion 
we concluded that the fault lay in the compiler; it was not 
generating correct VAX-11 native code to do multiple indirection. We don't
know what it IS doing, really, but we can see that it doesn't work properly.
It doesn't seem that the fault could be at the system service level, but it
is impossible to tell because we cannot see the generated object code (our
installation does not posess a disassembler).

Digital might say this is a feature to keep programmers from using excessive
indirection, but what pray tell is wrong with excessive indirection, 
especially when it allows manipulation of many interconnected lists?

Is there anyone out there who has encountered a similiar problem? Does
Unix C do this, too? We thought it might be the VAX hardware, but further
thought seemed to point to the compiler. There are already
more bugs than a bait store in the VAX C I/O library 
routines for fseek() and lseek(), but those are at least documented in the
VAX C manual. A limitation on indirection should be, too.

Would anyone at Digital care to comment on this? Are there any fixes?
We're both graduating and job hunting, so getting hold of us may be tricky.

Anyway, beware deep pointer indirection in VAX C. It's a killer.

Paul Frommeyer ({ihnp4,seismo}!{pur-ee,iuvax}!corwin)
Russ Walker ({ihnp4,seismo}!{pur-ee,iuvax}!mnp)
Computer Science Dept.
Ball State University
Muncie, IN 47306

gwyn@brl-smoke.ARPA (Doug Gwyn ) (05/11/87)

In article <582@bsu-cs.UUCP> corwin@bsu-cs.UUCP (Paul Frommeyer) writes:
>... When malloc was called with more than
>3 levels of pointer indirection, why, it just crashed!
>...
>Digital might say this is a feature to keep programmers from using excessive
>indirection, but what pray tell is wrong with excessive indirection, 
>especially when it allows manipulation of many interconnected lists?

For Chrissakes, malloc()'s internals don't know how you plan to use the
value returned from malloc()!  Whatever bug there is is most likely in
your use of pointers.  Are you aware that in
	stuff->link1->link2->link3->link4->link5->link6 = malloc(...)
all of  stuff  through  stuff->...link5  must have been correctly set up by
you beforehand?  malloc() cannot possibly take care of your links for you.

edw@ius2.cs.cmu.edu (Eddie Wyatt) (05/11/87)

   This letter really doesn't merit a reply, but I'll write one anyway.

   I take it that your machine is running VMS not Unix.  To say
you are working on a VAX really says nothing.

   I doubt very very very very much that there is anything wrong with
the VMS C compiler or malloc.  I personally don't work with a VMS
system but I have seen enough bus errors and segmentation faults to
guess at what your problem is.  Things to look at:

	o If you have a multitude of modules, make sure that they
	  are all recently compiled.  The problem that may arise here
	  is if you have made any changes to the struct lista then
	  those changes may not be reflected in code that was compiled
	  before the changes (ie sizeof(lista-old) != sizeof(lista)).

	o Do you ever free any objects. If so are you sure that you
	  are not accessing that object after you free it.

	o Do you ever allocate array in fact do you use any arrays?
	  If so are you sure that you do not over index the array.
	  The common problem here is:

			array = alloc_array(int,10);
			for (i = 0; i <= 10 ; i++)
				array[i] = 0;  /* boom */

	o A more bizzaire problem is over writing the code segments
	  of your program.  Fun bug to track down, let me tell
	  you.  It took me a whole day to find one in a program
	  that I did not write.  Anyway, problems can come from
	  accessing uninitialized pointer values. Example:

			foobar()
			    {
			    int *screw; /* :-) */

			    *screw = 2.0;
			    }

	  This may show up right away (segmentation fault or bus
	  error in Unix) or if you are so lucky it will show up
	  miles away in the in totally unrelated code (posibly illegal
	  instruction).

	o I haven't run up against this problem but I think you can even
	  do things like call free on memory not allocated with
	  malloc or calloc (like stack memory) and have things go crazy.

    Just because malloc doesn't return pointers inorder doesn't mean
there is anything wrong.  Malloc will search the free list before making
a system call for more memory (in Unix it is a call to sbrk which
increases the size of the programs data space).  I do believe that is
how "fast fit" works.

  Doesn't VMS have a symbolic debugger?  I thought it did, learn how
to use it fast.

-- 
					Eddie Wyatt

e-mail: edw@ius2.cs.cmu.edu