garry@batcomputer.tn.cornell.edu (Garry Wiegand) (05/01/88)
We are attempting - and have been attempting for some time now - to
port our large 3D graphics subroutine library onto the Mac. The Mac
is driving us rapidly nuts.
We have worked around the bugs in Quickdraw, the obscurities in the
documentation, the lack of a usable debugger, the solitary system
error message (a round black object with a fuse :-), and *some* of the
perversities of MPW/Aztec/Lightspeed/etc etc etc. We have not yet
succeeded in working around/through the bureaucracy at Apple so that
we can ask questions "officially". Now we've run into a wall, and we
could use some help.
The MPW C compiler is not very good, but unfortunately it is Apple's
choice, so it has to be the primary focus of our Mac support. MPW C
has the custom of loading the A5 register with a pointer to all of
"global data" during program start-up. All accesses are then via a
15-bit offset to A5, which never changes. ("Global data" consists of
everything that's not code and not dynamically allocated - global
variables, static variables, strings, and floating-point constants (12
bytes each!).) The bottom line is that with MPW you can only ever
have 32K (2^15) of fixed data. Doesn't matter if your Mac has
megabytes and megabytes of memory; 32K is the static data limit.
Question 1:
Anybody know of any magic workarounds to this limit, short of
damaging all of our source code?
Question 2:
Has anyone heard any rumors that this MPW bug might be fixed
sometime soon? (There's a rumored "3.0" release coming, but as
far as we can tell a "32K" fix is not included in it.)
Question 3:
We *might* be able to squeeze our own data into 32K, but the
user program that links to our library would surely push
things back over the limit. If we could separate the user's A5
usage from our library's, then we might be OK. Has anyone ever
gone through the exercise of segmenting a program into *multiple
separate code resources* on the Mac? (The system apparently
does this all the time, but the details of the "message-
passing" involved are obscure to us.)
Any help would be greatly appreciated.
garry wiegand (garry@oak.cadif.cornell.edu - ARPA)
(garry@crnlthry - BITNET)
PS:
As much as I'm complaining about Apple's software, they're the only
PC/workstation company I've encountered so far that actually *cares*
about software, and thinks about how things *ought* to be done - they
don't just copy boring 15-year-old Unix ideas. "Inside the Mac" is
interesting reading even if you never intend to use a Mac.
(Good ideas unfortunately do not automatically make a system "programmer
friendly". "Ghetto" debugging, complicated data structures, no error
checking, and no error messages - Apple, you're a bunch of nitwits!)wetter@tybalt.caltech.edu (Pierce T. Wetter) (05/01/88)
In article <4625@batcomputer.tn.cornell.edu> garry@oak.cadif.cornell.edu writes: >We have worked around the bugs in Quickdraw, the obscurities in the What bugs? >The MPW C compiler is not very good, but unfortunately it is Apple's >choice, so it has to be the primary focus of our Mac support. MPW C >has the custom of loading the A5 register with a pointer to all of >"global data" during program start-up. All accesses are then via a >15-bit offset to A5, which never changes. ("Global data" consists of >everything that's not code and not dynamically allocated - global >variables, static variables, strings, and floating-point constants (12 >bytes each!).) The bottom line is that with MPW you can only ever >have 32K (2^15) of fixed data. Doesn't matter if your Mac has >megabytes and megabytes of memory; 32K is the static data limit. Actually this is common to all languages under MPW. Global data is referenced as an offset from a5. On the 68000 thats 16 bits, on the 020 its 32 bits. Read the object code format description if you want more details. I agree it can be annoying, but there are some things to remember. 1. Static variables consist of a bunch of DC.L statements. Therefore 20K of empty static arrays will take up 20K of disk space. 2. Quickdraw globals have to be addressed this way anyways. > >Question 1: > Anybody know of any magic workarounds to this limit, short of > damaging all of our source code? If its for a 68020 machine, you might be able to ignore the messages produced by link, but don't quote me. >Question 2: > Has anyone heard any rumors that this MPW bug might be fixed > sometime soon? (There's a rumored "3.0" release coming, but as > far as we can tell a "32K" fix is not included in it.) >Question 3: > We *might* be able to squeeze our own data into 32K, but the > user program that links to our library would surely push > things back over the limit. If we could separate the user's A5 > usage from our library's, then we might be OK. Has anyone ever > gone through the exercise of segmenting a program into *multiple > separate code resources* on the Mac? (The system apparently > does this all the time, but the details of the "message- > passing" involved are obscure to us.) Segmenting a program simply requires a "#define ___SEG___ segname" message. All procedures following such a message will be placed in 'segname' by the linker. This won't help anyways. The real problem is that the 68000 can only have a 16 bit offset from a5. Note that the limit should be 64K and a5 should point to the center of your data, but no compiler writers are smart enough to think of that. As for your data, you're probably allocating large arrays. Why not just allocate a pointer and write some initilization code to allocate the space at startup. Since this is a library you might want to do the following: Replace static double MEMORY[10000000000] with static double *MEMORY=0; { if (MEMORY=NULL) MEMORY=malloc( 10000000000* sizeof(double)) MEMORY[5]=PI; } >(Good ideas unfortunately do not automatically make a system "programmer > friendly". "Ghetto" debugging, complicated data structures, no error > checking, and no error messages - Apple, you're a bunch of nitwits!) They do give you the number though, you just have to look it up. (If its listed. My Favorite Unlisted Error: -4001, LaserPrinter out of Paper) Pierce Wetter ---------------------------------------------------------------- wetter@tybalt.caltech.edu Race For Space Grand Prize Winner. ----------------------------------------------------------------- Useless Advice #986: Never sit on a Tack.
earleh@eleazar.Dartmouth.EDU (Earle R. Horton) (05/03/88)
In article <4625@batcomputer.tn.cornell.edu>, garry@batcomputer.tn.cornell.edu (Garry Wiegand) writes: > We are attempting - and have been attempting for some time now - to > port our large 3D graphics subroutine library onto the Mac. The Mac > is driving us rapidly nuts... You and everybody else! > bytes each!).) The bottom line is that with MPW you can only ever > have 32K (2^15) of fixed data. Doesn't matter if your Mac has > megabytes and megabytes of memory; 32K is the static data limit. > I have before me an ad by Manx Software Systems which advertises that their C compiler/development system is capable of "unlimited data size and unlimited program size", including static and global areas. If you get the "Developer System" you also get "MPW compatibility" whatever that means. (Presumably source level compatibility, maybe even the #include files have the same names!) I have no experience with this company, but for $299 for the Developer System, it certainly seems that investigating their claims might be profitable. Personally, I think that if you are programming in C or any other language which allows for dynamic memory allocation and data structures, then you really do not need even 32k of global data. Try using malloc() to allocate space for your floating point arrays. The problem is possibly more one of program style than of the limits imposed by MPW. You might also consider putting string constants in a resource; Apple provides the 'STR#' resource specifically for this purpose. This is no defense of Apple, I am merely trying to state that maybe if you thought things over, you might find that you don't need 32k of static space after all. You might even consider using "Handle"s to store your data and working with the Macintosh Memory Manager! > (Good ideas unfortunately do not automatically make a system "programmer > friendly". "Ghetto" debugging, complicated data structures, no error > checking, and no error messages - Apple, you're a bunch of nitwits!) Unfortunately, this is true. I have heard Macintosh users make fun of the IBM PC and MSDOS for the 640k memory limit, but the size limit for "small" model programs is 64k, not 32k, and most suppliers of development systems support the large data model, where static data can be as big as the machine can address. Also, the debugger supplied with MSDOS, "debug", sick as it is, can write out files and load a program, features not even considered for "MacsBug" on the Macintosh. If you think Macintosh data structures are complicated, try looking at some "X" code sometime! Yuucchhh! Try compiling a version 11 X program that you got from Charlie's BBS using the header files that came with the version 11 X that came with your machine. Chaos! At least with Macintosh you get a single point of control for the data structures that are used, even though nobody can agree on the size of an "int" or the names of #include files, and there is serious doubt sometimes that there is anybody home at the point of control. (If you want a real good laugh, take a look at Technical Note #88 "Hit Signals" sometime: SigBigA6 EQU $FFFFFFFF ;maximum positive A6 value ... MOVE.L #SigBigA6,A6 ;make A6 valid for Signal Indeed! ) From the user's perspective, the Macintosh is a friendly computer which is always responsive to his wishes through easy to use menus and controls, and which requires no tiresome memorization of commands to learn to use. From the programmer's perspective, it is either a challenge to program, or a never-ending nightmare of obscurity and anti-support from the manufacturer. Which one it is depends to some extent on the attitude of the programmer involved, I think. You have to realize that the Macintosh is a user machine, and not a programmer's machine. On the bright side, the level at which you do things on the Mac is usually so low, even compared with UNIX, that you really have a remarkable degree of control over what your program can do. I am primarilly a scientist, myself, and have been involved for the past six months in trying to write a simulation program for use in Pulsar research, using the Mac II. I am beginning to realize that having a beautiful user interface is maybe not so important for a scientist as is the ability to get something done in a reasonable amount of time, and with a reasonable amount of data security. The Macintosh certainly falls short in both of these areas. I will accept flames on all points except the last: There is no data security for your files on the Mac, and if you don't have three copies of every file which you use, you are courting disaster. As a computer user, I heartedly recommend the Mac for terminal emulation, word processing, spreadsheets, and most "user-type" applications. As a programmer and as a scientist, I cannot at this time recommend it for anything other than as a boat anchor. -- ********************************************************************* *Earle R. Horton, H.B. 8000, Dartmouth College, Hanover, NH 03755 * *********************************************************************
clive@drutx.ATT.COM (Clive Steward) (05/06/88)
From article <8816@eleazar.Dartmouth.EDU>, by earleh@eleazar.Dartmouth.EDU (Earle R. Horton): > Personally, I think that if you are programming in C or any other > language which allows for dynamic memory allocation and data > structures, then you really do not need even 32k of global data. Try > using malloc()... Nice idea, Earle, but there are some number of difficult situations for this. In particular, the very useful program generators yacc and lex generate data arrays, which are used for finite state machines. If the programs you generate with them are large, so are these arrays. I have a particular program in mind which has about 150k or so of them. There are solutions any programmer can think of, but none of them pretty. Clive
oster@dewey.soe.berkeley.edu (David Phillip Oster) (05/08/88)
In article <7327@drutx.ATT.COM> clive@drutx.ATT.COM (Clive Steward) writes: >for this. In particular, the very useful program generators yacc and >lex generate data arrays, which are used for finite state machines. So, write the data arrays to a resource using a tool. Then in your actual program, just as you init your unitinitialized arrays with NewPtr, you init your initialized arrays with: fooarray = (FooArrayType) * GetResource('GNRL', 128); Simple, no? just make the resource purgable and locked, and this gives you now problem. Sure it is a pain that the compiler didn't do it for you, but when the workaround takes one line, why bitch? Copyright (c) 1988 by David Phillip Oster, All Rights Reserved --- David Phillip Oster --When you asked me to live in sin with you Arpa: oster@dewey.soe.berkeley.edu --I didn't know you meant sloth. Uucp: {uwvax,decvax,ihnp4}!ucbvax!oster%dewey.soe.berkeley.edu
guido@cwi.nl (Guido van Rossum) (05/08/88)
In article <23952@ucbvax.BERKELEY.EDU> oster@dewey.soe.berkeley.edu.UUCP (David Phillip Oster) writes: >In article <7327@drutx.ATT.COM> clive@drutx.ATT.COM (Clive Steward) writes: >>for this. In particular, the very useful program generators yacc and >>lex generate data arrays, which are used for finite state machines. > >So, write the data arrays to a resource using a tool. [...] >Sure it is a pain that the compiler didn't do it for you, >but when the workaround takes one line, why bitch? Aren't we exaggerating a bit, Mr. Oster? The generators mentioned produce their output as C code (mostly data initializations), so how do you suggest we create the resource in the first place? It has to be run through a compiler at some point... Sure, the code generated is simple in structure so we could scan it by other means, but then it's no longer a one-liner, is it? The real problem here is not to get data loaded into your program (your trick dies just fine) but to translate a *large* initialized data statement into bits. -- Guido van Rossum, Centre for Mathematics and Computer Science (CWI), Amsterdam guido@piring.cwi.nl or mcvax!piring!guido or guido%piring.cwi.nl@uunet.uu.net
oster@dewey.soe.berkeley.edu (David Phillip Oster) (05/08/88)
In article <306@piring.cwi.nl> guido@cwi.nl (Guido van Rossum) writes: >The generators mentioned >produce their output as C code (mostly data initializations), so how do >you suggest we create the resource in the first place? It has to be run >through a compiler at some point... My apologies, you are right. I was confused because I've used this technique on a grammar for C and although _all_ the initialized structures were over 32k, no _single_ structure was anywhere close. If you really do have single structures over 32k, of course you can break them into smaller pieces and concatenate them in the tiny one-shot C programs that write their single large piece of initialized data into a resource fork (I usually just have them write to themselves, which is already open, and the default destination of AddResource anyway. Then I manually copy the resource to its final destination using ResEdit.) If the yacc grammar is changing, then I write a small tool to do the resource moving. More commonly, you are porting a working program from some other machine, so this job only needs to be done once. (Why, you could write a one-shot on that other machine that writes the data out as a binary file, port the binary file to the mac, and just copy it into a resource in a tiny program. (Beware of intel et al byte swapping if that other machine isn't a Vax or a 68000.)) It is a pain. The workaround is still simple and short.
dan@Apple.COM (Dan Allen) (05/09/88)
David Oster's advice is sound and in fact is what Apple has done for versions of Yacc and Lex that are used internally. They are built with the standard MPW C 2.0 and seem to work fine in building things like the CFront C++ preprocessor. DISCLIAMER: NO, these Apple internal versions of Lex and Yacc are NOT available for the public. YES, I wish they were. Lean on your favorite product marketing people and tell them you'd like to be able to buy Lex and Yacc and Awk for MPW and maybe they'd figure out a way to sell it. The reason that they are not available is simple: it is a legal issue. The lawyers are going to ruin this industry, I'll tell you, and here again we see a licensing problem rather than an engineering problem. Something to do with Unix and AT&T, if I recall right. GOOD NEWS: There is another way around global data problems with big arrays other than saving the arrays as locked resources. (By the way, you can create those arrays with Rez code that is almost identical to the C code you would have written anyway, and it works very nicely.) If you are running into lots of large arrays because you are doing numerical analysis stuff (finite element analysis, Runge-Kutta integration of sets of PDQs, etc.), then the latest version of "Numerical Recipes in C: The Art of Scientific Computing" by Press, Flannery, Teukolsky, and Vetterling (Cambridge Press) has a great solution to the conformant array problem which ends up putting all arrays on the heap using malloc. Not only does the solution fix the global data problem, but it also fixes the conformant array problem, all in about two lines of code. The crux of the solution is: double **array; array = SpecialRoutine(10,10); Where the SpecialRoutine allocates pointers to each row of the array. It takes up a little extra overhead in terms of memory (4 bytes per row of the matrix), but speeds up array indexing as well as solves the conformant array problem. Great stuff!!! Dan Allen Software Explorer Apple Computer #include <stddisclaimer.h>
wetter@tybalt.caltech.edu (Pierce T. Wetter) (05/10/88)
>DISCLIAMER: NO, these Apple internal versions of Lex and Yacc are NOT >available for the public. YES, I wish they were. Lean on your favorite >product marketing people and tell them you'd like to be able to buy Lex Bison which is Gnu's version of Yacc has been ported to MPW (by me) and will me posted to the net soon. As for lex, feel free to port flex, which was just posted to net.sources.unix. Pierce Wetter ---------------------------------------------------------------- wetter@tybalt.caltech.edu Race For Space Grand Prize Winner. ----------------------------------------------------------------- Useless Advice #986: Never sit on a Tack.
clive@drutx.UUCP (05/11/88)
From article <9395@apple.Apple.Com>, by dan@Apple.COM (Dan Allen): > David Oster's advice is sound Presumably with the small detail taken care of that's been reminded. > and in fact is what Apple has done for > versions of Yacc and Lex that are used internally. Do these internal versions automate resourcing the data arrays? In program development, which is much more interesting than porting, the code generating those arrays changes often. It's definitely worth it. > They are built with > the standard MPW C 2.0 and seem to work fine in building things like the > CFront C++ preprocessor. (awk also mentioned). When will we see this!!! Soon, please! All of them, please! Especially C++! Especially the C++ hooked up to MacApp that's been talked about... > > DISCLIAMER: NO, these Apple internal versions of Lex and Yacc are NOT > available for the public. YES, I wish they were. Lean on your favorite I'd like to think you'd find an adequate market. I would hope ownership wouldn't be a problem -- isn't it the code for these which is owned (hence you must be running workalikes with MPW), rather than the ideas? The tools have some evident usefulness for straight Mac program development, though syntactically driven solutions are usually (and properly) subordinate to visual ones here. I have some ideas about how the two can helpfully mix for applications other than word processors et al, though these may just be my own interests, and not main line. It's also true that lex can be used for pattern recognition on other data than words, and very effectively to build translators of all kinds, like across application environments. But also, a Mac with hard drive is quite an adequate single-person replacement for a typical Unix software development environment, in terms of power. I've written a lot of code on mine. With the items mentioned above integrated with MPW's toolset, the picture would be pretty complete for an affordable personal workstation, on which to do commercial work. At a very reasonable price. Presumably this is a market of interest. And so also might be the market for Macs created by software of interesting new kinds being written for them. Think about the next time you have to write a controller for XYZ industrial automation process. Wouldn't you like to use a Mac for the front-end, and probably processing engine too? Accelerators like yacc and lex, and a building tool like C++ with MacApp classes to get the basic Mac interface and utilities properties easily, would make this a pretty going proposition. And I'm sure an eminently more likeable one than O/P/S2. N'est-ce pas? Il faut user le Mac! (for work; play is another topic altogether, and I'm starting to be pretty sure the two don't much mix, in this life....) Clive Steward
alan@pdn.UUCP (Alan Lovejoy) (05/13/88)
In article <6333@cit-vax.Caltech.Edu> wetter@tybalt.caltech.edu.UUCP (Pierce T. Wetter) writes: > Actually this is common to all languages under MPW. Global data is >referenced as an offset from a5. On the 68000 thats 16 bits, on the 020 its >32 bits. Read the object code format description if you want more details. >I agree it can be annoying, but there are some things to remember. > ... The real problem is that the 68000 can only have >a 16 bit offset from a5. Note that the limit should be 64K and a5 should point >to the center of your data, but no compiler writers are smart enough to think >of that. Using A5 as the global frame pointer is standard practice on 68k systems. If effect, A5 is being used as a "segment register". Global variables are referenced using 16-bit *signed* displacements from the address in A5. This allows for easy relocation of the static data, and simplifies the linking process. The size of this displacement on the 68020 is also 16 bits (if the register indirect with displacement addressing mode is used). The code to add two global int variables into a third automatic variable would be: -- 68k assembly -- -- pseudo-C -- MOVE.L var1(A5),D0 ;D0 = *(A5 + var1) ADD.L var2(A5),D0 ;D0 += *(A5 + var2) MOVE.L D0,var3(A6) ;*(A6 + var3) = D0 It's true that the 68020 can have a 32-bit displacement from an address register, but it involves an addressing mode not available on the 68k. If you use that adressing mode, your code won't run on an mc68000 or mc68010. Also, the displacement immediately follows the instruction and takes up either 16 or 32 bits of memory. It has to be fetched by the processor as part of the instruction stream. It takes more cycles to fetch 32 bits than it does to fetch 16 bits (or else it uses up space in the pipeline that could otherwise have been used for additional work). And the 'address register indirect with 32-bit displacement' addressing mode requires 32 bits JUST FOR THE INSTRUCTION, plus another 32 bits for the displacement. Most 68k instructions fit in only 16 bits, apart form offsets and immediate (constant) values. Still, in many cases this may be a good solution for MacII owners or developers. There is a well-known technique that can and should be used by Mac compilers to 'break the 32k barrier'. Actually, there are several, but let me outline my favorite one: Imagine you are a linker stitching together an executable from a collection of object modules. You notice that one of the data structures in static (global) storage is a megabyte in size. If this is the only object in static storage, there is no problem. There is also no problem if the total size of the other global objects is less than 32k (or 64k if you're semi-intelligent). But what do you do when you encounter a second global object a megabyte in size? Let's assume that there are only two global objects, each one megabyte. The base address of the first will be put into A5. Where can we store the base address of the second so that the program can find it? We could assign another register to such duty, but we only have 8 address registers, one of which is already being used (by the hardware!) as the top-of-stack pointer, another must be used as the local frame pointer, two must be free at all times as work-registers, A5 is already spoken for, leaving only two "free" address registers (if you think this is a cramp, try programming an Intel CPU). Using extra registers can give us 192k of global memory space, but the cost is rather high because those address registers are needed for more efficient pointer and array manipulation. You probably wonder why the compiler didn't provide a fix for your problem, so that you (the linker) wouldn't have to. The C compiler *can't* fix this problem, because the C compiler usually can't see the program as a whole, but only a function or so at a time. It can't possibly know the total size of the global data, since each function linked in can declare its own global data without asking permission of the "main" function or any other. There is a reason why Pascal programs are structured the way they are! No, I'm not suggesting you switch to Pascal (or Modula-2, or Ada, or..). So even if you (the linker) decide to allocate additional address registers as global data pointers, you have yet another nasty problem: you'll have to go through each object module and fix all references to A5 so that they refer to the register you have chosen as the pointer to the segment containing the object being referenced by that instruction. Unfortunately, all the solutions suffer from this problem. The goal is to make such fix-up jobs as painless for the linker as possible. In my opinion, the best technique is as follows: instead of storing the data for the second megabyte-sized object at some offset from A5, store its address there instead. In other words, the linker can create a segment table or object table which is stored somewhere +/- 32k bytes from A5. So the 500000th byte of megabyte object #2 would be accessed thusly: MOVE.L bigObjectAddrsess(A5), A4; move the base address of a large ; object into register A4 MOVEI.L #500000,D0 ; initialize D0 to 500000 MOVE.B (A4,D0),D1 ; move the 500000th byte of the large ; object to register D1 This requires an additional 16-bit instruction with a 16-bit offset field (32 bits total) be inserted into the code by the linker for referencing the largest global objects (the smaller objects should be given first priority for allocation in the "main" segment). Hopefully, some compiler/linker suppliers will be moved to work on this problem. -- Alan Lovejoy; alan@pdn; 813-530-8241; Paradyne Corporation: Largo, Florida. Disclaimer: Do not confuse my views with the official views of Paradyne Corporation (regardless of how confusing those views may be). Motto: Never put off to run-time what you can do at compile-time!
lsr@Apple.COM (Larry Rosenstein) (05/17/88)
In article <7812@drutx.ATT.COM> clive@drutx.ATT.COM (Clive Steward) writes: > >When will we see this!!! Soon, please! All of them, please! Especially C++! >Especially the C++ hooked up to MacApp that's been talked about... MPW C++ and a compatible version of MacApp should arrive in the fall. You should be able to write MacApp programs in any mixture of Object Pascal and C++ (and Assembler). -- Larry Rosenstein, Object Specialist Apple Computer, Inc. 20525 Mariani Ave, MS 27-AJ Cupertino, CA 95014 AppleLink:Rosenstein1 domain:lsr@Apple.COM UUCP:{sun,voder,nsc,decwrl}!apple!lsr