[comp.sys.mac.hypercard] What to do about > 32K of data?

davef@jessica.stanford.edu (David Finkelstein) (03/17/89)

I'm writing a stack that has to deal with > 32K of data.  Basically
I've got a really large text file located on a file server that I need
to read in and store, so I can display any part of the file to the
user.  [I need to load the entire file into HyperCard because network
access is too slow to read through the file to get to each line of
information I want.]  No problem really; I just use three fields.

However, the user of the stack can choose to view any part of the text
information, meaning potentiall *all* of the information, or large
disjoint subsets, at once.  I can't just throw the information into a
scrolling field; I'm limited to 32K.

The solution I have so far is to place the scroll bar of a scrolling
field alongside a non-scrolling field.  I also place my own buttons on
top of the scroll buttons.  Then as the user scrolls the scrolling
field (which has the same number of lines as I wish my non-scrolling
field had), I manually fill in the lines of the non-scrolling field
with whatever information should be on that line (reading in from the
3 fields mentioned above).  It works, but it's slow, especially since
I have to watch for times when the user moves the scroll box or clicks
in the scroll bar (there's a noticable delay before my non-scrolling
field gets updated).

Any ideas on what I can do to display > 32K of information to the
user?  Any ideas on how I can improve the performance of the setup I
have?


Thanks...

***************************************************************
David Finkelstein		|davef@jessica.stanford.edu
Academic Information Resources	|
Stanford University		|     Just say "please."

dan@Apple.COM (Dan Allen) (03/23/89)

In article <938@Portia.Stanford.EDU> davef@jessica.stanford.edu (David Finkelstein) writes:
>I'm writing a stack that has to deal with > 32K of data.  Basically
>I've got a really large text file located on a file server that I need
>to read in and store, so I can display any part of the file to the
>user.  [I need to load the entire file into HyperCard because network
>access is too slow to read through the file to get to each line of
>information I want.]  No problem really; I just use three fields.
>
>However, the user of the stack can choose to view any part of the text
>information, meaning potentiall *all* of the information, or large
>disjoint subsets, at once.  I can't just throw the information into a
>scrolling field; I'm limited to 32K.

The problem is trying to squash linear text into a hypertext model.  I
think many people try to make HyperCard a word processor, which it is
not.

If the person has a really large text file to read, then read it with
the MPW Shell which can open and work with files in the megabytes
without any problem.

If someone want to use a hypertext system, then the information should
be broken down into hypertext sized chunks.  Hypertext is supposed to be
fast, and HyperCard has a card as the unit of information.  It may seem
limiting (and indeed it is when thought of as a word processor), but if
the new paradigm of hypertext is used, then having cards is seen as a
bonus.  Let me explain...

Each card in HyperCard should have a reasonable amount of information,
where reasonable has been tentatively defined as that information that
can all be seen at once in 512x342 pixels.  Scrolling fields extend this
range, but were put in by Bill Atkinson only under duress: his personal
stacks have no scrolling fields in them, and few of mine do.  Hypertext
has (in our implemetation in HyperCard) a different model than the
endless scrolling done in a word processor.  If you like scrolling, use
a word processor; if you like queries, use an SQL database; if you like
clicking and free associating, then use HyperCard.

There are great advantages to looking at your information in light of
card chunks of material.  Presenting the browser with more than a cards
worth of information is not good because the user gets overwhelmed: too
much information.  The best hypertext stacks present material in a
card's worth of space, and then the user is given the option to explore
different areas in greater detail.  If the topic at hand is music, for
example, a timeline of music could be shown with Classical, Jazz, Pop,
and Country shown.  The user could then explore more about the type of
music that he/she likes rather than having to explore the types they
don't like.

Hypertext gives people choices and freedom to move about, not the need
to read and wade through 32K chunks of text (at 2K per page thats 16
pages of text!) trying to find what they want.

Now I realize that we have archives of big chunks of text which are not
immediately and automatically convertible to hypertext-style stacks.
That's why we gave people scrolling fields in my mind: they are a
holding area while experts parse through the text and break it up into
non-scrolling fields worth of info.

END OF PERSONAL DAN ALLEN SOAPBOX ON THE PROPER USE OF HYPERTEXT.

Dan Allen
HyperCard Team
Apple Computer

bayes@hpfcdc.HP.COM (Scott Bayes) (03/24/89)

> 
> [...]
> 
> The problem is trying to squash linear text into a hypertext model.  I
> think many people try to make HyperCard a word processor, which it is
> not.
> 
> If the person has a really large text file to read, then read it with
> the MPW Shell which can open and work with files in the megabytes
> without any problem.
> 
> If someone want to use a hypertext system, then the information should
> be broken down into hypertext sized chunks.  Hypertext is supposed to be
> fast, and HyperCard has a card as the unit of information.  It may seem
> limiting (and indeed it is when thought of as a word processor), but if
> the new paradigm of hypertext is used, then having cards is seen as a
> bonus.  Let me explain...
> 
> [...]
> 
> Dan Allen
> HyperCard Team
> Apple Computer

Do you guys really expect to keep HyperCard viable? Or do you just want
people to move over to SuperCard and its ilk?

I can see either approach as being reasonable to you.  I can't see users
staying with HyperCard if it won't do things they want for no other
reasons than an ideology Apple holds on "what's right for HC."

If there are technical or compatibility problems, I can buy that.  If
you've just decided that you know what's RIGHT, forget it.  I'll use
something else.

Remember the first Macs.  Little non-expandable toasters.  Jobs may have
thought that was RIGHT for the Mac.  If he and Apple had stuck by that,
this newsgroup probably wouldn't have 10% of the traffic it does today.

Scott Bayes
concerned

dan@Apple.COM (Dan Allen) (03/25/89)

In article <10520021@hpfcdc.HP.COM> bayes@hpfcdc.HP.COM (Scott Bayes) writes:
>Do you guys really expect to keep HyperCard viable? Or do you just want
>people to move over to SuperCard and its ilk?
>
>I can see either approach as being reasonable to you.  I can't see users
>staying with HyperCard if it won't do things they want for no other
>reasons than an ideology Apple holds on "what's right for HC."
>
>If there are technical or compatibility problems, I can buy that.  If
>you've just decided that you know what's RIGHT, forget it.  I'll use
>something else.

Thank you for your concern, and as always, I was somewhat misunderstood.
What I meant to get across is that different types of applications have
different data models and are useful for different things.  Word
processors are great for vast quantities of linear text, databases are
great for record/field stuff, etc.  HyperCard is not a word processor or
a database, although it does have elements of each of these standard
applications.  HyperCard IS a hypertext application: something new.  All
I wanted people to understand is that this model is fundamentally
different than a word processing or database model.  Therefore the data
is to be treated differently.  Now that doesn't mean we don't want to
continue to improve HyperCard in any way.  We are improving HyperCard
dramatically.  Technical problems can and are being overcome.

As far as SuperCard goes, I really have no comment since I have not even
seen it.  If SuperCard is better, then I encourage everyone to go out
and buy it and use it and dump HyperCard.  Fine.  But I understand that
although it may have the look of HyperCard, it does not have the feel or
performance.  Before everyone gives up on HyperCard, just do two things
for me: a) use SuperCard, and b) use HyperCard 2.0 when it is released
later this year.  All I ask is a fair comparison.

In the meantime, continue to send those cards and letters of suggestions
and improvements.  I think you'll like 2.0.

Dan Allen

THESE COMMENTS ARE DAN ALLEN'S PERSONAL COMMENTS AND ARE IN NO WAY 
REPRESENTATIVE OF THE OFFICIAL VIEW OF APPLE COMPUTER.
  

ollef@osiris.sics.se (Olle Furberg) (03/26/89)

>Before everyone gives up on HyperCard, just do two things
>for me: a) use SuperCard, and b) use HyperCard 2.0 when it is released
>later this year.  All I ask is a fair comparison.
>I think you'll like 2.0.
>
>THESE COMMENTS ARE DAN ALLEN'S PERSONAL COMMENTS AND ARE IN NO WAY
>REPRESENTATIVE OF THE OFFICIAL VIEW OF APPLE COMPUTER.
>
 
I've never heard of 2.0. What's the main diff. between it and all these 1.x?
 
 
              /Olle

alan@mtxinu.COM (Alan Tobey) (03/28/89)

> Thank you for your concern, and as always, I was somewhat misunderstood.
> What I meant to get across is that different types of applications have
> different data models and are useful for different things.  Word
> processors are great for vast quantities of linear text, databases are
> great for record/field stuff, etc.  HyperCard is not a word processor or
> a database, although it does have elements of each of these standard
> applications.  HyperCard IS a hypertext application: something new.  All
> I wanted people to understand is that this model is fundamentally
> different than a word processing or database model.  Therefore the data
> is to be treated differently.  

Dan, I'm  sure that for YOU, and for the other developers of HyperCard,
"Hypercard is a hypertext application" is correct and complete.  But
for some of us that's a pretty narrow view.  For some of us, for
example, HyperCard is PRIMARILY "Macintosh programming for the rest
of us," and the fact that is IMPLEMENTED as a hypertext system is often
incidental.  Without HyperCard, there's NO WAY I could have produced any
of the dozens of personal Mac applications that I and my company use every
day.  Some of these applications USE the hypertext "features" (as I'd
prefer to describe them), some don't.  To FORCE HyperCard into being
ONLY a "hypertext application" is to ignore that it does some OTHER
things better than most other tools around.  As it seems to some of us,
arbitrarily crippling our use of the best tool we have (for example, by
limiting the amount of text we can handle in one bite) is both short-
sighted and unnecessary.  It's like telling a young girl that
she was conceived to grow up into a software engineer, so she shouldn't ever
bother with playing baseball (since we fixed her legs so she can't ever
run past second base!).

ech@pegasus.ATT.COM (Edward C Horvath) (03/28/89)

Dan Allen wrote,
> What I meant to get across is that different types of applications have
> different data models and are useful for different things...HyperCard
> is not a word processor or a database...

From article <807@mtxinu.UUCP>, by alan@mtxinu.COM (Alan Tobey):
> Dan, I'm  sure that for YOU, and for the other developers of HyperCard,
> "Hypercard is a hypertext application" is correct and complete.  But
> for some of us that's a pretty narrow view.  For some of us, for
> example, HyperCard is PRIMARILY "Macintosh programming for the rest
> of us,"...

But that does not place any requirement on Dan or the rest of the HC dev
team to try to satisfy all your program development needs.  In particular,
the XFCN/XCMD escapes allow third parties to add lots of of functionality to
your front ends, as Oracle has done with their database.  You may have to
learn SQL as well as HyperTalk, but there's good reason to expect that
any effort to make HT a "real" database language would add comparable
complexity which we ALL pay for.  "The rest of us" don't need to have
Apple spend its resources reinventing the database when Oracle already
has the expertise, inclination, and a shipping product.  Acius and others
are following suit if you have some religious objection to Oracle (hi, Mike!).

So be creative: the HyperCard platform, third-party back-ends, and your
own customized front-ends comprise an environment of extraordinary power
and flexibility.  All three components have a role to play.

=Ned Horvath=

bayes@hpfcdc.HP.COM (Scott Bayes) (03/29/89)

Dan, thank you for the response. Sorry if I came on too strong.

I have found some of the limitations of HC to seem somewhat arbritrary. Whether
they are arbitrary, or are dictated by the internal model is something I can by
definition have no competent opinion on. I DID think I heard you saying "It's
this way because that's how a HyperText system SHOULD be" rather than "because
that's the only reasonable way we can make it work."

BTW, to call HC, in its present incarnation, a "hypertext application", seems
a bit farfetched. Will 2.0 give us real text links. Huh? Huh? Please!

I don't currently use HC as a HyperText system, as it is so painful to do
hypertexty things with. Graphics seems well served (within the limitations of
a paint model, which I accept), whereas text seems somewhat more casually
dealt with. Excellent for a get-it-out-the-door-and-start-people-wanting-it
first revision, but to put hypertextual meat on the "for-the-rest-of-us" bones,
I think you gotta have text links.

Youse guys done good. I bought a Mac very soon after I saw _The Computer
Chronicles_ review of HC in late summer '87; HC was a very big reason.

Thanks
Scott Bayes

dan@Apple.COM (Dan Allen) (03/30/89)

In article <2606@osiris.sics.se> ollef@sics.se (Olle Furberg) writes:
>>Before everyone gives up on HyperCard, just do two things
>>for me: a) use SuperCard, and b) use HyperCard 2.0 when it is released
>>later this year.  All I ask is a fair comparison.
>>I think you'll like 2.0.
>>
>>THESE COMMENTS ARE DAN ALLEN'S PERSONAL COMMENTS AND ARE IN NO WAY
>>REPRESENTATIVE OF THE OFFICIAL VIEW OF APPLE COMPUTER.
>>
> 
>I've never heard of 2.0. What's the main diff. between it and all these 1.x?

GOOD!  2.0 is supposed to be a secret!  We are working on it right now
and it will be released/announced when we finish it, probably later this
year.  I really should not say much more about it ("We are not
authorized to comment on unannounced products") but I just wanted to let
the net readers know that we are not sitting idly by...

Dan Allen
HyperCard Team
Apple Computer

dan@Apple.COM (Dan Allen) (03/30/89)

In article <807@mtxinu.UUCP> alan@mtxinu.COM (Alan Tobey) writes:
>Dan, I'm  sure that for YOU, and for the other developers of HyperCard,
>"Hypercard is a hypertext application" is correct and complete.  But
>for some of us that's a pretty narrow view.  For some of us, for
>example, HyperCard is PRIMARILY "Macintosh programming for the rest
>of us," and the fact that is IMPLEMENTED as a hypertext system is often
>incidental.  Without HyperCard, there's NO WAY I could have produced any
>of the dozens of personal Mac applications that I and my company use every
>day.  Some of these applications USE the hypertext "features" (as I'd
>prefer to describe them), some don't.  To FORCE HyperCard into being
>ONLY a "hypertext application" is to ignore that it does some OTHER


Good point.  HyperCard in its original form (WildCard) had no
programming language in it.  It was strictly a fast text browser.  Bill
used Switcher and MacPaint to do the graphics, and then he said, "What
the heck" and put in the paint tools.  Then a little later someone
wanted it to print, so they put in (minimal) printing.  Finally someone
said, "Hey, put a language into it" and thus HyperTalk was added in the
final 9 months.

Now we find that many people are using it as a programming language
rather like MacApp was going to be.  With this new insight we are
rethinking things and some future unnumbered version of HyperCard or
HyperCard replacement (years from now pie in the sky kind of thing) may
be more of a programming environment with the other things added as
icing on the cake.

Thanks for the input.


***********************************************************
** Dan Allen             **   dan@apple.COM (Unix mail)  **
** Software Explorer     **   ALLEN.DAN (AppleLink)      **
** HyperCard Team        **   20525 Mariani Ave. MS 22AE **
** Apple Computer        **   Cupertino, CA 95014        **
***********************************************************
** Sam:    "You know what they say, 'You can catch more  **
**          flys with honey than with vinegar.'"         **
** Woody:  "I don't mean to butt in, but you can catch   **
**          the most with dead squirrels."               **
***********************************************************

dan@Apple.COM (Dan Allen) (03/30/89)

In article <10520022@hpfcdc.HP.COM> bayes@hpfcdc.HP.COM (Scott Bayes) writes:
>BTW, to call HC, in its present incarnation, a "hypertext application", seems
>a bit farfetched. Will 2.0 give us real text links. Huh? Huh? Please!

Let's say that you will like HC 2.0.  Thanks for your input.  We want to
make HC as best as it can be, but my earlier comments simply meant that
we do not want to become the next FullWrite (throw it all in for 750K of
code) by throwing in the kitchen sink.  We can't do this because we need
to still run in a 1 MB machine, since that's what MOST people have.
FullWrite, just to pick one of many apps, is having a very hard time of
this these days, for example.

Dan Allen
Apple

fozzard@boulder.Colorado.EDU (Richard Fozzard) (03/30/89)

In article <10520022@hpfcdc.HP.COM> bayes@hpfcdc.HP.COM (Scott Bayes) writes:
>
>BTW, to call HC, in its present incarnation, a "hypertext application", seems
>a bit farfetched. Will 2.0 give us real text links. Huh? Huh? Please!
>
I second the motion!


ps: how about a visual indicator of which text has links? I mean like
bold, greyed, dotted box, something other than OPTION-COMMAND (and 
preferably other than "normal" text styles).



========================================================================
Richard Fozzard
University of Colorado				"Serendipity empowers"
fozzard@boulder.colorado.edu

czei@quanta.eng.ohio-state.edu (Michael S. Czeiszperger) (03/30/89)

>In article <10520022@hpfcdc.HP.COM> bayes@hpfcdc.HP.COM (Scott Bayes) writes:
>
>BTW, to call HC, in its present incarnation, a "hypertext application", seems
>a bit farfetched. Will 2.0 give us real text links. Huh? Huh? Please!


How do you like the HyperText functions in the Developer's Stack?  
With that method, you can either double click or option click on
the linked word.  You can signify that the word is linked by making
it all CAPS, putting *'s around it, etc.  Not optimal, but certainly 
workable.  I havn't actually written any scripts this way, having
just gotten the stack today, but the examples they give work fine.

-- 
Michael S. Czeiszperger   | "He seemed to be able to see the threads
Systems Analyst           |  which bind the Universe together..."
The Ohio State University | 2015 Neil Ave, Columbus, OH 43210
ARPA:czei@icarus.eng.ohio-state.edu  PAN:CZEI (614) 292-0161

bayes@hpfcdc.HP.COM (Scott Bayes) (03/31/89)

>>In article <10520022@hpfcdc.HP.COM> bayes@hpfcdc.HP.COM (Scott Bayes) writes:
>>
>>BTW, to call HC, in its present incarnation, a "hypertext application", seems
>>a bit farfetched. Will 2.0 give us real text links. Huh? Huh? Please!
>
>
>How do you like the HyperText functions in the Developer's Stack?  
>With that method, you can either double click or option click on
>the linked word.  You can signify that the word is linked by making
>it all CAPS, putting *'s around it, etc.  Not optimal, but certainly 
>workable.  I havn't actually written any scripts this way, having
>just gotten the stack today, but the examples they give work fine.

Which functions were those? I browsed the stack, but nothing sprang
to my eye. They might do the job, but I really don't want to mess with
the body of the text; just perform some click on a selected word
and get a chance to define a link to some location in another card.

To Dan,

Though it may not sound like it, I'm not really after the kitchen sink. I
think a few well-considered enhancements can improve HC a lot. From my
particular bias:

on theMessage
  send BIAS to APPLE
end theMessage

	text links, on the order of ease-of-use of current button linkage
	remove 16-bit limits (32K fields, etc)
	local state for background buttons
	speed improvements esp. for large numbers of fields/buttons per card
	tighten up language, remove some accidental idioms
	resizable cards
	let DAs be non-modal (I have to close ScrapBook to access HC)
	Script Manager support (are you Script Manager compatible already?!?)
	user-definable objects (you figure out what that means. I can't yet)

Other stuff might be nice, but not necessary:
	
	spreadsheet type fields (tab columns or some such mechanism??)
	global state for background fields (can almost do with paint text)
	font/size/style control within fields
	object-oriented draw (can always use McDraw, SuperPaint, etc)
	
Some things I wouldn't want to see you waste your time on:

	more functions, etc (what are XFCNs/XCMDs for??)
	yet more variations on find, sort, etc
	arbitrary-shape buttons, fields, etc
	
My inner model for text links:

import a chunk of text, free-form stuff, into a field. Existing import 
  buttons OK
read through it.
click with some tool to start a textlink
got to the target card for the link, and click on something there to complete
  the link. The target is not just the card, but some object or part of one
  on the card.

When traversing links:

the link is visible, but text hasn't been modified (autohilite by HC is OK)
click with browser tool
get transported to target, with hilight on/in target object

The point is I don't need to "chunk up" the text, or modify it to show
existence of links. Just "point and shoot." I'm lazy.

>
>-- 
>Michael S. Czeiszperger   | "He seemed to be able to see the threads
>Systems Analyst           |  which bind the Universe together..."
>The Ohio State University | 2015 Neil Ave, Columbus, OH 43210
>ARPA:czei@icarus.eng.ohio-state.edu  PAN:CZEI (614) 292-0161

Thank you both for your indulgence.

Scott Bayes

science@nems.ARPA (Mark Zimmermann) (04/17/89)

((in reply to davef@jessica.stanford.edu query in HH V3.4))
you may want to check out my indexer/browser software, esp. TEXAS and/or
TEX in HyperCard ... look around archives, check CompuServe DL6, <info-mac>
or BMUG or BCS-MAC or send me a disk & self-addressed stamped envelope ...
essentially, my software builds complete inverted index to arbitrarily-large
text file (up to hundreds of megabytes, anyway) and lets you browse through
the index, call up key-word-in-context displays, or retrieve chunks of the
full text as desired ... might be adaptable to solve some of your problems
with 'What to do about > 32K of data?' that you asked about in HyperHackers
V3.4 ..... ^z
-------