[net.micro.cbm] Tokenize text files.

cuda@ihuxf.UUCP (Mike Nelson) (08/11/86)

I'm trying to figure out how to tokenize a text file that I downloaded
with the modem.  Right now it is a CBM ascii sequential file.  I under-
stand that there is a "tokenize basic text" vector at 772, 773. (I think
that is where it's at)  I tried a small program in the immediate mode 
that filled the region at 512 with a line number and a REM statement
and did a sys to $a560.  (The vector at 772,773 is actually $a57c.  Ray
Miller sent out a little blurb on tokenizing text by starting at location
$a560.)  

It seems that I remember reading somewhere that there is an easy way 
to do this.  The solution I have in mind is to write a machine language 
routine to open the file and run it through the keyboard buffer just 
as though it had been typed in.  Any other ways this could be done?  
What about the program or direct mode in register 157?  Any help would
be appreciated.

Mike Nelson
ihuxf!cuda
AT&T Bell Labs

fred@cbmvax.cbm.UUCP (Fred Bowen) (08/19/86)

> I'm trying to figure out how to tokenize a text file that I downloaded
> with the modem.  Right now it is a CBM ascii sequential file.
> ...
> The solution I have in mind is to write a machine language 
> routine to open the file and run it through the keyboard buffer just 
> as though it had been typed in.  Any other ways this could be done?  
> What about the program or direct mode in register 157?  Any help would
> be appreciated.
> 
> Mike Nelson
> ihuxf!cuda
> AT&T Bell Labs

I hate myself for it, but I just had to try it.  Sort of like a brain teaser
but worse.  The little program below does the trick on a c128.  Now you can
edit your BASIC programs with a real editor.  (The CBM Editor that comes with
the c64 assembler is not a real editor :-).  What it does is opens a channel
to a SEQ disk file, and makes it the default input channel.  BASIC then sucks
'keys' directly from the file instead of the keyboard/screen, merrily crunchin'
and a linkin' as she reads.  We simply have to wedge IMAIN to catch the status
set by reading the last line from disk so that normal I/O can be restored after
the 'load'.  I do not suggest you try this on a c64- the 64 does not do a
clean job of allowing I/O redirection!  Note this program expects everything,
including line numbers, to be in petscii (same for as if you had listed your
BASIC program to disk via OPEN/CMD/LIST).  I really hate myself for this.


1 REM  LOAD A BASIC LIST (SEQ) FILE, CRUNCH & LINK IT ON C128
2 REM  19AUGUST86   FRED BOWEN
3 REM
10 READ A$,B$: ADR=DEC(A$): PRG=ADR
20 DO: POKE ADR,DEC(B$): ADR=ADR+1: READB$: LOOP UNTIL B$="END"
30 INPUT "[cls]FILENAME TO LOAD & LINK"; F$
40 POKE DEC("34A"),13: POKEDEC("34B"),13: POKEDEC("D0"),2
50 PRINT "[cls][d][d]OPEN 8,8,8,";CHR$(34);F$;",S,R";CHR$(34);":SYS";PRG;",,8"
60 PRINT "CLOSE 8[home]";
70 NEW
100 DATA 1800
110 DATA 86,15,20,C6,FF,AD,02,03,AE,03,03,85,FB,86,FC,A9
120 DATA 20,A2,18,8D,02,03,8E,03,03,A2,00,86,7F,86,90,EA
130 DATA A5,90,D0,03,6C,FB,00,20,CC,FF,A9,00,85,15,A5,FB
140 DATA A6,FC,8D,02,03,8E,03,03,6C,02,03,END

where [cls] is CHR$(147) clear screen and [d] is CHR$(17) cursor down.

What the ML looks like:

	01800  86 15    STX $15		;BASIC's CHANNL
	01802  20 C6 FF JSR $FFC6	;CHKIN
	01805  AD 02 03 LDA $0302	;save BASIC's IMAIN vector
	01808  AE 03 03 LDX $0303
	0180B  85 FA    STA $FB
	0180D  86 FB    STX $FC
	0180F  A9 20    LDA #$20	;point IMAIN to us
	01811  A2 18    LDX #$18
	01813  8D 02 03 STA $0302
	01816  8E 03 03 STX $0303
	01819  A2 00    LDX #$00
	0181B  86 7F    STX $7F		;clear RUNMOD
	0181D  86 90    STX $90		;clear STATUS
	0181F  EA       NOP 

	01820  A5 90    LDA $90		;begin our IMAIN wedge- check STATUS
	01822  D0 03    BNE $1827 	;branch if not clear
	01824  6C FA 00 JMP ($00FB)	;else continue normal IMAIN

	01827  20 CC FF JSR $FFCC	;CLRCH
	0182A  A9 00    LDA #$00
	0182C  85 15    STA $15		;clear BASIC's CHANNL too
	0182E  A5 FA    LDA $FB		;restore IMAIN
	01830  A6 FB    LDX $FC
	01832  8D 02 03 STA $0302
	01835  8E 03 03 STX $0303
	01838  6C 02 03 JMP ($0302)	;done. return to READY
-- 
Fred Bowen			uucp:	{ihnp4|seismo|caip}!cbmvax!fred
				arpa:	cbmvax!fred@seismo.CSS.GOV
				tele:	215 431-9100

Commodore Electronics, Ltd.,  1200 Wilson Drive,  West Chester,  PA,  19380

fred@cbmvax.cbm.UUCP (Fred Bowen) (08/20/86)

The lastest issue of Transactor (Nov 86) contains a letter describing how
to do this (as well as execute command files!) via a one liner.  This little
ditty thanks to Frank DiGioia, Stone Mountain, GA:

	OPEN 2,8,2,"file,S,R": POKE 812,73: POKE 781,2: SYS 65478

Just hit RESTORE when it finishes, or end file data with the following (note
NO line number):

	CLOSE 2: POKE 812,47: SYS 65484

This will also work on a 128, if you do the following:

	OPEN 2,8,2,"file,S,R": POKE DEC("32C"),DEC("3C"): SYS DEC("FFC6"),,2
	...
	CLOSE 2: POKE DEC("32C"),DEC("22")
-- 
Fred Bowen			uucp:	{ihnp4|seismo|caip}!cbmvax!fred
				arpa:	cbmvax!fred@seismo.CSS.GOV
				tele:	215 431-9100

Commodore Electronics, Ltd.,  1200 Wilson Drive,  West Chester,  PA,  19380