gr47@sdcc12.ucsd.EDU (gr47) (11/12/87)
------------------------------------------------------------------------------
12-NOV-1987 12:48:14.90 PST
I am about to write a Cray Fortran program to convert Cray datatypes to VAX
binary representations. This is needed to avoid having to convert very large
binary output datasets to ascii and back to binary for the transfer from Cray
to VAX.
My tentative flow diagram is as follows:
1) Generate large Cray unformatted Fortran sequential model output file
2) Convert to VAX binary and write to a Cray absolute file (absolute =
with no Cray control bytes) -- write VAX record control bytes as
necessary in this step
3) Transport to VAX with NETOUT program -- for binary files, NETOUT
spits out only VAX files with fixed length records (choice of either
512 or 4192 bytes)
4) Diddle with the VAX file header to make the file either VARIABLE,
STREAM or SEGMENTED
In 3) & 4), I have ruled out FIXED-LENGTH type records since use of
these would entail significant changes to VAX application programs already in
place.
NETOUT is the only reliable way to transport files at this point; I may be
able to FTP files later, but I am taking NETOUT limitations (i.e., FIXED
record output) as a given at this point.
I want to do most or all of the necessary conversion on the Cray due to
space/time considerations in transport between the machines and due to
space limitations on the VAX. I want to avoid having to do any record
restructuring, thus file duplicating, on the VAX.
Questions:
Are there any problems with changing VAX file attributes from
FIXED-LENGTH records to VARIABLE, STREAM or SEGMENTED by playing only
with the VAX file header (not reading/writing the file), assuming
appropriate control bytes have been placed in the file beforehand ?
Can someone fill me in on SEGMENTED records? Apparently, for very long
records, these comprise 'segments' of 2044 bytes each. Each segment
begins with 2 control bytes that indicates whether that segment is
the first, last, or only of that logical record, or none of the above.
What tells RMS how long the last or only (this is often less than max
length) segment of a record is ?
I would prefer to use SEGMENTED records since this would entail no changes
to my existing VAX Fortran programs. Additionally, there is a limit on the
size of a VARIABLE record (8191 longwords = 32764 bytes) which may be
surpassed in my application or that of others ... STREAM would seem to
be more suitable for my application than VARIABLE since there is no such
size limit (correct?). However, I have never used STREAM records
(un-VAX-like, :^) ); can they be used for unformatted files and are
there any drawbacks in using them for large binary files?
Thanks for any help.
Steve Piper
scripps inst. of oceanography a-020
la jolla, ca 92093
cdrgmv::piper2 (SPAN/HEPNET)
piper2%cdrgmv.span@sdsc (BITNET I think)
gr47%sdcc12@sdcsvax.ucsd.edu (ARPA/Usenet)
------------------------------------------------------------------------------leichter@VENUS.YCC.YALE.EDU ("Jerry Leichter") (11/16/87)
Are there any problems with changing VAX file attributes from
FIXED-LENGTH records to VARIABLE, STREAM or SEGMENTED by playing only
with the VAX file header (not reading/writing the file), assuming
appropriate control bytes have been placed in the file beforehand ?
The internal structure of a STREAM file is identical to that of a FIXED-LENGTH
record file - i.e., all the bytes in the file are "just data" - so this will
normally present no problems (but see below). If you are careful, you can
get away with this for VARIABLE files, too - but be aware that you are doing
something unsupported. If you get something wrong, RMS will be quite
unforgiving. (Generally, your process will unceremoniously disappear after a
non-fatal bugcheck. The only way to get any indication of what happened is to
look at the process final completion status. If the top process goes away on
you, you'll have to look at the accounting record. Usually, the best way to
play around with this sort of stuff is in a spawned subprocess.)
Can someone fill me in on SEGMENTED records? Apparently, for very
long records, these comprise 'segments' of 2044 bytes each. Each
segment begins with 2 control bytes that indicates whether that
segment is the first, last, or only of that logical record, or none
of the above. What tells RMS how long the last or only (this is often
less than max length) segment of a record is ?
SEGMENTED records are a construct of the FORTRAN I/O system, ONLY. RMS sees
the file as a variable-length record file (with a fixed control field?) and is
quite unaware of any special significance the FORTRAN I/O system assigns to
the control bytes. Check the FORTRAN documentation for details.
I would prefer to use SEGMENTED records since this would entail no changes
to my existing VAX Fortran programs. Additionally, there is a limit on
the size of a VARIABLE record (8191 longwords = 32764 bytes) which may be
surpassed in my application or that of others ... STREAM would seem to be
more suitable for my application than VARIABLE since there is no such size
limit (correct?). However, I have never used STREAM records
(un-VAX-like, :^) ); can they be used for unformatted files and are
there any drawbacks in using them for large binary files?
VMS STREAM files are record oriented files - they just have a record's length
determined by the position of a termination character, rather than a count.
I believe they have the same size limitations as other "styles" of files.
-- Jerry
------