October 14, 1999
At Last - Virtual Memory
on the Apple II Computers
Unleashed to the General Public
By Charles T. 'Dr. Tom' Turley
Contributing Editor - GS WorldView
This article and it's reprint contents may be used for any needs without
restriction.
The other day I was browsing thru my old 1990 Nibble magazine issues
when I
noticed a Cover Article - "Virtual Memory" and simply had to check
it out. Well,
after reading it completely and doing a test run of the program, I
decided to
engage in some research to locate the author and get his premission
to include
it in this article as a complete reprint IN ASCII text and with the
ShrinkIt
archived inclusion of the program files also.
The ShrinkIt archive named: A2VM.BXY can be found within this folder.
Your
welcome to download the archive and use the program files for your
own needs in
anyway you see fit to use them. During my telephone conversation with
the
program author, Dr. John R. Vokey today obtaining his consent to make
the
reprint of his article and the program files archive available in the
October,
1999 issue of GS WorldView for use by the A2 community and its users,
he and I
discussed our hopes that some swift and motivated 'A2 Wizard' might
find this
article and program of some value in their efforts to develop 'Virtual
Memory'
in some fashion with GS/OS also, such as RAM Doubler and other like
programs
offer other computer platforms and their operating systems. Dr. Vokey
also
mentioned in our conversation that he still has an Apple IIgs that
he uses in
his University research efforts. We both agreed that 'Virtual Memory'
for the
GS/OS was indeed possible to do, should somebody care to go for it.
So, is their
anybody out there that cares to give it a try?
I've done an OCR to ASCII text of the entire article from the August,
1990 issue
(included below) and I also did an RGB color scan of the magazine cover,
as a
JPEG graphic file: Nibble.Cover.8.90.jpg - included in this folder.
I hope you
will find this article reprint and program files public release of
some value
with your needs.
For those of you who desire varification of consent from Dr. Vokey to
reprint
the article and distribute the program in the ShrinkIt archive, he
can be
reached by email at: vokey@hg.eleth.ca
------------------------------------------------------------------------------
Cover Feature Reprint from: August 1990 Nibble Magazlne
------------------------------------------------------------------------------
Virtual Memory
By John R. Vokey, Ph.D.
Applesoft BASIC was written in the days when Apple II computers shipped
with a
standard 16K of memory and when people used cassette tape for external
storage.
Consequently, it contains few facilities to take advantage of the extended
RAM
and disk capacities common today. In particular, no matter how much
memory is
available, Applesoft BASIC limits the theoretical size of any data
array to
64K, and the practical size to substantially less after memory for
the operating
system, file buffers, display pages, and the Applesoft BASIC interpreter
itself
is taken into account.
Virtual memory lets applications assume very large memory spaces Ñ usually
many
times larger than the working memory capacity of the computer. The
process is
common to many mainframe and minicomputer systems. The name derives
from the
fact that the application is able to function virtually, that is, as
if large
amounts of memory were available, when, in fact, only a small portion
of
memory is being used at any moment; the remainder of the application's
"memory"
is resident on some external storage device, such as a hard disk, and
portions
of it are swapped in and out of memory as they are needed.
Using the Program
Virtual Arrays patches Applesoft BASIC under ProDOS to allow this common
version
of virtual memory. The patch allows for both real and integer virtual
arrays of
up to 16 megabytes each, limited only by the capacity of the external
storage
device (i.e., hard disk, RAM disk, or floppy disk) used to hold the
contents of
the virtual arrays. Aside from the initial dimensioning of the virtual
arrays
(described below), most Applesoft programs should execute without modification,
typically requiring less RAM than before, and with the added advantage
of
automatic storage to disk of the array data. The patch will function
on any
Apple II (or compatible) using ProDOS and BASIC.SYSTEM. Further, it
is
transparent to normal (i.e., nonvirtual) arrays; only those integer
and real
arrays that are too large for main memory (or for which automatic storage
to
disk is desired) need be declared virtual, and all others may be dimensioned
and
used as usual.
__________________________________________________________________________________
Dr. John R. Vokey, Department of Psychology, University of Lethbridge,
Lethbridge,
Alberta, Canada T1K 3M4. This program is compatible with ProDOS.
To install the Virtual Array patch, use the command
BRUN VIRT.MEM.OBJ
in either immediate or deferred mode. The installer code relocates the
patch to
six free pages of high memory, and interfaces the patch to Applesoft
via the
ampersand vector, and installs a modification to Applesoft's CHRGET
routine.
It completes installation by displaying a copyright notice on the text
screen.
The installer automatically daisy-chains the ampersand vector so that
other
installed ampersand utilities will continue to function. It also checks
whether the patch has already been installed; if so, it foregoes installation
and redisplays the copyright notice.
The Virtual Array patch adds three new commands to the language. Some
examples
of their use are demonstrated in the BASIC program in Listing 4. (Contributing
Editors note: Listing 4 is the program file named VIRT.MEM.DEMO, which
can be
found in the ShrinkIt archive named: A@VM.BXY) Each command is invoked
with the
ampersand.
1. The syntax for the first command is
& SAVE pathname, arrayname{%} (dim {, dim})
The & SAVE command defines the specified real or integer array as
virtual, and
creates (or retrieves) a virtual array file under the specified ProDOS
pathname.
Pathname may be any string expression that evaluates to a legal ProDOS
pathname,
and any array declaration that would be legal in an Applesoft real
or integer
array DIMension statement may be used to dimension the array in the
& SAVE
statement, with the addition that array sizes greater than 64K, up
to 16
megabytes (disk space permitting), may be declared as virtual (see
note 1).
Because of Applesoft BASIC's internal handling of subscripts, however,
no single
dimension of the array may exceed 32767.
For example, the statement
& SAVE "/RAM/EXAMPLE", X(100, 100)
in either immediate or deferred mode will dimension the real array,
X, to
contain 50K (101 x 101 elements, with five bytes per element) of data,
create
the corresponding 50K file on the "/RAM" volume under the filename
"EXAMPLE",
and set each of the 10,201 elements of the array to zero. From that
point on,
the array may be used in exactly the same fashion as any normal, real
array in
Applesoft BASIC.
If a virtual array file of the specified pathname already exists the
data will
be retrieved from it, rather than a new fiie created. In this way,
virtual array
data may be shared across programs (or even within the same program
by array
variable "aliases"~. Virtual array files are given the "VAR" (i.e.,
$FD) file
type, with the auxiliary type code of "VA" ($D6Cl), for "Virtual Array."
Although the patch will correctly discriminate between VARiable files
created
with BASIC.SYSTEM's STORE command and those created by the virtual
array patch,
correctly rejecting the former, BASIC.SYSTEM does not, as it does not
check the
auxiliary code of these files.
Virtual array files contain only the numerical data for the virtual
array;
neither the array name, its dimensions, nor its type are stored in
the file.
Although this simplicity allows virtual arrays of different names (and
dimension
structure, size, or type) to access the same virtual array file, it
also
requires that you keep track of the dimension structure, size, and
type of the
virtual array files if they are to be reused or shared across programs.
For real
arrays, each element of the virtual array is stored to the disk file
in
Applesoft's five-byte, packed floating-point format; integer virtual
array data
require two bytes per element. These formats result in files that are
more
space-efficient and are more rapidly accessed than the more usual sequential
text file.
2. & STORE arrayname {, arraayname} The & STORE statement,in
either immediate or
deferred mode "flushes" the specified virtual arrays. That is, the
particular
blocks of data of the virtual arrays currently in memory are written
to disk,
usually preparatory to a program end. Normally, any changes to the
virtual array
data automatically are written to the array file as blocks of array
data are
swapped between the file and memory during the normal course of using
the
virtual array. However, because the block of array data currently in
memory is
not updated in the file unless another block from the file is swapped
into
memory, any changes to the last block accessed would not be saved to
the file.
The & STORE command may be used to ensure that the specified virtual
arrays are
updated before the end of a program (or before chaining or loading
another
program, or CLEARing the variables). For example, the statement
& STORE X, Y, Z%
will flush the virtual arrays X, Y, and Z%.
3. & CLEAR {var {,var}}. The & CLEAR statement, in either immediate
or deferred
mode, clears the specified variables of any type, simple or array (including
virtual array), from memory, freeing the memory formerly occupied for
other
data. If you specify no variables, the program performs a regular Applesoft
CLEAR. For virtual arrays, only the array in RAM is cleared; the virtual
array file is not disturbed. The command is adapted from the "DISPose"
utility
developed by Kaner and Vokey (see note 2). So that the command can
distinguish
between simple and array variables of the same name, arrays must include
array
indices, the values (expression or numerical) of the indices are immaterial,
but they are evaluated and therefore must be within the bounds declared
for the
array. If a specified variable does not exist, it is created and then
CLEARed.
For example, the statement
& CLEAR A, B%, C$, D(O,O)
will clear the values of the simple variables A, B %, and C$ and the
array
(normal or virtual) D from memory.
Error Handling
The virtual array patch supports all Applesoft and ProDOS BASIC.SYSTEM
error
messages and Applesoft error trapping using the "ONERR GOTO" statement.
Virtual
arrays function just like normal arrays, returning error messages like
"bad
subscript" and "redimensioned array" under the appropriate circumstances.
In
addition, the & STORE statement will return an "OUT OF DATA" error
if the
specified array does not exist (see note 3), and will return a new
error
message, "?ARRAY ERROR" (new error code 128), if the specified array
exists but
is not a virtual array.
ProDOS BASIC.SYSTEM errors behave similarly. Specifying an existing
pathname
that is not a virtual array filename in an & SAVE statement, for
example,
results in a "DUPLICATE FILE" error, and removing a disk volume containing
a
currently active virtual array file results in a "FILE NOT FOUND ERROR"
the next
time the virtual array file needs to be accessed.
Hints on Using Virtual Arrays
Although it is not required as long as a prefix has been set, you should
specify
complete pathnames in & SAVE statements; if you do, the virtual
array files will
continue to be accessed (assuming the volume remains online) even if
the default
prefix changes.
To maximize program speed, virtual array files should be created on
the fastest
storage device available, such as a RAM disk or a hard disk, and programs
should
be structured to minimize the swapping of data blocks between the array
file
and memory. Note that the data elements for Applesoft arrays (and,
hence,
virtual arrays) are stored so that the first index changes fastest,
while the
last index changes slowest, contrary to convention. Thus, statements
such as
"FOR I = 1 TO 100: FOR J = 1 TO 100: X(I,J) = 0: NEXT J,I" will access
the array
data out of order and, consequently, require more frequent swapping
of data
blocks than would the same statements with the indices of the array
variable
reversed (i.e., X(J,I)).
Although virtual arrays require only 512 bytes of user memory, because
they are
easily recovered from the virtual array file (with the & SAVE statement),
they
may be purged from memory (with the & CLEAR statement) when not
in active use to
free the memory for other purposes. Remember to & STORE the virtual
array files,
if necessary, before purging them to ensure that all data blocks have
been
updated.
Because of the convenience of automatic storage to a disk file, and
the ability
to recall the arrays easily both within and between programs, you may
often want
to make even small arrays virtual.
About the Virtual Memory Demo Program
The Applesoft BASIC program shown in Listing 4 provides some examples
on using
each of the commands of the virtual memory package. The program itself
does very
little; it installs the virtual memory patch, creates (or retrieves)
a 50K
(i.e., X(99,99)) real, virtual array file, and then allows you to view
and edit
the contents of the file. However, there are some features of the program
to
note.
First, the program attempts to create the virtual array file on the
"/RAM"
volume. Any failure to do that (e.g., no such volume, volume full,
etc.), will
result in an attempt to create the file on the default or currently
prefixed
volume. If that attempt also fails, the program halts. The delay encountered
while creating the array reflects the time it takes to write five-byte
zeros to
each of the 10,000 elements of the array file. Incidentally, the lower
right
corner of the display presents a continuous update of the amount of
user memory
available after various operations are performed. Note how small the
change is
in this value as the 50K virtual array is created.
Second, assuming that the virtual array file has not been deleted and
is still
online, re-running the program will demonstrate how the virtual memory
patch
uses an existing virtual array. In this case, note how the startup
is now
virtually instantaneous, and that all of the element of the array are
recovered
intact.
Third, the array is indexed backward, as X(column, rou rather than as
the
conventional X(row, column). This wa done for speed, as discussed earlier.
Fourth, when a shift in rows or columns is necessary, the updating of
the screen
display of the array data is somewhat slow; hence the "One moment,
please"
message each time it is required. This slowness is partially a function
of the
"Print Using" subroutine, written entirely in Applesoft (an adaptation
of one I
found in an old publication from Apple (see note 4)) used to format
the data,
and, particularly if the virtual array is stored on a floppy diskette,
partially
a result of the fact that each row of the data display requires that
a new block
of virtual array data be loaded from the virtual array file.
Last, throughout the program, ProDOS BASIC. SYSTEM errors, including
those
generated by the virtual memory commands, are trapped and handled with
BASIC's
ONERR GOTO and corresponding RESUME commands.
Entering the Program
The source code for the patch, written in 6502 assembly language, is
shown in
Listing 1. (Contributing Editors Note: Listing 1 is included as the
file:
VIRT.MEM.OBJ within the ShrinkIt archive: A2VM.BXY). The source code
makes
extensive use of macros, which are shown in Listing 2 (Contributing
Editors
Note: Listing 2 is included as the file: VIRT.MACS within the ShrinkIt
archive:
A2VM.BXY). If you do not have a macro assembler, either manually substitute
the
code for the macros at each point of their invocation in Listing 1
or directly
enter the resulting object code shown in Listing 3. (Contributing Editors
Note:
Listing 3 is included as the file: VIRT.MEM.OBJ within the ShrinkIt
archive:
A2VM.BXY). If you choose the latter route, note that there are two
code
sections: the virtual array patch installer from $4000-$41AE, and the
virtual
array patch itself from $4200-47F2. Despite the break between them
save them as
a single file with the command
BSAVE VIRT.MEM.OBJ,AS4000,LS7F2
For help entering Nibble listings, see the Typing Tips section in this
issue.
(Contributing Editors Note: This is of no value or need as the files
so listed
have been included within the ShrinkIt archive: A2VM.BXY).
HOW THE PR0GRAM WORKS
The virtual array patch works by intercepting Apple soft's attempts
to locate
array data. It accomplishes this interception through the aforementioned
modification to the interpreter's CHRGET routine (see note 5). If the
attempt
involves a virtual array, the patch first determine whether the particular
element to be located currently resides in memory; if so, it sets the
appropriate vectors to point to the element and exits to the interpreter.
If
not, it writes the current block of data in memory to the appropriate
section of
the virtual array file, thereby recording any changes to the data,
and then
reads in the correct block of data from the virtual array file, once
again
setting the appropriate vectors before exiting to the interpreter.
In either
case, Applesoft uses the array element as if it had located it itself.
If a
block of array data is swapped into memory in this way, the array element
requested by the interpreter is loaded as the middle element of the
block to
allow both forward and backward stepping from that point of about 50
real array
elements and over 125 integer array elements before a new block of
array data
will be required to be swapped into memory.
Memory Use
All of the parameters associated with a given virtual array, such as
its
dimensions, disk filename, and so on are stored within the array descriptor
in
memory. The initial few bytes of this descriptor Ñ the variable name
and offset
to the next variable Ñ are identical to that of nonvirtual arrays 90
that the
Applesoft memory management routines function normally. The first difference
occurs in the byte that normally contains the number of dimensions;
for virtual
arrays, this byte is set to zero to flag to the patch that the array
is indeed
virtual and needs special handling by the patch. This special handling
includes
not only swapping blocks of data between the virtual array file and
memory, but
also computing the desired array element to allow for array sizes greater
than
BASIC's 64K limit. The remainder of the descriptor consists of a byte
containing
the true number of dimensions, followed, using two bytes each, by the
sizes of
each of the dimensions in reverse order. The next three bytes contain
the record
number of the array element currently heading the data block in memory,
the
number (in two bytes) of bytes in the block in memory, followed by
the pathname
of the array file in Pascal string form (a length byte, followed by
the ASCII
codes of the string in successive bytes). The descriptor is followed
by a
512-byte block of memory to hold the current block of array data. That's
it! No
matter how large the virtual array, no further memory is used.
One further memory-saving trick was used. Normally, to read or write
any disk
file under ProDOS, a lK buffer to hold information about the file is
required to
be set aside when the file is first opened. To avoid this lK overhead
every time
a virtual array file is to be accessed, the patch uses the floating
buffer maintained
by BASIC.SYSTEM for its CAT and CATALOG commands and as a temporary
data buffer for user text-file READ and WRITE commands. Although the
patch's use
of this buffer does save memory, it comes at the cost of having to
check every
time as to whether it is currently buffering data for a user text file
Ñ and, if
so, having to write the buffered data to the file in question before
using it to
read or write a virtual array file. It also requires that the patch
maintain its own
ProDOS parameter tables, rather than use those located within BASIC.SYSTEM.
As a consequence, although they use the same data buffer, the patch
does not
interfere with BASIC.SYSTEM; reading and writing virtual array data
are
transparent to your concurrent reading and writing of disk files.
Notes
1. A single, 5.25-inch floppy diskette is limited under ProDOS to 280
blocks
(140K). Even this limited capacity, however, allows for a virtual array
of over
28,000 real or 71,000 integer elements, which is considerably more
than that
allowed for in Applesoft.
2. Kaner, H. C. & Vokey, J. R. Subroutine Master. Nibble, November
1985, pp.
46-75.
3. Applesoft's STORE statement, as with the SAVE statement, is a legacy
of the
early days of the Apple II when tape was the storage medium for programs
and
data. It is used (without the &) to store real and integer data
arrays to tape
through the tape port on the backplane of the Apple II, II Plus, and
IIe. (As a
BASIC.SYSTEM command, it also is used to STORE the variables of an
Applesoft
BASIC program.) Because of its similar purpose, both the syntax of
the command
and the error messages it invoked have been retained in its reincarnation
as a
virtual array command. The STORE statement, for example, also results
in an "out
of data" error if the specified array does not exist.
4. Grossley, J. (1979). Print Using Simulator, Contact, 6, 8.
5. Further details on this method of patching Applesoft may be found
in Kaner,
H. C., & Vokey, J. R. "Modify- ing Apple's floating point BASIC:
An &
interpreter without the &." Compute!, 1982.
-end of reprint: Nibble Magazine, August, 1990, pages 44 thru 47-