October 14, 1999
 
At Last - Virtual Memory on the Apple II Computers
Unleased to the General Public 

By Charles T. 'Dr. Tom' Turley

Contributing Editor - GS WorldView

This article and it's reprint contents may be used for any needs without
restriction. 

The other day I was browsing thru my old 1990 Nibble magazine issues when I
noticed a Cover Article - "Virtual Memory" and simply had to check it out. Well,
after reading it completely and doing a test run of the program, I decided to
engage in some research to locate the author and get his premission to include
it in this article as a complete reprint IN ASCII text and with the ShrinkIt
archived inclusion of the program files also.

The ShrinkIt archive named: A2VM.BXY can be found within this folder. Your
welcome to download the archive and use the program files for your own needs in
anyway you see fit to use them. During my telephone conversation with the
program author, Dr. John R. Vokey today obtaining his consent to make the
reprint of his article and the program files archive available in the October,
1999 issue of GS WorldView for use by the A2 community and its users, he and I
discussed our hopes that some swift and motivated 'A2 Wizard' might find this
article and program of some value in their efforts to develop 'Virtual Memory'
in some fashion with GS/OS also, such as RAM Doubler and other like programs
offer other computer platforms and their operating systems. Dr. Vokey also
mentioned in our conversation that he still has an Apple IIgs that he uses in
his University research efforts. We both agreed that 'Virtual Memory' for the
GS/OS was indeed possible to do, should somebody care to go for it. So, is their
anybody out there that cares to give it a try?

I've done an OCR to ASCII text of the entire article from the August, 1990 issue
(included below) and I also did an RGB color scan of the magazine cover, as a
JPEG graphic file: Nibble.Cover.8.90.jpg - included in this folder. I hope you
will find this article reprint and program files public release of some value
with your needs.

For those of you who desire varification of consent from Dr. Vokey to reprint
the article and distribute the program in the ShrinkIt archive, he can be
reached by email at: vokey@hg.eleth.ca

--------------------------------------------------------
Cover Feature Reprint from: August 1990 Nibble Magazlne
--------------------------------------------------------

Virtual Memory

By John R. Vokey, Ph.D.

Applesoft BASIC was written in the days when Apple II computers shipped with a
standard 16K of memory and when people used cassette tape for external storage.
Consequently, it contains few facilities to take advantage of the extended RAM
and disk capacities common today. In particular, no matter how much memory is
available, Applesoft BASIC limits the theoretical size of any data array to
64K, and the practical size to substantially less after memory for the operating
system, file buffers, display pages, and the Applesoft BASIC interpreter itself
is taken into account.

Virtual memory lets applications assume very large memory spaces Ñ usually many
times larger than the working memory capacity of the computer. The process is
common to many mainframe and minicomputer systems. The name derives from the
fact that the application is able to function virtually, that is, as if large
amounts of memory were available, when, in fact, only a small portion of
memory is being used at any moment; the remainder of the application's "memory"
is resident on some external storage device, such as a hard disk, and portions
of it are swapped in and out of memory as they are needed.

USING THE PR0GRAM

Virtual Arrays patches Applesoft BASIC under ProDOS to allow this common version
of virtual memory. The patch allows for both real and integer virtual arrays of
up to 16 megabytes each, limited only by the capacity of the external storage
device (i.e., hard disk, RAM disk, or floppy disk) used to hold the contents of
the virtual arrays. Aside from the initial dimensioning of the virtual arrays
(described below), most Applesoft programs should execute without modification,
typically requiring less RAM than before, and with the added advantage of
automatic storage to disk of the array data. The patch will function on any
Apple II (or compatible) using ProDOS and BASIC.SYSTEM. Further, it is
transparent to normal (i.e., nonvirtual) arrays; only those integer and real
arrays that are too large for main memory (or for which automatic storage to
disk is desired) need be declared virtual, and all others may be dimensioned and
used as usual.

__________________________________________________________________________________
Dr. John R. Vokey, Department of Psychology, University of Lethbridge, Lethbridge,
Alberta, Canada T1K 3M4. This program is compatible with ProDOS.



To install the Virtual Array patch, use the command

BRUN VIRT.MEM.OBJ

in either immediate or deferred mode. The installer code relocates the patch to
six free pages of high memory, and interfaces the patch to Applesoft via the
ampersand vector, and installs a modification to Applesoft's CHRGET routine.
It completes installation by displaying a copyright notice on the text screen.

The installer automatically daisy-chains the ampersand vector so that other
installed ampersand utilities will continue to function. It also checks
whether the patch has already been installed; if so, it foregoes installation
and redisplays the copyright notice.

The Virtual Array patch adds three new commands to the language. Some examples
of their use are demonstrated in the BASIC program in Listing 4. (Contributing
Editors note: Listing 4 is the program file named VIRT.MEM.DEMO, which can be
found in the ShrinkIt archive named: A@VM.BXY) Each command is invoked with the
ampersand.

1. The syntax for the first command is

& SAVE pathname, arrayname{%} (dim {, dim})

The & SAVE command defines the specified real or integer array as virtual, and
creates (or retrieves) a virtual array file under the specified ProDOS pathname.
Pathname may be any string expression that evaluates to a legal ProDOS pathname,
and any array declaration that would be legal in an Applesoft real or integer
array DIMension statement may be used to dimension the array in the & SAVE
statement, with the addition that array sizes greater than 64K, up to 16
megabytes (disk space permitting), may be declared as virtual (see note 1).
Because of Applesoft BASIC's internal handling of subscripts, however, no single
dimension of the array may exceed 32767.

For example, the statement

& SAVE "/RAM/EXAMPLE", X(100, 100)

in either immediate or deferred mode will dimension the real array, X, to
contain 50K (101 x 101 elements, with five bytes per element) of data, create
the corresponding 50K file on the "/RAM" volume under the filename "EXAMPLE",
and set each of the 10,201 elements of the array to zero. From that point on,
the array may be used in exactly the same fashion as any normal, real array in
Applesoft BASIC.

If a virtual array file of the specified pathname already exists the data will
be retrieved from it, rather than a new fiie created. In this way, virtual array
data may be shared across programs (or even within the same program by array
variable "aliases"~. Virtual array files are given the "VAR" (i.e., $FD) file
type, with the auxiliary type code of "VA" ($D6Cl), for "Virtual Array."
Although the patch will correctly discriminate between VARiable files created
with BASIC.SYSTEM's STORE command and those created by the virtual array patch,
correctly rejecting the former, BASIC.SYSTEM does not, as it does not check the
auxiliary code of these files.

Virtual array files contain only the numerical data for the virtual array;
neither the array name, its dimensions, nor its type are stored in the file.
Although this simplicity allows virtual arrays of different names (and dimension
structure, size, or type) to access the same virtual array file, it also
requires that you keep track of the dimension structure, size, and type of the
virtual array files if they are to be reused or shared across programs. For real
arrays, each element of the virtual array is stored to the disk file in
Applesoft's five-byte, packed floating-point format; integer virtual array data
require two bytes per element. These formats result in files that are more
space-efficient and are more rapidly accessed than the more usual sequential
text file.

2. & STORE arrayname {, arraayname} The & STORE statement,in either immediate or
deferred mode "flushes" the specified virtual arrays. That is, the particular
blocks of data of the virtual arrays currently in memory are written to disk,
usually preparatory to a program end. Normally, any changes to the virtual array
data automatically are written to the array file as blocks of array data are
swapped between the file and memory during the normal course of using the
virtual array. However, because the block of array data currently in memory is
not updated in the file unless another block from the file is swapped into
memory, any changes to the last block accessed would not be saved to the file.
The & STORE command may be used to ensure that the specified virtual arrays are
updated before the end of a program (or before chaining or loading another
program, or CLEARing the variables). For example, the statement

& STORE X, Y, Z%

will flush the virtual arrays X, Y, and Z%.

3. & CLEAR {var {,var}}. The & CLEAR statement, in either immediate or deferred
mode, clears the specified variables of any type, simple or array (including
virtual array), from memory, freeing the memory formerly occupied for other
data. If you specify no variables, the program performs a regular Applesoft
CLEAR. For virtual arrays, only the array in RAM is cleared; the virtual
array file is not disturbed. The command is adapted from the "DISPose" utility
developed by Kaner and Vokey (see note 2). So that the command can distinguish
between simple and array variables of the same name, arrays must include array
indices, the values (expression or numerical) of the indices are immaterial,
but they are evaluated and therefore must be within the bounds declared for the
array. If a specified variable does not exist, it is created and then CLEARed.
For example, the statement

& CLEAR A, B%, C$, D(O,O)

will clear the values of the simple variables A, B %, and C$ and the array
(normal or virtual) D from memory.


Error Handling

The virtual array patch supports all Applesoft and ProDOS BASIC.SYSTEM error
messages and Applesoft error trapping using the "ONERR GOTO" statement. Virtual
arrays function just like normal arrays, returning error messages like "bad
subscript" and "redimensioned array" under the appropriate circumstances. In
addition, the & STORE statement will return an "OUT OF DATA" error if the
specified array does not exist (see note 3), and will return a new error
message, "?ARRAY ERROR" (new error code 128), if the specified array exists but
is not a virtual array.

ProDOS BASIC.SYSTEM errors behave similarly. Specifying an existing pathname
that is not a virtual array filename in an & SAVE statement, for example,
results in a "DUPLICATE FILE" error, and removing a disk volume containing a
currently active virtual array file results in a "FILE NOT FOUND ERROR" the next
time the virtual array file needs to be accessed.


Hints on Using Virtual Arrays

Although it is not required as long as a prefix has been set, you should specify
complete pathnames in & SAVE statements; if you do, the virtual array files will
continue to be accessed (assuming the volume remains online) even if the default
prefix changes.

To maximize program speed, virtual array files should be created on the fastest
storage device available, such as a RAM disk or a hard disk, and programs should
be structured to minimize the swapping of data blocks between the array file
and memory. Note that the data elements for Applesoft arrays (and, hence,
virtual arrays) are stored so that the first index changes fastest, while the
last index changes slowest, contrary to convention. Thus, statements such as
"FOR I = 1 TO 100: FOR J = 1 TO 100: X(I,J) = 0: NEXT J,I" will access the array
data out of order and, consequently, require more frequent swapping of data
blocks than would the same statements with the indices of the array variable
reversed (i.e., X(J,I)).

Although virtual arrays require only 512 bytes of user memory, because they are
easily recovered from the virtual array file (with the & SAVE statement), they
may be purged from memory (with the & CLEAR statement) when not in active use to
free the memory for other purposes. Remember to & STORE the virtual array files,
if necessary, before purging them to ensure that all data blocks have been
updated.

Because of the convenience of automatic storage to a disk file, and the ability
to recall the arrays easily both within and between programs, you may often want
to make even small arrays virtual.


About the Virtual Memory Demo Program

The Applesoft BASIC program shown in Listing 4 provides some examples on using
each of the commands of the virtual memory package. The program itself does very
little; it installs the virtual memory patch, creates (or retrieves) a 50K
(i.e., X(99,99)) real, virtual array file, and then allows you to view and edit
the contents of the file. However, there are some features of the program to
note.

First, the program attempts to create the virtual array file on the "/RAM"
volume. Any failure to do that (e.g., no such volume, volume full, etc.), will
result in an attempt to create the file on the default or currently prefixed
volume. If that attempt also fails, the program halts. The delay encountered
while creating the array reflects the time it takes to write five-byte zeros to
each of the 10,000 elements of the array file. Incidentally, the lower right
corner of the display presents a continuous update of the amount of user memory
available after various operations are performed. Note how small the change is
in this value as the 50K virtual array is created.

Second, assuming that the virtual array file has not been deleted and is still
online, re-running the program will demonstrate how the virtual memory patch
uses an existing virtual array. In this case, note how the startup is now
virtually instantaneous, and that all of the element of the array are recovered
intact.

Third, the array is indexed backward, as X(column, rou rather than as the
conventional X(row, column). This wa done for speed, as discussed earlier.

Fourth, when a shift in rows or columns is necessary, the updating of the screen
display of the array data is somewhat slow; hence the "One moment, please"
message each time it is required. This slowness is partially a function of the
"Print Using" subroutine, written entirely in Applesoft (an adaptation of one I
found in an old publication from Apple (see note 4)) used to format the data,
and, particularly if the virtual array is stored on a floppy diskette, partially
a result of the fact that each row of the data display requires that a new block
of virtual array data be loaded from the virtual array file.

Last, throughout the program, ProDOS BASIC. SYSTEM errors, including those
generated by the virtual memory commands, are trapped and handled with BASIC's
ONERR GOTO and corresponding RESUME commands.


ENTERING THE PROGRAM

The source code for the patch, written in 6502 assembly language, is shown in
Listing 1. (Contributing Editors Note: Listing 1 is included as the file:
VIRT.MEM.OBJ within the ShrinkIt archive: A2VM.BXY). The source code makes
extensive use of macros, which are shown in Listing 2 (Contributing Editors
Note: Listing 2 is included as the file: VIRT.MACS within the ShrinkIt archive:
A2VM.BXY). If you do not have a macro assembler, either manually substitute the
code for the macros at each point of their invocation in Listing 1 or directly
enter the resulting object code shown in Listing 3. (Contributing Editors Note:
Listing 3 is included as the file: VIRT.MEM.OBJ within the ShrinkIt archive:
A2VM.BXY). If you choose the latter route, note that there are two code
sections: the virtual array patch installer from $4000-$41AE, and the virtual
array patch itself from $4200-47F2. Despite the break between them save them as
a single file with the command

BSAVE VIRT.MEM.OBJ,AS4000,LS7F2

For help entering Nibble listings, see the Typing Tips section in this issue.
(Contributing Editors Note: This is of no value or need as the files so listed
have been included within the ShrinkIt archive: A2VM.BXY).


HOW THE PR0GRAM WORKS

The virtual array patch works by intercepting Apple soft's attempts to locate
array data. It accomplishes this interception through the aforementioned
modification to the interpreter's CHRGET routine (see note 5). If the attempt
involves a virtual array, the patch first determine whether the particular
element to be located currently resides in memory; if so, it sets the
appropriate vectors to point to the element and exits to the interpreter. If
not, it writes the current block of data in memory to the appropriate section of
the virtual array file, thereby recording any changes to the data, and then
reads in the correct block of data from the virtual array file, once again
setting the appropriate vectors before exiting to the interpreter. In either
case, Applesoft uses the array element as if it had located it itself. If a
block of array data is swapped into memory in this way, the array element
requested by the interpreter is loaded as the middle element of the block to
allow both forward and backward stepping from that point of about 50 real array
elements and over 125 integer array elements before a new block of array data
will be required to be swapped into memory.


Memory Use

All of the parameters associated with a given virtual array, such as its
dimensions, disk filename, and so on are stored within the array descriptor in
memory. The initial few bytes of this descriptor Ñ the variable name and offset
to the next variable Ñ are identical to that of nonvirtual arrays 90 that the
Applesoft memory management routines function normally. The first difference
occurs in the byte that normally contains the number of dimensions; for virtual
arrays, this byte is set to zero to flag to the patch that the array is indeed
virtual and needs special handling by the patch. This special handling includes
not only swapping blocks of data between the virtual array file and memory, but
also computing the desired array element to allow for array sizes greater than
BASIC's 64K limit. The remainder of the descriptor consists of a byte containing
the true number of dimensions, followed, using two bytes each, by the sizes of
each of the dimensions in reverse order. The next three bytes contain the record
number of the array element currently heading the data block in memory, the
number (in two bytes) of bytes in the block in memory, followed by the pathname
of the array file in Pascal string form (a length byte, followed by the ASCII
codes of the string in successive bytes). The descriptor is followed by a
512-byte block of memory to hold the current block of array data. That's it! No
matter how large the virtual array, no further memory is used.

One further memory-saving trick was used. Normally, to read or write any disk
file under ProDOS, a lK buffer to hold information about the file is required to
be set aside when the file is first opened. To avoid this lK overhead every time
a virtual array file is to be accessed, the patch uses the floating buffer
maintained by BASIC.SYSTEM for its CAT and CATALOG commands and as a temporary
data buffer for user text-file READ and WRITE commands. Although the patch's use
of this buffer does save memory, it comes at the cost of having to check every
time as to whether it is currently buffering data for a user text file Ñ and, if
so, having to write the buffered data to the file in question before using it to
read or write a virtual array file. It also requires that the patch maintain its
own ProDOS parameter tables, rather than use those located within BASIC.SYSTEM.
As a consequence, although they use the same data buffer, the patch does not
interfere with BASIC.SYSTEM; reading and writing virtual array data are
transparent to your concurrent reading and writing of disk files.


N0TES

1. A single, 5.25-inch floppy diskette is limited under ProDOS to 280 blocks
(140K). Even this limited capacity, however, allows for a virtual array of over
28,000 real or 71,000 integer elements, which is considerably more than that
allowed for in Applesoft.

2. Kaner, H. C. & Vokey, J. R. Subroutine Master. Nibble, November 1985, pp.
46-75.

3. Applesoft's STORE statement, as with the SAVE statement, is a legacy of the
early days of the Apple II when tape was the storage medium for programs and
data. It is used (without the &) to store real and integer data arrays to tape
through the tape port on the backplane of the Apple II, II Plus, and IIe. (As a
BASIC.SYSTEM command, it also is used to STORE the variables of an Applesoft
BASIC program.) Because of its similar purpose, both the syntax of the command
and the error messages it invoked have been retained in its reincarnation as a
virtual array command. The STORE statement, for example, also results in an "out
of data" error if the specified array does not exist.

4. Grossley, J. (1979). Print Using Simulator, Contact, 6, 8. 

5. Further details on this method of patching Applesoft may be found in Kaner,
H. C., & Vokey, J. R. "Modify- ing Apple's floating point BASIC: An &
interpreter without the &."  Compute!, 1982.


-end of reprint: Nibble Magazine, August, 1990, pages 44 thru 47-