37. Maintaining Large Systems

When a program gets large, it is often desirable to split it up into several files. One reason for this is to help keep the parts of the program organized, to make things easier to find. It's also useful to have the program broken into small pieces that are more convenient to edit and compile. It is particularly important to avoid the need to recompile all of a large program every time any piece of it changes; if the program is broken up into many files, only the files that have changes in them need to be recompiled.

The apparent drawback to splitting up a program is that more "commands" are needed to manipulate it. To load the program, you now have to load several files separately, insead of just loading one file. To compile it, you have to figure out which files need compilation, by seeing which have been edited since they were last compiled, and then you have to compile those files.

What's even more complicated is that files can have interdependencies. You might have a file called "DEFS" that contains some macro definitions (or flavor or structure definitions), and functions in other files might use those macros. This means that in order to compile any of those other files, you must first load the file "DEFS" into the Lisp environment, so that the macros will be defined and can be expanded at compile time. You'd have to remember this whenever you compile any of those files. Furthermore, if "DEFS" has changed, other files of the program might need to be recompiled because the macros might have changed and need to be re-expanded.

This chapter describes the system facility, which takes care of all these things for you. The way it works is that you define a set of files to be a system, using the defsystem special form, described below. This system definition says which files make up the system, which ones depend on the presence of others, and so on. You put this system definition into its own little file, and then all you have to do is load that file and the Lisp environment will know about your system and what files are in it. You can then use the make-system function (see (make-system-fun)) to load in all the files of the system, or recompile all the files that need compiling, and so on.

The system facility is very general and extensible. This chapter explains how to use it and how to extend it. This chapter also explains the patch facility, which lets you conveniently update a large program with incremental changes, and how to save away Lisp environments in disk partitions.

37.1 Defining a System

Special Form: defsystem name (keyword args...)...

Defines a system named name. The options selected by the keywords are explained in detail later. In general, they fall into two categories: properties of the system and transformations. A transformation is an operation such as compiling or loading which takes one or more files and does something to them. The simplest system is a set of files and a transformation to be performed on them.

Here are a few examples.

(defsystem mysys (:compile-load ("AI: GEORGE; PROG1" "AI: GEORG2; PROG2"))) (defsystem zmail (:name "ZMail") (:pathname-default "AI: ZMAIL;") (:package zwei) (:module defs "DEFS") (:module mult "MULT" :package tv) (:module main ("TOP" "COMNDS" "MAIL" "USER" "WINDOW" "FILTER" mult "COMETH")) (:compile-load defs) (:compile-load main (:fasload defs))) (defsystem bar (:module reader-macros "RDMAC") (:module other-macros "MACROS") (:module main-program "MAIN") (:compile-load reader-macros) (:compile-load other-macros (:fasload reader-macros)) (:compile-load main-program (:fasload reader-macros other-macros)))

The first example defines a new system called mysys, which consists of two files, both of which are to be compiled and loaded. The second example is somewhat more complicated. What all the options mean will be specified shortly, but the primary difference is that there is a file defs which must be loaded before the rest of the files (main) can be compiled. The final example has two levels of dependency. reader-macros must be compiled and loaded before other-macros can be compiled. Both reader-macros and other-macros must then be loaded before main-program can be compiled.

The defsystem options other than transformations are:

.kitem :name Specifies a "pretty" version of the name for the system, for use in printing. .kitem :short-name Specified an abbreviated name used in constructing disk label comments and in patch file names for some file systems. .kitem :component-systems Specifies the names of other systems used to make up this system. Performing an operation on a system with component systems is equivalent to performing the same operation on all the individual systems. The format is (:component-systems names...). .kitem :package Specifies the package in which transformations are performed. A package specified here will override one in the 7-*- line of the file in question. .kitem :pathname-default Gives a local default within the definition of the system for strings to be parsed into pathnames. Typically this specifies the directory, when all the files of a system are on the same directory. .kitem :patchable Makes the system be a patchable system (see (patch-facility)). An optional argument specifies the directory to put patch files in. The default is the :pathname-default of the system. .kitem :initial-status Specifies what the status of the system should be when make-system is used to create a new major version. The default is :experimental. See (patch-system-status) for further details. .kitem :not-in-disk-label Make a patchable system not appear in the disk label comment. This should probably never be specified for a user system. It is used by patchable systems internal to the main Lisp system, to avoid cluttering up the label. .kitem :module Allows assigning a name to a set of files within the system. This name can then be used instead of repeating the filenames. The format is (:module name files options...). files is usually a list of filenames (strings). In general, it is a module-specification, which can be any of the following:

a string

This is a file name.

a symbol

This is a module name. It stands for all of the files which are in that module of this system.

an external module component

This is a list of the form (system-name module-names...), to specify modules in another system. It stands for all of the files which are in all of those modules.

a list of module components

A module component is any of the above, or the following:

a list of file names

This is used in the case where the names of the input and output files of a transformation are not related according to the standard naming conventions, for example when a QFASL file has a different name or resides on a different directory than the source file. The file names in the list are used from left to right, thus the first name is the source file. Each file name after the first in the list is defaulted from the previous one in the list.

To avoid syntactic ambiguity, this is allowed as a module component but not as a module specification.

The currently defined options for the :module clause are

:package

Overrides any package specified for the whole system for transformations performed on just this module.

In the second defsystem example above, there are three modules. The first two each have only one file, and the third one (main) is made up both of files and another module. To take examples of the other possibilities,

(:module prog (("AI: GEORGE; PROG" "AI: GEORG2; PROG"))) (:module foo (defs (zmail defs)))

The prog module consists of one file, but it lives in two directories, GEORGE and GEORG2. If this were a Lisp program, that would mean that the file "AI: GEORGE; PROG >" would be compiled into "AI: GEORG2; PROG QFASL". The foo module consists of two other modules, the defs module in the same system, and the defs module in the zmail system. It is not generally useful to compile files that belong to other systems, thus this foo module would not normally be the subject of a transformation. However, dependencies (defined below) use modules and need to be able to refer to (depend on) modules of other systems.

Function: si:set-system-source-file system-name filename

This function specifies which file contains the defsystem for the system system-name. filename can be a pathname object or a string.

Sometimes it is useful to say where the definition of a system can be found without taking time to load that file. If make-system is ever used on that system, the file whose name has been specified will be loaded automatically.

37.2 Transformations

Transformations are of two types, simple and complex. A simple transformation is a single operation on a file, such as compiling it or loading it. A complex transformation takes the output from one transformation and performs another transformation on it, for example, loading the results of compilation.

The general format of a simple transformation is (name input dependencies condition). input is usually a module specification or another transformation whose output is used. The transformation name is to be performed on all the files in the module, or all the output files of the other transformation.

dependencies and condition are optional.

dependencies is a transformation specification, either a list (transformation-name module-names...), or a list of such lists. A module-name is either a symbol which is the name of a module in the current system, or a list (system-name module-names...). A dependency declares that all of the indicated transformations must be performed on the indicated modules before the current transformation itself can take place. Thus in the zmail example above, the defs module must have the :fasload transformation performed on it before the :compile transformation can be performed on main.

The dependency has to be a tranformation that was explicitly specified as a transformation in the system definition, not just an action that might have been performed by anything. That is, if you have a dependency (:fasload foo), it means that (fasload foo) is a tranformation of your system and you depend on that tranformation; it does not simply mean that you depend on foo's being loaded. Furthermore, it doesn't work if (:fasload foo) was an implicit piece of another tranformation. For example, the following is correct and will work:

(defsystem foo (:module foo "FOO") (:module bar "BAR") (:compile-load (foo bar)))

but this doesn't work:

(defsystem foo (:module foo "FOO") (:module bar "BAR") (:module blort "BLORT") (:compile-load (foo bar)) (:compile-load blort (:fasload foo)))

because foo's :fasload is not mentioned explicitly (i.e. at top level) but is only implicit in the (:compile-load (foo bar)). One must instead write:

(defsystem foo (:module foo "FOO") (:module bar "BAR") (:module blort "BLORT") (:compile-load foo) (:compile-load bar) (:compile-load blort (:fasload foo)))

condition is a predicate which specifies when the transformation should take place. Generally it defaults according to the type of the transformation. Conditions are discussed further on (transformation-condition-discussion).

The defined simple transformations are:

:fasload

Calls the fasload function to load the indicated files, which must be QFASL files. The condition defaults to si:file-newer-than-installed-p, which is t if a newer version of the file exists on the file computer than was read into the current environment.

:readfile

Calls the readfile function to read in the indicated files. Use this for files that are not to be compiled. condition defaults to si:file-newer-than-installed-p.

:compile

Calls the qc-file function to compile the indicated files. condition defaults to si:file-newer-than-file-p which returns t if the source file has been written more recently than the binary file.

A special simple transformation is

:do-components

(:do-components dependencies) inside a system with component systems will cause the dependencies to be done before anything in the component systems. This is useful when you have a module of macro files used by all of the component systems.

The defined complex transformations are

:compile-load

(:compile-load input compile-dependencies load-dependencies compile-condition load-condition) is the same as (:fasload (:compile input compile-dependencies compile-condition) load-dependencies load-condition). This is the most commonly-used transformation. Everything after input is optional.

:compile-load-init

See (compile-load-init-transformation).

As was explained above, each filename in an input specification can in fact be a list of strings for the case where the source file of a program differs from the binary file in more than just the file type. In fact, every filename is treated as if it were an infinite list of filenames with the last filename, or in the case of a single string the only filename, repeated forever at the end. Each simple transformation takes some number of input filename arguments, and some number of output filename arguments. As transformations are performed, these arguments are taken from the front of the filename list. The input arguments are actually removed and the output arguments left as input arguments to the next higher transformation. To make this clearer, consider the prog module above having the :compile-load transformation performed on it. This means that prog is given as the input to the :compile transformation and the output from this transformation is given as the input to the :fasload transformation. The :compile transformation takes one input filename argument, the name of a lisp source file, and one output filename argument, the name of the qfasl file. The :fasload transformation takes one input filename argument, the name of a qfasl file, and no output filename arguments. So, for the first and only file in the prog module, the filename argument list looks like ("AI: GEORGE; PROG" "AI: GEORG2; PROG" "AI: GEORG2; PROG" ...). The :compile transformation is given arguments of "AI: GEORGE; PROG" and "AI: GEORG2; PROG" and the filename argument list which it outputs as the input to the :fasload transformation is ("AI: GEORG2; PROG" "AI: GEORG2; PROG" ...). The :fasload transformation then is given its one argument of "AI: GEORG2; PROG".

Note that dependencies are not "transitive" nor "inherited". For example, if module a depends on macros defined in module b, and therefore needs b to be loaded in order to compile, and b has a similar dependency on c, c will not be loaded during compilation of a. Transformations with these dependencies would be written

(:compile-load a (:fasload b)) (:compile-load b (:fasload c))

To say that compilation of a depends on both b and c, you would instead write

(:compile-load a (:fasload b c)) (:compile-load b (:fasload c))

If in addition a depended on c (but not b) during loading (perhaps a contains defvars whose initial values depend on functions or special variables defined in c) you would write the transformations

(:compile-load a (:fasload b c) (:fasload c)) (:compile-load b (:fasload c))



37.3 Making a System

Function: make-system name &rest keywords

The make-system function does the actual work of compiling and loading. In the example above, if PROG1 and PROG2 have both been compiled recently, then

(make-system 'mysys)

will load them as necessary. If either one might also need to be compiled, then

(make-system 'mysys ':compile)

will do that first as necessary.

The very first thing make-system does is check whether the file which contains the defsystem for the specified system has changed since it was loaded. If so, it offers to load the latest version, so that the remainder of the make-system can be done using the latest system definition. (This only happens if the filetype of that file is LISP.) After loading this file or not, make-system goes on to process the files that compose the system.

make-system lists what transformations it is going to perform on what files, asks the user for confirmation, then performs the transformations. Before each transformation a message is printed listing the transformation being performed, the file it is being done to, and the package. This behavior can be altered by keywords.

These are the keywords recognized by the make-system function and what they do.

.kitem :noconfirm Assumes a yes answer for all questions that would otherwise be asked of the user. .kitem :selective Asks the user whether or not to perform each transformation that appears to be needed for each file. .kitem :silent Avoids printing out each transformation as it is performed. .kitem :reload Bypasses the specified conditions for performing a transformation. Thus files are compiled even if they haven't changed and loaded even if they aren't newer than the installed version. .kitem :noload Does not load any files except those required by dependencies. For use in conjunction with the :compile option. .kitem :compile Compiles files also if need be. The default is to load but not compile. .kitem :recompile This is equivalent to a combination of :compile and :reload: it specifies compilation of all files, even those whose sources have not changed since last compiled. .kitem :no-increment-patch When given along with the :compile option, disables the automatic incrementing of the major system version that would otherwise take place. See (patch-facility). .kitem :increment-patch Increment a patchable system's major version without doing any compilations. See (patch-facility). .kitem :batch Allows a large compilation to be done unattended. It acts like :noconfirm with regard to questions, turns off more-processing and fdefine-warnings (see inhibit-fdefine-warnings, (inhibit-fdefine-warnings-var)), and saves the compiler warnings in an editor buffer and a file (it asks you for the name). .kitem :print-only Just prints out what transformations would be performed, does not actually do any compiling or loading. .kitem :noop Is ignored. This is mainly useful for programs that call make-system, so that such programs can include forms like

(make-system 'mysys (if compile-p ':compile ':noop))

37.4 Adding New Keywords to make-system

make-system keywords are defined as functions on the si:make-system-keyword property of the keyword. The functions are called with no arguments. Some of the relevant variables they can use are

Variable: si:*system-being-made*

The internal data structure which represents the system being made.

Variable: si:*make-system-forms-to-be-evaled-before*

A list of forms which are evaluated before the transformations are performed.

Variable: si:*make-system-forms-to-be-evaled-after*

A list of forms which are evaluated after the transformations have been performed.

Variable: si:*make-system-forms-to-be-evaled-finally*

A list of forms which are evaluated after the body of make-system has completed. This differs from si:*make-system-forms-to-be-evaled-after* in that these forms are evaluated outside of the "compiler context", which sometimes makes a difference.

Variable: si:*query-type*

Controls how questions are asked. Its normal value is :normal. :noconfirm means no questions will be asked and :selective asks a question for each individual file transformation.

Variable: si:*silent-p*

If t, no messages are printed out.

Variable: si:*batch-mode-p*

If t, :batch was specified.

Variable: si:*redo-all*

If t, all transformations are performed, regardless of the condition functions.

Variable: si:*top-level-transformations*

A list of the names of transformations that will be performed, such as (:fasload :readfile).

Variable: si:*file-transformation-function*

The actual function that gets called with the list of transformations that need to be performed. The default is si:do-file-transformations.

Special Form: si:define-make-system-special-variable variable value [defvar-p]

Causes variable to be bound to value, which is evaluated at make-system time, during the body of the call to make-system. This allows you to define new variables similar to those listed above. If defvar-p is specified as (or defaulted to) t, variable is defined with defvar. It is not given an initial value. If defvar-p is specified as nil, variable belongs to some other program and is not defvar'ed here.

The following simple example adds a new keyword to make-system called :just-warn, which means that fdefine warnings (see (fdefine-fun)) regarding functions being overwritten should be printed out, but the user should not be queried.

(si:define-make-system-special-variable inhibit-fdefine-warnings inhibit-fdefine-warnings nil) (defun (:just-warn si:make-system-keyword) () (setq inhibit-fdefine-warnings ':just-warn))

(See the description of the inhibit-fdefine-warnings variable, on (inhibit-fdefine-warnings-var).)

make-system keywords can have effect either directly when called, or by pushing a form to be evaluated onto si:*make-system-forms-to-be-evaled-after* or one of the other two similar lists. In general, the only useful thing to do is to set some special variable defined by si:define-make-system-special-variable. In addition to the ones mentioned above, user-defined transformations may have their behavior controlled by new special variables, which can be set by new keywords. If you want to get at the list of transformations to be performed, for example, the right way would be to set si:*file-transformation-function* to a new function, which then might call si:do-file-transformations with a possibly modified list. That is how the :print-only keyword works.

37.5 Adding New Options for defsystem

Options to defsystem are defined as macros on the si:defsystem-macro property of the option keyword. Such a macro can expand into an existing option or transformation, or it can have side effects and return nil. There are several variables they can use; the only one of general interest is

Variable: si:*system-being-defined*

The internal data structure which represents the system which is currently being constructed.

Special Form: si:define-defsystem-special-variable variable value

Causes value to be evaluated and variable to be bound to the result during the expansion of the defsystem special form. This allows you to define new variables similar to the one listed above.

Special Form: si:define-simple-transformation

This is the most convenient way to define a new simple transformation. The form is

(si:define-simple-transformation name function default-condition input-file-types output-file-types pretty-names compile-like load-like)

For example,

(si:define-simple-transformation :compile si:qc-file-1 si:file-newer-than-file-p ("LISP") ("QFASL"))

input-file-types and output-file-types are how a transformation specifies how many input filenames and output filenames it should receive as arguments, in this case one of each. They also, obviously, specify the default file type for these pathnames. The si:qc-file-1 function is mostly like qc-file, except for its interface to packages. It takes input-file and output-file arguments.

pretty-names, compile-like, and load-like are optional.

pretty-names specifies how messages printed for the user should print the name of the transformation. It can be a list of the imperative ("Compile"), the present participle ("Compiling"), and the past participle ("compiled"). Note that the past participle is not capitalized, because it is not used at the beginning of a sentence. pretty-names can be just a string, which is taken to be the imperative, and the system will conjugate the participles itself. If pretty-names is omitted or nil it defaults to the name of the transformation.

compile-like and load-like say when the transformation should be performed. Compile-like transformations are performed when the :compile keyword is given to make-system. Load-like transformations are performed unless the :noload keyword is given to make-system. By default compile-like is t but load-like is nil.

Complex transformations are just defined as normal macro expansions, for example,

(defmacro (:compile-load si:defsystem-macro) (input &optional com-dep load-dep com-cond load-cond) `(:fasload (:compile ,input ,com-dep ,com-cond) ,load-dep ,load-cond))

37.6 More Esoteric Transformations

It is sometimes useful to specify a transformation upon which something else can depend, but which is not performed by default, but rather only when requested because of that dependency. The transformation nevertheless occupies a specific place in the hierarchy. The :skip defsystem macro allows specifying a transformation of this type. For example, suppose there is a special compiler for the read table which is not ordinarily loaded into the system. The compiled version should still be kept up to date, and it needs to be loaded if ever the read table needs to be recompiled.

(defsystem reader (:pathname-default "AI: LMIO;") (:package system-internals) (:module defs "RDDEFS") (:module reader "READ") (:module read-table-compiler "RTC") (:module read-table "RDTBL") (:compile-load defs) (:compile-load reader (:fasload defs)) (:skip :fasload (:compile read-table-compiler)) (:rtc-compile-load read-table (:fasload read-table-compiler)))

Assume that there is a complex transformation :rtc-compile-load which is like :compile-load, except that is is built on a transformation called something like :rtc-compile, which uses the read table compiler rather than the Lisp compiler. In the above system, then, if the :rtc-compile transformation is to be performed, the :fasload transformation must be done on read-table-compiler first, that is the read table compiler must be loaded if the read table is to be recompiled. If you say (make-system 'reader ':compile), then the :compile transformation will still happen on the read-table-compiler module, compiling the read table compiler if need be. But if you say (make-system 'reader), the reader and the read table will be loaded, but the :skip keeps this from happening to the read table compiler.

.setq transformation-condition-discussion page So far nothing has been said about what can be given as a condition for a transformation except for the default functions which check for a source file being newer than the binary and so on. In general, any function which takes the same arguments as the transformation function (e.g. qc-file) and returns t if the transformation needs to be performed, can be in this place as a symbol, including for example a closure. To take an example, suppose there is a file which contains compile-flavor-methods for a system, and which should therefore be recompiled if any of the flavor method definitions change. In this case, the condition function for compiling that file should return t if either the source of that file itself or any of the files that define the flavors have changed. This is what the :compile-load-init complex transformation is for. It is defined like this:

.setq compile-load-init-transformation page (defmacro (:compile-load-init si:defsystem-macro) (input add-dep &optional com-dep load-dep &aux function) (setq function (let-closed ((*additional-dependent-modules* add-dep)) 'compile-load-init-condition)) `(:fasload (:compile ,input ,com-dep ,function) ,load-dep)) (defun compile-load-init-condition (source-file qfasl-file) (or (si:file-newer-than-file-p source-file qfasl-file) (local-declare ((special *additional-dependent-modules*)) (si:other-files-newer-than-file-p *additional-dependent-modules* qfasl-file))))

The condition function which will be generated when this macro is used returns t either if si:file-newer-than-file-p would with those arguments, or if any of the other files in add-dep, which presumably is a module specification, are newer than the qfasl file. Thus the file (or module) to which the :compile-load-init transformation applies will be compiled if it or any of the source files it depends on has been changed, and will be loaded under the normal conditions. In most (but not all cases), com-dep would be a :fasload transformation of the same files as add-dep specifies, so that all the files this one depends on would be loaded before compiling it. ; --> DSK:LMMAN;MAKSYS XGP (in :DOVERable format!) Processing file AI:LMMAN;MAKSYS 22 1. "Maintaining Large Systems" >>ERROR: MAKE-SYSTEM-FUN has no value in ^V Decimal file position=2355 in AI:LMMAN;MAKSYS 22 1.1. "Defining a System" >>ERROR: PATCH-FACILITY has no value in ^V Decimal file position=5709 in AI:LMMAN;MAKSYS 22 >>ERROR: PATCH-SYSTEM-STATUS has no value in ^V Decimal file position=6041 in AI:LMMAN;MAKSYS 22 1.2. "Transformations" >>ERROR: TRANSFORMATION-CONDITION-DISCUSSION has no value in ^V Decimal file position=11883 in AI:LMMAN;MAKSYS 22 >>ERROR: COMPILE-LOAD-INIT-TRANSFORMATION has no value in ^V Decimal file position=13509 in AI:LMMAN;MAKSYS 22 1.3. "Making a System" >>ERROR: PATCH-FACILITY has no value in ^V Decimal file position=18161 in AI:LMMAN;MAKSYS 22 >>ERROR: PATCH-FACILITY has no value in ^V Decimal file position=18291 in AI:LMMAN;MAKSYS 22 >>ERROR: INHIBIT-FDEFINE-WARNINGS-VAR has no value in ^V Decimal file position=18532 in AI:LMMAN;MAKSYS 22 1.4. "Adding New Keywords to make-system" >>ERROR: FDEFINE-FUN has no value in ^V Decimal file position=21551 in AI:LMMAN;MAKSYS 22 >>ERROR: INHIBIT-FDEFINE-WARNINGS-VAR has no value in ^V Decimal file position=21970 in AI:LMMAN;MAKSYS 22 1.5. "Adding New Options for defsystem" 1.6. "More Esoteric Transformations" Concept Index Keyword Index Variable Index Function Index

TABLE-OF-CONTENTS 15 (+2) pages generated. >> The following variables were undefined: FDEFINE-FUN, INHIBIT-FDEFINE-WARNINGS-VAR, PATCH-SYSTEM-STATUS, PATCH-FACILITY >> The following vars were referenced before being defined: COMPILE-LOAD-INIT-TRANSFORMATION, TRANSFORMATION-CONDITION-DISCUSSION, MAKE-SYSTEM-FUN Run a second pass?yes Processing file AI:LMMAN;MAKSYS 22 1. "Maintaining Large Systems" 1.1. "Defining a System" >>ERROR: PATCH-FACILITY has no value in ^V Decimal file position=5709 in AI:LMMAN;MAKSYS 22 >>ERROR: PATCH-SYSTEM-STATUS has no value in ^V Decimal file position=6041 in AI:LMMAN;MAKSYS 22 1.2. "Transformations" 1.3. "Making a System" >>ERROR: PATCH-FACILITY has no value in ^V Decimal file position=18161 in AI:LMMAN;MAKSYS 22 >>ERROR: PATCH-FACILITY has no value in ^V Decimal file position=18291 in AI:LMMAN;MAKSYS 22 >>ERROR: INHIBIT-FDEFINE-WARNINGS-VAR has no value in ^V Decimal file position=18532 in AI:LMMAN;MAKSYS 22 1.4. "Adding New Keywords to make-system" >>ERROR: FDEFINE-FUN has no value in ^V Decimal file position=21551 in AI:LMMAN;MAKSYS 22 >>ERROR: INHIBIT-FDEFINE-WARNINGS-VAR has no value in ^V Decimal file position=21970 in AI:LMMAN;MAKSYS 22 1.5. "Adding New Options for defsystem" 1.6. "More Esoteric Transformations" Concept Index Keyword Index Variable Index Function Index

TABLE-OF-CONTENTS 15 (+2) pages generated. Create variables file DSK:LMMAN;MAKSYS VARS?t no Create variables file DSK:LMMAN;MAKSYS VARS? Answer either "Yes" or "No": No ; Runtime = 2 minutes 23.978 seconds (plus 13.782 gctime) ; 32 swap-in requests; thrash coefficient = 0.203

37.7 The Patch Facility

The patch facility allows a system maintainer to manage new releases of a large system and issue patches to correct bugs. It is designed to be used to maintain both the Lisp Machine system itself, and applications systems that are large enough to be loaded up and saved on a disk partition.

When a system of programs is very large, it needs to be maintained. Often problems are found and need to be fixed, or other little changes need to be made. However, it takes a long time to load up all of the files that make up such a system, and so rather than having every user load up all the files every time he wants to use the system, usually the files just get loaded once into a Lisp world, and then the Lisp world is saved away on a disk partition. Users then use this disk partition, and copies of it are distributed. The problem is that since the users don't load up the system every time they want to use it, they don't get all the latest changes.

The purpose of the patch system is to solve this problem. A patch file is a little file that, when you load it, updates the old version of the system into the new version of the system. Most often, patch files just contain new function definitions; old functions are redefined to do their new thing. When you want to use a system, you first use the Lisp environment saved on the disk, and then you load all the latest patches. Patch files are very small, so loading them doesn't take much time. You can even load the saved environment, load up the latest patches, and then save it away, to save future users the trouble of even loading the patches. (Of course, new patches may be made later, and then these will have to be loaded if you want to get the very latest version.)

For every system, there is a series of patches that have been made to that system. To get the latest version of the system, you load each patch file in the series, in order. Sooner or later, the maintainer of a system will want to stop building more and more patches, and recompile everything, starting afresh. A complete recompilation is also necessary when a system is changed in a far-reaching way, that can't be done with a small patch; for example, if you completely reorganize a program, or change a lot of names or conventions, you might need to completely recompile it to make it work again. After a complete recompilation has been done, the old patch files are no longer suitable to use; loading them in might even break things.

The way all this is kept track of is by labelling each version of a system with a two-part number. The two parts are called the major version number and the minor version number. The minor version number is increased every time a new patch is made; the patch is identified by the major and minor version number together. The major version number is increased when the program is completely recompiled, and at that time the minor version number is reset to zero. A complete system version is identified by the major version number, followed by a dot, followed by the minor version number.

To clarify this, here is a typical scenario. A new system is created; its initial version number is 1.0. Then a patch file is created; the version of the program that results from loading the first patch file into version 1.0 is called 1.1. Then another patch file might be created, and loading that patch file into system 1.1 creates version 1.2. Then the entire system is recompiled, creating version 2.0 from scratch. Now the two patch files are irrelevant, because they fix old software; the changes that they reflect are integrated into system 2.0.

Note that the second patch file should only be loaded into system 1.1 in order to create system 1.2; you shouldn't load it into 1.0 or any other system besides 1.1. It is important that all the patch files be loaded in the proper order, for two reasons. First, it is very useful that any system numbered 1.1 be exactly the same software as any other system numbered 1.1, so that if somebody reports a bug in version 1.1, it is clear just which software is being complained about. Secondly, one patch might patch another patch; loading them in some other order might have the wrong effect.

The patch facility keeps track, in the file system, of all the patch files that exist, remembering which version each one creates. There is a separate numbered sequence of patch files for each major version of each system. All of them are stored in the file system, and the patch facility keeps track of where they all are. In addition to the patch files themselves, there are "patch directory" files which contain the patch facility's data base by which it keeps track of what minor versions exist for a major version, and what the last major version of a system is. These files and how to make them are described below.

In order to use the patch facility, you must define your system with defsystem (see (system-system)) and declare it as patchable with the :patchable option. When you load your system (with make-system, see (make-system-fun)) it is added to the list of all systems present in the world. The patch facility keeps track of which version of each patchable system is present, and where the data about that system reside in the file system. This information can be used to update the Lisp world automatically to the latest versions of all the systems it contains. Once a system is present, you can ask for the latest patches to be loaded, ask which patches are already loaded, and add new patches.

You can also load in patches or whole new systems and then save the entire Lisp environment away in a disk partition. This is explained on (disk-partition).

When a Lisp Machine is booted, it prints out a line of information telling you what systems are present, and which version of each system is loaded. This information is returned by the function si:system-version-info. It is followed by a text string containing any additional information that was requested by whomever created the current disk partition (see disk-save, (disk-save-fun)).

Function: print-system-modifications &rest system-names

With no arguments, this lists all the systems present in this world and, for each system, all the patches that have been loaded into this world. For each patch it shows the major version number (which will always be the same since a world can only contain one major version), the minor version number, and an explanation of what the patch does, as typed in by the person who made the patch.

If print-system-modifications is called with arguments, only the modifications to the systems named are listed.

Function: si:get-system-version &optional system

Returns two values, the major and minor version numbers of the version of system currently loaded into the machine, or nil if that system is not present. system defaults to "System".

Function: si:system-version-info &optional (brief-p nil)

This returns a string giving information about which systems and what versions of the systems are loaded into the machine, and what microcode version is running. A typical string for it to produce is:

"System 65.12, ZMail 19.1, Vision 10.23, microcode 739"

If brief-p is t, it uses short names, suppresses the microcode version, any systems which should not appear in the disk label comment, the name System, and the commas:

"65.12 Vis 10.23"

37.7.1 Defining a System

In order to use the patch facility, you must declare your system as patchable by giving the :patchable option to defsystem (see (system-system)). The major version of your system in the file system will be incremented whenever make-system is used to compile it. Thus a major version is associated with a set of QFASL files. The major version of your system that is remembered as having been loaded into the Lisp environment will be set to the major version in the file system whenever make-system is used to load your system and the major version in the file system is greater than what you had loaded before.

After loading your system, you can save it with the disk-save function (see (disk-save-fun)). disk-save will ask you for any additional information you want printed as part of the greeting when the machine is booted. This is in addition to the names and versions of all the systems present in this world. If the system version will not fit in the 16-character field allocated in the disk label, disk-save will ask you to type in an abbreviated form.

37.7.2 Patch files

The patch system will maintain several different types of files in the directory associated with your system. This directory is specified to defsystem via either the :patchable option or the :pathname-default option. These files are maintained automatically, but so that you will know what they are and when they are obsolete (because they are associated with an obsolete version of your system), they are described here.

The file that tells the system's current major version has a name of the form AI: MYDIR; PATCH (PDIR) (on Tops-20, EE:PS:<MYDIR>PATCH.DIRECTORY), where the host, device, and directory (AI:MYDIR; or EE:PS:<MYDIR> in this example) come from the system definition as explained above.

For each major version of the system, there is a patch directory file, of the form AI: MYDIR; PAT259 (PDIR), which describes the individual patches for that version, where 259 is the major version number in this example. (On Tops-20, this is EE:PS:<MYDIR>PATCH-259.DIRECTORY).

Then for each minor version of the system, the source of the patch file itself has a name of the form AI: MYDIR; P59.69 >, for minor version 69 of major version 259. Note that 259 has been truncated to 59 to fit into six characters for ITS. On Tops-20 this would be EE:PS:<MYDIR>PATCH-259-69.LISP. Patch files get compiled, so there will also be files like AI: MYDIR; P59.69 QFASL (on Tops-20, EE:PS:<MYDIR>PATCH-259-69.QFASL).

If the :patchable option to defsystem is given an argument, telling it to put the patch files in a different directory than the one which holds the other files of the system, then a slightly different set of file name conventions are used.

On ITS, the file that tells the current major version is of the form AI: PATDIR; system (PDIR), where system is the name of the system and PATDIR is the directory specified in the :patchable option to defsystem. The patch directory file for major version nnn is of the form AI: PATDIR; sysnnn (PDIR), where sys is the short name specified with the :short-name option to defsystem. A patch file has a name of the form AI: PATDIR; nnn.mm; note that the major version is truncated to three digits instead of two. In this set of file name conventions, the patch files don't all fall together in alphabetical order, as they do in the first set.

On TOPS-20, the file names take the forms EE:PS:<PATDIR>system.PATCH-DIRECTORY, EE:PS:<PATDIR>system-nnn.PATCH-DIRECTORY, and EE:PS:<PATDIR>system-nnn-mmm.LISP (or .QFASL). These file name conventions allow the patches for multiple systems to coexist in the same directory.

37.7.3 Loading Patches

Function: load-patches &rest options

This function is used to bring the current world up to the latest minor version of whichever major version it is, for all systems present, or for certain specified systems. If there are any patches available, load-patches will offer to read them in. With no arguments, load-patches updates all the systems present in this world.

options is a list of keywords. Some keywords are followed by an argument. The following options are accepted:

:systems list

list is a list of names of systems to be brought up to date. If this option is not specified, all systems are processed.

.kitem :verbose Print an explanation of what is being done. This is the default.

.kitem :selective For each patch, say what it is and then ask the user whether or not to load it. This is the default. If the user answers "P", selective mode is turned off for any remaining patches to the current system.

.kitem :noselective Turns off :selective.

.kitem :silent Turns off both :selective and :verbose. In :silent mode all necessary patches are loaded without printing anything and without querying the user.

Currently load-patches is not called automatically, but the system may be changed to offer to load patches when the user logs in, in order to keep things up to date.

37.7.4 Making Patches

There are two editor commands that are used to create patch files. During a typical maintenance session on a system you will make several edits to its source files. The patch system can be used to copy these edits into a patch file so that they can be automatically incorporated into the system to create a new minor version. Edits in a patch file can be modified function definitions, new functions, modified defvar's and defconst's, or arbitrary forms to be evaluated, even including load's of new files.

Meta-X Add Patch adds the region (if there is one) or else the current "defun" to the patch file currently being constructed. The first time you give this command it will ask you what system you are patching, allocate a new minor version number, and start constructing the patch file for that version. If you change a function, you should recompile it, test it, then once it works use Add Patch to put it in the patch file.

The patch file being constructed is in an editor buffer. If you mistakenly Add Patch something which doesn't work, you can select the buffer containing the patch file and delete it. Then later you can Add Patch the corrected version.

While you are making your patch file, the minor version number that has been allocated for you is reserved so that nobody else can use it. This way if two people are patching a system at the same time, they will not both get the same minor version number.

After making and testing all of your patches, use meta-X Finish Patch to install the patch file so that other users can load it. This will compile the patch file if you have not done so yourself (patches are always compiled). It will ask you for a comment describing the reason for the patch; load-patches and print-system-modifications print these comments.

After finishing your patch, if you do another Add Patch it will ask you which system again and start a new minor version. Note that you can only be putting together patches for one system at a time.

If you start making a patch file and for some reason never do a Finish Patch (you decide to give up or your machine crashes), the minor version number that you were working on will remain reserved. Since patch files must always be loaded in strictly sequential order, nobody will be able to load any further patches made to this major version past this point. You must manually edit the patch directory file for this major version, removing the line corresponding to the aborted patch. It is OK for a minor version number to be skipped.

37.7.5 System Status

The patch system has the concept of the "status" of a major version of a system. The status is displayed when the system version is displayed, in places such as the system greeting message and the disk partition comment. This status allows users of the system to know what is going on. The status of a system changes as patches are made to it.

The status is one of the following keywords:

:experimental

The system has been built but has not yet been fully debugged and released to users. This is the default status when a new major version is created, unless it is overridden with the :initial-status option to defsystem.

:released

The system is released for general use. This status produces no extra text in the system greeting and the disk partition comment.

:obsolete

The system is no longer supported.

:broken

This is like :experimental but is used when the system was thought incorrectly to have been debugged, and hence was :released for a while.

Function: si:set-system-status system status &optional major-version

Changes the status of a system. system is the name of the system. major-version is the number of the major version to be changed; if unsupplied it defaults to the version currently loaded into the Lisp world. status should be one of the keywords above.

37.8 Saving New Versions: Disk Partitions

37.8.1 Concepts

The make-system and load-patches functions, described above, load software into the Lisp world. This takes time; it is wasteful for everyone to sit through this loading of software every time the software is to be used. Usually someone loads up software into a Lisp world and then saves away the whole Lisp world in a partition on a disk. This section explains how to do this and other things.

A Lisp Machine disk is divided into several named partitions (also called "bands" sometimes). Partitions can be used for many things. Every disk has a partition named PAGE, which is used to implement the virtual memory of the Lisp Machine. When you run Lisp, this is where the Lisp world actually resides. There are also partitions that hold saved images of the Lisp Machine microcode, conventionally named MCRn (where n is a digit), and partitions that hold saved images of Lisp worlds, conventionally named LODn. A saved image of a Lisp world is also called a "virtual memory load" or "system load".

The directory of partitions is in a special block on the disk called the label. When you "cold-boot" a Lisp Machine by typing CTRL/META/CTRL/META-Rubout, the machine checks the label to see which two partitions contain two important "files": the current microcode load, and the current saved image of the Lisp world. These are kept separate so that the microcode can be easily changed without going through the time-consuming process of generating a new system load. When you "cold-boot", the contents of the current microcode band are loaded into the microcode memory, and then the contents of the current saved image of the Lisp world is copied into the PAGE partition. Then Lisp starts running.

For each partition, the directory of partitions contains a brief textual description of the contents of the partition. For microcode partitions, a typical description might be "UCADR 739"; this means that version 739 of the microcode is in the partition. For saved Lisp images, it is a little more complicated. Ideally, the description would say which versions of which systems are loaded into the band. Unfortunately, there isn't enough room for that in most cases. A typical description is "65.8 ZMail 19.1", meaning that this band contains version 65.8 of System and version 19.1 of ZMail. The description is created when a Lisp world is saved away by disk-save (see below).

37.8.2 Manipulating the Label

Function: print-disk-label &optional (unit 0) (stream standard-output)

Print a description of the label of the disk specified by unit onto stream. The description starts with the name of the disk pack, various information about the disk that is generally uninteresting, and the names of the two current load partitions (microcode and saved Lisp image). This is followed by one line of description for each partition. Each one has a name, disk address, size, and textual description. The two partitions that are the current load partitions, used when you cold-boot, are preceeded by asterisks. unit may be the unit number of the disk (most Lisp machines just have one unit, numbered 0), or the "host name" of another Lisp Machine on the Chaosnet (in which case the label of unit 0 on that machine will be printed, and the user of that machine will be notified that you are looking at his label).

Function: set-current-band partition-name

Set the current saved Lisp image partition to be partition-name. If partition-name is a number, the name LODn will be used.

Function: set-current-microload partition-name

Set the current microcode partition to be partition-name. If partition-name is a number, the name MCRn will be used.

When using the functions to set the current load partitions, be extra sure that you are specifying the correct partition. Having done it, cold-booting the machine will reload from those partitions. Some versions of the microcode will not work with some versions of the Lisp system, and if you set the two current partitions incompatibly, cold-booting the machine will fail; you will need an expert to fix this.

Function: si:edit-disk-label unit &optional init-p

This runs an interactive label editor on the specified unit. This editor allows you to change any field in the label. The HELP key documents the commands. You have to be an expert to need this and to understand what it does, so the commands are not documented here. Ask someone if you need help.

Function: print-loaded-band &optional format-dest

Tells you what you are currently running. This includes where it came from on the disk and what version of each system is present in your Lisp environment. format-dest defaults to t; if it is nil the answer will be returned as a string rather than printed out.

Function: disk-restore &optional partition

Allows booting from a band other than the current one. partition may be the name or the number of a disk partition containing a virtual-memory load, or nil or omitted, meaning to use the current partition. The specified partition is copied into the paging area of the disk and then started.

Although you can use this to boot a different Lisp image than the installed one, this does not provide a way to boot a different microcode image. disk-restore brings up the new band with the currently running microcode.

disk-restore asks the user for confirmation before doing it.

37.8.3 Updating Software

Of all the procedures described in this section, the most common one is to take a partition containing a Lisp image, update it to have all the latest patches, and save it away into a partition.

The way you do this is to start by cold-booting the machine, to get a fresh, empty system. Next, you must log in as something whose INIT file does not affect the Lisp world noticably (so that when you save away the Lisp image, the side-effects of the INIT file won't get saved too); you can log in as "LISPM". Now you can load in any new software you want; usually you just do (load-patches) and answer the questions, to bring all the present patchable systems up to date, but you might also add a new system and load it up. You may also want to call si:set-system-status to change the release status of the system.

When you're done loading everything, do (print-disk-label) to find a band in which to save your new Lisp world. It is best not to reuse the current band, since if something goes wrong during the saving of the partition, while you have written, say, half of the band that is current, it may be impossible to cold-boot the machine. Once you have found the partition, you use the disk-save function to save everything into that partition.

Function: disk-save partition-name

Save the current Lisp world in the designated partition. partition-name may be a partition name (a string), or it may be a number in which case the name LODn is used.

It first asks you for yes-or-no confirmation that you really want to reuse the named partition. Then it tries to figure out what to put into the textual description of the label. It starts with the brief version of si:system-version-info (see (si:system-version-info-fun)). Then it asks you for an "additional comment" to append to this; usually you just type a return here, but you can also add a comment that will be returned by si:system-version-info (and thus printed when the system is booted) from then on. If this doesn't fit into the fixed size available for the textual description, it asks you to retype the whole thing (the version info as well as your comment) in a compressed form that will fit. The compressed version will appear in the textual description in print-disk-label.

The Lisp environment is then saved away into the designated partition, and then the equivalent of a cold-boot from that partition is done.

Once the patched system has been successfully saved and the system comes back up, you can make it current with set-current-band.

Please don't save patched systems that have had the editor or the compiler run. This works, but it makes the saved system a lot bigger. You should try to do as little as possible between the time you cold-boot and the time you save the partition, in order to produce a clean saved environment.

Variable: si:login-history

The value of si:login-history is a list of entries, one for each person who has logged into this world since it was created. This makes it possible to tell who disk-saved a band with something broken in it. Each entry is a list of the user ID, the host logged into, the Lisp machine on which the world was being executed, and the date and time.

37.8.4 Installing New Software

The version numbers of the current microcode and system are announced to the INFO-LISPM mailing list. When a new system becomes available, mail is sent to the list explaining where to find the new system and what is new about it. Sometimes a microcode and a system go together, and the new system will not work with the old microcode and vice versa. When this happens extra care is required to avoid getting incompatible loads current at the same time so that the machine will not be able to boot itself.

All of the extant microcode versions can be found on the LISPM1 directory on AI. Microcode version nnn is in AI: LISPM1; UCADR nnnMCR. To copy a new microcode version into one of the microcode load partitions, first do a (print-disk-label) to ensure that the partition you intend to bash is not the current one; if it was, and something went wrong in the middle of loading the new microcode, it would be impossible to cold-boot, and this is hard to fix.

Then, install the microcode (on the non-current partition) by using si:load-mcr-file.

Function: si:load-mcr-file microcode-file partition

Load the contents of the file microcode-file into the designated partition. microcode-file is either the version number of the system microcode to be loaded, or the pathname of a file containing microcode (in "MCR" format), such as "AI: LISPM1; UCADR nnnMCR". partition is either the number of an MCR partition, or the name of one, such as "MCR1". This takes about 30 seconds.

The system load, unlike the microcode load, is much too large to fit in an AI file. Therefore, the only way to install an updated system on a machine is to copy it from another machine that already has it. So the first step is to find a machine that is not in use and has the desired system. We will call this the source machine. The machine where the new system will be installed is the target machine. You can see who is logged into which machines, see which ones are free, and use print-disk-label with an argument to examine the label of that machine's disk and see if it has the system you want.

The function for actually copying a system load partition off of another machine is called as follows. Before doing this, double-check the partition names by printing the labels of both machines, and make sure no one is using the source machine.

Function: si:receive-band source-host source-band target-band

Copy the partition on source-host's partition named source-band onto the local machine's partition named target-band. ("Band" means "partition".) This takes about ten minutes. It types out the size of the partition in pages, and types a number every 100 pages telling how far it has gotten. It puts up a display on the remote machine saying what's going on.

To go the other direction, use si:transmit-band.

Function: si:transmit-band source-band target-host target-band

This is just like si:receive-band, except you use it on the source machine instead of the target machine. It copies the local machine's partition named source-band onto target-machine's partition named target-band.

After transferring the band, it is good practice to make sure that it really was copied successfully by comparing the original and the copy. All of the known reasons for errors during band transfer have (of course) been corrected, but peace of mind is valuable. If the copy was not perfectly faithful, you might not find out about it until a long time later, when you use whatever part of the system that had not been copied properly.

Function: si:compare-band source-host source-band target-band

This is like si:receive-band, except that it does not change anything. It compares the two bands and complains about any differences.

Having gotten the current microcode load and system load copied into partitions on your machine, you can make them current using set-current-microload and set-current-band. Double-check everything with print-disk-label. Then cold-boot the machine, and the new system should come up in a half-minute or so.

If the microcode you installed is not the same version as was installed on the source machine from which you got the system load, you will need to follow the procedure given below under "installing new microcode". This can happen if someone hasn't installed the current microcode yet on that other machine.

37.8.5 Installing New Microcode

When an existing system is to be used with a new microcode, certain changes need to be made to the system, and it should then be dumped back out with the changes. Usually new microcode is released only along with a new system, so you hardly ever have to do this. The error handler has a table of errors that are detected by microcode. The hardware/microcode debugger (CC) has a microcode symbol table. These symbols are used when debugging other machines, and are also used by certain metering programs. These tables should be updated when a new microcode is installed.

The error-handler will automatically update its table (from a file on the AI:LISPM1; directory) when the machine is booted with the new microcode. The CC symbol table is updated by the following procedure:

(login 'lispm) (pkg-goto 'cadr) (cc-load-ucode-symbols "AI: LISPM1; UCADR nnnSYM") (pkg-goto)

where nnn is the microcode version number. This operation will take a minute or two; after it has read in most of the file the machine will stop for a long time while it sorts the symbols. It will look like it has crashed, but it hasn't, really, and will eventually come back.

After booting the system with the new microcode and following the above procedure, the updated system should be saved with disk-save as explained above. Note that this operation does not change the system version number. Once the new band is verified to work, the old band can be removed from the label with si:edit-disk-label if desired.