36. Maintaining Large Systems

When a program gets large, it is often desirable to split it up into several files. One reason for this is to help keep the parts of the program organized, to make things easier to find. It's also useful to have the program broken into small pieces that are more convenient to edit and compile. It is particularly important to avoid the need to recompile all of a large program every time any piece of it changes; if the program is broken up into many files, only the files that have changes in them need to be recompiled.

The apparent drawback to splitting up a program is that more "commands" are needed to manipulate it. To load the program, you now have to load several files separately, insead of just loading one file. To compile it, you have to figure out which files need compilation, by seeing which have been edited since they were last compiled, and then you have to compile those files.

What's even more complicated is that files can have interdependencies. You might have a file called "DEFS" that contains some macro definitions (or flavor or structure definitions), and functions in other files might use those macros. This means that in order to compile any of those other files, you must first load the file "DEFS" into the Lisp environment, so that the macros will be defined and can be expanded at compile time. You'd have to remember this whenever you compile any of those files. Furthermore, if "DEFS" has changed, other files of the program might need to be recompiled because the macros might have changed and need to be re-expanded.

This chapter describes the system facility, which takes care of all these things for you. The way it works is that you define a set of files to be a system, using the defsystem special form, described below. This system definition says which files make up the system, which ones depend on the presence of others, and so on. You put this system definition into its own little file, and then all you have to do is load that file and the Lisp environment will know about your system and what files are in it. You can then use the make-system function (see (make-system-fun)) to load in all the files of the system, or recompile all the files that need compiling, and so on.

The system facility is very general and extensible. This chapter explains how to use it and how to extend it. This chapter also explains the patch facility, which lets you conveniently update a large program with incremental changes, and how to save away Lisp environments in disk partitions.

36.1 Defining a System

Special Form: defsystem name (keyword args...)...

Defines a system named name. The options selected by the keywords are explained in detail later. In general, they fall into two categories: properties of the system and transformations. A transformation is an operation such as compiling or loading which takes one or more files and does something to them. The simplest system is a set of files and a transformation to be performed on them.

Here are a few examples.

(defsystem mysys (:compile-load ("AI: GEORGE; PROG1" "AI: GEORG2; PROG2"))) (defsystem zmail (:name "ZMail") (:pathname-default "AI: ZMAIL;") (:package zwei) (:module defs "DEFS") (:module mult "MULT" :package tv) (:module main ("TOP" "COMNDS" "MAIL" "USER" "WINDOW" "FILTER" mult "COMETH")) (:compile-load defs) (:compile-load main (:fasload defs))) (defsystem bar (:module reader-macros "RDMAC") (:module other-macros "MACROS") (:module main-program "MAIN") (:compile-load reader-macros) (:compile-load other-macros (:fasload reader-macros)) (:compile-load main-program (:fasload reader-macros other-macros)))

The first example defines a new system called mysys, which consists of two files, both of which are to be compiled and loaded. The second example is somewhat more complicated. What all the options mean will be specified shortly, but the primary difference is that there is a file defs which must be loaded before the rest of the files (main) can be compiled. The final example has two levels of dependency. reader-macros must be compiled and loaded before other-macros can be compiled. Both reader-macros and other-macros must then be loaded before main-program can be compiled.

The defsystem options other than transformations are:

.kitem :name Specifies a "pretty" version of the name for the system, for use in printing. .kitem :short-name Specified an abbreviated name used in constructing disk label comments and in patch file names for some file systems. .kitem :component-systems Specifies the names of other systems used to make up this system. Performing an operation on a system with component systems is equivalent to performing the same operation on all the individual systems. The format is (:component-systems names...). .kitem :package Specifies the package in which transformations are performed. A package specified here will override one in the 7-*- line of the file in question. .kitem :pathname-default Gives a local default within the definition of the system for strings to be parsed into pathnames. Typically this specifies the directory, when all the files of a system are on the same directory. .kitem :patchable Makes the system be a patchable system (see (patch-facility)). An optional argument specifies the directory to put patch files in. The default is the :pathname-default of the system. .kitem :initial-status Specifies what the status of the system should be when make-system is used to create a new major version. The default is :experimental. See (patch-system-status) for further details. .kitem :not-in-disk-label Make a patchable system not appear in the disk label comment. This should probably never be specified for a user system. It is used by patchable systems internal to the main Lisp system, to avoid cluttering up the label. .kitem :module Allows assigning a name to a set of files within the system. This name can then be used instead of repeating the filenames. The format is (:module name files options...). files is a module-specification, which can be any of the following:

a string

This is a file name.

a symbol

This is a module name. It stands for all of the files which are in that module of this system.

an external module component

This is a list of the form (system-name module-names...), to specify modules in another system. It stands for all of the files which are in all of those modules.

a list of module components

A module component is any of the above, or the following:

a list of file names

This is used in the case where the names of the input and output files of a transformation are not related according to the standard naming conventions, for example when a QFASL file has a different name or resides on a different directory than the source file. The file names in the list are used from left to right, thus the first name is the source file. Each file name after the first in the list is defaulted from the previous one in the list.

To avoid syntactic ambiguity, this is allowed as a module component but not as a module specification.

The currently defined options for the :module clause are

:package

Overrides any package specified for the whole system for transformations performed on just this module.

In the second defsystem example above, there are three modules. The first two each have only one file, and the third one (main) is made up both of files and another module. To take examples of the other possibilities,

(:module prog (("AI: GEORGE; PROG" "AI: GEORG2; PROG"))) (:module foo (defs (zmail defs)))

The prog module consists of one file, but it lives in two directories, GEORGE and GEORG2. If this were a Lisp program, that would mean that the file "AI: GEORGE; PROG >" would be compiled into "AI: GEORG2; PROG QFASL". The foo module consists of two other modules, the defs module in the same system, and the defs module in the zmail system. It is not generally useful to compile files that belong to other systems, thus this foo module would not normally be the subject of a transformation. However, dependencies (defined below) use modules and need to be able to refer to (depend on) modules of other systems.

36.2 Transformations

Transformations are of two types, simple and complex. A simple transformation is a single operation on a file, such as compiling it or loading it. A complex transformation takes the output from one transformation and performs another transformation on it, for example, loading the results of compilation.

The general format of a simple transformation is (name input dependencies condition). input is usually a module specification or another transformation whose output is used. The transformation name is to be performed on all the files in the module, or all the output files of the other transformation.

dependencies and condition are optional.

dependencies is a transformation specification, either a list (transformation-name module-names...), or a list of such lists. A module-name is either a symbol which is the name of a module in the current system, or a list (system-name module-names...). A dependency declares that all of the indicated transformations must be performed on the indicated modules before the current transformation itself can take place. Thus in the zmail example above, the defs module must have the :fasload transformation performed on it before the :compile transformation can be performed on main.

The dependency has to be a tranformation that was explicitly specified as a transformation in the system definition, not just an action that might have been performed by anything. That is, if you have a dependency (:fasload foo), it means that (fasload foo) is a tranformation of your system and you depend on that tranformation; it does not simply mean that you depend on foo's being loaded. Furthermore, it doesn't work if (:fasload foo) was an implicit piece of another tranformation. For example, the following is correct and will work:

(defsystem foo (:module foo "FOO") (:module bar "BAR") (:compile-load (foo bar)))

but this doesn't work:

(defsystem foo (:module foo "FOO") (:module bar "BAR") (:module blort "BLORT") (:compile-load (foo bar)) (:compile-load blort (:fasload foo)))

because foo's :fasload is not mentioned explicitly (i.e. at top level) but is only implicit in the (:compile-load (foo bar)). One must instead write:

(defsystem foo (:module foo "FOO") (:module bar "BAR") (:module blort "BLORT") (:compile-load foo) (:compile-load bar) (:compile-load blort (:fasload foo)))

condition is a predicate which specifies when the transformation should take place. Generally it defaults according to the type of the transformation. Conditions are discussed further on (transformation-condition-discussion).

The defined simple transformations are:

:fasload

Calls the fasload function to load the indicated files, which must be QFASL files. The condition defaults to si:file-newer-than-installed-p, which is t if a newer version of the file exists on the file computer than was read into the current environment.

:readfile

Calls the readfile function to read in the indicated files. Use this for files that are not to be compiled. condition defaults to si:file-newer-than-installed-p.

:compile

Calls the qc-file function to compile the indicated files. condition defaults to si:file-newer-than-file-p which returns t if the source file has been written more recently than the binary file.

A special simple transformation is

:do-components

(:do-components dependencies) inside a system with component systems will cause the dependencies to be done before anything in the component systems. This is useful when you have a module of macro files used by all of the component systems.

The defined complex transformations are

:compile-load

(:compile-load input compile-dependencies load-dependencies compile-condition load-condition) is the same as (:fasload (:compile input compile-dependencies compile-condition) load-dependencies load-condition). This is the most commonly-used transformation. Everything after input is optional.

:compile-load-init

See (compile-load-init-transformation).

As was explained above, each filename in an input specification can in fact be a list of strings for the case where the source file of a program differs from the binary file in more than just the file type. In fact, every filename is treated as if it were an infinite list of filenames with the last filename, or in the case of a single string the only filename, repeated forever at the end. Each simple transformation takes some number of input filename arguments, and some number of output filename arguments. As transformations are performed, these arguments are taken from the front of the filename list. The input arguments are actually removed and the output arguments left as input arguments to the next higher transformation. To make this clearer, consider the prog module above having the :compile-load transformation performed on it. This means that prog is given as the input to the :compile transformation and the output from this transformation is given as the input to the :fasload transformation. The :compile transformation takes one input filename argument, the name of a lisp source file, and one output filename argument, the name of the qfasl file. The :fasload transformation takes one input filename argument, the name of a qfasl file, and no output filename arguments. So, for the first and only file in the prog module, the filename argument list looks like ("AI: GEORGE; PROG" "AI: GEORG2; PROG" "AI: GEORG2; PROG" ...). The :compile transformation is given arguments of "AI: GEORGE; PROG" and "AI: GEORG2; PROG" and the filename argument list which it outputs as the input to the :fasload transformation is ("AI: GEORG2; PROG" "AI: GEORG2; PROG" ...). The :fasload transformation then is given its one argument of "AI: GEORG2; PROG".

Note that dependencies are not "transitive" nor "inherited". For example, if module a depends on macros defined in module b, and therefore needs b to be loaded in order to compile, and b has a similar dependency on c, c will not be loaded during compilation of a. Transformations with these dependencies would be written

(:compile-load a (:fasload b)) (:compile-load b (:fasload c))

To say that compilation of a depends on both b and c, you would instead write

(:compile-load a (:fasload b c)) (:compile-load b (:fasload c))

If in addition a depended on c (but not b) during loading (perhaps a contains defvars whose initial values depend on functions or special variables defined in c) you would write the transformations

(:compile-load a (:fasload b c) (:fasload c)) (:compile-load b (:fasload c))



36.3 Making a System

Function: make-system name &rest keywords

The make-system function does the actual work of compiling and loading. In the example above, if PROG1 and PROG2 have both been compiled recently, then

(make-system 'mysys)

will load them as necessary. If either one might also need to be compiled, then

(make-system 'mysys ':compile)

will do that first as necessary.

make-system lists what transformations it is going to perform on what files, asks the user for confirmation, then performs the transformations. Before each transformation a message is printed listing the transformation being performed, the file it is being done to, and the package. This behavior can be altered by keywords.

These are the keywords recognized by the make-system function and what they do.

.kitem :noconfirm Assumes a yes answer for all questions that would otherwise be asked of the user. .kitem :selective Asks the user whether or not to perform each transformation that appears to be needed for each file. .kitem :silent Avoids printing out each transformation as it is performed. .kitem :reload Bypasses the specified conditions for performing a transformation. Thus files are compiled even if they haven't changed and loaded even if they aren't newer than the installed version. .kitem :noload Does not load any files except those required by dependencies. For use in conjunction with the :compile option. .kitem :compile Compiles files also if need be. The default is to load but not compile. .kitem :no-increment-patch When given along with the :compile option, disables the automatic incrementing of the major system version that would otherwise take place. See (patch-facility). .kitem :increment-patch Increment a patchable system's major version without doing any compilations. See (patch-facility). .kitem :batch Allows a large compilation to be done unattended. It acts like :noconfirm with regard to questions, turns off more-processing and fdefine-warnings (see inhibit-fdefine-warnings, (inhibit-fdefine-warnings-var)), and saves the compiler warnings in an editor buffer and a file (it asks you for the name). .kitem :print-only Just prints out what transformations would be performed, does not actually do any compiling or loading. .kitem :noop Is ignored. This is mainly useful for programs that call make-system, so that such programs can include forms like

(make-system 'mysys (if compile-p ':compile ':noop))

36.4 Adding New Keywords to make-system

make-system keywords are defined as functions on the si:make-system-keyword property of the keyword. The functions are called with no arguments. Some of the relevant variables they can use are

Variable: si:*system-being-made*

The internal data structure which represents the system being made.

Variable: si:*make-system-forms-to-be-evaled-before*

A list of forms which are evaluated before the transformations are performed.

Variable: si:*make-system-forms-to-be-evaled-after*

A list of forms which are evaluated after the transformations have been performed.

Variable: si:*make-system-forms-to-be-evaled-finally*

A list of forms which are evaluated after the body of make-system has completed. This differs from si:*make-system-forms-to-be-evaled-after* in that these forms are evaluated outside of the "compiler context", which sometimes makes a difference.

Variable: si:*query-type*

Controls how questions are asked. Its normal value is :normal. :noconfirm means no questions will be asked and :selective asks a question for each individual file transformation.

Variable: si:*silent-p*

If t, no messages are printed out.

Variable: si:*batch-mode-p*

If t, :batch was specified.

Variable: si:*redo-all*

If t, all transformations are performed, regardless of the condition functions.

Variable: si:*top-level-transformations*

A list of the names of transformations that will be performed, such as (:fasload :readfile).

Variable: si:*file-transformation-function*

The actual function that gets called with the list of transformations that need to be performed. The default is si:do-file-transformations.

Special Form: si:define-make-system-special-variable variable value [defvar-p]

Causes variable to be bound to value, which is evaluated at make-system time, during the body of the call to make-system. This allows you to define new variables similar to those listed above. If defvar-p is specified as (or defaulted to) t, variable is defined with defvar. It is not given an initial value. If defvar-p is specified as nil, variable belongs to some other program and is not defvar'ed here.

The following simple example adds a new keyword to make-system called :just-warn, which means that fdefine warnings (see (fdefine-fun)) regarding functions being overwritten should be printed out, but the user should not be queried.

(si:define-make-system-special-variable inhibit-fdefine-warnings inhibit-fdefine-warnings nil) (defun (:just-warn si:make-system-keyword) () (setq inhibit-fdefine-warnings ':just-warn))

(See the description of the inhibit-fdefine-warnings variable, on (inhibit-fdefine-warnings-var).)

make-system keywords can have effect either directly when called, or by pushing a form to be evaluated onto si:*make-system-forms-to-be-evaled-after* or one of the other two similar lists. In general, the only useful thing to do is to set some special variable defined by si:define-make-system-special-variable. In addition to the ones mentioned above, user-defined transformations may have their behavior controlled by new special variables, which can be set by new keywords. If you want to get at the list of transformations to be performed, for example, the right way would be to set si:*file-transformation-function* to a new function, which then might call si:do-file-transformations with a possibly modified list. That is how the :print-only keyword works.

36.5 Adding New Options for defsystem

Options to defsystem are defined as macros on the si:defsystem-macro property of the option keyword. Such a macro can expand into an existing option or transformation, or it can have side effects and return nil. There are several variables they can use; the only one of general interest is

Variable: si:*system-being-defined*

The internal data structure which represents the system which is currently being constructed.

Special Form: si:define-defsystem-special-variable variable value

Causes value to be evaluated and variable to be bound to the result during the expansion of the defsystem special form. This allows you to define new variables similar to the one listed above.

Special Form: si:define-simple-transformation

This is the most convenient way to define a new simple transformation. The form is

(si:define-simple-transformation name function default-condition input-file-types output-file-types pretty-names compile-like load-like)

For example,

(si:define-simple-transformation :compile si:qc-file-1 si:file-newer-than-file-p ("LISP") ("QFASL"))

input-file-types and output-file-types are how a transformation specifies how many input filenames and output filenames it should receive as arguments, in this case one of each. They also, obviously, specify the default file type for these pathnames. The si:qc-file-1 function is mostly like qc-file, except for its interface to packages. It takes input-file and output-file arguments.

pretty-names, compile-like, and load-like are optional.

pretty-names specifies how messages printed for the user should print the name of the transformation. It can be a list of the imperative ("Compile"), the present participle ("Compiling"), and the past participle ("compiled"). Note that the past participle is not capitalized, because it is not used at the beginning of a sentence. pretty-names can be just a string, which is taken to be the imperative, and the system will conjugate the participles itself. If pretty-names is omitted or nil it defaults to the name of the transformation.

compile-like and load-like say when the transformation should be performed. Compile-like transformations are performed when the :compile keyword is given to make-system. Load-like transformations are performed unless the :noload keyword is given to make-system. By default compile-like is t but load-like is nil.

Complex transformations are just defined as normal macro expansions, for example,

(defmacro (:compile-load si:defsystem-macro) (input &optional com-dep load-dep com-cond load-cond) `(:fasload (:compile ,input ,com-dep ,com-cond) ,load-dep ,load-cond))

36.6 More Esoteric Transformations

It is sometimes useful to specify a transformation upon which something else can depend, but which is not performed by default, but rather only when requested because of that dependency. The transformation nevertheless occupies a specific place in the hierarchy. The :skip defsystem macro allows specifying a transformation of this type. For example, suppose there is a special compiler for the read table which is not ordinarily loaded into the system. The compiled version should still be kept up to date, and it needs to be loaded if ever the read table needs to be recompiled.

(defsystem reader (:pathname-default "AI: LMIO;") (:package system-internals) (:module defs "RDDEFS") (:module reader "READ") (:module read-table-compiler "RTC") (:module read-table "RDTBL") (:compile-load defs) (:compile-load reader (:fasload defs)) (:skip :fasload (:compile read-table-compiler)) (:rtc-compile-load read-table (:fasload read-table-compiler)))

Assume that there is a complex transformation :rtc-compile-load which is like :compile-load, except that is is built on a transformation called something like :rtc-compile, which uses the read table compiler rather than the Lisp compiler. In the above system, then, if the :rtc-compile transformation is to be performed, the :fasload transformation must be done on read-table-compiler first, that is the read table compiler must be loaded if the read table is to be recompiled. If you say (make-system 'reader ':compile), then the :compile transformation will still happen on the read-table-compiler module, compiling the read table compiler if need be. But if you say (make-system 'reader), the reader and the read table will be loaded, but the :skip keeps this from happening to the read table compiler.

.setq transformation-condition-discussion page So far nothing has been said about what can be given as a condition for a transformation except for the default functions which check for a source file being newer than the binary and so on. In general, any function which takes the same arguments as the transformation function (e.g. qc-file) and returns t if the transformation needs to be performed, can be in this place as a symbol, including for example a closure. To take an example, suppose there is a file which contains compile-flavor-methods for a system, and which should therefore be recompiled if any of the flavor method definitions change. In this case, the condition function for compiling that file should return t if either the source of that file itself or any of the files that define the flavors have changed. This is what the :compile-load-init complex transformation is for. It is defined like this:

.setq compile-load-init-transformation page (defmacro (:compile-load-init si:defsystem-macro) (input add-dep &optional com-dep load-dep &aux function) (setq function (let-closed ((*additional-dependent-modules* add-dep)) 'compile-load-init-condition)) `(:fasload (:compile ,input ,com-dep ,function) ,load-dep)) (defun compile-load-init-condition (source-file qfasl-file) (or (si:file-newer-than-file-p source-file qfasl-file) (local-declare ((special *additional-dependent-modules*)) (si:other-files-newer-than-file-p *additional-dependent-modules* qfasl-file))))

The condition function which will be generated when this macro is used returns t either if si:file-newer-than-file-p would with those arguments, or if any of the other files in add-dep, which presumably is a module specification, are newer than the qfasl file. Thus the file (or module) to which the :compile-load-init transformation applies will be compiled if it or any of the source files it depends on has been changed, and will be loaded under the normal conditions. In most (but not all cases), com-dep would be a :fasload transformation of the same files as add-dep specifies, so that all the files this one depends on would be loaded before compiling it.