\input texinfo @c -*- texinfo -*- @c %**start of header @setfilename cl-store.texi @settitle CL-STORE Manual @dircategory Software development @direntry * cl-store: (cl-store). CL Serialization Package @end direntry @copying Copyright @copyright{} (c) (C) 2004 Sean Ross All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The names of the authors and contributors may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. @end copying @c @titlepage @title CL-STORE: CL Serialization Package @page @vskip 0pt plus 1filll @insertcopying @end titlepage @contents @ifnottex @node Top @top CL-STORE: CL Serialization Package @insertcopying @menu * Introduction: Introduction * Getting Started: Getting Started * API: API * Customizing: Customizing * New Backends: New Backends * Notes: Notes * Credits: Credits * Index:: @end menu @end ifnottex @node Introduction @chapter Introduction CL-STORE is a portable serialization package for Common Lisp which allows the reading and writing of most objects found in Common Lisp resolving any circularities which it detects. It is intended to serve the same purpose as Java's ObjectOutput and ObjectInputStream, although it's somewhat more extensible. The CL-STORE Home Page is at @uref{http://common-lisp.net/project/cl-store} where one can find details about mailing lists, cvs repositories and various releases. This documentation is for CL-STORE version 0.6 . Enjoy Sean. @section Example @lisp (defclass myclass () ((a :accessor a :initarg :a))) (cl-store:store (make-instance 'myclass :a 3) "/tmp/test.out") (a (cl-store:restore "/tmp/test.out")) @end lisp @section Supported Objects @itemize @bullet @item Numbers (floats, integers, complex, NaN floats, rationals) @item Strings (Supports Unicode Strings) @item Characters @item Symbols @item Packages @item HashTables @item Lists @item Vectors And Arrays @item Instances of CLOS Classes @item CLOS Classes @item Structure Instances @item Structure Definitions (CMUCL and SBCL only) @item Functions (stores the function name) @item Generic Functions (stores generic-function-name) @end itemize @section Supported Implementations @itemize @bullet @item SBCL @item CMUCL @item CLISP @item Lispworks @item Allegro CL @item OpenMCL @item ECL @end itemize @node Getting Started @chapter Getting Started CL-STORE uses @uref{http://cliki.net/asdf,,asdf} as it's system definition tool and is required whenever you load the package. You will need to download it, or if you have @uref{http://sbcl.org,,sbcl} @code{(require 'asdf)} @section Downloading @itemize @item ASDF-INSTALL CL-STORE is available through asdf-install. If you are new to Common Lisp this is the suggested download method. With asdf-install loaded run @code{(asdf-install:install :cl-store)} This will download and install the package for you. Asdf-install will try to verify that the package signature is correct and that you trust the author. If the key is not found or the trust level is not sufficient a continuable error will be signalled. You can choose to ignore the error and continue to install the package. See the documentation of asdf-install for more details. @item DOWNLOAD The latest cl-store release will always be available from @uref{http://common-lisp.net,,cl.net}. Download and untar in an appropriate directory then symlink @file{cl-store.asd} to a directory on @code{asdf:*central-registry*} (see the documentation for asdf for details about setting up asdf). @item CVS If you feel the need to be on the bleeding edge you can use anonymous CVS access, see the @uref{http://common-lisp.net/project/cl-store,,Home Page} for more details for accessing the archive. Once downloaded follow the symlink instructions above. @end itemize @section Installing Once downloaded and symlinked you can load CL-STORE at anytime using @code{(asdf:oos 'asdf:load-op :cl-store)} This will compile CL-STORE the first time it is loaded. @section Testing Once installed you can run the regression tests for it. The tests depend on the @uref{http://cliki.net/rt,,Regression Tests} asdf package which is asdf-installable. The tests can be run be executing @code{(asdf:oos 'asdf:test-op :cl-store)} If any tests fail please send a message to one of the Mailing Lists. @node API @chapter API @section Variables @anchor{Variable *nuke-existing-classes*} @vindex *nuke-existing-classes* @deftp {Variable} *nuke-existing-classes* @emph{Default NIL} Determines wether or not to override existing classes when restoring a CLOS Class. If @code{*nuke-existing-classes*} is not NIL the current definition will be overridden. @end deftp @anchor{Variable *store-class-superclasses*} @vindex *store-class-superclasses* @deftp {Variable} *store-class-superclasses* @emph{Default NIL} If @code{*store-class-superclasses*} is not NIL when storing a CLOS Class all superclasses will be stored. @end deftp @anchor{Variable *store-class-slots*} @vindex *store-class-slots* @deftp {Variable} *store-class-slots* @emph{Default T} If @code{*store-class-slots*} is NIL slots which are class allocated will not be serialized when storing objects. @end deftp @anchor{Variable *nuke-existing-packages*} @vindex *nuke-existing-packages* @deftp {Variable} *nuke-existing-packages* @emph{Default NIL} If @code{*nuke-existing-packages*} is non-nil then packages which already exist will be deleted when restoring packages. @end deftp @anchor{Variable *store-used-packages*} @vindex *store-used-packages* @deftp {Variable} *store-used-packages* @emph{Default NIL} The variable determines how packages on a package use list will be serialized. If non-nil the the package will be fully serialized, otherwise only the name will be stored. @end deftp @anchor{Variable *store-hash-size*} @vindex *store-hash-size* @deftp {Variable} *store-hash-size* @emph{Default 50} The default size of the hash-table created to keep track of objects which have already been stored. By binding the variable to a suitable value you can avoid the consing involved by rehashing hash-tables. @end deftp @anchor{Variable *restore-hash-size*} @vindex *restore-hash-size* @deftp {Variable} *restore-hash-size* @emph{Default 50} The default size of the hash-table created to keep track of objects which have already been restored. By binding the variable to a suitable value you can avoid the consing involved by rehashing hash-tables. @end deftp @anchor{Variable *check-for-circs*} @vindex *check-for-circs* @deftp {Variable} *check-for-circs* @emph{Default t} Binding this variable to nil when storing or restoring an object inhibits all checks for circularities which gives a severe boost to performance. The downside of this is that no restored objects will be eq and attempting to store circular objects will hang. The speed improvements are definitely worth it if you know that there will be no circularities or shared references in your data (eg spam-filter hash-tables). @end deftp @anchor{Variable *default-backend*} @vindex *default-backend* @deftp {Variable} *default-backend* The backend that will be used by default. @end deftp @section Functions @anchor{Generic store} @deffn {Generic} store object place &optional (backend *default-backend*) Stores @emph{object} into @emph{place} using @emph{backend}. @emph{Place} must be either a @code{stream} or a @code{pathname-designator}. All conditions signalled from store can be handled by catching @code{store-error}. If the @code{store-error} is not handled the causing error will be signalled. @end deffn @anchor{Generic restore} @deffn {Generic} restore place &optional (backend *default-backend*) Restores an object serialized using @code{store} from @emph{place} using @emph{backend}. @emph{Place} must be either a @code{stream} or a @code{pathname-designator}. Restore is setffable eg. @lisp (store 0 "/tmp/counter") (incf (restore "/tmp/counter")) @end lisp All conditions signalled from restore can be handled by catching @code{restore-error}. If the @code{restore-error} is not handled the causing error will be signalled. @end deffn @anchor{Function find-backend} @deffn {Function} find-backend name &optional (errorp nil) Return backup called @emph{name}. If there is no such backend NIL is returned if @emph{errorp} is false, otherwise an error is signalled. @end deffn @anchor{Function caused-by} @deffn {Function} caused-by cl-store-error Returns the @code{condition} which caused @code{cl-store-error} to be signalled. @end deffn @section Macros @anchor{Macro with-backend} @deffn {Macro} with-backend backend &body body Execute @emph{body} with @code{*default-backend*} bound to the backend designated by @emph{backend}. @end deffn @section Conditions @anchor{Condition cl-store-error} @deftp {Condition} cl-store-error Class Precedence: @code{condition} Root CL-STORE Condition all errors occuring while storing or restoring can be handled by catching @code{cl-store-error} @end deftp @anchor{Condition store-error} @deftp {Condition} store-error Class Precedence: @code{cl-store-error} A @code{store-error} will be signalled when an error occurs within @code{store} or @code{multiple-value-store}. The causing error can be obtained using @code{(caused-by condition)} @end deftp @anchor{Condition restore-error} @deftp {Condition} restore-error Class Precedence: @code{cl-store-error} A @code{restore-error} will be signalled when an error occurs within @code{restore}. The causing error can be obtained using @code{(caused-by condition)} @end deftp @node Customizing @chapter Customizing @section About Customizing Each backend in CL-STORE can be customized to store various values in a custom manner. By using the @code{defstore-} and @code{defrestore-} macros you can define your own methods for storing various objects. This may require a marginal understanding of the backend you wish to extend. eg. @lisp (in-package :cl-user) (use-package :cl-store) (setf *default-backend* (find-backend 'cl-store)) ;; Create the custom class (defclass random-obj () ((a :accessor a :initarg :a))) ;; Register random object. This is specific to the ;; cl-store-backend. (defvar *random-obj-code* (register-code 110 'random-obj)) ;; Create a custom storing method for random-obj ;; outputting the code previously registered. (defstore-cl-store (obj random-obj stream) (output-type-code *random-obj-code* stream) (store-object (a obj) stream)) ;; Define a restoring method. (defrestore-cl-store (random-obj stream) (random (restore-object stream))) ;; Test it out. (store (make-instance 'random-obj :a 10) "/tmp/random") (restore "/tmp/random") => ; some number from 0 to 9 @end lisp If you need to get fancier take a look at the macroexpansion of the customizing macros. @vskip 0pt plus 1filll @section Customizing API This API is primarily concerned with the cl-store-backend although other backends will be similar in structure. @subsection Functions @anchor{Function register-code} @deffn {Function} register-code code name &optional (errorp t) Registers @emph{name} under the code @emph{code} into the cl-store-backend. The backend will use this mapping when restoring values. Will signal an error if code is already registered and @emph{errorp} is not NIL. Currently codes 1 through 35 are in use. @end deffn @anchor{Function output-type-code} @deffn {Function} output-type-code type-code stream Writes @emph{type-code} into @emph{stream}. This must be done when writing out objects so that the type of the object can be identified on deserialization. @end deffn @anchor{Function store-32-bit} @deffn {Function} store-32-bit integer stream Outputs the the low 32 bits from @emph{integer} into @emph{stream}. @end deffn @anchor{Function read-32-bit} @deffn {Function} read-32-bit stream Reads a 32-bit integer from @emph{stream}. @end deffn @anchor{Generic store-object} @deffn {Generic} store-object object place Stores @emph{object} into @emph{place}. This should be used inside @code{defstore-cl-store} to output parts of objects. @code{store} should not be used. @end deffn @anchor{Generic restore-object} @deffn {Generic} restore-object place Restore an object, written out using @code{store-object} from @emph{place}. @end deffn @anchor{Generic get-slot-details} @deffn {Generic} get-slot-details slot-definition Generic function which returns a list of slots details which can be used as an argument to @code{ensure-class}. Currently it is only specialized on slot-definition @end deffn @anchor{Generic serializable-slots} @deffn {Generic} serializable-slots object Method which returns a list of slot-definition objects which will be serialized for @emph{object}. The default is to call @code{serializable-slots-using-class}. @end deffn @anchor{Generic serializable-slots-using-class} @deffn {Generic} serializable-slots-using-class object class Returns a list of slot-definition objects which will be serialized for object and class. Example. When serializing cl-sql objects to disk or to another lisp session the view-database slot should not be serialized. Instead of specializing serializable-slots for each view-class created you can do this. @lisp (defmethod serializable-slots-using-class ((object t) (class clsql-sys::standard-db-class)) (delete 'clsql-sys::view-database (call-next-method) :key 'slot-definition-name)) @end lisp @end deffn @vskip 0pt plus 1filll @subsection Macros @anchor{Macro defstore-cl-store} @deffn {Macro} defstore-cl-store (var type stream &key qualifier) &body body Create a custom storing mechanism for @emph{type} which must be a legal Class Name. @emph{Body} will be called when an object of class @emph{type} is stored using @code{store-object} with @emph{var} bound to the object to be stored and @emph{stream} bound to the stream to output to. If @emph{qualifier} is given it must be a legal qualifier to @code{defmethod}. Example. @lisp (defstore-cl-store (obj ratio stream) (output-type-code +ratio-code+ stream) (store-object (numerator obj) stream) (store-object (denominator obj) stream)) @end lisp @end deffn @anchor{Macro defrestore-cl-store} @deffn {Macro} defrestore-cl-store (type stream) &body body Create a custom restoring mechanism for the @emph{type} registered using @code{register-code}.@emph{Body} will be executed with @emph{stream} being the input stream to restore an object from. Example. @lisp (defrestore-cl-store (ratio stream) (/ (restore-object stream) (restore-object stream))) @end lisp @end deffn @anchor{Macro resolving-object} @deffn {Macro} resolving-object (var create) &body body Executes @emph{body} resolving circularities detected in @emph{object}. Resolving-object works by creating a closure, containing code to set a particular place in @emph{object}, which is then pushed onto a list. Once the object has been fully restored all functions on this list are called and the circularities are resolved. Example. @lisp (defrestore-cl-store (cons stream) (resolving-object (object (cons nil nil)) (setting (car object) (restore-object stream)) (setting (cdr object) (restore-object stream)))) @end lisp @end deffn @vskip 0pt plus 1filll @anchor{Macro setting} @deffn {Macro} setting place get This macro can only be used inside @code{resolving-object}. It sets the value designated by @emph{place} to @emph{get} for the object that is being resolved. Example. @lisp (defrestore-cl-store (simple-vector stream) (let* ((size (restore-object stream)) (res (make-array size))) (resolving-object (object res) (loop repeat size for i from 0 do ;; we need to copy the index so that ;; it's value is preserved for after the loop. (let ((x i)) (setting (aref object x) (restore-object stream))))) res)) @end lisp @end deffn @anchor{Macro setting-hash} @deffn {Macro} setting-hash getting-key getting-value @code{setting-hash} works identically to setting although it is used exclusively on hash-tables due to the fact that both the key and the value being restored could be a circular reference. Example. @lisp (defrestore-cl-store (hash-table stream) (let ((rehash-size (restore-object stream)) (rehash-threshold (restore-object stream)) (size (restore-object stream)) (test (restore-object stream)) (count (restore-object stream))) (let ((hash (make-hash-table :test (symbol-function test) :rehash-size rehash-size :rehash-threshold rehash-threshold :size size))) (resolving-object (obj hash) (loop repeat count do (setting-hash (restore-object stream) (restore-object stream)))) hash))) @end lisp @end deffn @node New Backends @chapter New Backends @section About You can define your own backends in cl-store to do custom object I/O. Theoretically one can add a backend that can do socket based communication with any language provided you know the correct format to output objects in. If the framework is not sufficient to add your own backend just drop me a line and we will see what we can do about it. @section The Process @subsection Add the backend Use @code{defbackend} to define the new backend choosing the output format, an optional magic number, extra fields for the backend and a backend to extend which defaults to the base backend. eg. (from the cl-store-backend) @lisp (defbackend cl-store :magic-number 1347643724 :stream-type '(unsigned-byte 8) :old-magic-numbers (1912923 1886611788 1347635532) :extends resolving-backend :fields ((restorers :accessor restorers :initform (make-hash-table)))) @end lisp @subsection Recognizing Objects. Decide how to recognize objects on restoration. When restoring objects the backend has a responsibility to return a symbol identifying the @code{defrestore} method to call by overriding the @code{get-next-reader} method. In the cl-store backend this is done by keeping a mapping of type codes to symbols. When storing an object the type code is written down the stream first and then the restoring details for that particular object. The @code{get-next-reader} method is then specialized to read the type code and look up the symbol in a hash-table kept on the backend. eg. (from the cl-store-backend) @lisp (defvar *cl-store-backend* (find-backend 'cl-store)) ;; This is a util method to register the code with a symbol (defun register-code (code name &optional (errorp t)) (aif (and (gethash code (restorers *cl-store-backend*)) errorp) (error "Code ~A is already defined for ~A." code name) (setf (gethash code (restorers *cl-store-backend*)) name)) code) ;; An example of registering the code 7 with ratio (defconstant +ratio-code+ (register-code 7 'ratio)) ;; Extending the get-next-reader method (defmethod get-next-reader ((backend cl-store) (stream stream)) (let ((type-code (read-type-code stream))) (or (gethash type-code (restorers backend)) (values nil (format nil "Type ~A" type-code))))) (defstore-cl-store (obj ratio stream) (output-type-code +ratio-code+ stream) ;; output the type code (store-object (numerator obj) stream) (store-object (denominator obj) stream)) @end lisp @subsection Extending the Resolving backend If you are extending the @code{resolving-backend} you have a couple of extra responsibilities to ensure that circular references are resolved correctly. @code{Store-referrer} must be extended for your backend to output the referrer code. This must be done as if it were a @code{defstore} for a referrer. A @code{defrestore-} must also be defined for the referrer which must return a referrer created with @code{make-referrer}. Once that is done you can use @code{resolving-object} and @code{setting} to resolve circularities in objects. eg (from the cl-store backend) @lisp (defconstant +referrer-code+ (register-code 1 'referrer nil)) (defmethod store-referrer (ref stream (backend cl-store)) (output-type-code +referrer-code+ stream) (store-32-bit ref stream)) (defrestore-cl-store (referrer stream) (make-referrer :val (read-32-bit stream nil))) @end lisp @section Example: Simple Pickle Format As a short example we will define a backend that can handle simple objects using the python pickle format. @subsection Define the backend @lisp (in-package :cl-user) (use-package :cl-store) (defbackend pickle :stream-type 'character) @end lisp @vskip 0pt plus 2filll @subsection Recognize Objects This is just a simple example to be able to handle single strings stored with Python's pickle module. @lisp (defvar *pickle-mapping* '((#\S . string))) (defmethod get-next-reader ((backend pickle) (stream stream)) (let ((type-code (read-char stream))) (or (cdr (assoc type-code *pickle-mapping*)) (values nil (format nil "Type ~A" type-code))))) (defrestore-pickle (noop stream)) (defstore-pickle (obj string stream) (format stream "S'~A'~%p0~%." obj)) (defrestore-pickle (string stream) (let ((val (read-line stream))) (read-line stream) ;; remove the PUSH op (read-line stream) ;; remove the END op (subseq val 1 (1- (length val))))) @end lisp @subsection Test the new Backend. This can be tested with the code @lisp Python >>> import pickle >>> pickle.dump('Foobar', open('/tmp/foo.p', 'w')) Lisp * (cl-store:restore "/tmp/foo.p" 'pickle) => "Foobar" And Lisp * (cl-store:store "BarFoo" "/tmp/foo.p" 'pickle) Python >>> pickle.load(open('/tmp/foo.p')) 'BarFoo' @end lisp @vskip 0pt plus 2filll @section API @subsection Functions @anchor{Generic backend-restore} @deffn {Generic} backend-restore backend place Restore the object found in stream @emph{place} using backend @emph{backend}. Checks the magic-number and invokes @code{backend-restore-object}. Called by @code{restore}, override for custom restoring. @end deffn @anchor{Generic backend-restore-object} @deffn {Generic} backend-restore backend place Find the next function to call to restore the next object with @emph{backend} and invoke it with @emph{place}. Called by @code{restore-object}, override this method to do custom restoring (see @file{circularities.lisp} for an example). @end deffn @anchor{Generic backend-store} @deffn {Generic} backend-store backend place obj Stores the backend code and calls @code{store-object}. This is called by @code{store}. Override for custom storing. @end deffn @anchor{Generic backend-store-object} @deffn {Generic} backend-store-object backend obj place Called by @code{store-object}, override this to do custom storing (see @file{circularities.lisp} for an example). @end deffn @anchor{Generic get-next-reader} @deffn {Generic} get-next-reader backend place Method which must be specialized for @emph{backend} to return the next symbol designating a @code{defrestore} instance to restore an object from @emph{place}. If no reader is found return a second value which will be included in the error. @end deffn @subsection Macros @anchor{Macro defbackend} @deffn {Macro} defbackend name &key (stream-type (required-arg "stream-type")) magic-number fields (extends 'backend) old-magic-numbers eg. @code{(defbackend pickle :stream-type 'character)} This creates a new backend called @emph{name}, @emph{stream-type} describes the type of stream that the backend will serialize to which must be suitable as an argument to open. @emph{Magic-number}, when present, must be of type (unsigned-byte 32) which will be written as a verifier for the backend. @emph{Fields} are extra fields to be added to the new class which will be created. By default the @emph{extends} keyword is @emph{backend},the root backend, but this can be any legal backend. @emph{Old-magic-numbers} holds previous magic-numbers that have been used by the backend to identify incompatible versions of objects stored. @end deffn @node Notes @chapter Notes @section Backend Designators The @emph{backend} argument to store, restore and with-backend is a backend designator which can be one of. @itemize @bullet @item A backend returned by @code{(find-backend name)} @item A symbol designating a backend (the first argument to defbackend). @end itemize @section Known Issues @itemize @bullet @item CLISP, OpenMCL, Allegro CL cannot store structure instances. @item Structure definitions are only supported in SBCL and CMUCL. @item Due to the fact that function's aren't fully supported CLOS Classes initfunction slot cannot be serialized. @end itemize @section Delivery with Lispworks Restoring lists in delivered images can be problematic since the tree shaker can remove the symbol cl:nil (this seems to only happen with delivery-level > 4). To work around this add the following keywords to the delivery call. @lisp :packages-to-keep '(:cl) :keep-symbols '(cl:nil) @end lisp @section Regarding String Serialization Users are required to be extremely careful when serializing strings from one lisp implementation to another since the array-element-type will be tracked for strings and the Hyperspec does not specify an upper limit for base-chars. This can be a problem if you serialize a simple-base-string containing wide characters, in an implementation which specifies no limit on base-char, to an implementation with a limit. If you have a solution I would be happy to hear it. @node Credits @chapter Credits Thanks To @itemize @bullet @item Common-Lisp.net: For project hosting. @item Alain Picard : Structure Storing and support for Infinite Floats for Lispworks. @item Robert Sedgewick: Package Imports for OpenMCL and suggesting Multiple Backends. @item Thomas Stenhaug: Comprehensive package storing and miscellaneous improvements. @item Killian Sprotte: Type specification fixups. @end itemize @node Index @chapter Index @section Function Index @printindex fn @section Variable Index @printindex vr @bye