rsync-backup.in, rsync-backup.8: Trim caches on fshash mismatch.

[rsync-backup] / rsync-backup.8

diff --git a/rsync-backup.8 b/rsync-backup.8
index 3d2cffc..fcbf197 100644 (file)
--- a/rsync-backup.8
+++ b/rsync-backup.8
@@ -1,7 +1,15 @@
+.ie t .ds o \(bu
+.el .ds o o
+.de hP
+.IP
+\h'-\w'\fB\\$1\ \fP'u'\fB\\$1\ \fP\c
+..

 .TH rsync-backup 8 "7 October 2012" rsync-backup
 .TH rsync-backup 8 "7 October 2012" rsync-backup

+.SH NAME
+rsync-backup \- back up files using rsync

 .SH SYNOPSIS
 .B rsync-backup
 .SH SYNOPSIS
 .B rsync-backup

-.RB [ \-v ]
+.RB [ \-nv ]

 .RB [ \-c
 .IR config-file ]
 .SH DESCRIPTION
 .RB [ \-c
 .IR config-file ]
 .SH DESCRIPTION


@@ -40,27 +48,140 @@ security disaster.  Remember that the backup server is, in the end,
 responsible for the integrity of the backup data.  A dishonest backup
 server can easily compromise a client which is being restored from
 corrupt backup data.)
 responsible for the integrity of the backup data.  A dishonest backup
 server can easily compromise a client which is being restored from
 corrupt backup data.)

+.SS Command-line options
+Most of the behaviour of
+.B rsync-backup
+is controlled by a configuration file, described starting with the
+section named
+.B Configuration commands
+below.
+But a few features are controlled by command-line options.
+.TP
+.B \-h
+Show a brief help message for the program, and exit successfully.
+.TP
+.B \-V
+Show
+.BR rsync-backup 's
+version number and some choice pieces of build-time configuration, and
+exit successfully.
+.TP
+.BI "\-c " conf
+Read
+.I conf
+instead of the default configuration file (shown as
+.B conf
+in the
+.B \-V
+output).
+.TP
+.B \-n
+Don't actually take a backup, or write proper logs: instead, write a
+description of what would be done to standard error.
+.TP
+.B \-v
+Produce verbose progress information on standard output while the backup
+is running.  This keeps one amused while running a backup
+interactively.  In any event,
+.B rsync-backup
+will report failures to standard error, and otherwise run silently, so
+it doesn't annoy unnecessarily if run by
+.BR cron (8).
+.SS Backup process
+Backing up a filesystem works as follows.
+.hP \*o
+Make a snapshot of the filesystem on the client, and ensure that the
+snapshot is mounted.  There are some `trivial' snapshot types which use
+the existing mounted filesystem, and either prevent processes writing to
+it during the backup, or just hope for the best.  Other snapshot types
+require the snapshot to be mounted somewhere distinct from the main
+filesystem, so that the latter can continue being used.
+.hP \*o
+Run
+.B rsync
+to copy the snapshot to the backup volume \(en specifically, to
+.IB host / fs / new \fR.
+If this directory already exists, then it's presumed to be debris from a
+previous attempt to dump this filesystem:
+.B rsync
+will update it appropriately, by adding, deleting or modifying the
+files.  This means that retrying a failed dump \(en after fixing whatever
+caused it to go wrong, obviously! \(en is usually fairly quick.
+.hP \*o
+Run
+.B fshash
+on the client to generate a `digest' describing the contents of the
+filesystem, and send this to the server as
+.IB host / fs / new .fshash \fR.
+.hP \*o
+Release the snapshot: we don't need it any more.
+.hP \*o
+Run
+.B fshash
+over the new backup; specifically, to
+.BI tmp/fshash. host . fs . date \fR.
+This gives us a digest for what the backup volume actually stored.
+.hP \*o
+Compare the two
+.B fshash
+digests.  If they differ then dump the differences to the log file and
+report a backup failure.  (Backups aren't any good if they don't
+actually back up the right thing.  And you stand a better chance of
+fixing them if you know that they're going wrong.)
+.hP \*o
+Commit the backup, by renaming the dump directory to
+.IB host / fs / date
+and the
+.B fshash
+digest file to
+.IB host / fs / date .fshash \fR.

 .PP
 .PP

-The 
-
-
+The backup is now complete.

 .SS Configuration commands
 The configuration file is simply a Bash shell fragment: configuration
 commands are shell functions.
 .TP
 .SS Configuration commands
 The configuration file is simply a Bash shell fragment: configuration
 commands are shell functions.
 .TP

+.BI "addhook " hook " " command
+Arrange that the named
+.I hook
+runs the given
+.IR command .
+See
+.B runhook
+for more details.
+.TP

 .BI "backup " "fs\fR[:\fIfsarg\fR] ..."
 Back up the named filesystems.  The corresponding
 .IR fsarg s
 may be required by the snapshot type.
 .TP
 .BI "backup " "fs\fR[:\fIfsarg\fR] ..."
 Back up the named filesystems.  The corresponding
 .IR fsarg s
 may be required by the snapshot type.
 .TP

+.BI "defhook " hook
+Define a new hook named
+.IR hook .
+See
+.B addhook
+and
+.B runhook
+for more information.
+.TP

 .BI "host " host
 Future
 .B backup
 commands will back up filesystems on the named
 .IR host .
 .BI "host " host
 Future
 .B backup
 commands will back up filesystems on the named
 .IR host .

-This clears the
+To back up filesystems on the backup server itself, use its hostname:
+.B rsync-backup
+will avoid inefficient and pointless messing about
+.BR ssh (1)
+in this case.
+This command clears the

 .B like
 .B like

-list.
+list, the remote
+.B user
+name, and resets the retention policy to its default (i.e., the to
+policy defined prior to the first
+.B host
+command).

 .TP
 .BI "like " "host\fR ..."
 Declare that subsequent filesystems are `similar' to like-named
 .TP
 .BI "like " "host\fR ..."
 Declare that subsequent filesystems are `similar' to like-named


@@ -87,7 +208,7 @@ can be
 .BR weekly ,
 .BR monthly ,
 or
 .BR weekly ,
 .BR monthly ,
 or

-.B annually 
+.B annually

 (or
 .BR yearly ,
 which means the same); the
 (or
 .BR yearly ,
 which means the same); the


@@ -101,28 +222,134 @@ or
 Expiry considers each existing dump against the policy lines in order:
 the last applicable line determines the dump's fate \(en so you should
 probably write the lines in decreasing order of duration.
 Expiry considers each existing dump against the policy lines in order:
 the last applicable line determines the dump's fate \(en so you should
 probably write the lines in decreasing order of duration.

+.RS
+.PP
+Groups of
+.B retain
+commands between
+.B host
+and/or
+.B backup
+commands collectively define a retention policy.  Once a policy is
+defined, subsequent
+.B backup
+operations use the policy.  The first
+.B retain
+command after a
+.B host
+or
+.B backup
+command clears the policy and starts defining a new one.  The policy
+defined before the first
+.B host
+is the
+.I default
+policy: at the start of each
+.B host
+stanza, the policy is reset to the default.
+.RE
+.TP
+.BI "retry " count
+The
+.B live
+snapshot type (see below) doesn't prevent a filesystem from being
+modified while it's being backed up.  If this happens, the
+.B fshash
+pass will detect the difference and fail.  If the filesystem in question
+is relatively quiescent, then maybe retrying the backup will result in a
+successful consistent copy.  Following this command, a backup which
+results in an
+.B fshash
+mismatch will be retried up to
+.I count
+times before being declared a failure.  The default is to retry once,
+clearing mismatching files from the
+.BR fshash (1)
+caches before the second attempt.
+.TP
+.BI "runhook " hook " " args\fR...
+Invoke the named
+.IR hook .
+The individual commands on the hook are run, in order, as
+.RS
+.IP
+.I command
+.IR args ...
+.PP
+If any command fails (returns nonzero) then no other hooks are run and
+.B runhook
+fails with the same exit code.
+.RE

 .TP
 .BI "snap " type " " \fR[\fIargs\fR...]
 Use the snapshot
 .I type
 for subsequent backups.  Some snapshot types require additional
 .TP
 .BI "snap " type " " \fR[\fIargs\fR...]
 Use the snapshot
 .I type
 for subsequent backups.  Some snapshot types require additional

-arguments, which may be supplied here.
+arguments, which may be supplied here.  This command clears the
+.B retry
+counter.
+.TP
+.BI "user " name
+Specify the user name on the remote host.  Without this, calls to
+.BR ssh (1)
+and
+.BR rsync (1)
+won't specify any user name, so the default (probably from the
+.BR ssh_config (5)
+file) will apply.

 .SS Configuration variables
 The following shell variables may be overridden by the configuration
 file.
 .TP
 .SS Configuration variables
 The following shell variables may be overridden by the configuration
 file.
 .TP

+.B HASH
+The hash function to use for verifying archive integrity.  This is
+passed to the
+.B \-H
+option of
+.BR fshash ,
+so it must name one of the hash functions supported by your Python's
+.B hashlib
+module.
+The default is
+.BR sha256 .
+.TP
+.B INDEXDB
+The name of a SQLite database initialized by
+.BR update-bkp-index (8)
+in which an index is maintained of which dumps are on which backup
+volumes.  If the file doesn't exist, then no index is maintained.  The
+default is
+.IB localstatedir /lib/bkp/index.db
+where
+.I localstatedir
+is the state directory configured at build time.
+.TP

 .B MAXLOG
 The number of log files to be kept for each filesystem.  Old logfiles
 are deleted to keep the total number below this bound.  The default
 value is 14.
 .TP
 .B MAXLOG
 The number of log files to be kept for each filesystem.  Old logfiles
 are deleted to keep the total number below this bound.  The default
 value is 14.
 .TP

+.B METADIR
+The metadata directory for the currently mounted backup volume.
+The default is
+.IB mntbkpdir /meta
+where
+.I mntbkpdir
+is the backup mount directory configured at build time.
+.TP

 .B RSYNCOPTS
 Command-line options to pass to
 .BR rsync (1)
 in addition to the basic set:
 .B RSYNCOPTS
 Command-line options to pass to
 .BR rsync (1)
 in addition to the basic set:

-.B \-\-archive \-\-hard-links \-\-numeric-ids \-\-del \-\-sparse
-.B \-\-compress \-\-one-file-system \-\-partial
-.B \-\-filter="dir-merge .rsync-backup"
+.B \-\-archive
+.B \-\-hard-links
+.B \-\-numeric-ids
+.B \-\-del
+.B \-\-sparse
+.B \-\-compress
+.B \-\-one-file-system
+.B \-\-partial
+.BR "\-\-filter=""dir-merge .rsync-backup""" .

 The default is
 .BR \-\-verbose .
 .TP
 The default is
 .BR \-\-verbose .
 .TP


@@ -155,21 +382,33 @@ where
 .I mntbkpdir
 is the backup mount directory configured at build time.
 .TP
 .I mntbkpdir
 is the backup mount directory configured at build time.
 .TP

-.B HASH
-The hash function to use for verifying archive integrity.  This is
-passed to the
-.B \-H
-option of
-.BR fshash ,
-so it must name one of the hash functions supported by your Python's
-.B hashlib
-module.  The default is
-.BR sha256 .
-.SS Hook functions
-The configuration file may define shell functions to perform custom
-actions at various points in the backup process.
+.B VOLUME
+The name of the current volume.  If this is left unset, the volume name
+is read from the file
+.IB METADIR /volume
+once at the start of the backup run.
+.SS Hooks
+The configuration file can modify the behaviour of the backup in two
+main ways: by adding commands to hooks (see the
+.B addhook
+command); and by redefining shell functions.
+.PP
+The following hooks are defined.

 .TP
 .TP

-.BI "backup_precommit_hook " host " " fs " " date
+.BI "commit " host " " fs " " date
+Called during the commit procedure.  The backup tree and manifest have
+been renamed into their proper places.  Typically one would use this
+hook to rename files created in a corresponding
+.B precommit
+command.
+.TP
+.BI "end " rc
+The backup has completed;
+.B rsync-backup
+will exit with status
+.IR rc .
+.TP
+.BI "precommit " host " " fs " " date

 Called after a backup has been verified complete and about to be
 committed.  The backup tree is in
 .B new
 Called after a backup has been verified complete and about to be
 committed.  The backup tree is in
 .B new


@@ -180,12 +419,26 @@ manifest is in
 A typical action would be to create a digital signature on the
 manifest.
 .TP
 A typical action would be to create a digital signature on the
 manifest.
 .TP

+.BI "setup " host " " fs " " date
+Called when a backup of a particular filesystem is about to start.  It
+can return with code 99 to skip the backup.
+.TP
+.B "start"
+Invoked before performing any actual dumps (the first time
+.B host
+is run).
+.PP
+The following shell functions can be redefined by users.
+.TP

 .BI "backup_commit_hook " host " " fs " " date
 .BI "backup_commit_hook " host " " fs " " date

-Called during the commit procedure.  The backup tree and manifest have
-been renamed into their proper places.  Typically one would use this
-hook to rename files created by the
-.B backup_precommit_hook
-function.
+Called from the
+.B commit
+hook for compatibility.
+.TP
+.BI "backup_precommit_hook " host " " fs " " date
+Called from the
+.B precommit
+hook for compatibility.

 .TP
 .BR "whine " [ \-n ] " " \fItext\fR...
 Called to report `interesting' events when the
 .TP
 .BR "whine " [ \-n ] " " \fItext\fR...
 Called to report `interesting' events when the


@@ -261,3 +514,52 @@ type handlers: please see the script for details.  Please send the
 author interesting snapshot handlers for inclusion in the main
 distribution.
 .SS Archive structure
 author interesting snapshot handlers for inclusion in the main
 distribution.
 .SS Archive structure

+Backup trees are stored in a fairly straightforward directory tree.
+.PP
+At the top level is one directory for each client host.  There are also
+some special entries:
+.TP
+.B \&.rsync-backup-store
+This file must be present in order to indicate that a backup volume is
+present (and not just an empty mount point).
+.TP
+.B fshash.cache
+The cache database used for improving performance of local file
+hashing.  There may be other
+.B fshash.cache-*
+files used by SQLite for its own purposes.
+.TP
+.B lost+found
+Part of the filesystem used on the backup volume.  You don't want to
+mess with this.
+.TP
+.B tmp
+Used to store temporary files during the backup process.  (Some of them
+want to be on the same filesystem as the rest of the backup.)  When
+things go wrong, files are left behind in the hope that they might help
+someone debug the mess.  It's always safe to delete the files in here
+when no backup is running.
+.PP
+So don't use those names for your hosts.
+.PP
+The next layer down contains a directory for each filesystem on the
+given host.
+.PP
+The bottom layer contains a directory for each dump of that filesystem,
+named with the date at which the dump was started (in ISO8601
+.IB yyyy \(en mm \(en dd
+format), together with associated files named
+.IB date .* \fR.
+There is also a symbolic link
+.B last
+referring to the most recent backup of the filesystem.
+.SH SEE ALSO
+.BR check-bkp-status (8),
+.BR fshash (1),
+.BR lvm (8),
+.BR rfreezefs (8),
+.BR rsync (1),
+.BR ssh (1),
+.BR update-bkp-index (8).
+.SH AUTHOR
+Mark Wooding, <mdw@distorted.org.uk>