Commit | Line | Data |
---|---|---|
69305044 MW |
1 | .ie t .ds o \(bu |
2 | .el .ds o o | |
3 | .de hP | |
4 | .IP | |
5 | \h'-\w'\fB\\$1\ \fP'u'\fB\\$1\ \fP\c | |
6 | .. | |
f6b4ffdc MW |
7 | .TH rsync-backup 8 "7 October 2012" rsync-backup |
8 | .SH SYNOPSIS | |
9 | .B rsync-backup | |
10 | .RB [ \-v ] | |
11 | .RB [ \-c | |
12 | .IR config-file ] | |
13 | .SH DESCRIPTION | |
14 | The | |
15 | .B rsync-backup | |
16 | script is a backup program of the currently popular | |
17 | .RB ` rsync (1) | |
18 | .BR \-\-link-dest ' | |
19 | variety. It uses | |
20 | .BR rsync 's | |
21 | ability to create hardlinks from (apparently) similar existing local | |
22 | trees to make incremental dumps efficient, even from remote sources. | |
23 | Restoring files is easy because the backups created are just directories | |
24 | full of files, exactly as they were on the source \(en and this is | |
25 | verified using the | |
26 | .BR fshash (1) | |
27 | program. | |
28 | .PP | |
29 | The script does more than just running | |
30 | .BR rsync . | |
31 | It is also responsible for creating and removing snapshots of volumes to | |
32 | be backed up, and expiring old dumps according to a user-specified | |
33 | retention policy. | |
34 | .SS Installation | |
35 | The idea is that the | |
36 | .B rsync-backup | |
37 | script should be installed and run on a central backup server with local | |
38 | access to the backup volumes. | |
39 | .PP | |
40 | The script should be run with full (root) privileges, so that it can | |
41 | correctly record file ownership information. The server should also be | |
42 | able to connect via | |
43 | .BR ssh (1) | |
44 | to the client machines, and run processes there as root. (This is not a | |
45 | security disaster. Remember that the backup server is, in the end, | |
46 | responsible for the integrity of the backup data. A dishonest backup | |
47 | server can easily compromise a client which is being restored from | |
48 | corrupt backup data.) | |
69305044 MW |
49 | .SS Command-line options |
50 | Most of the behaviour of | |
51 | .B rsync-backup | |
52 | is controlled by a configuration file, described starting with the | |
53 | section named | |
54 | .B Configuration commands | |
55 | below. | |
56 | But a few features are controlled by command-line options. | |
57 | .TP | |
58 | .B \-h | |
59 | Show a brief help message for the program, and exit successfully. | |
60 | .TP | |
61 | .B \-V | |
62 | Show | |
63 | .BR rsync-backup 's | |
64 | version number and some choice pieces of build-time configuration, and | |
65 | exit successfully. | |
66 | .TP | |
67 | .BI "\-c " conf | |
68 | Read | |
69 | .I conf | |
70 | instead of the default configuration file (shown as | |
71 | .B conf | |
72 | in the | |
73 | .B \-V | |
74 | output). | |
75 | .TP | |
76 | .B \-v | |
77 | Produce verbose progress information on standard output while the backup | |
78 | is running. This keeps one amused while running a backup | |
79 | interactively. In any event, | |
80 | .B rsync-backup | |
81 | will report failures to standard error, and otherwise run silently, so | |
82 | it doesn't annoy unnecessarily if run by | |
83 | .BR cron (8). | |
84 | .SS Backup process | |
85 | Backing up a filesystem works as follows. | |
86 | .hP \*o | |
87 | Make a snapshot of the filesystem on the client, and ensure that the | |
88 | snapshot is mounted. There are some `trivial' snapshot types which use | |
89 | the existing mounted filesystem, and either prevent processes writing to | |
90 | it during the backup, or just hope for the best. Other snapshot types | |
91 | require the snapshot to be mounted somewhere distinct from the main | |
92 | filesystem, so that the latter can continue being used. | |
93 | .hP \*o | |
94 | Run | |
95 | .B rsync | |
96 | to copy the snapshot to the backup volume \(en specifically, to | |
97 | .IB host / fs / new \fR. | |
98 | If this directory already exists, then it's presumed to be debris from a | |
99 | previous attempt to dump this filesystem: | |
100 | .B rsync | |
101 | will update it appropriately, by adding, deleting or modifying the | |
102 | files. This means that retrying a failed dump \(en after fixing whatever | |
103 | caused it to go wrong, obviously! \(en is usually fairly quick. | |
104 | .hP \*o | |
105 | Run | |
106 | .B fshash | |
107 | on the client to generate a `digest' describing the contents of the | |
108 | filesystem, and send this to the server as | |
109 | .IB host / fs / new .fshash \fR. | |
110 | .hP \*o | |
111 | Release the snapshot: we don't need it any more. | |
112 | .hP \*o | |
113 | Run | |
114 | .B fshash | |
115 | over the new backup; specifically, to | |
116 | .BI tmp/fshash. host . fs . date \fR. | |
117 | This gives us a digest for what the backup volume actually stored. | |
118 | .hP \*o | |
119 | Compare the two | |
120 | .B fshash | |
121 | digests. If they differ then dump the differences to the log file and | |
122 | report a backup failure. (Backups aren't any good if they don't | |
123 | actually back up the right thing. And you stand a better chance of | |
124 | fixing them if you know that they're going wrong.) | |
125 | .hP \*o | |
126 | Commit the backup, by renaming the dump directory to | |
127 | .IB host / fs / date | |
128 | and the | |
129 | .B fshash | |
130 | digest file to | |
131 | .IB host / fs / date .fshash \fR. | |
f6b4ffdc | 132 | .PP |
69305044 | 133 | The backup is now complete. |
f6b4ffdc MW |
134 | .SS Configuration commands |
135 | The configuration file is simply a Bash shell fragment: configuration | |
136 | commands are shell functions. | |
137 | .TP | |
138 | .BI "backup " "fs\fR[:\fIfsarg\fR] ..." | |
139 | Back up the named filesystems. The corresponding | |
140 | .IR fsarg s | |
141 | may be required by the snapshot type. | |
142 | .TP | |
143 | .BI "host " host | |
144 | Future | |
145 | .B backup | |
146 | commands will back up filesystems on the named | |
147 | .IR host . | |
69305044 MW |
148 | To back up filesystems on the backup server itself, use its hostname: |
149 | .B rsync-backup | |
150 | will avoid inefficient and pointless messing about | |
151 | .BR ssh (1) | |
152 | in this case. | |
153 | This command clears the | |
f6b4ffdc MW |
154 | .B like |
155 | list. | |
156 | .TP | |
157 | .BI "like " "host\fR ..." | |
158 | Declare that subsequent filesystems are `similar' to like-named | |
159 | filesystems on the named | |
160 | .IR host s, | |
161 | and that | |
162 | .B rsync | |
163 | should use those trees as potential sources of hardlinkable files. Be | |
164 | careful when using this option without | |
165 | .BR rsync 's | |
166 | .B \-\-checksum | |
167 | option: an erroneous hardlink will cause the backup to fail. (The | |
168 | backup won't be left silently incorrect.) | |
169 | .TP | |
170 | .BI "retain " frequency " " duration | |
171 | Define part a backup retention policy: backup trees of the | |
172 | .I frequency | |
173 | should be kept for the | |
174 | .IR duration . | |
175 | The | |
176 | .I frequency | |
177 | can be | |
178 | .BR daily , | |
179 | .BR weekly , | |
180 | .BR monthly , | |
181 | or | |
69305044 | 182 | .B annually |
f6b4ffdc MW |
183 | (or |
184 | .BR yearly , | |
185 | which means the same); the | |
186 | .I duration | |
187 | may be any of | |
188 | .BR week , | |
189 | .BR month , | |
190 | .BR year , | |
191 | or | |
192 | .BR forever . | |
193 | Expiry considers each existing dump against the policy lines in order: | |
194 | the last applicable line determines the dump's fate \(en so you should | |
195 | probably write the lines in decreasing order of duration. | |
196 | .TP | |
197 | .BI "snap " type " " \fR[\fIargs\fR...] | |
198 | Use the snapshot | |
199 | .I type | |
200 | for subsequent backups. Some snapshot types require additional | |
201 | arguments, which may be supplied here. | |
202 | .SS Configuration variables | |
203 | The following shell variables may be overridden by the configuration | |
204 | file. | |
205 | .TP | |
206 | .B MAXLOG | |
207 | The number of log files to be kept for each filesystem. Old logfiles | |
208 | are deleted to keep the total number below this bound. The default | |
209 | value is 14. | |
210 | .TP | |
211 | .B RSYNCOPTS | |
212 | Command-line options to pass to | |
213 | .BR rsync (1) | |
214 | in addition to the basic set: | |
69305044 MW |
215 | .B \-\-archive |
216 | .B \-\-hard-links | |
217 | .B \-\-numeric-ids | |
218 | .B \-\-del | |
219 | .B \-\-sparse | |
220 | .B \-\-compress | |
221 | .B \-\-one-file-system | |
222 | .B \-\-partial | |
223 | .BR "\-\-filter=""dir-merge .rsync-backup""" . | |
f6b4ffdc MW |
224 | The default is |
225 | .BR \-\-verbose . | |
226 | .TP | |
227 | .B SNAPDIR | |
228 | LVM (and | |
229 | .BR rfreezefs ) | |
230 | snapshots are mounted on subdirectories below the | |
231 | .B SNAPDIR | |
232 | .IR "on backup clients" . | |
233 | The default is | |
234 | .IB mntbkpdir /snap | |
235 | where | |
236 | .I mntbkpdir | |
237 | is the backup mount directory configured at build time. | |
238 | .TP | |
239 | .B SNAPSIZE | |
240 | The volume size option to pass to | |
241 | .BR lvcreate (8) | |
242 | when creating a snapshot. The default is | |
243 | .B \-l10%ORIGIN | |
244 | which seems to work fairly well. | |
245 | .TP | |
246 | .B STOREDIR | |
247 | Where the actual backup trees should be stored. See the section on | |
248 | .B Archive structure | |
249 | below. | |
250 | The default is | |
251 | .IB mntbkpdir /store | |
252 | where | |
253 | .I mntbkpdir | |
254 | is the backup mount directory configured at build time. | |
255 | .TP | |
256 | .B HASH | |
257 | The hash function to use for verifying archive integrity. This is | |
258 | passed to the | |
259 | .B \-H | |
260 | option of | |
261 | .BR fshash , | |
262 | so it must name one of the hash functions supported by your Python's | |
263 | .B hashlib | |
264 | module. The default is | |
265 | .BR sha256 . | |
266 | .SS Hook functions | |
267 | The configuration file may define shell functions to perform custom | |
268 | actions at various points in the backup process. | |
269 | .TP | |
270 | .BI "backup_precommit_hook " host " " fs " " date | |
271 | Called after a backup has been verified complete and about to be | |
272 | committed. The backup tree is in | |
273 | .B new | |
274 | in the current directory, and the | |
275 | .B fshash | |
276 | manifest is in | |
277 | .BR new.fshash . | |
278 | A typical action would be to create a digital signature on the | |
279 | manifest. | |
280 | .TP | |
281 | .BI "backup_commit_hook " host " " fs " " date | |
282 | Called during the commit procedure. The backup tree and manifest have | |
283 | been renamed into their proper places. Typically one would use this | |
284 | hook to rename files created by the | |
285 | .B backup_precommit_hook | |
286 | function. | |
287 | .TP | |
288 | .BR "whine " [ \-n ] " " \fItext\fR... | |
289 | Called to report `interesting' events when the | |
290 | .B \-v | |
291 | option is in force. The default action is to echo the | |
292 | .I text | |
293 | to (what was initially) standard output, followed by a newline unless | |
294 | .B \-n | |
295 | is given. | |
296 | .SS Snapshot types | |
297 | The following snapshot types are available. | |
298 | .TP | |
299 | .B live | |
300 | A trivial snapshot type: attempts to back up a live filesystem. How | |
301 | well this works depends on how active the filesystem is. If files | |
302 | change while the dump is in progress then the | |
303 | .B fshash | |
304 | verification will likely fail. Backups using this snapshot type must | |
305 | specify the filesystem mount point as the | |
306 | .IR fsarg . | |
307 | .TP | |
308 | .B ro | |
309 | A slightly less trivial snapshot type: make the filesystem read-only | |
310 | while the dump is in progress. Backups using this snapshot type must | |
311 | specify the filesystem mount point as the | |
312 | .IR fsarg . | |
313 | .TP | |
314 | .BI "lvm " vg | |
315 | Create snapshots using LVM. The snapshot argument is interpreted as the | |
316 | relevant volume group. The filesystem name is interpreted as the origin | |
317 | volume name; the snapshot will be called | |
318 | .IB fs .bkp | |
319 | and mounted on | |
320 | .IB SNAPDIR / fs \fR; | |
321 | space will be allocated to it according to the | |
322 | .I SNAPSIZE | |
323 | variable. | |
324 | .TP | |
325 | .BI "rfreezefs " client " " vg | |
326 | This gets complicated. Suppose that a server has an LVM volume group, | |
327 | and exports (somehow) a logical volume to a client. Examples are a host | |
328 | providing a virtual disk to a guest, or a server providing | |
329 | network-attached storage to a client. The server can create a snapshot | |
330 | of the volume using LVM, but must synchronize with the client to ensure | |
331 | that the filesystem image captured in the snapshot is clean. The | |
332 | .BR rfreezefs (8) | |
333 | program should be installed on the client to perform this rather | |
334 | delicate synchronization. Declare the server using the | |
335 | .B host | |
336 | command as usual; pass the client's name as the | |
337 | .I client | |
338 | and the | |
339 | server's volume group name as the | |
340 | .I vg | |
341 | snapshot arguments. Finally, backups using this snapshot type must | |
342 | specify the filesystem mount point (or, actually, any file in the | |
343 | filesystem) on the client, as the | |
344 | .IR fsarg . | |
345 | .PP | |
346 | Additional snapshot types can be defined in the configuration file. A | |
347 | snapshot type requires two shell functions. | |
348 | .TP | |
349 | .BI snap_ type " " snapargs " " fs " " fsarg | |
350 | Create the snapshot, and write the mountpoint (on the client host) to | |
351 | standard output, in a form suitable as an argument to | |
352 | .BR rsync . | |
353 | .TP | |
354 | .BI unsnap_ type " " snapargs " " fs " " fsarg | |
355 | Remove the snapshot. | |
356 | .PP | |
357 | There are a number of utility functions which can be used by snapshot | |
358 | type handlers: please see the script for details. Please send the | |
359 | author interesting snapshot handlers for inclusion in the main | |
360 | distribution. | |
361 | .SS Archive structure | |
69305044 MW |
362 | Backup trees are stored in a fairly straightforward directory tree. |
363 | .PP | |
364 | At the top level is one directory for each client host. There are also | |
365 | some special entries: | |
366 | .TP | |
367 | .B fshash.cache | |
368 | The cache database used for improving performance of local file | |
369 | hashing. There may be other | |
370 | .B fshash.cache-* | |
371 | files used by SQLite for its own purposes. | |
372 | .TP | |
373 | .B lost+found | |
374 | Part of the filesystem used on the backup volume. You don't want to | |
375 | mess with this. | |
376 | .TP | |
377 | .B tmp | |
378 | Used to store temporary files during the backup process. (Some of them | |
379 | want to be on the same filesystem as the rest of the backup.) When | |
380 | things go wrong, files are left behind in the hope that they might help | |
381 | someone debug the mess. It's always safe to delete the files in here | |
382 | when no backup is running. | |
383 | .PP | |
384 | So don't use those names for your hosts. | |
385 | .PP | |
386 | The next layer down contains a directory for each filesystem on the given host. | |
387 | .PP | |
388 | The bottom layer contains a directory for each dump of that filesystem, | |
389 | named with the date at which the dump was started (in ISO8601 | |
390 | .IB yyyy \(en mm \(en dd | |
391 | format), together with associated files named | |
392 | .IB date .* \fR. | |
393 | .SH SEE ALSO | |
394 | .BR fshash (1), | |
395 | .BR lvm (8), | |
396 | .BR rfreezefs (8), | |
397 | .BR rsync (1), | |
398 | .BR ssh (1). | |
399 | .SH AUTHOR | |
400 | Mark Wooding, <mdw@distorted.org.uk> |