X-Git-Url: https://git.distorted.org.uk/~mdw/sgt/agedu/blobdiff_plain/671599440ffaee87de2dd6fd8236d84d0c85bca0..HEAD:/agedu.but diff --git a/agedu.but b/agedu.but index de47dd2..eebb174 100644 --- a/agedu.but +++ b/agedu.but @@ -6,8 +6,8 @@ \U NAME -\cw{agedu} - correlate disk usage with last-access times to identify -large and disused data +\cw{agedu} \dash correlate disk usage with last-access times to +identify large and disused data \U SYNOPSIS @@ -65,10 +65,13 @@ mode: which will print (among other messages) a URL on its standard output along the lines of -\c URL: http://127.164.152.163:48638/ +\c URL: http://127.0.0.1:48638/ (That URL will always begin with \cq{127.}, meaning that it's in the -\cw{localhost} address space.) +\cw{localhost} address space. So only processes running on the same +computer can even try to connect to that web server, and also there +is access control to prevent other users from seeing it \dash see +below for more detail.) Now paste that URL into your web browser, and you will be shown a graphical representation of the disk usage in \cw{/home/fred} and @@ -163,10 +166,11 @@ default it invents its own URL and prints it out. \lcont{ -The web server runs until \cw{agedu} receives an end-of-file event -on its standard input. (The expected usage is that you run it from -the command line, immediately browse web pages until you're -satisfied, and then press Ctrl-D.) +The web server runs until \cw{agedu} receives an end-of-file event on +its standard input. (The expected usage is that you run it from the +command line, immediately browse web pages until you're satisfied, and +then press Ctrl-D.) To disable the EOF behaviour, use the +\cw{--no-eof} option. In case the index file contains any confidential information about your file system, the web server protects the pages it serves from @@ -187,10 +191,10 @@ completely) and a username and password of your choice. \dd In this mode, \cw{agedu} generates a textual report on standard output, listing the disk usage in the specified directory and all -its subdirectories down to a fixed depth. By default that depth is +its subdirectories down to a given depth. By default that depth is 1, so that you see a report for \e{directory} itself and all of its -immediate subdirectories. You can configure a different depth using -\cw{-d}, described in the next section. +immediate subdirectories. You can configure a different depth (or no +depth limit) using \cw{-d}, described in the next section. \lcont{ @@ -241,9 +245,11 @@ whereas \cw{-S} will not. \lcont{ -(Actually, the output will not be exactly identical, due to a -difference in treatment of last-access times on directories. See the -documentation of the \cw{--dir-atime} option in the next section. +(The output will not be exactly \e{identical}, due to a +difference in treatment of last-access times on directories. +However, it should be effectively equivalent for most purposes. See +the documentation of the \cw{--dir-atime} option in the next section +for further detail.) } @@ -252,9 +258,47 @@ documentation of the \cw{--dir-atime} option in the next section. \dd In this mode, \cw{agedu} will generate an HTML report of the disk usage in the specified directory and its immediate subdirectories, in the same form that it serves from its web server -in \cw{-w} mode. However, this time, a single HTML report will be -generated and simply written to standard output, with no hyperlinks -pointing to other similar pages. +in \cw{-w} mode. + +\lcont{ + +By default, a single HTML report will be generated and simply +written to standard output, with no hyperlinks pointing to other +similar pages. If you also specify the \cw{-d} option (see below), +\cw{agedu} will instead write out a collection of HTML files with +hyperlinks between them, and call the top-level file +\cw{index.html}. + +} + +\dt \cw{--cgi} + +\dd In this mode, \cw{agedu} will run as the bulk of a CGI script +which provides the same set of web pages as the built-in web server +would. It will read the usual CGI environment variables, and write +CGI-style data to its standard output. + +\lcont{ + +The actual CGI program itself should be a tiny wrapper around +\cw{agedu} which passes it the \cw{--cgi} option, and also +(probably) \cw{-f} to locate the index file. \cw{agedu} will do +everything else. + +No access control is performed in this mode: restricting access to +CGI scripts is assumed to be the job of the web server. + +} + +\dt \cw{-h} or \cw{--help} + +\dd Causes \cw{agedu} to print some help text and terminate +immediately. + +\dt \cw{-V} or \cw{--version} + +\dd Causes \cw{agedu} to print its version number and terminate +immediately. \U OPTIONS @@ -438,8 +482,91 @@ complexity.) } -The following options affect the web server mode \cw{-w}, and in one -case also the stand-along HTML generation mode \cw{-H}: +\dt \cw{--mtime} + +\dd This option causes \cw{agedu} to index files by their last +modification time instead of their last access time. You might want +to use this if your last access times were completely useless for +some reason: for example, if you had recently searched every file on +your system, the system would have lost all the information about +what files you hadn't recently accessed before then. Using this +option is liable to be less effective at finding genuinely wasted +space than the normal mode (that is, it will be more likely to flag +things as disused when they're not, so you will have more candidates +to go through by hand looking for data you don't need), but may be +better than nothing if your last-access times are unhelpful. + +\lcont{ + +Another use for this mode might be to find \e{recently created} +large data. If your disk has been gradually filling up for years, +the default mode of \cw{agedu} will let you find unused data to +delete; but if you know your disk had plenty of space recently and +now it's suddenly full, and you suspect that some rogue program has +left a large core dump or output file, then \cw{agedu --mtime} might +be a convenient way to locate the culprit. + +} + +The following option affects all the modes that generate reports: +the web server mode \cw{-w}, the stand-alone HTML generation mode +\cw{-H} and the text report mode \cw{-t}. + +\dt \cw{--files} + +\dd This option causes \cw{agedu}'s reports to list the individual +files in each directory, instead of just giving a combined report +for everything that's not in a subdirectory. + +The following option affects the text report mode \cw{-t}. + +\dt \cw{-a} \e{age} or \cw{--age} \e{age} + +\dd This option tells \cw{agedu} to report only files of at least the +specified age. An age is specified as a number, followed by one of +\cq{y} (years), \cq{m} (months), \cq{w} (weeks) or \cq{d} (days). +(This syntax is also used by the \cw{-r} option.) For example, \cw{-a +6m} will produce a text report which includes only files at least six +months old. + +The following options affect the stand-alone HTML generation mode +\cw{-H} and the text report mode \cw{-t}. + +\dt \cw{-d} \e{depth} or \cw{--depth} \e{depth} + +\dd This option controls the maximum depth to which \cw{agedu} +recurses when generating a text or HTML report. + +\lcont{ + +In text mode, the default is 1, meaning that the report will include +the directory given on the command line and all of its immediate +subdirectories. A depth of two includes another level below that, +and so on; a depth of zero means \e{only} the directory on the +command line. + +In HTML mode, specifying this option switches \cw{agedu} from +writing out a single HTML file to writing out multiple files which +link to each other. A depth of 1 means \cw{agedu} will write out an +HTML file for the given directory and also one for each of its +immediate subdirectories. + +If you want \cw{agedu} to recurse as deeply as possible, give the +special word \cq{max} as an argument to \cw{-d}. + +} + +\dt \cw{-o} \e{filename} or \cw{--output} \e{filename} + +\dd This option is used to specify an output file for \cw{agedu} to +write its report to. In text mode or single-file HTML mode, the +argument is treated as the name of a file. In multiple-file HTML +mode, the argument is treated as the name of a directory: the +directory will be created if it does not already exist, and the +output HTML files will be created inside it. + +The following options affect the web server mode \cw{-w}, and in some +cases also the stand-alone HTML generation mode \cw{-H}: \dt \cw{-r} \e{age range} or \cw{--age-range} \e{age range} @@ -456,11 +583,11 @@ limits, and you can do this using \cw{-r}. The argument to \cw{-r} consists of a single age, or two ages separated by a minus sign. An age is a number, followed by one of \cq{y} (years), \cq{m} (months), \cq{w} (weeks) or \cq{d} (days). -The first age in the range represents the oldest data, and will be -coloured red in the HTML; the second age represents the newest, -coloured green. If the second age is not specified, it will default -to zero (so that green means data which has been accessed \e{just -now}). +(This syntax is also used by the \cw{-a} option.) The first age in the +range represents the oldest data, and will be coloured red in the +HTML; the second age represents the newest, coloured green. If the +second age is not specified, it will default to zero (so that green +means data which has been accessed \e{just now}). For example, \cw{-r 2y} will mark data in red if it has been unused for two years or more, and green if it has been accessed just now. @@ -472,12 +599,11 @@ three months ago or later. \dt \cw{--address} \e{addr}[\cw{:}\e{port}] -\dd Specifies the network address and port number on which -\cw{agedu} should listen when running its web server. If you want -\cw{agedu} to listen for connections coming in from any source, you -should probably specify the special IP address \cw{0.0.0.0}. If the -port number is omitted, it will be assumed to be 80 (for which -\cw{agedu} will probably need to be running as a privileged user). +\dd Specifies the network address and port number on which \cw{agedu} +should listen when running its web server. If you want \cw{agedu} to +listen for connections coming in from any source, specify the address +as the special value \cw{ANY}. If the port number is omitted, an +arbitrary unused port will be chosen for you and displayed. \lcont{ @@ -545,6 +671,20 @@ consist of the username, followed by a colon, followed by the password, followed \e{immediately} by end of file (no trailing newline, or else it will be considered part of the password). +\dt \cw{--title} \e{title} + +\dd Specify the string that appears at the start of the \cw{} +section of the output HTML pages. The default is \cq{agedu}. This +title is followed by a colon and then the path you're viewing within +the index file. You might use this option if you were serving +\cw{agedu} reports for several different servers and wanted to make it +clearer which one a user was looking at. + +\dt \cw{--no-eof} + +\dd Stop \cw{agedu} in web server mode from looking for end-of-file on +standard input and treating it as a signal to terminate. + \U LIMITATIONS The data file is pretty large. The core of \cw{agedu} is the @@ -553,13 +693,13 @@ efficiently perform the queries it needs; this data structure requires \cw{O(N log N)} storage. This is larger than you might expect; a scan of my own home directory, containing half a million files and directories and about 20Gb of data, produced an index file -nearly a third of a Gb in size. Furthermore, since the data file -must be memory-mapped during most processing, it can never grow -larger than available address space, which means that any use of -\cw{agedu} on a seriously large file system is probably going to -have to be done on a 64-bit computer. - -Also, \cw{agedu} has not yet been \e{tested} on a 64-bit computer. +over 60Mb in size. Furthermore, since the data file must be +memory-mapped during most processing, it can never grow larger than +available address space, so a \e{really} big filesystem may need to +be indexed on a 64-bit computer. (This is one reason for the +existence of the \cw{-D} and \cw{-L} options: you can do the +scanning on the machine with access to the filesystem, and the +indexing on a machine big enough to handle it.) The data structure also does not usefully permit access control within the data file, so it would be difficult \dash even given the @@ -567,6 +707,29 @@ willingness to do additional coding \dash to run a system-wide \cw{agedu} scan on a \cw{cron} job and serve the right subset of reports to each user. +In certain circumstances, \cw{agedu} can report false positives +(reporting files as disused which are in fact in use) as well as the +more benign false negatives (reporting files as in use which are +not). This arises when a file is, semantically speaking, \q{read} +without actually being physically \e{read}. Typically this occurs +when a program checks whether the file's mtime has changed and only +bothers re-reading it if it has; programs which do this include +\cw{rsync}(\e{1}) and \cw{make}(\e{1}). Such programs will fail to +update the atime of unmodified files despite depending on their +continued existence; a directory full of such files will be reported +as disused by \cw{agedu} even in situations where deleting them will +cause trouble. + +Finally, of course, \cw{agedu}'s normal usage mode depends critically +on the OS providing last-access times which are at least approximately +right. So a file system mounted with Linux's \cq{noatime} option, or +the equivalent on any other OS, will not give useful results! +(However, the Linux mount option \cq{relatime}, which distributions +now tend to use by default, should be fine for all but specialist +purposes: it reduces the accuracy of last-access times so that they +might be wrong by up to 24 hours, but if you're looking for files that +have been unused for months or years, that's not a problem.) + \U LICENCE \cw{agedu} is free software, distributed under the MIT licence. Type