I've just realised that there was a lurking gotcha in the naming of

[sgt/agedu] / agedu.but

\cfg{man-identity}{agedu}{1}{2008-11-02}{Simon Tatham}{Simon Tatham}

\define{dash} \u2013{-}

\title Man page for \cw{agedu}

\U NAME

\cw{agedu} \dash correlate disk usage with last-access times to
identify large and disused data

\U SYNOPSIS

\c agedu [ options ] action [action...]
\e bbbbb   iiiiiii   iiiiii  iiiiii

\U DESCRIPTION

\cw{agedu} scans a directory tree and produces reports about how
much disk space is used in each directory and subdirectory, and also
how that usage of disk space corresponds to files with last-access
times a long time ago.

In other words, \cw{agedu} is a tool you might use to help you free
up disk space. It lets you see which directories are taking up the
most space, as \cw{du} does; but unlike \cw{du}, it also
distinguishes between large collections of data which are still in
use and ones which have not been accessed in months or years \dash
for instance, large archives downloaded, unpacked, used once, and
never cleaned up. Where \cw{du} helps you find what's using your
disk space, \cw{agedu} helps you find what's \e{wasting} your disk
space.

\cw{agedu} has several operating modes. In one mode, it scans your
disk and builds an index file containing a data structure which
allows it to efficiently retrieve any information it might need.
Typically, you would use it in this mode first, and then run it in
one of a number of \q{query} modes to display a report of the disk
space usage of a particular directory and its subdirectories. Those
reports can be produced as plain text (much like \cw{du}) or as
HTML. \cw{agedu} can even run as a miniature web server, presenting
each directory's HTML report with hyperlinks to let you navigate
around the file system to similar reports for other directories.

So you would typically start using \cw{agedu} by telling it to do a
scan of a directory tree and build an index. This is done with a
command such as

\c $ agedu -s /home/fred
\e   bbbbbbbbbbbbbbbbbbb

which will build a large data file called \c{agedu.dat} in your
current directory. (If that current directory is \e{inside}
\cw{/home/fred}, don't worry \dash \cw{agedu} is smart enough to
discount its own index file.)

Having built the index, you would now query it for reports of disk
space usage. If you have a graphical web browser, the simplest and
nicest way to query the index is by running \cw{agedu} in web server
mode:

\c $ agedu -w
\e   bbbbbbbb

which will print (among other messages) a URL on its standard output
along the lines of

\c URL: http://127.0.0.1:48638/

(That URL will always begin with \cq{127.}, meaning that it's in the
\cw{localhost} address space. So only processes running on the same
computer can even try to connect to that web server, and also there
is access control to prevent other users from seeing it \dash see
below for more detail.)

Now paste that URL into your web browser, and you will be shown a
graphical representation of the disk usage in \cw{/home/fred} and
its immediate subdirectories, with varying colours used to show the
difference between disused and recently-accessed data. Click on any
subdirectory to descend into it and see a report for its
subdirectories in turn; click on parts of the pathname at the top of
any page to return to higher-level directories. When you've finished
browsing, you can just press Ctrl-D to send an end-of-file
indication to \cw{agedu}, and it will shut down.

After that, you probably want to delete the data file
\cw{agedu.dat}, since it's pretty large. In fact, the command
\cw{agedu -R} will do this for you; and you can chain \cw{agedu}
commands on the same command line, so that instead of the above you
could have done

\c $ agedu -s /home/fred -w -R
\e   bbbbbbbbbbbbbbbbbbbbbbbbb

for a single self-contained run of \cw{agedu} which builds its
index, serves web pages from it, and cleans it up when finished.

If you don't have a graphical web browser, you can do text-based
queries as well. Having scanned \cw{/home/fred} as above, you might
run

\c $ agedu -t /home/fred
\e   bbbbbbbbbbbbbbbbbbb

which again gives a summary of the disk usage in \cw{/home/fred} and
its immediate subdirectories; but this time \cw{agedu} will print it
on standard output, in much the same format as \cw{du}. If you then
want to find out how much \e{old} data is there, you can add the
\cw{-a} option to show only files last accessed a certain length of
time ago. For example, to show only files which haven't been looked
at in six months or more:

\c $ agedu -t /home/fred -a 6m
\e   bbbbbbbbbbbbbbbbbbbbbbbbb

That's the essence of what \cw{agedu} does. It has other modes of
operation for more complex situations, and the usual array of
configurable options. The following sections contain a complete
reference for all its functionality.

\U OPERATING MODES

This section describes the operating modes supported by \cw{agedu}.
Each of these is in the form of a command-line option, sometimes
with an argument. Multiple operating-mode options may appear on the
command line, in which case \cw{agedu} will perform the specified
actions one after another. For instance, as shown in the previous
section, you might want to perform a disk scan and immediately
launch a web server giving reports from that scan.

\dt \cw{-s} \e{directory} or \cw{--scan} \e{directory}

\dd In this mode, \cw{agedu} scans the file system starting at the
specified directory, and indexes the results of the scan into a
large data file which other operating modes can query.

\lcont{

By default, the scan is restricted to a single file system (since
the expected use of \cw{agedu} is that you would probably use it
because a particular disk partition was running low on space). You
can remove that restriction using the \cw{--cross-fs} option; other
configuration options allow you to include or exclude files or
entire subdirectories from the scan. See the next section for full
details of the configurable options.

The index file is created with restrictive permissions, in case the
file system you are scanning contains confidential information in
its structure.

Index files are dependent on the characteristics of the CPU
architecture you created them on. You should not expect to be able
to move an index file between different types of computer and have
it continue to work. If you need to transfer the results of a disk
scan to a different kind of computer, see the \cw{-D} and \cw{-L}
options below.

}

\dt \cw{-w} or \cw{--web}

\dd In this mode, \cw{agedu} expects to find an index file already
written. It allocates a network port, and starts up a web server on
that port which serves reports generated from the index file. By
default it invents its own URL and prints it out.

\lcont{

The web server runs until \cw{agedu} receives an end-of-file event on
its standard input. (The expected usage is that you run it from the
command line, immediately browse web pages until you're satisfied, and
then press Ctrl-D.) To disable the EOF behaviour, use the
\cw{--no-eof} option.

In case the index file contains any confidential information about
your file system, the web server protects the pages it serves from
access by other people. On Linux, this is done transparently by
means of using \cw{/proc/net/tcp} to check the owner of each
incoming connection; failing that, the web server will require a
password to view the reports, and \cw{agedu} will print the password
it invented on standard output along with the URL.

Configurable options for this mode let you specify your own address
and port number to listen on, and also specify your own choice of
authentication method (including turning authentication off
completely) and a username and password of your choice.

}

\dt \cw{-t} \e{directory} or \cw{--text} \e{directory}

\dd In this mode, \cw{agedu} generates a textual report on standard
output, listing the disk usage in the specified directory and all
its subdirectories down to a given depth. By default that depth is
1, so that you see a report for \e{directory} itself and all of its
immediate subdirectories. You can configure a different depth (or no
depth limit) using \cw{-d}, described in the next section.

\lcont{

Used on its own, \cw{-t} merely lists the \e{total} disk usage in
each subdirectory; \cw{agedu}'s additional ability to distinguish
unused from recently-used data is not activated. To activate it, use
the \cw{-a} option to specify a minimum age.

The directory structure stored in \cw{agedu}'s index file is treated
as a set of literal strings. This means that you cannot refer to
directories by synonyms. So if you ran \cw{agedu -s .}, then all the
path names you later pass to the \cw{-t} option must be either
\cq{.} or begin with \cq{./}. Similarly, symbolic links within the
directory you scanned will not be followed; you must refer to each
directory by its canonical, symlink-free pathname.

}

\dt \cw{-R} or \cw{--remove}

\dd In this mode, \cw{agedu} deletes its index file. Running just
\cw{agedu -R} on its own is therefore equivalent to typing \cw{rm
agedu.dat}. However, you can also put \cw{-R} on the end of a
command line to indicate that \cw{agedu} should delete its index
file after it finishes performing other operations.

\dt \cw{-D} or \cw{--dump}

\dd In this mode, \cw{agedu} reads an existing index file and
produces a dump of its contents on standard output. This dump can
later be loaded into a new index file, perhaps on another computer.

\dt \cw{-L} or \cw{--load}

\dd In this mode, \cw{agedu} expects to read a dump produced by the
\cw{-D} option from its standard input. It constructs an index file
from that dump, exactly as it would have if it had read the same
data from a disk scan in \cw{-s} mode.

\dt \cw{-S} \e{directory} or \cw{--scan-dump} \e{directory}

\dd In this mode, \cw{agedu} will scan a directory tree and convert
the results straight into a dump on standard output, without
generating an index file at all. So running \cw{agedu -S /path}
should produce equivalent output to that of \cw{agedu -s /path -D},
except that the latter will produce an index file as a side effect
whereas \cw{-S} will not.

\lcont{

(The output will not be exactly \e{identical}, due to a
difference in treatment of last-access times on directories.
However, it should be effectively equivalent for most purposes. See
the documentation of the \cw{--dir-atime} option in the next section
for further detail.)

}

\dt \cw{-H} \e{directory} or \cw{--html} \e{directory}

\dd In this mode, \cw{agedu} will generate an HTML report of the
disk usage in the specified directory and its immediate
subdirectories, in the same form that it serves from its web server
in \cw{-w} mode.

\lcont{

By default, a single HTML report will be generated and simply
written to standard output, with no hyperlinks pointing to other
similar pages. If you also specify the \cw{-d} option (see below),
\cw{agedu} will instead write out a collection of HTML files with
hyperlinks between them, and call the top-level file
\cw{index.html}.

}

\dt \cw{--cgi}

\dd In this mode, \cw{agedu} will run as the bulk of a CGI script
which provides the same set of web pages as the built-in web server
would. It will read the usual CGI environment variables, and write
CGI-style data to its standard output.

\lcont{

The actual CGI program itself should be a tiny wrapper around
\cw{agedu} which passes it the \cw{--cgi} option, and also
(probably) \cw{-f} to locate the index file. \cw{agedu} will do
everything else.

No access control is performed in this mode: restricting access to
CGI scripts is assumed to be the job of the web server.

}

\U OPTIONS

This section describes the various configuration options that affect
\cw{agedu}'s operation in one mode or another.

The following option affects nearly all modes (except \cw{-S}):

\dt \cw{-f} \e{filename} or \cw{--file} \e{filename}

\dd Specifies the location of the index file which \cw{agedu}
creates, reads or removes depending on its operating mode. By
default, this is simply \cq{agedu.dat}, in whatever is the current
working directory when you run \cw{agedu}.

The following options affect the disk-scanning modes, \cw{-s} and
\cw{-S}:

\dt \cw{--cross-fs} and \cw{--no-cross-fs}

\dd These configure whether or not the disk scan is permitted to
cross between different file systems. The default is not to:
\cw{agedu} will normally skip over subdirectories on which a
different file system is mounted. This makes it convenient when you
want to free up space on a particular file system which is running
low. However, in other circumstances you might wish to see general
information about the use of space no matter which file system it's
on (for instance, if your real concern is your backup media running
out of space, and if your backups do not treat different file
systems specially); in that situation, use \cw{--cross-fs}.

\lcont{

(Note that this default is the opposite way round from the
corresponding option in \cw{du}.)

}

\dt \cw{--prune} \e{wildcard} and \cw{--prune-path} \e{wildcard}

\dd These cause particular files or directories to be omitted
entirely from the scan. If \cw{agedu}'s scan encounters a file or
directory whose name matches the wildcard provided to the
\cw{--prune} option, it will not include that file in its index, and
also if it's a directory it will skip over it and not scan its
contents.

\lcont{

Note that in most Unix shells, wildcards will probably need to be
escaped on the command line, to prevent the shell from expanding the
wildcard before \cw{agedu} sees it.

\cw{--prune-path} is similar to \cw{--prune}, except that the
wildcard is matched against the entire pathname instead of just the
filename at the end of it. So whereas \cw{--prune *a*b*} will match
any file whose actual name contains an \cw{a} somewhere before a
\cw{b}, \cw{--prune-path *a*b*} will also match a file whose name
contains \cw{b} and which is inside a directory containing an
\cw{a}, or any file inside a directory of that form, and so on.

}

\dt \cw{--exclude} \e{wildcard} and \cw{--exclude-path} \e{wildcard}

\dd These cause particular files or directories to be omitted from
the index, but not from the scan. If \cw{agedu}'s scan encounters a
file or directory whose name matches the wildcard provided to the
\cw{--exclude} option, it will not include that file in its index
\dash but unlike \cw{--prune}, if the file in question is a
directory it will still scan its contents and index them if they are
not ruled out themselves by \cw{--exclude} options.

\lcont{

As above, \cw{--exclude-path} is similar to \cw{--exclude}, except
that the wildcard is matched against the entire pathname.

}

\dt \cw{--include} \e{wildcard} and \cw{--include-path} \e{wildcard}

\dd These cause particular files or directories to be re-included in
the index and the scan, if they had previously been ruled out by one
of the above exclude or prune options. You can interleave include,
exclude and prune options as you wish on the command line, and if
more than one of them applies to a file then the last one takes
priority.

\lcont{

For example, if you wanted to see only the disk space taken up by
MP3 files, you might run

\c $ agedu -s . --exclude '*' --include '*.mp3'
\e   bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

which will cause everything to be omitted from the scan, but then
the MP3 files to be put back in. If you then wanted only a subset of
those MP3s, you could then exclude some of them again by adding,
say, \cq{--exclude-path './queen/*'} (or, more efficiently,
\cq{--prune ./queen}) on the end of that command.

As with the previous two options, \cw{--include-path} is similar to
\cw{--include} except that the wildcard is matched against the
entire pathname.

}

\dt \cw{--progress}, \cw{--no-progress} and \cw{--tty-progress}

\dd When \cw{agedu} is scanning a directory tree, it will typically
print a one-line progress report every second showing where it has
reached in the scan, so you can have some idea of how much longer it
will take. (Of course, it can't predict \e{exactly} how long it will
take, since it doesn't know which of the directories it hasn't
scanned yet will turn out to be huge.)

\lcont{

By default, those progress reports are displayed on \cw{agedu}'s
standard error channel, if that channel points to a terminal device.
If you need to manually enable or disable them, you can use the
above three options to do so: \cw{--progress} unconditionally
enables the progress reports, \cw{--no-progress} unconditionally
disables them, and \cw{--tty-progress} reverts to the default
behaviour which is conditional on standard error being a terminal.

}

\dt \cw{--dir-atime} and \cw{--no-dir-atime}

\dd In normal operation, \cw{agedu} ignores the atimes (last access
times) on the \e{directories} it scans: it only pays attention to
the atimes of the \e{files} inside those directories. This is
because directory atimes tend to be reset by a lot of system
administrative tasks, such as \cw{cron} jobs which scan the file
system for one reason or another \dash or even other invocations of
\cw{agedu} itself, though it tries to avoid modifying any atimes if
possible. So the literal atimes on directories are typically not
representative of how long ago the data in question was last
accessed with real intent to use that data in particular.

\lcont{

Instead, \cw{agedu} makes up a fake atime for every directory it
scans, which is equal to the newest atime of any file in or below
that directory (or the directory's last \e{modification} time,
whichever is newest). This is based on the assumption that all
\e{important} accesses to directories are actually accesses to the
files inside those directories, so that when any file is accessed
all the directories on the path leading to it should be considered
to have been accessed as well.

In unusual cases it is possible that a directory itself might embody
important data which is accessed by reading the directory. In that
situation, \cw{agedu}'s atime-faking policy will misreport the
directory as disused. In the unlikely event that such directories
form a significant part of your disk space usage, you might want to
turn off the faking. The \cw{--dir-atime} option does this: it
causes the disk scan to read the original atimes of the directories
it scans.

The faking of atimes on directories also requires a processing pass
over the index file after the main disk scan is complete.
\cw{--dir-atime} also turns this pass off. Hence, this option
affects the \cw{-L} option as well as \cw{-s} and \cw{-S}.

(The previous section mentioned that there might be subtle
differences between the output of \cw{agedu -s /path -D} and
\cw{agedu -S /path}. This is why. Doing a scan with \cw{-s} and then
dumping it with \cw{-D} will dump the fully faked atimes on the
directories, whereas doing a scan-to-dump with \cw{-S} will dump
only \e{partially} faked atimes \dash specifically, each directory's
last modification time \dash since the subsequent processing pass
will not have had a chance to take place. However, loading either of
the resulting dump files with \cw{-L} will perform the atime-faking
processing pass, leading to the same data in the index file in each
case. In normal usage it should be safe to ignore all of this
complexity.)

}

\dt \cw{--mtime}

\dd This option causes \cw{agedu} to index files by their last
modification time instead of their last access time. You might want
to use this if your last access times were completely useless for
some reason: for example, if you had recently searched every file on
your system, the system would have lost all the information about
what files you hadn't recently accessed before then. Using this
option is liable to be less effective at finding genuinely wasted
space than the normal mode (that is, it will be more likely to flag
things as disused when they're not, so you will have more candidates
to go through by hand looking for data you don't need), but may be
better than nothing if your last-access times are unhelpful.

\lcont{

Another use for this mode might be to find \e{recently created}
large data. If your disk has been gradually filling up for years,
the default mode of \cw{agedu} will let you find unused data to
delete; but if you know your disk had plenty of space recently and
now it's suddenly full, and you suspect that some rogue program has
left a large core dump or output file, then \cw{agedu --mtime} might
be a convenient way to locate the culprit.

}

The following option affects all the modes that generate reports:
the web server mode \cw{-w}, the stand-alone HTML generation mode
\cw{-H} and the text report mode \cw{-t}.

\dt \cw{--files}

\dd This option causes \cw{agedu}'s reports to list the individual
files in each directory, instead of just giving a combined report
for everything that's not in a subdirectory.

The following options affect the stand-alone HTML generation mode
\cw{-H} and the text report mode \cw{-t}.

\dt \cw{-d} \e{depth} or \cw{--depth} \e{depth}

\dd This option controls the maximum depth to which \cw{agedu}
recurses when generating a text or HTML report.

\lcont{

In text mode, the default is 1, meaning that the report will include
the directory given on the command line and all of its immediate
subdirectories. A depth of two includes another level below that,
and so on; a depth of zero means \e{only} the directory on the
command line.

In HTML mode, specifying this option switches \cw{agedu} from
writing out a single HTML file to writing out multiple files which
link to each other. A depth of 1 means \cw{agedu} will write out an
HTML file for the given directory and also one for each of its
immediate subdirectories.

If you want \cw{agedu} to recurse as deeply as possible, give the
special word \cq{max} as an argument to \cw{-d}.

}

\dt \cw{-o} \e{filename} or \cw{--output} \e{filename}

\dd This option is used to specify an output file for \cw{agedu} to
write its report to. In text mode or single-file HTML mode, the
argument is treated as the name of a file. In multiple-file HTML
mode, the argument is treated as the name of a directory: the
directory will be created if it does not already exist, and the
output HTML files will be created inside it.

The following options affect the web server mode \cw{-w}, and in some
cases also the stand-alone HTML generation mode \cw{-H}:

\dt \cw{-r} \e{age range} or \cw{--age-range} \e{age range}

\dd The HTML reports produced by \cw{agedu} use a range of colours
to indicate how long ago data was last accessed, running from red
(representing the most disused data) to green (representing the
newest). By default, the lengths of time represented by the two ends
of that spectrum are chosen by examining the data file to see what
range of ages appears in it. However, you might want to set your own
limits, and you can do this using \cw{-r}.

\lcont{

The argument to \cw{-r} consists of a single age, or two ages
separated by a minus sign. An age is a number, followed by one of
\cq{y} (years), \cq{m} (months), \cq{w} (weeks) or \cq{d} (days).
The first age in the range represents the oldest data, and will be
coloured red in the HTML; the second age represents the newest,
coloured green. If the second age is not specified, it will default
to zero (so that green means data which has been accessed \e{just
now}).

For example, \cw{-r 2y} will mark data in red if it has been unused
for two years or more, and green if it has been accessed just now.
\cw{-r 2y-3m} will similarly mark data red if it has been unused for
two years or more, but will mark it green if it has been accessed
three months ago or later.

}

\dt \cw{--address} \e{addr}[\cw{:}\e{port}]

\dd Specifies the network address and port number on which
\cw{agedu} should listen when running its web server. If you want
\cw{agedu} to listen for connections coming in from any source, you
should probably specify the special IP address \cw{0.0.0.0}. If the
port number is omitted, an arbitrary unused port will be chosen for
you and displayed.

\lcont{

If you specify this option, \cw{agedu} will not print its URL on
standard output (since you are expected to know what address you
told it to listen to).

}

\dt \cw{--auth} \e{auth-type}

\dd Specifies how \cw{agedu} should control access to the web pages
it serves. The options are as follows:

\lcont{

\dt \cw{magic}

\dd This option only works on Linux, and only when the incoming
connection is from the same machine that \cw{agedu} is running on.
On Linux, the special file \cw{/proc/net/tcp} contains a list of
network connections currently known to the operating system kernel,
including which user id created them. So \cw{agedu} will look up
each incoming connection in that file, and allow access if it comes
from the same user id under which \cw{agedu} itself is running.
Therefore, in \cw{agedu}'s normal web server mode, you can safely
run it on a multi-user machine and no other user will be able to
read data out of your index file.

\dt \cw{basic}

\dd In this mode, \cw{agedu} will use HTTP Basic authentication: the
user will have to provide a username and password via their browser.
\cw{agedu} will normally make up a username and password for the
purpose, but you can specify your own; see below.

\dt \cw{none}

\dd In this mode, the web server is unauthenticated: anyone
connecting to it has full access to the reports generated by
\cw{agedu}. Do not do this unless there is nothing confidential at
all in your index file, or unless you are certain that nobody but
you can run processes on your computer.

\dt \cw{default}

\dd This is the default mode if you do not specify one of the above.
In this mode, \cw{agedu} will attempt to use Linux magic
authentication, but if it detects at startup time that
\cw{/proc/net/tcp} is absent or non-functional then it will fall
back to using HTTP Basic authentication and invent a user name and
password.

}

\dt \cw{--auth-file} \e{filename} or \cw{--auth-fd} \e{fd}

\dd When \cw{agedu} is using HTTP Basic authentication, these
options allow you to specify your own user name and password. If you
specify \cw{--auth-file}, these will be read from the specified
file; if you specify \cw{--auth-fd} they will instead be read from a
given file descriptor which you should have arranged to pass to
\cw{agedu}. In either case, the authentication details should
consist of the username, followed by a colon, followed by the
password, followed \e{immediately} by end of file (no trailing
newline, or else it will be considered part of the password).

\dt \cw{--title} \e{title}

\dd Specify the string that appears at the start of the \cw{<title>}
section of the output HTML pages. The default is \cq{agedu}. This
title is followed by a colon and then the path you're viewing within
the index file. You might use this option if you were serving
\cw{agedu} reports for several different servers and wanted to make it
clearer which one a user was looking at.

\dt \cw{--no-eof}

\dd Stop \cw{agedu} in web server mode from looking for end-of-file on
standard input and treating it as a signal to terminate.

\U LIMITATIONS

The data file is pretty large. The core of \cw{agedu} is the
tree-based data structure it uses in its index in order to
efficiently perform the queries it needs; this data structure
requires \cw{O(N log N)} storage. This is larger than you might
expect; a scan of my own home directory, containing half a million
files and directories and about 20Gb of data, produced an index file
over 60Mb in size. Furthermore, since the data file must be
memory-mapped during most processing, it can never grow larger than
available address space, so a \e{really} big filesystem may need to
be indexed on a 64-bit computer. (This is one reason for the
existence of the \cw{-D} and \cw{-L} options: you can do the
scanning on the machine with access to the filesystem, and the
indexing on a machine big enough to handle it.)

The data structure also does not usefully permit access control
within the data file, so it would be difficult \dash even given the
willingness to do additional coding \dash to run a system-wide
\cw{agedu} scan on a \cw{cron} job and serve the right subset of
reports to each user.

In certain circumstances, \cw{agedu} can report false positives
(reporting files as disused which are in fact in use) as well as the
more benign false negatives (reporting files as in use which are
not). This arises when a file is, semantically speaking, \q{read}
without actually being physically \e{read}. Typically this occurs
when a program checks whether the file's mtime has changed and only
bothers re-reading it if it has; programs which do this include
\cw{rsync}(\e{1}) and \cw{make}(\e{1}). Such programs will fail to
update the atime of unmodified files despite depending on their
continued existence; a directory full of such files will be reported
as disused by \cw{agedu} but deleting them will cause trouble.

\U LICENCE

\cw{agedu} is free software, distributed under the MIT licence. Type
\cw{agedu --licence} to see the full licence text.

\versionid $Id$