TODO list for agedu
===================
-Before it's non-embarrassingly releasable:
+ - flexibility in the HTML report output mode: expose the internal
+ mechanism for configuring the output filenames, and allow the
+ user to request individual files with hyperlinks as if the other
+ files existed. (In particular, functionality of this kind would
+ enable other modes of use like the built-in --cgi mode, without
+ me having to anticipate them in detail.)
- - add more configurable options
- + server address in httpd mode
- + HTTP authentication: specify username and/or password, the
- latter by at least some means which doesn't involve it showing
- up in "ps"
+ - non-ASCII character set support
+ + could usefully apply to --title and also to file names
+ + how do we determine the input charset? Via locale, presumably.
+ + how do we do translation? Importing my charset library is one
+ heavyweight option; alternatively, does the native C locale
+ mechanism provide enough functionality to do the job by itself?
+ + in HTML, we would need to decide on an _output_ character set,
+ specify it in a <meta http-equiv> tag, and translate to it from
+ the input locale
+ - one option is to make the output charset the same as the
+ input one, in which case all we need is to identify its name
+ for the <meta> tag
+ - the other option is to make the output charset UTF-8 always
+ and translate to that from everything else
+ - in the web server and CGI modes, it would probably be nicer
+ to move that <meta> tag into a proper HTTP header
+ + even in text mode we would want to parse the filenames in some
+ fashion, due to the unhelpful corner case of Shift-JIS Windows
+ (in which backslashes in the input string must be classified as
+ path separators or the second byte of a two-byte character)
+ - that's really painful, since it will impact string processing
+ of filenames throughout the code
+ - so perhaps a better approach would be to do locale processing
+ of filenames at _scan_ time, and normalise to UTF-8 in both
+ the index and dump files?
+ + involves incrementing the version of the dump-file format
+ + then paths given on the command line are translated
+ quickly to UTF-8 before comparing them against index paths
+ + and now the HTML output side becomes easy, though the text
+ output involves translating back again
+ + but what if the filenames aren't intended to be
+ interpreted in any particular character set (old-style
+ Unix semantics) or in a consistent one?
- - more flexible running modes
- + combined scan+dump mode which doesn't even generate an index
- file (nearly indistinguishable from find(1))
- + load mode which reads a dump from standard input and builds
- the index (need to nail down a perfectly general dump format)
- + at least some ability to chain actions within the same run:
- "agedu -s dirname -w" would seem handy.
-
- - work out what to do about atimes on directories in the absence of
- the Linux syscall magic
- * one option is to read them during the scan and reinstate them
- after each recursion pop. Race-condition prone.
- * marking them in a distinctive colour in the reports is another
- option.
- * a third option is simply to ignore space taken up by
- directories in the first place; inaccurate but terribly simple.
- * incidentally, sometimes open(...,O_NOATIME) will fail, and
- then we have to fall back to ordinary open. Be prepared to do
- this, which probably means getting rid of the icky macro
- hackery in du.c and turning it into a more sensible run-time
- abstraction layer.
-
- - polish the plain-text output to make it look more like du
- + configurable recursive output depth
- + show the right bits last
-
- - figure out what to do about scans starting in the root directory
- + Currently we end up with a double leading slash on the
- pathnames, which is ugly, and we also get a zero-length href
- in between those slashes which means the web interface doesn't
- let you click back up to the top level at all.
- + One big problem here is that a lot of the code assumes that
- you can find the extent of a pathname by searching for "foo"
- and "foo^A", trusting that anything inside the directory will
- begin "foo/". So I'd need to consistently fix this everywhere
- so that a trailing slash is disregarded while doing it, but
- not actually removed.
- + The text output gets it all wrong.
- + The HTML output is fiddly even at the design stage: where
- would I _ideally_ put the link to click on to get back to /?
- It's unclear!
-
- - cross-Unix portability:
- + use autoconf
- * configure use of stat64
- * configure use of /proc/net/tcp
- * configure use of /dev/random
- * configure use of Linux syscall magic replacing readdir
- + later glibcs have fdopendir, hooray! So we can use that
- too, if it's available and O_NOATIME is too.
- * what do we do elsewhere about _GNU_SOURCE?
-
- - prepare a little in advance for a potential future Windows port:
- + store the separator character in the index file when writing
- it, and be prepared to cope on reading if it isn't a slash
- + store literal byte sizes in all the size fields, instead of
- Unixoid 512-byte sectors
-
- - man page, licence.
-
-Future directions:
+ - we could still be using more of the information coming from
+ autoconf. Our config.h is defining a whole bunch of HAVE_FOOs for
+ particular functions (e.g. HAVE_INET_NTOA, HAVE_MEMCHR,
+ HAVE_FNMATCH). We could usefully supply alternatives for some of
+ these functions (e.g. cannibalise the PuTTY wildcard matcher for
+ use in the absence of fnmatch, switch to vanilla truncate() in
+ the absence of ftruncate); where we don't have alternative code,
+ it would perhaps be polite to throw an error at configure time
+ rather than allowing the subsequent build to fail.
+ + however, I don't see anything here that looks very
+ controversial; IIRC it's all in POSIX, for one thing. So more
+ likely this should simply wait until somebody complains.
- run-time configuration in the HTTP server
* I think this probably works by having a configuration form, or
straight to terminfo: generate lines of attribute-interleaved
text and display them, so we only really need the sequences
"go here and display stuff", "scroll up", "scroll down".
- + I think the attribute-interleaved text might be possible to do
- cunningly, as well: we autodetect a basically VT-style
- terminal, and add 256-colour sequences on the end. So, for
- instance, we might set ANSI-yellow foreground, set ANSI-red
- background, _then_ set both foreground and background to the
- appropriate xterm 256-colour, and then display some
- appropriate character which would have given the right blend
- of the ANSI-16 fore and background colours. Then the same
- display code should gracefully degrade in the face of a
- terminal which doesn't support xterm-256.
- * current best plan is to simulate the xterm-256 shading from
- 0/5 to 5/5 by doing space, colon and hash in colour A on
- colour B background, then hash, colon and space in B on A
- background.
+ Infrastructure work before doing any of this would be to split
html.c into two: one part to prepare an abstract data
structure describing an HTML-like report (in particular, all
HTML. Then the former can be reused to produce very similar
reports in coloured plain text.
- - http://msdn.microsoft.com/en-us/library/ms724290.aspx suggest
- modern Windowses support atime-equivalents, so a Windows port is
- possible in principle. Would need to modify the current structure
- a lot, to abstract away (at least) memory-mapping of files,
- details of disk scan procedure, networking for httpd. Unclear
- what the right UI would be on Windows, too; command-line exactly
- as now might be considered just a _little_ unfriendly. Or perhaps
- not.
+ - abstracting away all the Unix calls so as to enable a full
+ Windows port. We can already do the difficult bit on Windows
+ (scanning the filesystem and retrieving atime-analogues).
+ Everything else is just coding - albeit quite a _lot_ of coding,
+ since the Unix assumptions are woven quite tightly into the
+ current code.
+ + If nothing else, it's unclear what the user interface properly
+ ought to be in a Windows port of agedu. A command-line job
+ exactly like the Unix version might be useful to some people,
+ but would certainly be strange and confusing to others.
+
+ - it might conceivably be useful to support a choice of indexing
+ strategies. The current "continuous index" mechanism's tradeoff of
+ taking O(N log N) space in order to be able to support any age
+ cutoff you like is not going to be ideal for everybody. A second
+ more conventional "discrete index" mechanism which allows the
+ user to specify a number of fixed cutoffs and just indexes each
+ directory on those alone would undoubtedly be a useful thing for
+ large-scale users. This will require considerable thought about
+ how to make the indexers pluggable at both index-generation time
+ and query time.
+ * however, now we have the cut-down version of the continuous
+ index, the space saving is less compelling.
+
+ - A user requested what's essentially a VFS layer: given multiple
+ index files and a map of how they fit into an overall namespace,
+ we should be able to construct the right answers for any query
+ about the resulting aggregated hierarchy by doing at most
+ O(number of indexes * normal number of queries) work.
+
+ - Support for filtering the scan by ownership and permissions. The
+ index data structure can't handle this, so we can't build a
+ single index file admitting multiple subset views; but a user
+ suggested that the scan phase could record information about
+ ownership and permissions in the dump file, and then the indexing
+ phase could filter down to a particular sub-view - which would at
+ least allow the construction of various subset indices from one
+ dump file, without having to redo the full disk scan which is the
+ most time-consuming part of all.
~mdw / sgt / agedu / blobdiff