--- /dev/null
+.\" -*-nroff-*-
+.\"
+.\" Manual for the audio conversion gremlin
+.\"
+.\" (c) 2016 Mark Wooding
+.\"
+.
+.\"----- Licensing notice ---------------------------------------------------
+.\"
+.\" This file is part of the `autoys' audio tools collection.
+.\"
+.\" `autoys' is free software; you can redistribute it and/or modify
+.\" it under the terms of the GNU General Public License as published by
+.\" the Free Software Foundation; either version 2 of the License, or
+.\" (at your option) any later version.
+.\"
+.\" `autoys' is distributed in the hope that it will be useful,
+.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
+.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+.\" GNU General Public License for more details.
+.\"
+.\" You should have received a copy of the GNU General Public License
+.\" along with `autoys'; if not, write to the Free Software Foundation,
+.\" Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+.
+.TH gremlin 1 "13 February 2016" "Mark Wooding" "autoys"
+.
+.\"--------------------------------------------------------------------------
+.SH NAME
+gremlin \- batch audio file converter
+.
+.SH SYNOPSIS
+.B gremlin
+.RB [ \-in ]
+.RB [ \-T
+.IR timeout ]
+.RB [ \-t
+.IR timeout ]
+.I config
+.
+.\"--------------------------------------------------------------------------
+.SH DESCRIPTION
+.
+The
+.B gremlin
+program converts audio files
+in an input `master' directory tree,
+which presumably contains
+high-quality (ideally lossless) encodings
+of interesting audio,
+writing corresponding converted files
+to a collection of output directory trees.
+It's non-interactive, idempotent, and restartable;
+it never modifies its master tree.
+It's exactly the sort of thing you want to
+install as a daily cron job.
+.PP
+The
+.B gremlin
+reads a configuration file
+which describes the conversion policy for each of the output trees.
+The policy can say things like:
+copy MP3 files up to 160kb/s,
+or Ogg Vorbis files up to 128kb/s;
+and convert everything else to 128kb/s Ogg Vorbis.
+.PP
+The
+.B gremlin
+can also convert image files, such as cover art.
+.PP
+Input files can be anything which
+GStreamer and/or the Python Imaging Library can understand;
+output files are more constrained,
+because the
+.B gremlin
+has to be able to understand
+their relevant properties.
+The currently supported audio formats are
+Ogg Vorbis and
+MP3;
+image formats are
+JPEG,
+PNG, and
+BMP.
+.PP
+In a little more detail:
+the
+.B gremlin
+works through its input master tree,
+one directory at a time.
+For each master directory,
+it tries to write a converted version
+to a corresponding output directory
+in each of the output trees.
+For each file in the master directory,
+it determines which files should be made
+in each output directory:
+if those files exist,
+and are not older than the master file,
+then they're left alone on the assumption that they're up-to-date;
+otherwise, the
+.B gremlin
+will make the output files by converting the master file.
+.PP
+Any other files or directories in the output directory
+will be
+.IR deleted .
+The
+.B gremlin
+assumes that its output trees belong entirely to it,
+to maintain according to its configuration,
+and that unexpected files are either
+debris left over from an earlier failure
+or a result of a policy change,
+and in either case the right thing to do is
+to delete the offending files.
+.
+.SS "Command line syntax"
+The following options are recognized.
+.TP
+.B "\-h, \-\-help"
+Write a help message to standard output
+describing the
+.BR gremlin 's
+command-line options,
+and exit with status zero.
+.TP
+.B "\-\-version"
+Write the
+.BR gremlin 's
+version number to standard output
+and exit with status zero.
+.TP
+.B "\-i, \-\-interactive"
+Write progress eyecandy to standard output while running.
+While walking the master tree,
+the
+.B gremlin
+shows which directory it's currently examining.
+While converting audio files,
+it shows a progress meter showing
+a bar chart of the job in progress,
+the percentage of the job which is complete,
+and an estimated time to completion.
+(This last starts out rather inaccurate,
+but seems to be pretty good after a couple of seconds.)
+All this is done automatically if standard output is a terminal;
+this option can be used to turn it on under other circumstances.
+.TP
+.B "\-n, \-\-no-act"
+Don't actually modify the filesystem.
+No files will be created or removed.
+.TP
+.BI "\-t, \-\-timeout=" timeout
+Only run for about
+.I timeout
+seconds.
+Once the timeout has expired,
+.B gremlin
+will try to finish what it's doing
+and then exit with status zero.
+.IP
+(This might seem a surprising choice of exit status.
+The idea is that the
+.B gremlin
+was asked to spend some amount of time converting files,
+and it has done that successfully.)
+.TP
+.BI "\-T, \-\-timeout-nasty=" timeout
+If the timeout set by the
+.B \-t
+option (above) has expired,
+and a further
+.I timeout
+seconds have elapsed
+but the
+.B gremlin
+still hasn't managed to wrap things up,
+then exit immediately with status 3,
+possibly leaving files partially converted,
+or other kinds of incompleteness.
+(A future run of the
+.B gremlin
+will notice this wreckage and clean it up.)
+.
+.\"--------------------------------------------------------------------------
+.SH CONFIGURATION FILE
+.
+.SS "Lexical syntax"
+The
+.BR gremlin 's
+configuration file has a simple token-oriented lexical syntax.
+Whitespace acts to separate tokens but has no other meaning.
+A hash sign
+.RB ` # '
+outside of a quoted string introduces a comment
+which extends to the end of the line;
+newlines otherwise just separate tokens, just like other whitespace.
+There are no `reserved words',
+but some names have special meanings,
+depending on the context.
+.PP
+Integers are written in decimal.
+(There is no provision for entering numbers in hex or octal.)
+.IP
+.I int
+::=
+.I digit
+\&...
+.br
+.I digit
+::=
+.B 0
+|
+.B 1
+|
+.B 2
+|
+.B 3
+|
+.B 4
+|
+.B 5
+|
+.B 6
+|
+.B 7
+|
+.B 8
+|
+.B 9
+.PP
+Strings (mostly used for pathnames and suchlike)
+are enclosed in double quotes
+.RB ` """" ';
+quotes and backslashes to be included in the string
+must be escaped by preceding them with a backslash
+.RB ` \e '.
+.IP
+.I string
+::=
+.B """"
+.IR string-char ...\&
+.B """"
+.br
+.I string-char
+::=
+any character other than
+.B """"
+or
+.B \e
+.br
+\h'4m'|
+.B "\e"""
+|
+.B \e\e
+.
+.SS "Top-level syntax"
+At a high level,
+the configuration consists of a sequence of
+.IR "top-level items" .
+.IP
+.I config
+::=
+.I toplevel-item
+\&...
+.
+.SS "Global settings"
+Miscellaneous configuration for the whole program
+goes in a top-level
+.B vars
+section.
+.IP
+.I toplevel-item
+::=
+.I vars-section
+.br
+.I vars-section
+::=
+.B vars
+.B {
+.IR var-setting
+\&...\&
+.B }
+.PP
+There may be multiple such sections.
+The same variable may be set more than once;
+if that happens,
+only the last such setting has affect.
+.IP
+.I var-setting
+::=
+.B master
+.B =
+.I path
+.br
+.I path
+::=
+.I string
+.PP
+The
+.B master
+variable holds the pathname of the top of the master tree.
+.PP
+There are, at present, no other global settings.
+.
+.SS "Target definitions"
+The other kind of top-level configuration item
+defines a target directory
+to be constructed or updated
+by the
+.BR gremlin .
+.IP
+.I toplevel-item
+::=
+.I target-def
+.br
+.I target-def
+::=
+.B target
+.I path
+.B {
+.I type-clause
+\&...\&
+.B }
+.br
+.I type-clause
+::=
+.B type
+.I type
+.B {
+.I policy
+\&...\&
+.B }
+.PP
+A
+.B target
+definition tells the
+.B gremlin
+to populate a directory tree,
+named rooted at the given
+.IR path .
+The body of the target definition consists of
+a sequence of
+.B type
+clauses
+which explain what to do with different kinds of file.
+The possible
+.I type
+tokens are as follows.
+.TP
+.B audio
+Encoded audio files,
+which can be decoded by the GStreamer library.
+.TP
+.B image
+Image files,
+which can be decoded by the Python Imaging Library.
+.PP
+The body of the type clause defines a
+.I policy
+for converting files of that type.
+.
+.SS "Policy descriptions"
+There are two kinds of
+.I primitive
+policies,
+which are described in full below:
+.BR accept ,
+which copies (or links) a master file
+if its format is appropriate,
+or does nothing;
+and
+.BR convert ,
+which converts a master file into a chosen format,
+and (in principle) should always succeed.
+There are also two ways to build up
+.I compound
+policies from simpler ones.
+.IP
+.I policy
+::=
+.B and
+.B {
+.I policy
+\&...\&
+.B }
+.br
+\h'4m'|
+.B or
+.B {
+.I policy
+\&...\&
+.B }
+.PP
+The
+.B and
+policy applies
+.I all
+of its operand policies,
+potentially producing multiple output files.
+.PP
+The body of a
+.I type-clause
+consists of a sequence of policies
+which are implicitly combined together in this way.
+.PP
+The
+.B or
+policy
+tries its operand policies in turn,
+in the order specified,
+until one of them succeeds;
+no more policies are tried after this.
+.IP
+.I policy
+::=
+.B accept
+.I format-spec
+.br
+\h'4m'|
+.B convert
+.I format-spec
+.IP
+.I format-spec
+::=
+.I format-name
+.br
+\h'4m'|
+.I format-name
+.B {
+.I format-prop
+\&...\&
+.B }
+.PP
+(The possible
+.IR format-name s
+and the corresponding
+.IR format-spec s
+are described in the section below.)
+.PP
+The
+.B convert
+policy converts a file to the specified format.
+More specifically:
+if the file's format already matches the
+.I format-spec
+then it is copied to the target directory.
+(Indeed, if possible,
+the file is hard linked into the target directory.)
+If the file's format doesn't match,
+then the
+.B gremlin
+converts it,
+producing an output file of the requested format.
+.PP
+The
+.B accept
+policy copies or links a file if its format matches the
+.IR format-spec ,
+just as
+.B convert
+does.
+However, if the file doesn't match then
+.B accept
+fails.
+.PP
+The usual use of
+.B accept
+is within an
+.B or
+block.
+For example, suppose that the master tree mostly contains
+losslessly encoded files, such as FLAC,
+and we usually want to produce Ogg Vorbis
+for use on devices with limited storage capacity;
+but some of the master files are only available as MP3,
+and re-encoding MP3 as Ogg Vorbis won't be good for sound quality.
+Therefore, you can say something like
+.IP
+.nf
+.ft B
+or {
+ accept mp3 { bitrate = 160 }
+ convert ogg-vorbis { bitrate = 128 }
+}
+.fi
+.ft P
+.PP
+which means:
+if a master file is an MP3 file with bitrate approximately 160kb/s or less,
+then copy it;
+otherwise, convert the file to Ogg Vorbis, at about 128kb/s.
+.PP
+It's possible that even a simple policy
+acting on the files in a master directory
+will come up with multiple ways
+to produce the same output file.
+The rule used to decide is as follows:
+if the
+.B gremlin
+can make the output file by copying one of the master files
+then it does that;
+otherwise it converts one of the inputs chosen arbitrarily.
+For example,
+suppose that a policy for
+.B audio
+files says
+.IP
+.B convert ogg-vorbis
+.PP
+and the master directory contains
+.B foo.flac
+and
+.BR foo.ogg ;
+then it will copy
+.B foo.ogg
+and ignore
+.BR foo.flac .
+If, instead, the master contains
+.B foo.flac
+and
+.BR foo.mp3 ,
+then one of these will be converted,
+but it's hard to predict which.
+.
+.SS "Audio formats"
+Two audio
+.IR format-type s
+are defined.
+.PP
+All audio formats support a
+.B bitrate
+property.
+.IP
+.I format-prop
+::=
+.B bitrate
+.B =
+.I int
+.PP
+The bitrate is expressed in kilobits per second.
+For an existing file to match a
+.I format-spec
+containing a
+.B bitrate
+property,
+the file's bitrate must be less than
+the specified bitrate times a fudge factor
+(currently sqrt(2)).
+(The
+.B bitrate
+property is notionally the desired
+.I output
+bitrate;
+the
+.B gremlin
+assumes that it's better to make output files a bit larger
+than to re-encode an already lossily compressed master file.)
+.PP
+At present, the audio formats define no other properties.
+.TP
+.B mp3
+The MP3 format that everyone knows and loves.
+For encoding, the
+.B gremlin
+uses Lame,
+and stores metadata in an ID3v2 tag;
+it also tries to store an ID3v1.1 tag,
+but this can fail for a number of reasons
+(e.g., if the genre can't be represented,
+or text contains characters outside of the ISO 8859-1 character set
+used in ID3v1 tags).
+.TP
+.B ogg-vorbis
+Vorbis-encoded audio in an Ogg container,
+as defined by the Xiph.Org Foundation.
+On encoding, the
+.B bitrate
+parameter is actually mapped to a quality setting
+chosen to produce approximately the right bitrate.
+.
+.SS "Image formats"
+Three image
+.IR format-type s
+are defined.
+.PP
+All image formats support a
+.B size
+property.
+.IP
+.I format-prop
+::=
+.B size
+.B =
+.I int
+.PP
+The size provides an upper bound on the width and height of the image.
+A master file will only match if
+both its width and height are
+less than the stated size.
+On output, the image will be scaled to the right size,
+preserving its aspect ratio.
+.TP
+.B jpeg
+The JFIF format, defined by the Joint Photographic Experts Group.
+The following additional properties can be set;
+they affect output only.
+.RS
+.TP
+.B optimize
+Spend longer to select optimal encoder settings.
+.TP
+.B progressive
+Make a progressively-rendering output file.
+This isn't usually a good idea.
+.TP
+.BI "quality = " int
+Set the image quality (at the expense of file size).
+This is a percentage; the default is 75.
+.RE
+.TP
+.B png
+The Portable Network Graphics format,
+originally defined in RFC2083.
+The following additional properties can be set;
+they affect output only.
+.RS
+.TP
+.B optimize
+Spend longer to try to make the output file smaller.
+.RE
+.TP
+.B bmp
+The Windows BMP format.
+There are no additional properties.
+.
+.SS "Example file"
+The following is the author's configuration file.
+I have an archive which mostly consists of FLAC files,
+with a few MP3 files where I've been unable to obtain physical CDs.
+I generate two output trees.
+One mostly contains Ogg Vorbis files,
+but tolerates occasional MP3
+rather than suffer the quality loss of re-encoding.
+It also generates small BMP-format images from cover art,
+because I have an old portable audio player
+which runs the free RockBox firmware,
+whose player is only capable of displaying such images.
+.IP
+.nf
+.ft B
+### -*-conf-*-
+
+vars {
+ master = "/mnt/jb/master"
+}
+
+target "/mnt/jb/gremlin/ogg-vorbis-128" {
+ type audio {
+ or {
+ accept mp3 { bitrate = 160 }
+ convert ogg-vorbis { bitrate = 128 }
+ }
+ }
+ type image {
+ or {
+ accept png
+ convert jpeg { quality = 7 }
+ }
+ convert bmp { size = 75 }
+ }
+}
+
+target "/mnt/jb/gremlin/mp3-160" {
+ type audio {
+ convert mp3 { bitrate = 160 }
+ }
+ type image {
+ or {
+ accept png
+ convert jpeg { quality = 7 }
+ }
+ }
+}
+.fi
+.ft P
+.
+.\"--------------------------------------------------------------------------
+.SH BUGS
+.
+The
+.B gremlin
+makes no effort to process more than one file at a time.
+.PP
+It should probably support more audio formats.
+They're quite easy to add,
+but I don't have a good feel for which formats are good.
+Patches and advice are welcome.
+.PP
+The
+.B and
+and
+.B or
+policy names are possibly confusing.
+They suggest that they work like the standard logical operators;
+while
+.B or
+sort of does, if you squint a bit,
+.B and
+certainly doesn't;
+on the other hand, it does try to do all of the things you ask of it.
+.PP
+.B gremlin
+is a very unhelpful name for the program.
+.
+.\"--------------------------------------------------------------------------
+.SH AUTHOR
+Mark Wooding, <mdw@distorted.org.uk>
+.
+.SH SEE ALSO
+.BR hush (1),
+.BR rsync (1).
+.
+.\"----- That's all, folks --------------------------------------------------
~mdw / autoys / commitdiff