gremlin/gremlin.in: Use `locale.getpreferredencoding'.

[autoys] / gremlin / gremlin.1

.\" -*-nroff-*-
.\"
.\" Manual for the audio conversion gremlin
.\"
.\" (c) 2016 Mark Wooding
.\"
.
.\"----- Licensing notice ---------------------------------------------------
.\"
.\" This file is part of the `autoys' audio tools collection.
.\"
.\" `autoys' is free software; you can redistribute it and/or modify
.\" it under the terms of the GNU General Public License as published by
.\" the Free Software Foundation; either version 2 of the License, or
.\" (at your option) any later version.
.\"
.\" `autoys' is distributed in the hope that it will be useful,
.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
.\" GNU General Public License for more details.
.\"
.\" You should have received a copy of the GNU General Public License
.\" along with `autoys'; if not, write to the Free Software Foundation,
.\" Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
.
.TH gremlin 1 "13 February 2016" "Mark Wooding" "autoys"
.
.\"--------------------------------------------------------------------------
.SH NAME
gremlin \- batch audio file converter
.
.SH SYNOPSIS
.B gremlin
.RB [ \-in ]
.RB [ \-T
.IR timeout ]
.RB [ \-t
.IR timeout ]
.I config
.
.\"--------------------------------------------------------------------------
.SH DESCRIPTION
.
The
.B gremlin
program converts audio files
in an input `master' directory tree,
which presumably contains
high-quality (ideally lossless) encodings
of interesting audio,
writing corresponding converted files
to a collection of output directory trees.
It's non-interactive, idempotent, and restartable;
it never modifies its master tree.
It's exactly the sort of thing you want to
install as a daily cron job.
.PP
The
.B gremlin
reads a configuration file
which describes the conversion policy for each of the output trees.
The policy can say things like:
copy MP3 files up to 160kb/s,
or Ogg Vorbis files up to 128kb/s;
and convert everything else to 128kb/s Ogg Vorbis.
.PP
The
.B gremlin
can also convert image files, such as cover art.
.PP
Input files can be anything which
GStreamer and/or the Python Imaging Library can understand;
output files are more constrained,
because the
.B gremlin
has to be able to understand
their relevant properties.
The currently supported audio formats are
Ogg Vorbis and
MP3;
image formats are
JPEG,
PNG, and
BMP.
.PP
In a little more detail:
the
.B gremlin
works through its input master tree,
one directory at a time.
For each master directory,
it tries to write a converted version
to a corresponding output directory
in each of the output trees.
For each file in the master directory,
it determines which files should be made
in each output directory:
if those files exist,
and are not older than the master file,
then they're left alone on the assumption that they're up-to-date;
otherwise, the
.B gremlin
will make the output files by converting the master file.
.PP
Any other files or directories in the output directory
will be
.IR deleted .
The
.B gremlin
assumes that its output trees belong entirely to it,
to maintain according to its configuration,
and that unexpected files are either
debris left over from an earlier failure
or a result of a policy change,
and in either case the right thing to do is
to delete the offending files.
.
.SS "Command line syntax"
The following options are recognized.
.TP
.B "\-h, \-\-help"
Write a help message to standard output
describing the
.BR gremlin 's
command-line options,
and exit with status zero.
.TP
.B "\-\-version"
Write the
.BR gremlin 's
version number to standard output
and exit with status zero.
.TP
.B "\-i, \-\-interactive"
Write progress eyecandy to standard output while running.
While walking the master tree,
the
.B gremlin
shows which directory it's currently examining.
While converting audio files,
it shows a progress meter showing
a bar chart of the job in progress,
the percentage of the job which is complete,
and an estimated time to completion.
(This last starts out rather inaccurate,
but seems to be pretty good after a couple of seconds.)
All this is done automatically if standard output is a terminal;
this option can be used to turn it on under other circumstances.
.TP
.B "\-n, \-\-no-act"
Don't actually modify the filesystem.
No files will be created or removed.
.TP
.BI "\-t, \-\-timeout=" timeout
Only run for about
.I timeout
seconds.
Once the timeout has expired,
.B gremlin
will try to finish what it's doing
and then exit with status zero.
.IP
(This might seem a surprising choice of exit status.
The idea is that the
.B gremlin
was asked to spend some amount of time converting files,
and it has done that successfully.)
.TP
.BI "\-T, \-\-timeout-nasty=" timeout
If the timeout set by the
.B \-t
option (above) has expired,
and a further
.I timeout
seconds have elapsed
but the
.B gremlin
still hasn't managed to wrap things up,
then exit immediately with status 3,
possibly leaving files partially converted,
or other kinds of incompleteness.
(A future run of the
.B gremlin
will notice this wreckage and clean it up.)
.
.\"--------------------------------------------------------------------------
.SH CONFIGURATION FILE
.
.SS "Lexical syntax"
The
.BR gremlin 's
configuration file has a simple token-oriented lexical syntax.
Whitespace acts to separate tokens but has no other meaning.
A hash sign
.RB ` # '
outside of a quoted string introduces a comment
which extends to the end of the line;
newlines otherwise just separate tokens, just like other whitespace.
There are no `reserved words',
but some names have special meanings,
depending on the context.
.PP
Integers are written in decimal.
(There is no provision for entering numbers in hex or octal.)
.IP
.I int
::=
.I digit
\&...
.br
.I digit
::=
.B 0
|
.B 1
|
.B 2
|
.B 3
|
.B 4
|
.B 5
|
.B 6
|
.B 7
|
.B 8
|
.B 9
.PP
Strings (mostly used for pathnames and suchlike)
are enclosed in double quotes
.RB ` """" ';
quotes and backslashes to be included in the string
must be escaped by preceding them with a backslash
.RB ` \e '.
.IP
.I string
::=
.B """"
.IR string-char ...\&
.B """"
.br
.I string-char
::=
any character other than
.B """"
or
.B \e
.br
\h'4m'|
.B "\e"""
|
.B \e\e
.
.SS "Top-level syntax"
At a high level,
the configuration consists of a sequence of
.IR "top-level items" .
.IP
.I config
::=
.I toplevel-item
\&...
.
.SS "Global settings"
Miscellaneous configuration for the whole program
goes in a top-level
.B vars
section.
.IP
.I toplevel-item
::=
.I vars-section
.br
.I vars-section
::=
.B vars
.B {
.IR var-setting
\&...\&
.B }
.PP
There may be multiple such sections.
The same variable may be set more than once;
if that happens,
only the last such setting has affect.
.IP
.I var-setting
::=
.B master
.B =
.I path
.br
.I path
::=
.I string
.PP
The
.B master
variable holds the pathname of the top of the master tree.
.PP
There are, at present, no other global settings.
.
.SS "Target definitions"
The other kind of top-level configuration item
defines a target directory
to be constructed or updated
by the
.BR gremlin .
.IP
.I toplevel-item
::=
.I target-def
.br
.I target-def
::=
.B target
.I path
.B {
.I type-clause
\&...\&
.B }
.br
.I type-clause
::=
.B type
.I type
.B {
.I policy
\&...\&
.B }
.PP
A
.B target
definition tells the
.B gremlin
to populate a directory tree,
named rooted at the given
.IR path .
The body of the target definition consists of
a sequence of
.B type
clauses
which explain what to do with different kinds of file.
The possible
.I type
tokens are as follows.
.TP
.B audio
Encoded audio files,
which can be decoded by the GStreamer library.
.TP
.B image
Image files,
which can be decoded by the Python Imaging Library.
.PP
The body of the type clause defines a
.I policy
for converting files of that type.
.
.SS "Policy descriptions"
There are two kinds of
.I primitive
policies,
which are described in full below:
.BR accept ,
which copies (or links) a master file
if its format is appropriate,
or does nothing;
and
.BR convert ,
which converts a master file into a chosen format,
and (in principle) should always succeed.
There are also two ways to build up
.I compound
policies from simpler ones.
.IP
.I policy
::=
.B and
.B {
.I policy
\&...\&
.B }
.br
\h'4m'|
.B or
.B {
.I policy
\&...\&
.B }
.PP
The
.B and
policy applies
.I all
of its operand policies,
potentially producing multiple output files.
.PP
The body of a
.I type-clause
consists of a sequence of policies
which are implicitly combined together in this way.
.PP
The
.B or
policy
tries its operand policies in turn,
in the order specified,
until one of them succeeds;
no more policies are tried after this.
.IP
.I policy
::=
.B accept
.I format-spec
.br
\h'4m'|
.B convert
.I format-spec
.IP
.I format-spec
::=
.I format-name
.br
\h'4m'|
.I format-name
.B {
.I format-prop
\&...\&
.B }
.PP
(The possible
.IR format-name s
and the corresponding
.IR format-spec s
are described in the section below.)
.PP
The
.B convert
policy converts a file to the specified format.
More specifically:
if the file's format already matches the
.I format-spec
then it is copied to the target directory.
(Indeed, if possible,
the file is hard linked into the target directory.)
If the file's format doesn't match,
then the
.B gremlin
converts it,
producing an output file of the requested format.
.PP
The
.B accept
policy copies or links a file if its format matches the
.IR format-spec ,
just as
.B convert
does.
However, if the file doesn't match then
.B accept
fails.
.PP
The usual use of
.B accept
is within an
.B or
block.
For example, suppose that the master tree mostly contains
losslessly encoded files, such as FLAC,
and we usually want to produce Ogg Vorbis
for use on devices with limited storage capacity;
but some of the master files are only available as MP3,
and re-encoding MP3 as Ogg Vorbis won't be good for sound quality.
Therefore, you can say something like
.IP
.nf
.ft B
or {
	accept mp3 { bitrate = 160 }
	convert ogg-vorbis { bitrate = 128 }
}
.fi
.ft P
.PP
which means:
if a master file is an MP3 file with bitrate approximately 160kb/s or less,
then copy it;
otherwise, convert the file to Ogg Vorbis, at about 128kb/s.
.PP
It's possible that even a simple policy
acting on the files in a master directory
will come up with multiple ways
to produce the same output file.
The rule used to decide is as follows:
if the
.B gremlin
can make the output file by copying one of the master files
then it does that;
otherwise it converts one of the inputs chosen arbitrarily.
For example,
suppose that a policy for
.B audio
files says
.IP
.B convert ogg-vorbis
.PP
and the master directory contains
.B foo.flac
and
.BR foo.ogg ;
then it will copy
.B foo.ogg
and ignore
.BR foo.flac .
If, instead, the master contains
.B foo.flac
and
.BR foo.mp3 ,
then one of these will be converted,
but it's hard to predict which.
.
.SS "Audio formats"
Two audio
.IR format-type s
are defined.
.PP
All audio formats support a
.B bitrate
property.
.IP
.I format-prop
::=
.B bitrate
.B =
.I int
.PP
The bitrate is expressed in kilobits per second.
For an existing file to match a
.I format-spec
containing a
.B bitrate
property,
the file's bitrate must be less than
the specified bitrate times a fudge factor
(currently sqrt(2)).
(The
.B bitrate
property is notionally the desired
.I output
bitrate;
the
.B gremlin
assumes that it's better to make output files a bit larger
than to re-encode an already lossily compressed master file.)
.PP
At present, the audio formats define no other properties.
.TP
.B mp3
The MP3 format that everyone knows and loves.
For encoding, the
.B gremlin
uses Lame,
and stores metadata in an ID3v2 tag;
it also tries to store an ID3v1.1 tag,
but this can fail for a number of reasons
(e.g., if the genre can't be represented,
or text contains characters outside of the ISO 8859-1 character set
used in ID3v1 tags).
.TP
.B ogg-vorbis
Vorbis-encoded audio in an Ogg container,
as defined by the Xiph.Org Foundation.
On encoding, the
.B bitrate
parameter is actually mapped to a quality setting
chosen to produce approximately the right bitrate.
.
.SS "Image formats"
Three image
.IR format-type s
are defined.
.PP
All image formats support a
.B size
property.
.IP
.I format-prop
::=
.B size
.B =
.I int
.PP
The size provides an upper bound on the width and height of the image.
A master file will only match if
both its width and height are
less than the stated size.
On output, the image will be scaled to the right size,
preserving its aspect ratio.
.TP
.B jpeg
The JFIF format, defined by the Joint Photographic Experts Group.
The following additional properties can be set;
they affect output only.
.RS
.TP
.B optimize
Spend longer to select optimal encoder settings.
.TP
.B progressive
Make a progressively-rendering output file.
This isn't usually a good idea.
.TP
.BI "quality = " int
Set the image quality (at the expense of file size).
This is a percentage; the default is 75.
.RE
.TP
.B png
The Portable Network Graphics format,
originally defined in RFC2083.
The following additional properties can be set;
they affect output only.
.RS
.TP
.B optimize
Spend longer to try to make the output file smaller.
.RE
.TP
.B bmp
The Windows BMP format.
There are no additional properties.
.
.SS "Example file"
The following is the author's configuration file.
I have an archive which mostly consists of FLAC files,
with a few MP3 files where I've been unable to obtain physical CDs.
I generate two output trees.
One mostly contains Ogg Vorbis files,
but tolerates occasional MP3
rather than suffer the quality loss of re-encoding.
It also generates small BMP-format images from cover art,
because I have an old portable audio player
which runs the free RockBox firmware,
whose player is only capable of displaying such images.
.IP
.nf
.ft B
### -*-conf-*-

vars {
	master = "/mnt/jb/master"
}

target "/mnt/jb/gremlin/ogg-vorbis-128" {
	type audio {
		or {
			accept mp3 { bitrate = 160 }
			convert ogg-vorbis { bitrate = 128 }
		}
	}
	type image {
		or {
			accept png
			convert jpeg { quality = 7 }
		}
		convert bmp { size = 75 }
	}
}

target "/mnt/jb/gremlin/mp3-160" {
	type audio {
		convert mp3 { bitrate = 160 }
	}
	type image {
		or {
			accept png
			convert jpeg { quality = 7 }
		}
	}
}
.fi
.ft P
.
.\"--------------------------------------------------------------------------
.SH BUGS
.
The
.B gremlin
makes no effort to process more than one file at a time.
.PP
It should probably support more audio formats.
They're quite easy to add,
but I don't have a good feel for which formats are good.
Patches and advice are welcome.
.PP
The
.B and
and
.B or
policy names are possibly confusing.
They suggest that they work like the standard logical operators;
while
.B or
sort of does, if you squint a bit,
.B and
certainly doesn't;
on the other hand, it does try to do all of the things you ask of it.
.PP
.B gremlin
is a very unhelpful name for the program.
.
.\"--------------------------------------------------------------------------
.SH AUTHOR
Mark Wooding, <mdw@distorted.org.uk>
.
.SH SEE ALSO
.BR hush (1),
.BR rsync (1).
.
.\"----- That's all, folks --------------------------------------------------