3 .\" Manual for the audio conversion gremlin
5 .\" (c) 2016 Mark Wooding
8 .\"----- Licensing notice ---------------------------------------------------
10 .\" This file is part of the `autoys' audio tools collection.
12 .\" `autoys' is free software; you can redistribute it and/or modify
13 .\" it under the terms of the GNU General Public License as published by
14 .\" the Free Software Foundation; either version 2 of the License, or
15 .\" (at your option) any later version.
17 .\" `autoys' is distributed in the hope that it will be useful,
18 .\" but WITHOUT ANY WARRANTY; without even the implied warranty of
19 .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
20 .\" GNU General Public License for more details.
22 .\" You should have received a copy of the GNU General Public License
23 .\" along with `autoys'; if not, write to the Free Software Foundation,
24 .\" Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
26 .TH gremlin 1 "13 February 2016" "Mark Wooding" "autoys"
28 .\"--------------------------------------------------------------------------
30 gremlin \- batch audio file converter
41 .\"--------------------------------------------------------------------------
46 program converts audio files
47 in an input `master' directory tree,
48 which presumably contains
49 high-quality (ideally lossless) encodings
51 writing corresponding converted files
52 to a collection of output directory trees.
53 It's non-interactive, idempotent, and restartable;
54 it never modifies its master tree.
55 It's exactly the sort of thing you want to
56 install as a daily cron job.
60 reads a configuration file
61 which describes the conversion policy for each of the output trees.
62 The policy can say things like:
63 copy MP3 files up to 160kb/s,
64 or Ogg Vorbis files up to 128kb/s;
65 and convert everything else to 128kb/s Ogg Vorbis.
69 can also convert image files, such as cover art.
71 Input files can be anything which
72 GStreamer and/or the Python Imaging Library can understand;
73 output files are more constrained,
76 has to be able to understand
77 their relevant properties.
78 The currently supported audio formats are
86 In a little more detail:
89 works through its input master tree,
90 one directory at a time.
91 For each master directory,
92 it tries to write a converted version
93 to a corresponding output directory
94 in each of the output trees.
95 For each file in the master directory,
96 it determines which files should be made
97 in each output directory:
99 and are not older than the master file,
100 then they're left alone on the assumption that they're up-to-date;
103 will make the output files by converting the master file.
105 Any other files or directories in the output directory
110 assumes that its output trees belong entirely to it,
111 to maintain according to its configuration,
112 and that unexpected files are either
113 debris left over from an earlier failure
114 or a result of a policy change,
115 and in either case the right thing to do is
116 to delete the offending files.
118 .SS "Command line syntax"
119 The following options are recognized.
122 Write a help message to standard output
125 command-line options,
126 and exit with status zero.
131 version number to standard output
132 and exit with status zero.
134 .B "\-i, \-\-interactive"
135 Write progress eyecandy to standard output while running.
136 While walking the master tree,
139 shows which directory it's currently examining.
140 While converting audio files,
141 it shows a progress meter showing
142 a bar chart of the job in progress,
143 the percentage of the job which is complete,
144 and an estimated time to completion.
145 (This last starts out rather inaccurate,
146 but seems to be pretty good after a couple of seconds.)
147 All this is done automatically if standard output is a terminal;
148 this option can be used to turn it on under other circumstances.
151 Don't actually modify the filesystem.
152 No files will be created or removed.
154 .BI "\-t, \-\-timeout=" timeout
158 Once the timeout has expired,
160 will try to finish what it's doing
161 and then exit with status zero.
163 (This might seem a surprising choice of exit status.
166 was asked to spend some amount of time converting files,
167 and it has done that successfully.)
169 .BI "\-T, \-\-timeout-nasty=" timeout
170 If the timeout set by the
172 option (above) has expired,
178 still hasn't managed to wrap things up,
179 then exit immediately with status 3,
180 possibly leaving files partially converted,
181 or other kinds of incompleteness.
184 will notice this wreckage and clean it up.)
186 .\"--------------------------------------------------------------------------
187 .SH CONFIGURATION FILE
192 configuration file has a simple token-oriented lexical syntax.
193 Whitespace acts to separate tokens but has no other meaning.
196 outside of a quoted string introduces a comment
197 which extends to the end of the line;
198 newlines otherwise just separate tokens, just like other whitespace.
199 There are no `reserved words',
200 but some names have special meanings,
201 depending on the context.
203 Integers are written in decimal.
204 (There is no provision for entering numbers in hex or octal.)
233 Strings (mostly used for pathnames and suchlike)
234 are enclosed in double quotes
236 quotes and backslashes to be included in the string
237 must be escaped by preceding them with a backslash
243 .IR string-char ...\&
248 any character other than
258 .SS "Top-level syntax"
260 the configuration consists of a sequence of
261 .IR "top-level items" .
268 .SS "Global settings"
269 Miscellaneous configuration for the whole program
286 There may be multiple such sections.
287 The same variable may be set more than once;
289 only the last such setting has affect.
303 variable holds the pathname of the top of the master tree.
305 There are, at present, no other global settings.
307 .SS "Target definitions"
308 The other kind of top-level configuration item
309 defines a target directory
310 to be constructed or updated
340 to populate a directory tree,
341 named rooted at the given
343 The body of the target definition consists of
347 which explain what to do with different kinds of file.
350 tokens are as follows.
354 which can be decoded by the GStreamer library.
358 which can be decoded by the Python Imaging Library.
360 The body of the type clause defines a
362 for converting files of that type.
364 .SS "Policy descriptions"
365 There are two kinds of
368 which are described in full below:
370 which copies (or links) a master file
371 if its format is appropriate,
375 which converts a master file into a chosen format,
376 and (in principle) should always succeed.
377 There are also two ways to build up
379 policies from simpler ones.
400 of its operand policies,
401 potentially producing multiple output files.
405 consists of a sequence of policies
406 which are implicitly combined together in this way.
411 tries its operand policies in turn,
412 in the order specified,
413 until one of them succeeds;
414 no more policies are tried after this.
438 and the corresponding
440 are described in the section below.)
444 policy converts a file to the specified format.
446 if the file's format already matches the
448 then it is copied to the target directory.
449 (Indeed, if possible,
450 the file is hard linked into the target directory.)
451 If the file's format doesn't match,
455 producing an output file of the requested format.
459 policy copies or links a file if its format matches the
464 However, if the file doesn't match then
473 For example, suppose that the master tree mostly contains
474 losslessly encoded files, such as FLAC,
475 and we usually want to produce Ogg Vorbis
476 for use on devices with limited storage capacity;
477 but some of the master files are only available as MP3,
478 and re-encoding MP3 as Ogg Vorbis won't be good for sound quality.
479 Therefore, you can say something like
484 accept mp3 { bitrate = 160 }
485 convert ogg-vorbis { bitrate = 128 }
491 if a master file is an MP3 file with bitrate approximately 160kb/s or less,
493 otherwise, convert the file to Ogg Vorbis, at about 128kb/s.
495 It's possible that even a simple policy
496 acting on the files in a master directory
497 will come up with multiple ways
498 to produce the same output file.
499 The rule used to decide is as follows:
502 can make the output file by copying one of the master files
504 otherwise it converts one of the inputs chosen arbitrarily.
506 suppose that a policy for
510 .B convert ogg-vorbis
512 and the master directory contains
520 If, instead, the master contains
524 then one of these will be converted,
525 but it's hard to predict which.
532 All audio formats support a
542 The bitrate is expressed in kilobits per second.
543 For an existing file to match a
548 the file's bitrate must be less than
549 the specified bitrate times a fudge factor
553 property is notionally the desired
558 assumes that it's better to make output files a bit larger
559 than to re-encode an already lossily compressed master file.)
561 At present, the audio formats define no other properties.
564 The MP3 format that everyone knows and loves.
568 and stores metadata in an ID3v2 tag;
569 it also tries to store an ID3v1.1 tag,
570 but this can fail for a number of reasons
571 (e.g., if the genre can't be represented,
572 or text contains characters outside of the ISO 8859-1 character set
576 Vorbis-encoded audio in an Ogg container,
577 as defined by the Xiph.Org Foundation.
580 parameter is actually mapped to a quality setting
581 chosen to produce approximately the right bitrate.
588 All image formats support a
598 The size provides an upper bound on the width and height of the image.
599 A master file will only match if
600 both its width and height are
601 less than the stated size.
602 On output, the image will be scaled to the right size,
603 preserving its aspect ratio.
606 The JFIF format, defined by the Joint Photographic Experts Group.
607 The following additional properties can be set;
608 they affect output only.
612 Spend longer to select optimal encoder settings.
615 Make a progressively-rendering output file.
616 This isn't usually a good idea.
619 Set the image quality (at the expense of file size).
620 This is a percentage; the default is 75.
624 The Portable Network Graphics format,
625 originally defined in RFC2083.
626 The following additional properties can be set;
627 they affect output only.
631 Spend longer to try to make the output file smaller.
635 The Windows BMP format.
636 There are no additional properties.
639 The following is the author's configuration file.
640 I have an archive which mostly consists of FLAC files,
641 with a few MP3 files where I've been unable to obtain physical CDs.
642 I generate two output trees.
643 One mostly contains Ogg Vorbis files,
644 but tolerates occasional MP3
645 rather than suffer the quality loss of re-encoding.
646 It also generates small BMP-format images from cover art,
647 because I have an old portable audio player
648 which runs the free RockBox firmware,
649 whose player is only capable of displaying such images.
656 master = "/mnt/jb/master"
659 target "/mnt/jb/gremlin/ogg-vorbis-128" {
662 accept mp3 { bitrate = 160 }
663 convert ogg-vorbis { bitrate = 128 }
669 convert jpeg { quality = 7 }
671 convert bmp { size = 75 }
675 target "/mnt/jb/gremlin/mp3-160" {
677 convert mp3 { bitrate = 160 }
682 convert jpeg { quality = 7 }
689 .\"--------------------------------------------------------------------------
694 makes no effort to process more than one file at a time.
696 It should probably support more audio formats.
697 They're quite easy to add,
698 but I don't have a good feel for which formats are good.
699 Patches and advice are welcome.
705 policy names are possibly confusing.
706 They suggest that they work like the standard logical operators;
709 sort of does, if you squint a bit,
712 on the other hand, it does try to do all of the things you ask of it.
715 is a very unhelpful name for the program.
717 .\"--------------------------------------------------------------------------
719 Mark Wooding, <mdw@distorted.org.uk>
725 .\"----- That's all, folks --------------------------------------------------
~mdw / autoys / blob