Ian Jackson [Tue, 22 Aug 2017 16:53:56 +0000 (17:53 +0100)]
ResponseConsumer: tolerate daft exception PotentialDataLoss
The docs for this say:
PotentialDataLoss may be raised by a transfer encoding decoder's
noMoreData method to indicate that it cannot be determined if the
entire response body has been delivered. This only occurs when
making requests to HTTP servers which do not set Content-Length or a
Transfer-Encoding in the response because in this case the end of
the response is indicated by the connection being closed, an event
which may also be due to a transient network problem or other error.
[1] https://twistedmatrix.com/documents/16.6.0/api/twisted.web.http.PotentialDataLoss.html retrieved 22.8.2017
If the origin server (or proxy) chooses to use `Connection: close' and
not to provide a Content-Length, then this exception will always be
raised. hippotatd uses Twisted's http server so does not trigger
this, but there might be an (intercepting) proxy. If so, then
hippotat will simply not work.
Additionally, the statement in the Twisted documentation that
connection close might be due to a "transient problem or other error"
is not correct. TCP is able to distinguish a graceful shutdown from
an ungraceful close, and origin servers ought to arrange that they do
not inappropriately perform a graceful close in error situations.
Of course a badly-implemented proxy might get this wrong, but in that
case a general web client would have no way to detect this situation
if it wants to function at all. So certainly turning this situation
into an exception by default seems wrong.
For hippotat, ignoring the potential problem hippotat just means
potentially processing a truncated response, which might mean lost
packets (which should be retransmitted) or truncated packets (which
will be detected by the IP header length field, and retransmitted).
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 22 Aug 2017 16:37:06 +0000 (17:37 +0100)]
ResponseConsumer: break out connectionLostOK into superclass
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 22 Aug 2017 16:32:00 +0000 (17:32 +0100)]
ResponseConsumer: move _resp initialisation to superclass constructor (nfc)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 22 Aug 2017 16:31:46 +0000 (17:31 +0100)]
ResponseConsumer: pass resp to superclass constructor (nfc)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 22 Aug 2017 16:31:08 +0000 (17:31 +0100)]
ResponseConsumer: pass resp to constructor (nfc)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 22 Aug 2017 16:24:48 +0000 (17:24 +0100)]
do not crash on unexpected connectionLost
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 25 Apr 2017 14:34:13 +0000 (15:34 +0100)]
ensure mtu is in the ipif substitution set
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 25 Apr 2017 14:33:33 +0000 (15:33 +0100)]
report success after failure
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 25 Apr 2017 14:33:19 +0000 (15:33 +0100)]
replace plaintext secret transmission with time-limited hmac-based bearer tokens
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 25 Apr 2017 13:50:01 +0000 (14:50 +0100)]
rename "password" to "secret" everywhere
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 25 Apr 2017 13:48:57 +0000 (14:48 +0100)]
document that nonce-based authentication is for the future
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 25 Apr 2017 13:48:38 +0000 (14:48 +0100)]
better document routing assistance
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 13:31:48 +0000 (14:31 +0100)]
hippotatd: do not crash on some unexpected requests
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 12:26:00 +0000 (13:26 +0100)]
process putatives: Introduce dbg()
No functional change.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 12:23:18 +0000 (13:23 +0100)]
process putatives: Properly report regexp mismatch ignores
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 12:23:03 +0000 (13:23 +0100)]
process putatives: Fix processing of [<servername> <client>] sections
Convert only the client part to an ip address.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 12:22:36 +0000 (13:22 +0100)]
process putatives: Add many log_debug_config calls
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 12:21:50 +0000 (13:21 +0100)]
process putatives: break out log_ignore
No significant functional change.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 12:20:51 +0000 (13:20 +0100)]
do actually process server-specific [<server> LIMIT] sections
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 12:20:29 +0000 (13:20 +0100)]
better message from client on http errors
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 00:36:35 +0000 (01:36 +0100)]
daemon: record the deferred for a request in the queue, too
So we can check nf.called, rather than request.finished. The latter
can be false even after the request has failed and is unfinishable.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 24 Apr 2017 00:00:29 +0000 (01:00 +0100)]
config: Abolish use of ConfigParser's DEFAULT feature
ConfigParser provides no way to find out which sections a key is
_really_ in: in all of its views, keys in DEFAULT appear aliased in
all sections.
So instead we use a COMMON section, and implement the defaulting
ourselves. This involves:
* Changing the cfg.get[int] calls to use cfg_search (these are
the things which implictly use DEFAULT and now need to be
explicit).
* Changing all our explicit references to and definitions of
DEFAULT options to COMMON.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 23 Apr 2017 23:53:00 +0000 (00:53 +0100)]
config: replace open-coded cfg.get[int]
We are going to want to do something more sophisticated.
Centralise things. No functional change.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 23 Apr 2017 23:50:13 +0000 (00:50 +0100)]
move cfg_get_raw into hippotatlib
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 23 Apr 2017 23:42:56 +0000 (00:42 +0100)]
rename cfg_process_common to cfg_process_general
No functional change.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 23 Apr 2017 23:40:29 +0000 (00:40 +0100)]
config reading: log correct filenames
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 23 Apr 2017 19:51:21 +0000 (20:51 +0100)]
hippotatd: catch errors from request.finish in _req_cancel
It is not completely clear when it is necessary to call
request.finish. Doing so when it is not necessary has a tendency to
throw an exception.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 23 Apr 2017 19:51:14 +0000 (20:51 +0100)]
logging: if logevent_is_boringtwisted fails, print why when we print the exception
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 23 Apr 2017 19:51:06 +0000 (20:51 +0100)]
syslog handling: print tracebacks when logging failure objects
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Fri, 21 Apr 2017 21:20:07 +0000 (22:20 +0100)]
fix crash on discarding packets
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Fri, 21 Apr 2017 21:19:22 +0000 (22:19 +0100)]
fix typo
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Fri, 21 Apr 2017 19:56:47 +0000 (20:56 +0100)]
ipif: set interface names
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Fri, 21 Apr 2017 19:55:45 +0000 (20:55 +0100)]
debian: Recommend userv-utils, now that it has sane ipif
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 01:24:05 +0000 (02:24 +0100)]
config: fix client config (no SERVER)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 01:23:56 +0000 (02:23 +0100)]
config: a bit more debugging
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:47:06 +0000 (01:47 +0100)]
cope with jesse's logger, which lacks --id=VALUE
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:46:54 +0000 (01:46 +0100)]
fix pidfile handling
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:46:29 +0000 (01:46 +0100)]
suppress boring twisted messages to syslog, too
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:46:13 +0000 (01:46 +0100)]
fix exit status
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:46:07 +0000 (01:46 +0100)]
fix cfg_process_saddrs ServerAddr
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:45:34 +0000 (01:45 +0100)]
logevent_is_boringtwisted: break out as a function from LogNotBoringTwisted (nfc)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:45:03 +0000 (01:45 +0100)]
cope with jesse's signal module (which lacks sig.name)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Mon, 10 Apr 2017 00:44:46 +0000 (01:44 +0100)]
init script fixes
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 22:44:24 +0000 (23:44 +0100)]
config dir reading fix
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 22:44:17 +0000 (23:44 +0100)]
packaging tidy
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 22:43:56 +0000 (23:43 +0100)]
packaging fixes
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 18:27:30 +0000 (19:27 +0100)]
packaging fixes - really now it installs on xenophobe
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 17:52:35 +0000 (18:52 +0100)]
packaging fixes - now it installs on xenophobe
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 15:13:27 +0000 (16:13 +0100)]
config: better examples/docs (update to how things are now)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 15:13:03 +0000 (16:13 +0100)]
config: sort out directory handling
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 15:12:23 +0000 (16:12 +0100)]
config: Some new options
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 15:11:12 +0000 (16:11 +0100)]
Revert "config: Check args after reading config"
This is total nonsense. We want to parse the arguments first.
This reverts commit
e5f6fff02ac195add8cf14ad4a597d58c6613a9b.
Ian Jackson [Sun, 9 Apr 2017 15:10:29 +0000 (16:10 +0100)]
config: Check args after reading config
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 15:10:04 +0000 (16:10 +0100)]
config: Break out read_config (and change default config files a bit)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 15:09:13 +0000 (16:09 +0100)]
rules: introduce $i
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 14:49:14 +0000 (15:49 +0100)]
Fix up sys.path in scripts
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 14:11:40 +0000 (15:11 +0100)]
control: demote authbind to Suggests
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 12:03:22 +0000 (13:03 +0100)]
initscript: check firewall
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 12:02:57 +0000 (13:02 +0100)]
initscript: use authbind
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 12:01:58 +0000 (13:01 +0100)]
initscript: lift out DESCRIPTION
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 12:01:17 +0000 (13:01 +0100)]
hippotatd: --pidfile improvement
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 12:01:08 +0000 (13:01 +0100)]
hippotatd: support --print-config
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 10:13:11 +0000 (11:13 +0100)]
init script fixes
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 01:31:40 +0000 (02:31 +0100)]
start daemon in init script
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sun, 9 Apr 2017 01:02:31 +0000 (02:02 +0100)]
hippotatd: support --pidfile
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 23:01:09 +0000 (00:01 +0100)]
hippotatd: support daemonisation
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 22:59:17 +0000 (23:59 +0100)]
hippotatd: support syslog
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 18:53:57 +0000 (19:53 +0100)]
packaging: create the user
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 18:44:05 +0000 (19:44 +0100)]
packaging: install more stuff
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 18:30:57 +0000 (19:30 +0100)]
packaging: create debian/copyright
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 18:28:16 +0000 (19:28 +0100)]
packaging: improve description
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 18:27:51 +0000 (19:27 +0100)]
packaging: remove evidently-uneeded file
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 18:21:10 +0000 (19:21 +0100)]
get dependencies right
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Sat, 8 Apr 2017 18:04:14 +0000 (19:04 +0100)]
wip packaging
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Wed, 5 Apr 2017 13:01:45 +0000 (14:01 +0100)]
hippotatd: --ownsource-local etc. options
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Wed, 5 Apr 2017 13:01:15 +0000 (14:01 +0100)]
ownsource: do not crash if not s.download_packages
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Wed, 5 Apr 2017 13:01:02 +0000 (14:01 +0100)]
pass opts to process_cfg
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Wed, 5 Apr 2017 12:45:02 +0000 (13:45 +0100)]
ownsource download names are internal names
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Wed, 5 Apr 2017 12:41:29 +0000 (13:41 +0100)]
rename fullsrcbomb.tar (internal filename only for now)
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Wed, 5 Apr 2017 12:31:45 +0000 (13:31 +0100)]
ownsource: process .gitignore properly
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Wed, 5 Apr 2017 00:13:15 +0000 (01:13 +0100)]
ownsrc debugging
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 23:50:34 +0000 (00:50 +0100)]
licence: make client GPLv3+, not AGPLv3+exception
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 23:41:03 +0000 (00:41 +0100)]
ownsrc fixes
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 23:37:43 +0000 (00:37 +0100)]
ownsource: do not include files more than once in srcbomb
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 23:33:26 +0000 (00:33 +0100)]
wip new packages download
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 18:29:25 +0000 (19:29 +0100)]
ownsource: wip packages
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 17:25:12 +0000 (18:25 +0100)]
hippotatd: add a startup message
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 17:17:00 +0000 (18:17 +0100)]
copyright notices
Ian Jackson [Tue, 4 Apr 2017 17:03:33 +0000 (18:03 +0100)]
DCO
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 17:00:21 +0000 (18:00 +0100)]
AGPLv3+CAFv2
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Ian Jackson [Tue, 4 Apr 2017 16:51:57 +0000 (17:51 +0100)]
COPYING (AGPL-3)
Ian Jackson [Tue, 4 Apr 2017 16:51:02 +0000 (17:51 +0100)]
source download plumbed in
Ian Jackson [Tue, 4 Apr 2017 16:46:47 +0000 (17:46 +0100)]
ownsource: tie in (part 1) and cleanup stuff
Ian Jackson [Tue, 4 Apr 2017 16:27:43 +0000 (17:27 +0100)]
ownsource: logging etc.
Ian Jackson [Tue, 4 Apr 2017 16:15:28 +0000 (17:15 +0100)]
server: new SourceResource (skeleton)
Ian Jackson [Tue, 4 Apr 2017 16:15:07 +0000 (17:15 +0100)]
server: new NotStupidResource
Ian Jackson [Tue, 4 Apr 2017 15:50:40 +0000 (16:50 +0100)]
ownsource: try to honour .gitignore
Ian Jackson [Tue, 4 Apr 2017 15:43:01 +0000 (16:43 +0100)]
ownsource fixes
Ian Jackson [Tue, 4 Apr 2017 14:53:01 +0000 (15:53 +0100)]
ownsource does dpkg stuff
Ian Jackson [Tue, 4 Apr 2017 14:17:18 +0000 (15:17 +0100)]
ownsrc wip package handling etc.
~mdw / hippotat / log