Page MenuHomePhabricator
Paste P17472

(An Untitled Masterwork)
ActivePublic

Authored by jbond on Wed, Oct 13, 2:24 PM.
$ time /usr/bin/debmonitor-client-unpriv --debug
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='debmonitor.discovery.wmnet', port=443): Read timed out. (read timeout=3)")': /hosts/sretest1001.eqiad.wmnet/update
INFO:debmonitor:Successfully sent the full update to the DebMonitor server
real 2m4.537s
user 0m1.113s
sys 0m0.069s

Event Timeline

from the Apache logs we see a http 500

2021-10-13T14:27:52     72350   10.64.48.138    proxy-server/500        128     POST    http://debmonitor.discovery.wmnet/hosts/sretest1001.eqiad.wmnet/update -       text/plain      -       -       debmonitor-client/0.3client1 +https://wikitech.wikimedia.org/wiki/Debmonitor   -       -       -       -       10.64.48.138    -     10.64.48.138

debmon logs show

[2021-10-13T14:27:52] Unable to update host 'sretest1001.eqiad.wmnet'
[2021-10-13T14:27:52] Internal Server Error: /hosts/sretest1001.eqiad.wmnet/update

then after that we get s low stream of successful posts

2021-10-13T14:27:49     13954115        10.64.48.138    proxy-server/201        0       POST    http://debmonitor.discovery.wmnet/hosts/sretest1001.eqiad.wmnet/update -       text/plain      -       -       debmonitor-client/0.3client1 +https://wikitech.wikimedia.org/wiki/Debmonitor   -       -       -       -       10.64.48.138  -10.64.48.138
2021-10-13T14:29:14     138813  10.64.48.138    proxy-server/201        0       POST    http://debmonitor.discovery.wmnet/hosts/sretest1001.eqiad.wmnet/update -       text/plain      -       -       debmonitor-client/0.3client1 +https://wikitech.wikimedia.org/wiki/Debmonitor   -       -       -       -       10.64.48.138    -     10.64.48.138
2021-10-13T14:29:52     145809  10.64.48.138    proxy-server/201        0       POST    http://debmonitor.discovery.wmnet/hosts/sretest1001.eqiad.wmnet/update -       text/plain      -       -       debmonitor-client/0.3client1 +https://wikitech.wikimedia.org/wiki/Debmonitor   -       -       -       -       10.64.48.138    -     10.64.48.138

debug logging

su: warning: cannot change directory to /nonexistent: No such file or directory
INFO:debmonitor:Found 536 installed binary packages
INFO:debmonitor:Found 0 upgradable binary packages (including new dependencies)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): debmonitor.discovery.wmnet:443
DEBUG:urllib3.util.retry:Incremented Retry for (url='/hosts/sretest1001.eqiad.wmnet/update'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='debmonitor.discovery.wmnet', port=443): Read timed out. (read timeout=3)")': /hosts/sretest1001.eqiad.wmnet/update
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): debmonitor.discovery.wmnet:443
DEBUG:urllib3.connectionpool:https://debmonitor.discovery.wmnet:443 "POST /hosts/sretest1001.eqiad.wmnet/update HTTP/1.1" 500 128
DEBUG:urllib3.util.retry:Incremented Retry for (url='/hosts/sretest1001.eqiad.wmnet/update'): Retry(total=1, connect=None, read=None, redirect=None, status=None)


DEBUG:urllib3.connectionpool:Retry: /hosts/sretest1001.eqiad.wmnet/update
DEBUG:urllib3.connectionpool:Resetting dropped connection: debmonitor.discovery.wmnet
DEBUG:urllib3.connectionpool:https://debmonitor.discovery.wmnet:443 "POST /hosts/sretest1001.eqiad.wmnet/update HTTP/1.1" 201 0
INFO:debmonitor:Successfully sent the full update to the DebMonitor server