Page MenuHomePhabricator
Create Task
Maniphest T221035

scap no longer !log'ging to server admin log
Closed, ResolvedPublic
Actions

Description

Multiple users have reported that scap is not performing !log actions to record deploys in the server admin log, since apparently the past weekend.

Last seen SCAP action logged:

2019-04-13 00:32:44	+logmsgbot	!log krinkle@deploy1001 Synchronized php-1.33.0-wmf.25/includes/: Idc19cc29764a / T220854 - hot fix (duration: 05m 37s)

Related Objects

Event Timeline

CDanis created this task.Apr 15 2019, 7:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 15 2019, 7:54 PM
Marostegui added projects: Scap, Release-Engineering-Team (Backlog).Apr 15 2019, 7:56 PM
Peachey88 updated the task description. (Show Details)Apr 15 2019, 8:17 PM
bd808 subscribed.Apr 15 2019, 11:23 PM
[23:18]  <    bd808>	logmsgbot isn't here?
[23:19]  <    bd808>	it should be running from icinga.wikimedia.org I think based on a quick grep of ops/puppet
[23:19]  <   cdanis>	okay
[23:19]  <   cdanis>	it thinks it is running there
[23:19]  <   cdanis>	and it even has your recent deploy message in its logging output
[23:20]  <   cdanis>	I'm going to restart it and see what happens I guess
[23:20]  <   cdanis>	!log cdanis@icinga1001.wikimedia.org ~ % sudo systemctl restart tcpircbot-logmsgbot.service 
[23:20]  < stashbot>	Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:20] 	logmsgbot (~logmsgbot@wikimedia/bot/logmsgbot) joined the channel
[23:20] ChanServ sets mode +v logmsgbot
[23:21]  <    bd808>	cdanis: that join looks promising :)
CDanis closed this task as Resolved.Apr 15 2019, 11:33 PM
CDanis claimed this task.

looks like logmsgbot was happily chattering away in #wikimedia-overload because of some race condition (within IRC services?) reconnecting to freenode around Apr 13 17:06.

P8402 Masterwork From Distant Lands
1-- Logs begin at Fri 2019-04-12 17:55:29 UTC, end at Mon 2019-04-15 23:22:15 UTC. --
2Apr 12 18:59:13 icinga1001 python[213132]: 2019-04-12 18:59:13,554 Connection from ('::ffff:10.64.32.25', 37716, 0, 0)
3Apr 12 18:59:13 icinga1001 python[213132]: 2019-04-12 18:59:13,554 TCP ('::ffff:10.64.32.25', 37716, 0, 0): "!log dzahn@cumin1001 START - Cookbook sre.hosts.decommission"
4Apr 12 18:59:19 icinga1001 python[213132]: 2019-04-12 18:59:19,540 Connection from ('::ffff:10.64.32.25', 37736, 0, 0)
5Apr 12 18:59:19 icinga1001 python[213132]: 2019-04-12 18:59:19,541 TCP ('::ffff:10.64.32.25', 37736, 0, 0): "!log dzahn@cumin1001 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)"
6Apr 12 19:24:33 icinga1001 python[213132]: 2019-04-12 19:24:33,577 Connection from ('::ffff:10.64.32.25', 37770, 0, 0)
7Apr 12 19:24:33 icinga1001 python[213132]: 2019-04-12 19:24:33,577 TCP ('::ffff:10.64.32.25', 37770, 0, 0): "!log dzahn@cumin1001 START - Cookbook sre.hosts.decommission"
8Apr 12 19:24:33 icinga1001 python[213132]: 2019-04-12 19:24:33,738 Connection from ('::ffff:10.64.32.25', 37776, 0, 0)
9Apr 12 19:24:33 icinga1001 python[213132]: 2019-04-12 19:24:33,738 TCP ('::ffff:10.64.32.25', 37776, 0, 0): "!log dzahn@cumin1001 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)"
10Apr 12 19:27:49 icinga1001 python[213132]: 2019-04-12 19:27:49,510 Connection from ('::ffff:10.64.32.25', 37778, 0, 0)
11Apr 12 19:27:49 icinga1001 python[213132]: 2019-04-12 19:27:49,510 TCP ('::ffff:10.64.32.25', 37778, 0, 0): "!log dzahn@cumin1001 START - Cookbook sre.hosts.decommission"
12Apr 12 19:27:49 icinga1001 python[213132]: 2019-04-12 19:27:49,666 Connection from ('::ffff:10.64.32.25', 37784, 0, 0)
13Apr 12 19:27:49 icinga1001 python[213132]: 2019-04-12 19:27:49,666 TCP ('::ffff:10.64.32.25', 37784, 0, 0): "!log dzahn@cumin1001 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)"
14Apr 12 21:10:51 icinga1001 python[213132]: 2019-04-12 21:10:51,647 Connection from ('::ffff:10.64.32.16', 56566, 0, 0)
15Apr 12 21:10:51 icinga1001 python[213132]: 2019-04-12 21:10:51,647 TCP ('::ffff:10.64.32.16', 56566, 0, 0): "!log krinkle@deploy1001 Synchronized php-1.33.0-wmf.25/extensions/ImageMap/includes/ImageMap.php: I0ee84f059da / T217087 (duration: 05m 12s)"
16Apr 13 00:32:43 icinga1001 python[213132]: 2019-04-13 00:32:43,945 Connection from ('::ffff:10.64.32.16', 60914, 0, 0)
17Apr 13 00:32:43 icinga1001 python[213132]: 2019-04-13 00:32:43,946 TCP ('::ffff:10.64.32.16', 60914, 0, 0): "!log krinkle@deploy1001 Synchronized php-1.33.0-wmf.25/includes/: Idc19cc29764a / T220854 - hot fix (duration: 05m 37s)"
18Apr 13 17:05:12 icinga1001 python[213132]: 2019-04-13 17:05:12,165 emerson!~emerson@freenode/staff/emerson [u'[Server Notice] This server is shutting down because of network issues, please reconnect to chat.freenode.net']
19Apr 13 17:05:52 icinga1001 python[213132]: 2019-04-13 17:05:52,219 cherryh.freenode.net [u'Server Terminating. Received SIGTERM']
20Apr 13 17:05:53 icinga1001 python[213132]: Traceback (most recent call last):
21Apr 13 17:05:53 icinga1001 python[213132]: File "tcpircbot.py", line 147, in <module>
22Apr 13 17:05:53 icinga1001 python[213132]: readable, _, _ = select.select([bot.connection.socket] + files, [], [])
23Apr 13 17:05:53 icinga1001 python[213132]: TypeError: argument must be an int, or have a fileno() method.
24Apr 13 17:05:53 icinga1001 systemd[1]: tcpircbot-logmsgbot.service: Main process exited, code=exited, status=1/FAILURE
25Apr 13 17:05:53 icinga1001 systemd[1]: tcpircbot-logmsgbot.service: Unit entered failed state.
26Apr 13 17:05:53 icinga1001 systemd[1]: tcpircbot-logmsgbot.service: Failed with result 'exit-code'.
27Apr 13 17:05:58 icinga1001 systemd[1]: tcpircbot-logmsgbot.service: Service hold-off time over, scheduling restart.
28Apr 13 17:05:58 icinga1001 systemd[1]: Stopped TCP socket to IRC bot: tcpircbot-logmsgbot.
29Apr 13 17:05:58 icinga1001 systemd[1]: Started TCP socket to IRC bot: tcpircbot-logmsgbot.
30Apr 13 17:05:58 icinga1001 python[100554]: 2019-04-13 17:05:58,795 niven.freenode.net [u'*** Looking up your hostname...']
31Apr 13 17:05:58 icinga1001 python[100554]: 2019-04-13 17:05:58,796 niven.freenode.net [u'*** Checking Ident']
32Apr 13 17:05:58 icinga1001 python[100554]: 2019-04-13 17:05:58,796 niven.freenode.net [u'*** Found your hostname']
33Apr 13 17:06:06 icinga1001 python[100554]: 2019-04-13 17:06:06,561 niven.freenode.net [u'*** No Ident response']
34Apr 13 17:06:06 icinga1001 python[100554]: 2019-04-13 17:06:06,561 welcome [niven.freenode.net -> logmsgbot]
35Apr 13 17:06:18 icinga1001 python[100554]: 2019-04-13 17:06:18,523 NickServ!NickServ@services. [u'This nickname is registered. Please choose a different nickname, or identify via \x02/msg NickServ identify <password>\x02.']
36Apr 13 17:06:18 icinga1001 python[100554]: 2019-04-13 17:06:18,599 NickServ!NickServ@services. [u'You are now identified for \x02logmsgbot\x02.']
37Apr 13 17:06:30 icinga1001 python[100554]: 2019-04-13 17:06:30,474 ChanServ!ChanServ@services. [u'[#wikimedia-overflow] PM ops in here if you want to be invited to the channel you were trying to join.']
38Apr 13 17:08:57 icinga1001 python[100554]: 2019-04-13 17:08:57,712 freenode-connect!frigg@freenode/utility-bot/frigg [u'Welcome to freenode. To protect the network all new connections will be scanned for vulnerabilities. This will not harm your computer, and vulnerable hosts will be notified.']
39Apr 13 20:44:31 icinga1001 python[100554]: 2019-04-13 20:44:31,755 Connection from ('::ffff:10.64.32.16', 50938, 0, 0)
40Apr 13 20:44:31 icinga1001 python[100554]: 2019-04-13 20:44:31,756 TCP ('::ffff:10.64.32.16', 50938, 0, 0): "!log gilles@deploy1001 Started deploy [performance/navtiming@c2b2d14]: (no justification provided)"
41Apr 13 20:44:36 icinga1001 python[100554]: 2019-04-13 20:44:36,852 Connection from ('::ffff:10.64.32.16', 50956, 0, 0)
42Apr 13 20:44:36 icinga1001 python[100554]: 2019-04-13 20:44:36,852 TCP ('::ffff:10.64.32.16', 50956, 0, 0): "!log gilles@deploy1001 Finished deploy [performance/navtiming@c2b2d14]: (no justification provided) (duration: 00m 05s)"
43Apr 14 05:02:06 icinga1001 python[100554]: 2019-04-14 05:02:06,014 Connection from ('::ffff:10.64.32.16', 59730, 0, 0)
44Apr 14 05:02:06 icinga1001 python[100554]: 2019-04-14 05:02:06,015 TCP ('::ffff:10.64.32.16', 59730, 0, 0): "!log ebernhardson@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Split expensive elasticsearch queries to codfw due to individual node overloads (duration: 05m 17s)"
45Apr 14 05:58:00 icinga1001 python[100554]: 2019-04-14 05:58:00,664 Connection from ('::ffff:10.64.32.16', 33600, 0, 0)
46Apr 14 05:58:00 icinga1001 python[100554]: 2019-04-14 05:58:00,665 TCP ('::ffff:10.64.32.16', 33600, 0, 0): "!log ebernhardson@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Switch wikidata and enwiki search traffic to codfw, excluding comp_suggest and prefix (duration: 05m 15s)"
47Apr 14 06:05:12 icinga1001 python[100554]: 2019-04-14 06:05:12,775 Connection from ('::ffff:10.64.32.16', 34958, 0, 0)
48Apr 14 06:05:12 icinga1001 python[100554]: 2019-04-14 06:05:12,775 TCP ('::ffff:10.64.32.16', 34958, 0, 0): "!log ebernhardson@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Switch wikidata elasticsearch back to eqiad to get a more even split between eqiad and codfw (duration: 05m 16s)"
49Apr 15 05:04:59 icinga1001 python[100554]: 2019-04-15 05:04:59,113 Connection from ('::ffff:10.64.32.16', 56458, 0, 0)
50Apr 15 05:04:59 icinga1001 python[100554]: 2019-04-15 05:04:59,113 TCP ('::ffff:10.64.32.16', 56458, 0, 0): "!log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool pc1007 (duration: 04m 55s)"
51Apr 15 05:31:25 icinga1001 python[100554]: 2019-04-15 05:31:25,458 Connection from ('::ffff:10.64.32.16', 57986, 0, 0)
52Apr 15 05:31:25 icinga1001 python[100554]: 2019-04-15 05:31:25,458 TCP ('::ffff:10.64.32.16', 57986, 0, 0): "!log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 04m 48s)"
53Apr 15 05:53:08 icinga1001 python[100554]: 2019-04-15 05:53:08,084 Connection from ('::ffff:10.64.32.16', 59542, 0, 0)
54Apr 15 05:53:08 icinga1001 python[100554]: 2019-04-15 05:53:08,084 TCP ('::ffff:10.64.32.16', 59542, 0, 0): "!log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Slowly repool db1100 (duration: 05m 11s)"
55Apr 15 06:14:56 icinga1001 python[100554]: 2019-04-15 06:14:56,752 Connection from ('::ffff:10.64.32.16', 32886, 0, 0)
56Apr 15 06:14:56 icinga1001 python[100554]: 2019-04-15 06:14:56,752 TCP ('::ffff:10.64.32.16', 32886, 0, 0): "!log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Fully repool db1100 (duration: 05m 11s)"
57Apr 15 10:48:58 icinga1001 python[100554]: 2019-04-15 10:48:58,249 Connection from ('::ffff:10.64.32.16', 38312, 0, 0)
58Apr 15 10:48:58 icinga1001 python[100554]: 2019-04-15 10:48:58,249 TCP ('::ffff:10.64.32.16', 38312, 0, 0): "!log jdrewniak@deploy1001 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:503957| Bumping portals to master (T128546)]] (duration: 05m 13s)"
59Apr 15 11:14:21 icinga1001 python[100554]: 2019-04-15 11:14:21,751 Connection from ('::ffff:10.64.32.16', 39956, 0, 0)
60Apr 15 11:14:21 icinga1001 python[100554]: 2019-04-15 11:14:21,752 TCP ('::ffff:10.64.32.16', 39956, 0, 0): "!log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:492942|Add enwiki to azwiki import source (T217104)]] (duration: 05m 11s)"
61Apr 15 11:26:06 icinga1001 python[100554]: 2019-04-15 11:26:06,335 Connection from ('::ffff:10.64.32.16', 41426, 0, 0)
62Apr 15 11:26:06 icinga1001 python[100554]: 2019-04-15 11:26:06,335 TCP ('::ffff:10.64.32.16', 41426, 0, 0): "!log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:503753|Add botadmin group on itwiki (T220915)]] (duration: 05m 09s)"
63Apr 15 11:33:45 icinga1001 python[100554]: 2019-04-15 11:33:45,059 Connection from ('::ffff:10.64.32.16', 42786, 0, 0)
64Apr 15 11:33:45 icinga1001 python[100554]: 2019-04-15 11:33:45,059 TCP ('::ffff:10.64.32.16', 42786, 0, 0): "!log ladsgroup@deploy1001 Synchronized wmf-config/InterwikiSortOrders.php: SWAT: [[gerrit:503270|Add Western Armenian Wikipedia to wmf-config/InterwikiSortOrders.php (T219871)]] (duration: 05m 10s)"
65Apr 15 13:35:28 icinga1001 python[100554]: 2019-04-15 13:35:28,696 Connection from ('::ffff:10.64.32.16', 45986, 0, 0)
66Apr 15 13:35:28 icinga1001 python[100554]: 2019-04-15 13:35:28,696 TCP ('::ffff:10.64.32.16', 45986, 0, 0): "!log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Test deploy with no changes (duration: 00m 52s)"
67Apr 15 13:47:48 icinga1001 python[100554]: 2019-04-15 13:47:48,521 Connection from ('::ffff:10.64.32.16', 47416, 0, 0)
68Apr 15 13:47:48 icinga1001 python[100554]: 2019-04-15 13:47:48,521 TCP ('::ffff:10.64.32.16', 47416, 0, 0): "!log anomie@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-new on group1 (T188327) (duration: 00m 52s)"
69Apr 15 14:16:20 icinga1001 python[100554]: 2019-04-15 14:16:20,775 Connection from ('::ffff:10.64.16.73', 38858, 0, 0)
70Apr 15 14:16:20 icinga1001 python[100554]: 2019-04-15 14:16:20,775 TCP ('::ffff:10.64.16.73', 38858, 0, 0): "!log oblivian@puppetmaster1001 conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=kartotherian"
71Apr 15 14:26:42 icinga1001 python[100554]: 2019-04-15 14:26:42,748 Connection from ('::ffff:10.64.32.16', 48034, 0, 0)
72Apr 15 14:26:42 icinga1001 python[100554]: 2019-04-15 14:26:42,748 TCP ('::ffff:10.64.32.16', 48034, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]"
73Apr 15 14:26:45 icinga1001 python[100554]: 2019-04-15 14:26:45,161 Connection from ('::ffff:10.64.32.16', 48046, 0, 0)
74Apr 15 14:26:45 icinga1001 python[100554]: 2019-04-15 14:26:45,162 TCP ('::ffff:10.64.32.16', 48046, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed"
75Apr 15 14:26:45 icinga1001 python[100554]: 2019-04-15 14:26:45,166 Connection from ('::ffff:10.64.32.16', 48048, 0, 0)
76Apr 15 14:26:45 icinga1001 python[100554]: 2019-04-15 14:26:45,166 TCP ('::ffff:10.64.32.16', 48048, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics finished"
77Apr 15 14:44:20 icinga1001 python[100554]: 2019-04-15 14:44:20,180 Connection from ('::ffff:10.64.32.16', 48294, 0, 0)
78Apr 15 14:44:20 icinga1001 python[100554]: 2019-04-15 14:44:20,180 TCP ('::ffff:10.64.32.16', 48294, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]"
79Apr 15 14:44:22 icinga1001 python[100554]: 2019-04-15 14:44:22,936 Connection from ('::ffff:10.64.32.16', 48306, 0, 0)
80Apr 15 14:44:22 icinga1001 python[100554]: 2019-04-15 14:44:22,937 TCP ('::ffff:10.64.32.16', 48306, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed"
81Apr 15 14:44:22 icinga1001 python[100554]: 2019-04-15 14:44:22,957 Connection from ('::ffff:10.64.32.16', 48308, 0, 0)
82Apr 15 14:44:22 icinga1001 python[100554]: 2019-04-15 14:44:22,957 TCP ('::ffff:10.64.32.16', 48308, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics finished"
83Apr 15 14:54:10 icinga1001 python[100554]: 2019-04-15 14:54:10,837 Connection from ('::ffff:10.64.32.16', 51348, 0, 0)
84Apr 15 14:54:10 icinga1001 python[100554]: 2019-04-15 14:54:10,837 TCP ('::ffff:10.64.32.16', 51348, 0, 0): "!log gehel@deploy1001 Started deploy [tilerator/deploy@fac7e5e] (stretch): check deployment is working - T220982"
85Apr 15 14:54:14 icinga1001 python[100554]: 2019-04-15 14:54:14,699 Connection from ('::ffff:10.64.32.16', 51358, 0, 0)
86Apr 15 14:54:14 icinga1001 python[100554]: 2019-04-15 14:54:14,699 TCP ('::ffff:10.64.32.16', 51358, 0, 0): "!log gehel@deploy1001 Finished deploy [tilerator/deploy@fac7e5e] (stretch): check deployment is working - T220982 (duration: 00m 04s)"
87Apr 15 14:59:02 icinga1001 python[100554]: 2019-04-15 14:59:02,583 Connection from ('::ffff:10.64.32.16', 51458, 0, 0)
88Apr 15 14:59:02 icinga1001 python[100554]: 2019-04-15 14:59:02,583 TCP ('::ffff:10.64.32.16', 51458, 0, 0): "!log gehel@deploy1001 Started deploy [tilerator/deploy@fac7e5e] (stretch): check deployment is working - T220982"
89Apr 15 14:59:28 icinga1001 python[100554]: 2019-04-15 14:59:28,295 Connection from ('::ffff:10.64.32.16', 51472, 0, 0)
90Apr 15 14:59:28 icinga1001 python[100554]: 2019-04-15 14:59:28,295 TCP ('::ffff:10.64.32.16', 51472, 0, 0): "!log gehel@deploy1001 Finished deploy [tilerator/deploy@fac7e5e] (stretch): check deployment is working - T220982 (duration: 00m 26s)"
91Apr 15 15:08:11 icinga1001 python[100554]: 2019-04-15 15:08:11,504 Connection from ('::ffff:10.64.32.16', 51604, 0, 0)
92Apr 15 15:08:11 icinga1001 python[100554]: 2019-04-15 15:08:11,505 TCP ('::ffff:10.64.32.16', 51604, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]"
93Apr 15 15:09:50 icinga1001 python[100554]: 2019-04-15 15:09:50,082 Connection from ('::ffff:10.64.32.16', 51626, 0, 0)
94Apr 15 15:09:50 icinga1001 python[100554]: 2019-04-15 15:09:50,083 TCP ('::ffff:10.64.32.16', 51626, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]"
95Apr 15 15:09:52 icinga1001 python[100554]: 2019-04-15 15:09:52,548 Connection from ('::ffff:10.64.32.16', 51638, 0, 0)
96Apr 15 15:09:52 icinga1001 python[100554]: 2019-04-15 15:09:52,548 TCP ('::ffff:10.64.32.16', 51638, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed"
97Apr 15 15:09:52 icinga1001 python[100554]: 2019-04-15 15:09:52,554 Connection from ('::ffff:10.64.32.16', 51640, 0, 0)
98Apr 15 15:09:52 icinga1001 python[100554]: 2019-04-15 15:09:52,554 TCP ('::ffff:10.64.32.16', 51640, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics finished"
99Apr 15 15:13:39 icinga1001 python[100554]: 2019-04-15 15:13:39,469 Connection from ('::ffff:10.64.32.16', 51698, 0, 0)
100Apr 15 15:13:39 icinga1001 python[100554]: 2019-04-15 15:13:39,470 TCP ('::ffff:10.64.32.16', 51698, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]"
101Apr 15 15:13:40 icinga1001 python[100554]: 2019-04-15 15:13:40,700 Connection from ('::ffff:10.64.32.16', 51710, 0, 0)
102Apr 15 15:13:40 icinga1001 python[100554]: 2019-04-15 15:13:40,700 TCP ('::ffff:10.64.32.16', 51710, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed"
103Apr 15 15:13:40 icinga1001 python[100554]: 2019-04-15 15:13:40,705 Connection from ('::ffff:10.64.32.16', 51712, 0, 0)
104Apr 15 15:13:40 icinga1001 python[100554]: 2019-04-15 15:13:40,705 TCP ('::ffff:10.64.32.16', 51712, 0, 0): "!log otto@deploy1001 scap-helm eventgate-analytics finished"
105Apr 15 15:53:52 icinga1001 python[100554]: 2019-04-15 15:53:52,867 Connection from ('::ffff:10.64.32.16', 60042, 0, 0)
106Apr 15 15:53:52 icinga1001 python[100554]: 2019-04-15 15:53:52,868 TCP ('::ffff:10.64.32.16', 60042, 0, 0): "!log reedy@deploy1001 Synchronized docroot/wikimedia.org: Standardise wikimedia.org docroot folder (duration: 00m 55s)"
107Apr 15 17:04:15 icinga1001 python[100554]: 2019-04-15 17:04:15,675 Connection from ('::ffff:10.64.32.16', 32906, 0, 0)
108Apr 15 17:04:15 icinga1001 python[100554]: 2019-04-15 17:04:15,676 TCP ('::ffff:10.64.32.16', 32906, 0, 0): "!log onimisionipe@deploy1001 Started deploy [wdqs/wdqs@b14fb69]: GUI updates"
109Apr 15 17:14:19 icinga1001 python[100554]: 2019-04-15 17:14:19,316 Connection from ('::ffff:10.64.32.16', 33150, 0, 0)
110Apr 15 17:14:19 icinga1001 python[100554]: 2019-04-15 17:14:19,316 TCP ('::ffff:10.64.32.16', 33150, 0, 0): "!log onimisionipe@deploy1001 Finished deploy [wdqs/wdqs@b14fb69]: GUI updates (duration: 10m 04s)"
111Apr 15 17:40:13 icinga1001 python[100554]: 2019-04-15 17:40:13,093 Connection from ('::ffff:10.64.32.16', 34798, 0, 0)
112Apr 15 17:40:13 icinga1001 python[100554]: 2019-04-15 17:40:13,093 TCP ('::ffff:10.64.32.16', 34798, 0, 0): "!log anomie@deploy1001 Synchronized php-1.33.0-wmf.25/includes/specials/pagers/ContribsPager.php: Backporting fix for T220991 (duration: 00m 53s)"
113Apr 15 17:41:51 icinga1001 python[100554]: 2019-04-15 17:41:51,334 Connection from ('::ffff:10.64.32.16', 36068, 0, 0)
114Apr 15 17:41:51 icinga1001 python[100554]: 2019-04-15 17:41:51,334 TCP ('::ffff:10.64.32.16', 36068, 0, 0): "!log anomie@deploy1001 Synchronized php-1.33.0-wmf.25/includes/specials/pagers/ContribsPager.php: Backporting fix for T220991 (duration: 00m 51s)"
115Apr 15 17:45:05 icinga1001 python[100554]: 2019-04-15 17:45:05,183 Connection from ('::ffff:10.64.32.16', 37364, 0, 0)
116Apr 15 17:45:05 icinga1001 python[100554]: 2019-04-15 17:45:05,183 TCP ('::ffff:10.64.32.16', 37364, 0, 0): "!log anomie@deploy1001 Synchronized php-1.33.0-wmf.25/includes/specials/pagers/ContribsPager.php: Backporting fix for T220991 (duration: 00m 51s)"
117Apr 15 18:30:12 icinga1001 python[100554]: 2019-04-15 18:30:12,700 Connection from ('::ffff:10.64.32.16', 46346, 0, 0)
118Apr 15 18:30:12 icinga1001 python[100554]: 2019-04-15 18:30:12,700 TCP ('::ffff:10.64.32.16', 46346, 0, 0): "!log ladsgroup@deploy1001 Synchronized static/apple-touch/wiktionary/de.png: SWAT: [[gerrit:503638|Upload a new Apple Touch icon for German Wiktionary (T202902)]] (duration: 00m 51s)"
119Apr 15 18:37:31 icinga1001 python[100554]: 2019-04-15 18:37:31,145 Connection from ('::ffff:10.64.32.16', 47710, 0, 0)
120Apr 15 18:37:31 icinga1001 python[100554]: 2019-04-15 18:37:31,145 TCP ('::ffff:10.64.32.16', 47710, 0, 0): "!log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:503638|Use the new Apple Touch icon for German Wiktionary (T202902)]] (duration: 00m 53s)"
121Apr 15 20:05:14 icinga1001 python[100554]: 2019-04-15 20:05:14,936 Connection from ('::ffff:10.64.32.16', 51612, 0, 0)
122Apr 15 20:05:14 icinga1001 python[100554]: 2019-04-15 20:05:14,936 TCP ('::ffff:10.64.32.16', 51612, 0, 0): "!log ladsgroup@deploy1001 Started deploy [ores/deploy@8f01d40]: T214841 T219913 T179631"
123Apr 15 20:19:45 icinga1001 python[100554]: 2019-04-15 20:19:45,800 Connection from ('::ffff:10.64.32.16', 51958, 0, 0)
124Apr 15 20:19:45 icinga1001 python[100554]: 2019-04-15 20:19:45,801 TCP ('::ffff:10.64.32.16', 51958, 0, 0): "!log ladsgroup@deploy1001 Finished deploy [ores/deploy@8f01d40]: T214841 T219913 T179631 (duration: 14m 31s)"
125Apr 15 20:23:10 icinga1001 python[100554]: 2019-04-15 20:23:10,916 Connection from ('::ffff:10.64.32.16', 52100, 0, 0)
126Apr 15 20:23:10 icinga1001 python[100554]: 2019-04-15 20:23:10,916 TCP ('::ffff:10.64.32.16', 52100, 0, 0): "!log arlolra@deploy1001 Started deploy [parsoid/deploy@abec3f9]: Updating Parsoid to 83c17fc9"
127Apr 15 20:28:40 icinga1001 python[100554]: 2019-04-15 20:28:40,633 Connection from ('::ffff:10.64.32.16', 52866, 0, 0)
128Apr 15 20:28:40 icinga1001 python[100554]: 2019-04-15 20:28:40,633 TCP ('::ffff:10.64.32.16', 52866, 0, 0): "!log arlolra@deploy1001 Finished deploy [parsoid/deploy@abec3f9]: Updating Parsoid to 83c17fc9 (duration: 05m 30s)"
129Apr 15 23:15:46 icinga1001 python[100554]: 2019-04-15 23:15:46,132 Connection from ('::ffff:10.64.32.16', 56666, 0, 0)
130Apr 15 23:15:46 icinga1001 python[100554]: 2019-04-15 23:15:46,132 TCP ('::ffff:10.64.32.16', 56666, 0, 0): "!log bd808@deploy1001 Synchronized wmf-config/wikitech.php: SWAT: [[gerrit:497423|wikitech: Use cn:caseExactMatch: as account search filter]] (T165795) (duration: 00m 52s)"
131Apr 15 23:20:10 icinga1001 systemd[1]: Stopping TCP socket to IRC bot: tcpircbot-logmsgbot...
132Apr 15 23:20:10 icinga1001 systemd[1]: Stopped TCP socket to IRC bot: tcpircbot-logmsgbot.
133Apr 15 23:20:10 icinga1001 systemd[1]: Started TCP socket to IRC bot: tcpircbot-logmsgbot.
134Apr 15 23:20:11 icinga1001 python[239987]: 2019-04-15 23:20:11,410 wolfe.freenode.net [u'*** Looking up your hostname...']
135Apr 15 23:20:11 icinga1001 python[239987]: 2019-04-15 23:20:11,411 wolfe.freenode.net [u'*** Checking Ident']
136Apr 15 23:20:11 icinga1001 python[239987]: 2019-04-15 23:20:11,411 wolfe.freenode.net [u'*** Found your hostname']
137Apr 15 23:20:20 icinga1001 python[239987]: 2019-04-15 23:20:20,751 wolfe.freenode.net [u'*** No Ident response']
138Apr 15 23:20:20 icinga1001 python[239987]: 2019-04-15 23:20:20,751 welcome [wolfe.freenode.net -> logmsgbot]
139Apr 15 23:20:20 icinga1001 python[239987]: 2019-04-15 23:20:20,859 NickServ!NickServ@services. [u'This nickname is registered. Please choose a different nickname, or identify via \x02/msg NickServ identify <password>\x02.']
140Apr 15 23:20:20 icinga1001 python[239987]: 2019-04-15 23:20:20,859 NickServ!NickServ@services. [u'You are now identified for \x02logmsgbot\x02.']
141Apr 15 23:20:21 icinga1001 python[239987]: 2019-04-15 23:20:21,010 freenode-connect!frigg@freenode/utility-bot/frigg [u'Welcome to freenode. To protect the network all new connections will be scanned for vulnerabilities. This will not harm your computer, and vulnerable hosts will be notified.']

Anyway, fixed now with a simple systemctl restart tcpircbot-logmsgbot on icinga1001.

% echo 'cdanis testing hopefully-restored logmsgbot' | nc localhost 9200

<+logmsgbot> cdanis testing hopefully-restored logmsgbot

greg edited projects, added Stashbot, Release-Engineering-Team (Watching / External); removed Release-Engineering-Team (Backlog).Apr 15 2019, 11:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL