Page MenuHomePhabricator

deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd
Closed, DeclinedPublic

Description

Error message
[2019-12-27T11:00:49,493][ERROR][logstash.inputs.udp      ] UDP listener died {:exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:161:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:116:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"]}
Impact

16500+ errors as of now just for today at this moment: https://logstash-beta.wmflabs.org/goto/446f9bb7bb29f5fcef023321ac1e7003. 630,000+ errors in December.

Event Timeline

1maurelio@deployment-logstash03:~$ sudo netstat -tulpen
2Active Internet connections (only servers)
3Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
4tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 13818629 27507/sshd
5tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 16077 799/exim4
6tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN 116 19929 407/node
7tcp 0 0 172.16.1.184:5666 0.0.0.0:* LISTEN 105 13773317 22894/nrpe
8tcp6 0 0 :::9092 :::* LISTEN 496 105240881 32049/java
9tcp6 0 0 :::9093 :::* LISTEN 496 105240886 32049/java
10tcp6 0 0 :::9125 :::* LISTEN 108 6305560 32621/prometheus-st
11tcp6 0 0 :::35495 :::* LISTEN 496 105153420 32049/java
12tcp6 0 0 :::9100 :::* LISTEN 110 11130 393/prometheus-node
13tcp6 0 0 :::5229 :::* LISTEN 497 104961872 31671/java
14tcp6 0 0 :::9999 :::* LISTEN 496 105153416 32049/java
15tcp6 0 0 172.16.1.184:9200 :::* LISTEN 117 20818 597/java
16tcp6 0 0 127.0.0.1:9200 :::* LISTEN 117 20817 597/java
17tcp6 0 0 ::1:9200 :::* LISTEN 117 20816 597/java
18tcp6 0 0 :::80 :::* LISTEN 0 17015 802/apache2
19tcp6 0 0 :::43089 :::* LISTEN 496 105153415 32049/java
20tcp6 0 0 :::10514 :::* LISTEN 497 104264651 31671/java
21tcp6 0 0 172.16.1.184:9300 :::* LISTEN 117 16242 597/java
22tcp6 0 0 127.0.0.1:9300 :::* LISTEN 117 16241 597/java
23tcp6 0 0 ::1:9300 :::* LISTEN 117 20800 597/java
24tcp6 0 0 :::9108 :::* LISTEN 110 17410 387/prometheus-elas
25tcp6 0 0 :::22 :::* LISTEN 0 13818631 27507/sshd
26tcp6 0 0 172.16.1.184:7800 :::* LISTEN 496 105241682 32049/java
27tcp6 0 0 :::9112 :::* LISTEN 108 6316243 32621/prometheus-st
28tcp6 0 0 172.16.1.184:7801 :::* LISTEN 497 104265499 31671/java
29tcp6 0 0 ::1:25 :::* LISTEN 0 16078 799/exim4
30tcp6 0 0 :::11514 :::* LISTEN 497 104264589 31671/java
31tcp6 0 0 :::3903 :::* LISTEN 121 21326 1954/mtail
32tcp6 0 0 127.0.0.1:9600 :::* LISTEN 497 104242875 31671/java
33udp 0 0 127.0.0.1:11514 0.0.0.0:* 0 9803466 1863/rsyslogd
34udp 0 0 0.0.0.0:68 0.0.0.0:* 0 11103 491/dhclient
35udp6 0 0 :::10514 :::* 497 104242873 31671/java
36udp6 0 0 :::43717 :::* 497 105240005 31671/java
37udp6 0 0 ::1:11514 :::* 0 9803465 1863/rsyslogd
38udp6 0 0 :::12201 :::* 497 104264650 31671/java
39udp6 0 0 :::45359 :::* 497 105243136 31671/java
40udp6 0 0 :::46520 :::* 497 105242978 31671/java
41udp6 0 0 :::38982 :::* 497 105240357 31671/java
42udp6 0 0 :::48279 :::* 497 105242968 31671/java
43udp6 0 0 :::57350 :::* 497 105242979 31671/java
44udp6 0 0 :::8324 :::* 497 104265520 31671/java
45udp6 0 0 :::9125 :::* 108 6305559 32621/prometheus-st
46udp6 0 0 :::50453 :::* 497 105240020 31671/java
47udp6 0 0 :::50993 :::* 497 105240018 31671/java

taavi renamed this task from deployment-logstash03: UDP listener died EADDRINUSE to deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd.Mar 5 2021, 8:17 AM

Mentioned in SAL (#wikimedia-releng) [2021-03-05T08:32:26Z] <Majavah> deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2

Jdforrester-WMF changed the subtype of this task from "Production Error" to "Task".Mar 5 2021, 8:16 PM
taavi added a subscriber: taavi.

This machine will be decommissioned shortly (T283013)