deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	MarcoAurelio
	Dec 27 2019, 11:16 AM

Description

Error message

[2019-12-27T11:00:49,493][ERROR][logstash.inputs.udp      ] UDP listener died {:exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:161:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:116:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"]}

Impact

16500+ errors as of now just for today at this moment: https://logstash-beta.wmflabs.org/goto/446f9bb7bb29f5fcef023321ac1e7003. 630,000+ errors in December.

Related Objects
Search...

Status	Subtype	Assigned	Task
Invalid		None	T197804 Puppet: forbid new Python2 code
Open		None	T218426 Upgrade various Cloud VPS Python 2 scripts to Python 3
Resolved	BUG REPORT	• Bstorm	T218423 Add python 3 packages to openstack::clientpackages::common
Resolved		MoritzMuehlenhoff	T232677 Remove support for Debian Jessie in Cloud Services
Duplicate		None	T236575 "deployment-prep" Cloud VPS project jessie deprecation
Resolved		None	T218729 Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster
Resolved		None	T238707 Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch)
Declined		None	T241481 deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd

Event Timeline

MarcoAurelio created this task.Dec 27 2019, 11:16 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 27 2019, 11:16 AM

MarcoAurelio updated the task description. (Show Details)Dec 27 2019, 11:16 AM

MarcoAurelio edited Stack Trace. (Show Details)

P10165 (An Untitled Masterwork)

2Active 3Proto 4tcp 5tcp 6tcp 7tcp 8tcp6 9tcp6 10tcp6 11tcp6 12tcp6 13tcp6 14tcp6 15tcp6 16tcp6 17tcp6 18tcp6 19tcp6 20tcp6 21tcp6 22tcp6 23tcp6 24tcp6 25tcp6 26tcp6 27tcp6 28tcp6 29tcp6 30tcp6 31tcp6 32tcp6 33udp 34udp 35udp6 36udp6 37udp6 38udp6 39udp6 40udp6 41udp6 42udp6 43udp6 44udp6 45udp6 46udp6 47udp6 class="paste-embed-body" style="max-height: 27.6em;">

1	maurelio@deployment-logstash03:~$ sudo netstat -tulpen Internet connections (only servers) Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 13818629 27507/sshd 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 16077 799/exim4 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN 116 19929 407/node 0 0 172.16.1.184:5666 0.0.0.0:* LISTEN 105 13773317 22894/nrpe 0 0 :::9092 :::* LISTEN 496 105240881 32049/java 0 0 :::9093 :::* LISTEN 496 105240886 32049/java 0 0 :::9125 :::* LISTEN 108 6305560 32621/prometheus-st 0 0 :::35495 :::* LISTEN 496 105153420 32049/java 0 0 :::9100 :::* LISTEN 110 11130 393/prometheus-node 0 0 :::5229 :::* LISTEN 497 104961872 31671/java 0 0 :::9999 :::* LISTEN 496 105153416 32049/java 0 0 172.16.1.184:9200 :::* LISTEN 117 20818 597/java 0 0 127.0.0.1:9200 :::* LISTEN 117 20817 597/java 0 0 ::1:9200 :::* LISTEN 117 20816 597/java 0 0 :::80 :::* LISTEN 0 17015 802/apache2 0 0 :::43089 :::* LISTEN 496 105153415 32049/java 0 0 :::10514 :::* LISTEN 497 104264651 31671/java 0 0 172.16.1.184:9300 :::* LISTEN 117 16242 597/java 0 0 127.0.0.1:9300 :::* LISTEN 117 16241 597/java 0 0 ::1:9300 :::* LISTEN 117 20800 597/java 0 0 :::9108 :::* LISTEN 110 17410 387/prometheus-elas 0 0 :::22 :::* LISTEN 0 13818631 27507/sshd 0 0 172.16.1.184:7800 :::* LISTEN 496 105241682 32049/java 0 0 :::9112 :::* LISTEN 108 6316243 32621/prometheus-st 0 0 172.16.1.184:7801 :::* LISTEN 497 104265499 31671/java 0 0 ::1:25 :::* LISTEN 0 16078 799/exim4 0 0 :::11514 :::* LISTEN 497 104264589 31671/java 0 0 :::3903 :::* LISTEN 121 21326 1954/mtail 0 0 127.0.0.1:9600 :::* LISTEN 497 104242875 31671/java 0 0 127.0.0.1:11514 0.0.0.0:* 0 9803466 1863/rsyslogd 0 0 0.0.0.0:68 0.0.0.0:* 0 11103 491/dhclient 0 0 :::10514 :::* 497 104242873 31671/java 0 0 :::43717 :::* 497 105240005 31671/java 0 0 ::1:11514 :::* 0 9803465 1863/rsyslogd 0 0 :::12201 :::* 497 104264650 31671/java 0 0 :::45359 :::* 497 105243136 31671/java 0 0 :::46520 :::* 497 105242978 31671/java 0 0 :::38982 :::* 497 105240357 31671/java 0 0 :::48279 :::* 497 105242968 31671/java 0 0 :::57350 :::* 497 105242979 31671/java 0 0 :::8324 :::* 497 104265520 31671/java 0 0 :::9125 :::* 108 6305559 32621/prometheus-st 0 0 :::50453 :::* 497 105240020 31671/java 0 0 :::50993 :::* 497 105240018 31671/java

maurelio@deployment-logstash03:~$ sudo netstat -tulpen Internet connections (only servers) Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 13818629 27507/sshd 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 16077 799/exim4 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN 116 19929 407/node 0 0 172.16.1.184:5666 0.0.0.0:* LISTEN 105 13773317 22894/nrpe 0 0 :::9092 :::* LISTEN 496 105240881 32049/java 0 0 :::9093 :::* LISTEN 496 105240886 32049/java 0 0 :::9125 :::* LISTEN 108 6305560 32621/prometheus-st 0 0 :::35495 :::* LISTEN 496 105153420 32049/java 0 0 :::9100 :::* LISTEN 110 11130 393/prometheus-node 0 0 :::5229 :::* LISTEN 497 104961872 31671/java 0 0 :::9999 :::* LISTEN 496 105153416 32049/java 0 0 172.16.1.184:9200 :::* LISTEN 117 20818 597/java 0 0 127.0.0.1:9200 :::* LISTEN 117 20817 597/java 0 0 ::1:9200 :::* LISTEN 117 20816 597/java 0 0 :::80 :::* LISTEN 0 17015 802/apache2 0 0 :::43089 :::* LISTEN 496 105153415 32049/java 0 0 :::10514 :::* LISTEN 497 104264651 31671/java 0 0 172.16.1.184:9300 :::* LISTEN 117 16242 597/java 0 0 127.0.0.1:9300 :::* LISTEN 117 16241 597/java 0 0 ::1:9300 :::* LISTEN 117 20800 597/java 0 0 :::9108 :::* LISTEN 110 17410 387/prometheus-elas 0 0 :::22 :::* LISTEN 0 13818631 27507/sshd 0 0 172.16.1.184:7800 :::* LISTEN 496 105241682 32049/java 0 0 :::9112 :::* LISTEN 108 6316243 32621/prometheus-st 0 0 172.16.1.184:7801 :::* LISTEN 497 104265499 31671/java 0 0 ::1:25 :::* LISTEN 0 16078 799/exim4 0 0 :::11514 :::* LISTEN 497 104264589 31671/java 0 0 :::3903 :::* LISTEN 121 21326 1954/mtail 0 0 127.0.0.1:9600 :::* LISTEN 497 104242875 31671/java 0 0 127.0.0.1:11514 0.0.0.0:* 0 9803466 1863/rsyslogd 0 0 0.0.0.0:68 0.0.0.0:* 0 11103 491/dhclient 0 0 :::10514 :::* 497 104242873 31671/java 0 0 :::43717 :::* 497 105240005 31671/java 0 0 ::1:11514 :::* 0 9803465 1863/rsyslogd 0 0 :::12201 :::* 497 104264650 31671/java 0 0 :::45359 :::* 497 105243136 31671/java 0 0 :::46520 :::* 497 105242978 31671/java 0 0 :::38982 :::* 497 105240357 31671/java 0 0 :::48279 :::* 497 105242968 31671/java 0 0 :::57350 :::* 497 105242979 31671/java 0 0 :::8324 :::* 497 104265520 31671/java 0 0 :::9125 :::* 108 6305559 32621/prometheus-st 0 0 :::50453 :::* 497 105240020 31671/java 0 0 :::50993 :::* 497 105240018 31671/java

Reedy added a project: SRE.Jan 17 2020, 1:47 AM

Reedy added a parent task: T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

MoritzMuehlenhoff triaged this task as Medium priority.Jan 17 2020, 8:10 AM

Krenair subscribed.Mar 8 2020, 9:51 PM

taavi mentioned this in T276521: deployment-logstash03 puppet errors.Mar 5 2021, 8:13 AM

taavi renamed this task from deployment-logstash03: UDP listener died EADDRINUSE to deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd.Mar 5 2021, 8:17 AM

Mentioned in SAL (#wikimedia-releng) [2021-03-05T08:32:26Z] <Majavah> deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2

Jdforrester-WMF changed the subtype of this task from "Production Error" to "Task".Mar 5 2021, 8:16 PM

This machine will be decommissioned shortly (T283013)

deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogdClosed, DeclinedPublicActions

Description

Impact

Related ObjectsSearch...

Event Timeline

deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd
Closed, DeclinedPublic
Actions

Related Objects
Search...