Page MenuHomePhabricator
Create Task
Maniphest T438375

rsyslog ratelimit / mediawiki log producers ratelimit
Open, Needs TriagePublic
Actions

Description

Rsyslog has introduced ratelimit in version 8.2602.0 (available in forky)

This has potential to be a drop-in improvement to the long standing issue where an error/bug/etc will spike log volume on the mw-k8s topic and cause log ingest to lag across all mediawiki channels and levels.

With this we could start setting rate limits before logs are produced to Kafka, based on combinations of log fields. e.g. each unique [channel]+[level] combination should not exceed N logs per M minutes.

Rough todo

  • backport >= 8.2602.0 package to trixie
  • update rsyslog in container
  • draft/test POC ratelimit configuration
  • establish sensible log rate limit scheme
  • set up metrics/monitoring as related to ratelimit, consider alerting/logging/etc when enforced etc.
  • deploy initial ratelimit config

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Test new rsyslog image on mw-web canaryHerronoperations/deployment-chartsmaster+4 -0
Test new rsyslog image on mw-experimentalHerronoperations/deployment-chartsmaster+4 -0
upgrade rsyslog to 8.2608.0-4Herronoperations/docker-images/production-imagesmaster+6 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

herron created this task.Thu, Sep 17, 4:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Sep 17, 4:00 PM
herron added a parent task: T390529: Reduce the amount of messages sent through channel:Memcached during failures.Thu, Sep 17, 4:01 PM
hnowlan subscribed.Fri, Sep 18, 4:57 PM

This would have been useful during T438387

herron added a comment.EditedFri, Sep 18, 7:05 PM

Initial backport of rsyslog 8.2608.0-4 to trixie is building via CI: https://gitlab.wikimedia.org/repos/sre/rsyslog/-/jobs/977217/artifacts/browse/WMF_BUILD_DIR

herron updated the task description. (Show Details)Fri, Sep 18, 7:05 PM
herron updated the task description. (Show Details)Wed, Sep 23, 1:27 PM
herron mentioned this in T438570: Scap's logstash checker for channels outside "error" and "exception".Wed, Sep 23, 1:52 PM
hnowlan moved this task from Inbox to Backlog on the SRE Observability board.Wed, Sep 23, 2:52 PM
herron updated the task description. (Show Details)Mon, Sep 28, 8:35 PM
Stashbot added a comment.Mon, Sep 28, 8:47 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-28T20:47:47Z] <herron> uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia T438375

gerritbot added a comment.Mon, Sep 28, 9:16 PM

Change #1345655 had a related patch set uploaded (by Herron; author: Herron):

[operations/docker-images/production-images@master] upgrade rsyslog to 8.2608.0-4

https://gerrit.wikimedia.org/r/1345655

gerritbot added a project: Patch-For-Review.Mon, Sep 28, 9:16 PM
MoritzMuehlenhoff subscribed.Tue, Sep 29, 12:57 PM

@herron The new rsyslog deb breaks the installation of new trixie hosts. The failure in rsyslog.service isn't very informative, but you can have a closer look at it at testvm2001.codfw.wmnet:

Sep 29 12:54:36 testvm2001 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 5.
Sep 29 12:54:36 testvm2001 systemd[1]: rsyslog.service: Start request repeated too quickly.
Sep 29 12:54:36 testvm2001 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Sep 29 12:54:36 testvm2001 systemd[1]: Failed to start rsyslog.service - System Logging Service.
herron added a comment.Tue, Sep 29, 1:48 PM
In T438375#12374174, @MoritzMuehlenhoff wrote:

@herron The new rsyslog deb breaks the installation of new trixie hosts. The failure in rsyslog.service isn't very informative, but you can have a closer look at it at testvm2001.codfw.wmnet:

Sep 29 12:54:36 testvm2001 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 5.
Sep 29 12:54:36 testvm2001 systemd[1]: rsyslog.service: Start request repeated too quickly.
Sep 29 12:54:36 testvm2001 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Sep 29 12:54:36 testvm2001 systemd[1]: Failed to start rsyslog.service - System Logging Service.

Thanks, this is a straight ahead backport from forky but seems package dependencies weren't installed. Installing libestr0 libfastjson4 and librdkafka1 get the service up and running. Will have a closer look at dependencies

MoritzMuehlenhoff added a comment.Tue, Sep 29, 1:58 PM

Indeed, it links against those, but doesn't have any Depends: for them:

jmm@testvm2001:~$ ldd /usr/sbin/rsyslogd

linux-vdso.so.1 (0x00007f3e6e3ed000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3e6e2c9000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f3e6e1ff000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3e6e10f000)
libestr.so.0 => /lib/x86_64-linux-gnu/libestr.so.0 (0x00007f3e6e10a000)
libfastjson.so.4 => /lib/x86_64-linux-gnu/libfastjson.so.4 (0x00007f3e6e0fa000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f3e6dfe4000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f3e6dfda000)
libyaml-0.so.2 => /lib/x86_64-linux-gnu/libyaml-0.so.2 (0x00007f3e6dfb9000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f3e6dfa7000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3e6ddb3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3e6e3ef000)
libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f3e6dda5000)

jmm@testvm2001:~$

MoritzMuehlenhoff added a comment.Tue, Sep 29, 2:17 PM

There only seems to be a binary deb uploaded to apt.wikimedia.org? Please always include the full changes files with source.

The reason the Depends aren't auto-generated is likely because you are missing

Depends: ${misc:Depends},  ${shlibs:Depends}

in debian/control. The latter will auto-create Depends: based on the shlibs inforrmation from the library packages.

herron added a comment.Tue, Sep 29, 2:39 PM

Fwiw it was imported like this:

apt1002:~/rsyslog/WMF_BUILD_DIR$ sudo -E reprepro -C main include trixie-wikimedia rsyslog_8.2608.0-4~wmf13+1_amd64.changes

Here's where I'm at currently: https://gitlab.wikimedia.org/repos/sre/rsyslog/-/commits/trixie-wikimedia which looks much better

WMF_BUILD_DIR$ dpkg-deb -f rsyslog_8.2608.0-4~wmf13+1_amd64.deb Depends
libc6 (>= 2.38), libestr0 (>= 0.1.9), libfastjson4 (>= 0.99.8), liblognorm5 (>= 2.0.3), libsystemd0 (>= 256~rc1), libuuid1 (>= 2.16), libyaml-0-2, libzstd1 (>= 1.5.5), zlib1g (>= 1:1.1.4)

Just bumped the changelog and rebuilding, would you like to have a look at the updated deb there before uploading? Should be finished in a minute https://gitlab.wikimedia.org/repos/sre/rsyslog/-/jobs/978140

herron added a comment.Tue, Sep 29, 3:36 PM

Went ahead and uploaded rsyslog 8.2608.0-4~wmf13+1.1 to fix fresh installs on trixie. Happy to take suggestions to generate/upload source packages, but so far have been following the rationale in https://wikitech.wikimedia.org/wiki/Debian_packaging_with_dgit_and_CI#Rationale_/_Further_Details

Dzahn mentioned this in T439499: PuppetFailure - zuul1005.Tue, Sep 29, 4:49 PM
Dzahn mentioned this in T439498: SystemdUnitFailed - zuul1005.Tue, Sep 29, 4:52 PM
gerritbot added a comment.Tue, Sep 29, 5:26 PM

Change #1345655 merged by Herron:

[operations/docker-images/production-images@master] upgrade rsyslog to 8.2608.0-4

https://gerrit.wikimedia.org/r/1345655

Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 29, 5:31 PM
MLechvien-WMF mentioned this in T390529: Reduce the amount of messages sent through channel:Memcached during failures.Wed, Sep 30, 12:56 PM
MoritzMuehlenhoff added a comment.Wed, Sep 30, 2:45 PM
In T438375#12375198, @herron wrote:

https://gitlab.wikimedia.org/repos/sre/rsyslog/-/commit/89b1ea3b68876e783252f05902e2b0deb95e7a2d looks great and I can confirm the resulting deb is also fine.

gerritbot added a comment.Wed, Sep 30, 5:12 PM

Change #1346712 had a related patch set uploaded (by Herron; author: Herron):

[operations/deployment-charts@master] Test new rsyslog image on mw-experimental and mw-web canary

https://gerrit.wikimedia.org/r/1346712

gerritbot added a project: Patch-For-Review.Wed, Sep 30, 5:12 PM
MLechvien-WMF merged a task: T390529: Reduce the amount of messages sent through channel:Memcached during failures.Fri, Oct 2, 8:04 AM
MLechvien-WMF added a project: Sustainability (Incident Followup).
MLechvien-WMF added subscribers: jijiki, Krinkle, MSantos and 3 others.
MLechvien-WMF added a comment.Fri, Oct 2, 8:07 AM

Porting over the incident follow-up tracking: this was noted as part of 2025-03-12_ExternalStorage_Database_Cluster_Overload

gerritbot added a comment.Tue, Oct 6, 2:24 PM

Change #1352121 had a related patch set uploaded (by Herron; author: Herron):

[operations/deployment-charts@master] Test new rsyslog image on mw-experimental

https://gerrit.wikimedia.org/r/1352121

gerritbot added a comment.Thu, Oct 8, 9:59 PM

Change #1352121 merged by jenkins-bot:

[operations/deployment-charts@master] Test new rsyslog image on mw-experimental

https://gerrit.wikimedia.org/r/1352121

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits