Page MenuHomePhabricator
Create Task
Maniphest T326794

Ingest php-slowlog in logstash
Closed, ResolvedPublic
Actions

Description

We are currently sending php-slowlog to the mediawiki.php-fpm.slowlog kafka topic.
Set up the logstash ingestion and dashboards for it.
Since it should be low traffic, we do not want to sample it if possible.

Details

ProjectBranchLines +/-Subject
operations/deployment-chartsmaster+3 -2mediawiki: Fix slowlog rsyslog message
operations/deployment-chartsmaster+1 -3mediawiki: Fix duplicate host.name property in log
operations/deployment-chartsmaster+2 -2mediawiki: Fix php-slowlog rsyslog json
operations/deployment-chartsmaster+11 -7mediawiki: Fix syslog json
operations/deployment-chartsmaster+3 -3mediawiki: Fix syslog json
operations/docker-images/production-imagesmaster+3 -1httpd-fcgi: Fix system logs test
operations/puppetproduction+315 -3logstash: enable filters for ecs 1.11.0
operations/deployment-chartsmaster+3 -3mediawiki: Switch php-slowlog to ecs format
operations/puppetproduction+28 -0P:logstash::production: mediawiki-php-fpm-slowlog
Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert changed the task status from Open to In Progress.Jan 12 2023, 11:24 AM
Clement_Goubert triaged this task as Medium priority.
Clement_Goubert created this task.
Clement_Goubert moved this task from Backlog to In Progress on the MW-on-K8s board.
Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.
Clement_Goubert added a comment.Jan 12 2023, 11:29 AM

The retention of the kafka topic is currently the default 7 days. This will be reduced once logstash ingestion is setup.

gerritbot added a comment.Jan 12 2023, 11:32 AM

Change 879417 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] P:logstash::production: mediawiki-php-fpm-slowlog

https://gerrit.wikimedia.org/r/879417

gerritbot added a project: Patch-For-Review.Jan 12 2023, 11:32 AM
gerritbot added a comment.Jan 17 2023, 9:56 AM

Change 879417 merged by Clément Goubert:

[operations/puppet@production] P:logstash::production: mediawiki-php-fpm-slowlog

https://gerrit.wikimedia.org/r/879417

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2023, 10:30 AM
gerritbot added a comment.Jan 18 2023, 5:09 PM

Change 881454 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Switch php-slowlog to ecs format

https://gerrit.wikimedia.org/r/881454

gerritbot added a project: Patch-For-Review.Jan 18 2023, 5:09 PM
gerritbot added a comment.Jan 19 2023, 10:25 AM

Change 881454 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Switch php-slowlog to ecs format

https://gerrit.wikimedia.org/r/881454

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2023, 10:31 AM
lmata moved this task from Inbox to Radar on the SRE Observability board.Jan 20 2023, 2:08 PM
gerritbot added a comment.Jan 20 2023, 6:31 PM

Change 881812 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: enable filters for ecs 1.11.0

https://gerrit.wikimedia.org/r/881812

gerritbot added a project: Patch-For-Review.Jan 20 2023, 6:31 PM
gerritbot added a comment.Jan 23 2023, 9:49 PM

Change 881812 merged by Cwhite:

[operations/puppet@production] logstash: enable filters for ecs 1.11.0

https://gerrit.wikimedia.org/r/881812

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2023, 10:30 PM
gerritbot added a comment.Jan 26 2023, 4:04 PM

Change 883978 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/docker-images/production-images@master] httpd-fcgi: Fix system logs test

https://gerrit.wikimedia.org/r/883978

gerritbot added a project: Patch-For-Review.Jan 26 2023, 4:04 PM

Change 883978 merged by Clément Goubert:

[operations/docker-images/production-images@master] httpd-fcgi: Fix system logs test

https://gerrit.wikimedia.org/r/883978

dancy removed a subscriber: dancy.Jan 26 2023, 4:22 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 26 2023, 4:31 PM
Stashbot added a comment.Jan 26 2023, 4:53 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-26T16:53:42Z] <claime> Running scap sync-file -D php_fpm_restart_script:/bin/true tox.ini "Rebuilding mediawiki-webserver image" - T326794

gerritbot added a comment.Jan 26 2023, 5:35 PM

Change 884045 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884045

gerritbot added a project: Patch-For-Review.Jan 26 2023, 5:35 PM
gerritbot added a comment.Jan 26 2023, 5:46 PM

Change 884045 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884045

gerritbot added a comment.Jan 26 2023, 5:52 PM

Change 884046 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884046

gerritbot added a comment.Jan 26 2023, 6:06 PM

Change 884046 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884046

Maintenance_bot removed a project: Patch-For-Review.Jan 26 2023, 6:30 PM
gerritbot added a comment.Jan 26 2023, 6:59 PM

Change 884051 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix php-slowlog rsyslog json

https://gerrit.wikimedia.org/r/884051

gerritbot added a project: Patch-For-Review.Jan 26 2023, 6:59 PM
gerritbot added a comment.Jan 26 2023, 7:16 PM

Change 884051 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix php-slowlog rsyslog json

https://gerrit.wikimedia.org/r/884051

Maintenance_bot removed a project: Patch-For-Review.Jan 26 2023, 7:30 PM
gerritbot added a comment.Jan 27 2023, 10:09 AM

Change 884273 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix duplicate host.name property in log

https://gerrit.wikimedia.org/r/884273

gerritbot added a project: Patch-For-Review.Jan 27 2023, 10:09 AM
gerritbot added a comment.Jan 27 2023, 10:27 AM

Change 884273 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix duplicate host.name property in log

https://gerrit.wikimedia.org/r/884273

Maintenance_bot removed a project: Patch-For-Review.Jan 27 2023, 10:31 AM
gerritbot added a comment.Feb 1 2023, 4:46 PM

Change 885838 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix slowlog rsyslog message

https://gerrit.wikimedia.org/r/885838

gerritbot added a project: Patch-For-Review.Feb 1 2023, 4:46 PM
gerritbot added a comment.Feb 1 2023, 5:23 PM

Change 885838 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix slowlog rsyslog message

https://gerrit.wikimedia.org/r/885838

Maintenance_bot removed a project: Patch-For-Review.Feb 1 2023, 5:31 PM
Clement_Goubert added a comment.Feb 3 2023, 12:16 PM

We are now correctly sending, ingesting and storing slowlogs in ECS format. Next step, dashboards.

Clement_Goubert added a comment.Feb 9 2023, 3:51 PM

Dashboard available: https://logstash.wikimedia.org/app/dashboards#/view/74557260-a88f-11ed-96bb-4b4732aa077a

Clement_Goubert closed this task as Resolved.Feb 17 2023, 11:17 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL