Page MenuHomePhabricator

Ingest php-slowlog in logstash
Closed, ResolvedPublic

Description

We are currently sending php-slowlog to the mediawiki.php-fpm.slowlog kafka topic.
Set up the logstash ingestion and dashboards for it.
Since it should be low traffic, we do not want to sample it if possible.

Event Timeline

Clement_Goubert changed the task status from Open to In Progress.Jan 12 2023, 11:24 AM
Clement_Goubert triaged this task as Medium priority.
Clement_Goubert created this task.
Clement_Goubert moved this task from Backlog to In Progress on the MW-on-K8s board.
Clement_Goubert moved this task from Incoming đŸ« to Doing 😎 on the serviceops board.

The retention of the kafka topic is currently the default 7 days. This will be reduced once logstash ingestion is setup.

Change 879417 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] P:logstash::production: mediawiki-php-fpm-slowlog

https://gerrit.wikimedia.org/r/879417

Change 879417 merged by Clément Goubert:

[operations/puppet@production] P:logstash::production: mediawiki-php-fpm-slowlog

https://gerrit.wikimedia.org/r/879417

Change 881454 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Switch php-slowlog to ecs format

https://gerrit.wikimedia.org/r/881454

Change 881454 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Switch php-slowlog to ecs format

https://gerrit.wikimedia.org/r/881454

Change 881812 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: enable filters for ecs 1.11.0

https://gerrit.wikimedia.org/r/881812

Change 881812 merged by Cwhite:

[operations/puppet@production] logstash: enable filters for ecs 1.11.0

https://gerrit.wikimedia.org/r/881812

Change 883978 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/docker-images/production-images@master] httpd-fcgi: Fix system logs test

https://gerrit.wikimedia.org/r/883978

Change 883978 merged by Clément Goubert:

[operations/docker-images/production-images@master] httpd-fcgi: Fix system logs test

https://gerrit.wikimedia.org/r/883978

Mentioned in SAL (#wikimedia-operations) [2023-01-26T16:53:42Z] <claime> Running scap sync-file -D php_fpm_restart_script:/bin/true tox.ini "Rebuilding mediawiki-webserver image" - T326794

Change 884045 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884045

Change 884045 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884045

Change 884046 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884046

Change 884046 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix syslog json

https://gerrit.wikimedia.org/r/884046

Change 884051 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix php-slowlog rsyslog json

https://gerrit.wikimedia.org/r/884051

Change 884051 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix php-slowlog rsyslog json

https://gerrit.wikimedia.org/r/884051

Change 884273 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix duplicate host.name property in log

https://gerrit.wikimedia.org/r/884273

Change 884273 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix duplicate host.name property in log

https://gerrit.wikimedia.org/r/884273

Change 885838 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix slowlog rsyslog message

https://gerrit.wikimedia.org/r/885838

Change 885838 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix slowlog rsyslog message

https://gerrit.wikimedia.org/r/885838

We are now correctly sending, ingesting and storing slowlogs in ECS format. Next step, dashboards.