Ingest php-slowlog in logstash
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Clement_Goubert
	Jan 12 2023, 11:24 AM

Description

We are currently sending php-slowlog to the mediawiki.php-fpm.slowlog kafka topic.
Set up the logstash ingestion and dashboards for it.
Since it should be low traffic, we do not want to sample it if possible.

Details

Subject	Repo	Branch	Lines +/-
mediawiki: Fix slowlog rsyslog message	operations/deployment-charts	master	+3 -2
mediawiki: Fix duplicate host.name property in log	operations/deployment-charts	master	+1 -3
mediawiki: Fix php-slowlog rsyslog json	operations/deployment-charts	master	+2 -2
mediawiki: Fix syslog json	operations/deployment-charts	master	+11 -7
mediawiki: Fix syslog json	operations/deployment-charts	master	+3 -3
httpd-fcgi: Fix system logs test	operations/docker-images/production-images	master	+3 -1
logstash: enable filters for ecs 1.11.0	operations/puppet	production	+315 -3
mediawiki: Switch php-slowlog to ecs format	operations/deployment-charts	master	+3 -3
P:logstash::production: mediawiki-php-fpm-slowlog	operations/puppet	production	+28 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T255792 Quibble runs core:unit tests twice!
Open	None	T328919 Upgrade to PHPUnit 10
Open	None	T338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
Open	None	T328921 Drop PHP 7.4 support from MediaWiki
Stalled	None	T334726 Use return type `never` in Wikibase
Open	None	T328922 Drop PHP 8.0 support from MediaWiki
Stalled	None	T319055 Upgrade to psr/container 2.x
Stalled	Krinkle	T319432 Migrate WMF production from PHP 7.4 to PHP 8.1
Open	None	T291916 Tracking task for Bullseye migrations in production
Stalled	None	T356293 Migrate MW appservers' base images to bullseye
Open	None	T290536 Serve production traffic via Kubernetes
Resolved	Joe	T283056 Create a mwdebug deployment for mediawiki on kubernetes
Resolved	• dpifke	T288164 Ensure WikimediaDebug "log" and "profile" features work with k8s-mwdebug
Resolved	Joe	T288851 Make logging work for mediawiki in k8s
Resolved	Clement_Goubert	T326794 Ingest php-slowlog in logstash

Event Timeline

Clement_Goubert changed the task status from Open to In Progress.Jan 12 2023, 11:24 AM

Clement_Goubert triaged this task as Medium priority.

Clement_Goubert created this task.

Clement_Goubert moved this task from Backlog to In Progress on the MW-on-K8s board.

Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.

The retention of the kafka topic is currently the default 7 days. This will be reduced once logstash ingestion is setup.

Change 879417 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] P:logstash::production: mediawiki-php-fpm-slowlog

https://gerrit.wikimedia.org/r/879417

gerritbot added a project: Patch-For-Review.Jan 12 2023, 11:32 AM

Change 879417 merged by Clément Goubert:

[operations/puppet@production] P:logstash::production: mediawiki-php-fpm-slowlog

https://gerrit.wikimedia.org/r/879417

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2023, 10:30 AM

Change 881454 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Switch php-slowlog to ecs format

https://gerrit.wikimedia.org/r/881454

gerritbot added a project: Patch-For-Review.Jan 18 2023, 5:09 PM

Change 881454 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Switch php-slowlog to ecs format

https://gerrit.wikimedia.org/r/881454

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2023, 10:31 AM

lmata moved this task from Inbox to Radar on the SRE Observability board.Jan 20 2023, 2:08 PM

Change 881812 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: enable filters for ecs 1.11.0

https://gerrit.wikimedia.org/r/881812

gerritbot added a project: Patch-For-Review.Jan 20 2023, 6:31 PM

Change 881812 merged by Cwhite:

[operations/puppet@production] logstash: enable filters for ecs 1.11.0

https://gerrit.wikimedia.org/r/881812

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2023, 10:30 PM

Change 883978 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/docker-images/production-images@master] httpd-fcgi: Fix system logs test

https://gerrit.wikimedia.org/r/883978

Change 883978 merged by Clément Goubert:

[operations/docker-images/production-images@master] httpd-fcgi: Fix system logs test

https://gerrit.wikimedia.org/r/883978

dancy unsubscribed.Jan 26 2023, 4:22 PM

Maintenance_bot removed a project: Patch-For-Review.Jan 26 2023, 4:31 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-26T16:53:42Z] <claime> Running scap sync-file -D php_fpm_restart_script:/bin/true tox.ini "Rebuilding mediawiki-webserver image" - T326794