Page MenuHomePhabricator
Create Task
Maniphest T268640

Log termbox SSR errors with MediaWiki request ID
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

As a developer investigating termbox problems, I want to be able to track which termbox SSR requests belong to which MediaWiki request.

Problem:
Currently, we (campsite crew) assume this is not possible beyond matching timestamps.

Example:
This came up during the discussion of T255436: Termbox Error Logging Should Differentiate between RemoteRenderer and Service timeouts..

Acceptance criteria:

  • From the logstash perspective, termbox-logged errors have a reqId field matching the original MediaWiki request ID.

Hints

  • README of the termbox code https://github.com/wikimedia/wikibase-termbox#wikibase-termbox (You'll need to install and run termbox using these instructions)
  • This code is included as a submodule of Wikibase.git for non SSR cases too
  • It is possible to run the termbox SSR locally for testing and examining the log output of the service locally

Open questions:

  • This probably means passing the WebRequest::getRequestId() to the SSR service, and then something in the service runner? Talk to WMF service-ops people about this.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Update termbox to 2021-03-01-112916-productionJakoboperations/deployment-chartsmaster+2 -2
Update termbox to 2021-03-01-112916-productionJakoboperations/deployment-chartsmaster+2 -2
Log termbox SSR errors with MediaWiki request IDJakobwikibase/termboxmaster+69 -11
Customize query in gerrit

Related Objects

Event Timeline

Lucas_Werkmeister_WMDE created this task.Nov 24 2020, 3:43 PM
Herald added a subscriber: Aklapper. · View Herald TranscriptNov 24 2020, 3:43 PM
Lucas_Werkmeister_WMDE mentioned this in T255436: Termbox Error Logging Should Differentiate between RemoteRenderer and Service timeouts..Nov 24 2020, 3:43 PM
• Pablo-WMDE added a project: User-Pablo-WMDE.Nov 24 2020, 3:47 PM
• Pablo-WMDE moved this task from Incoming to Medium term on the User-Pablo-WMDE board.
• Pablo-WMDE subscribed.
Addshore subscribed.Jan 5 2021, 11:17 AM
WMDE-leszek added a project: Wikidata-Campsite.Jan 5 2021, 12:29 PM
WMDE-leszek moved this task from Incoming to Prioritized Wikidata Tech Backlog (prioritised from top to bottom) on the Wikidata-Campsite board.
Addshore added a project: [DEPRECATED] wdwb-tech (legacy-backlog).Jan 22 2021, 11:05 PM
Addshore moved this task from Incoming to 1. Quick Wins (+I -E) on the [DEPRECATED] wdwb-tech (legacy-backlog) board.Jan 22 2021, 11:10 PM
Addshore updated the task description. (Show Details)Feb 9 2021, 1:20 PM
Addshore updated the task description. (Show Details)
Addshore updated the task description. (Show Details)Feb 9 2021, 1:23 PM
darthmon_wmde set the point value for this task to 8.Feb 10 2021, 1:23 PM
darthmon_wmde moved this task from Prioritized Wikidata Tech Backlog (prioritised from top to bottom) to Blocked / Waiting / External on the Wikidata-Campsite board.
darthmon_wmde moved this task from Blocked / Waiting / External to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.
darthmon_wmde edited projects, added: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed: Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Feb 10 2021, 2:15 PM
Jakob_WMDE claimed this task.Feb 25 2021, 8:33 AM
Jakob_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Feb 26 2021, 5:11 PM

Change 667206 had a related patch set uploaded (by Jakob; owner: Jakob):
[wikibase/termbox@master] Log termbox SSR errors with MediaWiki request ID

https://gerrit.wikimedia.org/r/667206

gerritbot added a project: Patch-For-Review.Feb 26 2021, 5:11 PM
Jakob_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 26 2021, 5:25 PM
Legoktm subscribed.Feb 26 2021, 6:00 PM

Just to follow up from IRC, yes, the standard MediaWiki HTTP wrappers should always pass along X-Request-Id. But the application shouldn't require it, since other things that hit the service directly (like monitoring, etc.) might not have the header.

Jakob_WMDE added a comment.Mar 1 2021, 9:07 AM

Thanks @Legoktm for following up! Yes, I also remembered that e.g. the kubernetes health checks will hit the same endpoint without that header. The patch I uploaded handles that just fine, so we should be ok.

gerritbot added a comment.Mar 1 2021, 11:29 AM

Change 667206 merged by jenkins-bot:
[wikibase/termbox@master] Log termbox SSR errors with MediaWiki request ID

https://gerrit.wikimedia.org/r/667206

Jakob_WMDE moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 1 2021, 11:42 AM

Moving it back to doing since this needs manual deployment. I'll do that today or tomorrow.

Maintenance_bot removed a project: Patch-For-Review.Mar 1 2021, 12:10 PM
akosiaris awarded a token.Mar 1 2021, 2:28 PM
Jakob_WMDE mentioned this in rWBTB1e779dcd8c6a: Log termbox SSR errors with MediaWiki request ID.Mar 1 2021, 2:52 PM
Addshore moved this task from 1. Quick Wins (+I -E) to Active on the [DEPRECATED] wdwb-tech (legacy-backlog) board.Mar 2 2021, 8:59 AM
gerritbot added a comment.Mar 4 2021, 11:13 AM

Change 668364 had a related patch set uploaded (by Jakob; owner: Jakob):
[operations/deployment-charts@master] Update termbox to 2021-03-01-112916-production

https://gerrit.wikimedia.org/r/668364

gerritbot added a project: Patch-For-Review.Mar 4 2021, 11:13 AM
gerritbot added a comment.Mar 4 2021, 11:31 AM

Change 668364 merged by jenkins-bot:
[operations/deployment-charts@master] Update termbox to 2021-03-01-112916-production

https://gerrit.wikimedia.org/r/668364

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2021, 12:10 PM
gerritbot added a comment.Mar 5 2021, 9:07 AM

Change 668651 had a related patch set uploaded (by Jakob; owner: Jakob):
[operations/deployment-charts@master] Update termbox to 2021-03-01-112916-production

https://gerrit.wikimedia.org/r/668651

gerritbot added a project: Patch-For-Review.Mar 5 2021, 9:07 AM
gerritbot added a comment.Mar 5 2021, 9:18 AM

Change 668651 merged by jenkins-bot:
[operations/deployment-charts@master] Update termbox to 2021-03-01-112916-production

https://gerrit.wikimedia.org/r/668651

Maintenance_bot removed a project: Patch-For-Review.Mar 5 2021, 10:10 AM
Jakob_WMDE mentioned this in T276550: Missing alerts for Termbox staging and test services.Mar 5 2021, 11:42 AM
Jakob_WMDE moved this task from Doing to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 5 2021, 1:14 PM

This is deployed now. Here is a logged error with the new reqId field: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-syslog-2021.03.05?id=5CH_AXgBg_6mGHGh_myf

Addshore closed this task as Resolved.Mar 5 2021, 2:41 PM
wiese moved this task from Medium term to Done on the User-Pablo-WMDE board.Mar 18 2022, 2:52 PM
Maintenance_bot moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 18 2022, 3:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits