Page MenuHomePhabricator
Create Task
Maniphest T255436

Termbox Error Logging Should Differentiate between RemoteRenderer and Service timeouts.
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

While investigating T255410 we concluded that we should try to make it clearer to people investigating issues if the timeout was due to:
A) failure to get a response from the termbox Kubernetes service in time
OR IF
B) we actually got the response from termbox saying it was unable to contact Special:EntityData in time.

We intend the issue in situation A should remain an ERROR from the mediawiki TermboxRemoteRenderer but the B) should become only a NOTICE because an ERROR will already have been emitted by the Kubernetes services (A).

Conceptually the RemoteRenderer (Wikibase service calling the SSR - Kubernetes - service) does the correct thing and falls back to the clientside rendered version.

Notes:

  • Error logging is currently happening in both wikibase and termbox SSR service. At most one error due to a timeout should normally be logged (either from termbox or from wikibase but not both)

Questions/Details requested:

  • link to "failure to get a response from the termbox Kubernetes service in time" error in logstash

https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-mediawiki-2020.11.20/mediawiki?id=AXXkYGPvKLIuIzERk2pj&_g=h@1251ff0

Acceptance Criteria

  • Issues with Termbox not being able to contact or get response in time from Special:EntityData (or Mediawiki API in general) are logged (by MediaWiki) using NOTICE error level

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
TermboxRemoteRenderer: Change log level from error to noticemediawiki/extensions/Wikibasemaster+3 -3
Customize query in gerrit

Related Objects

Event Timeline

Tarrow created this task.Jun 15 2020, 1:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 15 2020, 1:19 PM
Addshore added a project: Sustainability (Incident Followup).Jun 15 2020, 1:27 PM
Addshore subscribed.
Michael mentioned this in T255410: Termbox SSR connection terminated very often.Jun 15 2020, 2:41 PM
Addshore added a project: Wikidata-Campsite.Sep 29 2020, 11:14 AM
Addshore moved this task from Incoming to Prioritized Wikidata Tech Backlog (prioritised from top to bottom) on the Wikidata-Campsite board.
WMDE-leszek updated the task description. (Show Details)Oct 28 2020, 10:11 AM
WMDE-leszek updated the task description. (Show Details)Oct 28 2020, 1:37 PM
WMDE-leszek updated the task description. (Show Details)Oct 28 2020, 1:48 PM
Tarrow updated the task description. (Show Details)Nov 17 2020, 2:27 PM
WMDE-leszek updated the task description. (Show Details)Nov 20 2020, 11:06 AM
WMDE-leszek updated the task description. (Show Details)Nov 20 2020, 11:08 AM
WMDE-leszek updated the task description. (Show Details)
Tarrow updated the task description. (Show Details)Nov 20 2020, 11:29 AM
Tarrow updated the task description. (Show Details)Nov 20 2020, 11:31 AM
WMDE-leszek updated the task description. (Show Details)Nov 24 2020, 2:32 PM
WMDE-leszek subscribed.EditedNov 24 2020, 2:36 PM

During the team discussion it has been pointed out that there should be made possible to track what Termbox SSR - MediaWiki requests relates to what Mediawiki-Termbox SSR request.
Currently the requests, and their errors, could only be matched by having happened in the same time frame.
This change is out of scope of this task

WMDE-leszek updated the task description. (Show Details)Nov 24 2020, 2:45 PM
WMDE-leszek updated the task description. (Show Details)Nov 24 2020, 2:48 PM
WMDE-leszek updated the task description. (Show Details)Nov 24 2020, 2:52 PM
Lucas_Werkmeister_WMDE subscribed.EditedNov 24 2020, 3:34 PM

Task inspection notes:

The RemoteRenderer in PHP is TermboxRemoteRenderer.php; it logs “case A” (timeout) in line 61 and “case B” (SSR service got error from MediaWiki) in line 63.

We also agreed that all kinds of errors in the SSR service should already be logged – the termbox SSR service can instruct the service runner to log an error, or the service runner will itself log an error if it can’t even start the service properly – but in any case, in PHP land we can assume that an error we get was already logged, and we can safely downgrade to “notice” all the time. (Edit: therefore we decided to set the story points to 2 after all, even though we had previously estimated this around 5 – we are now all convinced that changing the word “error” to “notice” is all that’s needed, except maybe updating a test case.)

Lucas_Werkmeister_WMDE set the point value for this task to 2.Nov 24 2020, 3:36 PM
Lucas_Werkmeister_WMDE mentioned this in T268640: Log termbox SSR errors with MediaWiki request ID.Nov 24 2020, 3:43 PM
In T255436#6645149, @WMDE-leszek wrote:

During the team discussion it has been pointed out that there should be made possible to track what Termbox SSR - MediaWiki requests relates to what Mediawiki-Termbox SSR request.
Currently the requests, and their errors, could only be matched by having happened in the same time frame.
This change is out of scope of this task

T268640: Log termbox SSR errors with MediaWiki request ID

Lucas_Werkmeister_WMDE moved this task from Prioritized Wikidata Tech Backlog (prioritised from top to bottom) to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Nov 24 2020, 3:43 PM
Lucas_Werkmeister_WMDE edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Nov 24 2020, 4:15 PM
• Tonina_Zhelyazkova_WMDE claimed this task.Nov 25 2020, 10:38 AM
• Tonina_Zhelyazkova_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Nov 25 2020, 2:26 PM

Change 643506 had a related patch set uploaded (by Tonina Zhelyazkova; owner: Tonina Zhelyazkova):
[mediawiki/extensions/Wikibase@master] TermboxRemoteRenderer: Change log level from error to notice

https://gerrit.wikimedia.org/r/643506

gerritbot added a project: Patch-For-Review.Nov 25 2020, 2:26 PM
• Tonina_Zhelyazkova_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 25 2020, 3:47 PM
gerritbot added a comment.Nov 26 2020, 3:11 PM

Change 643506 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] TermboxRemoteRenderer: Change log level from error to notice

https://gerrit.wikimedia.org/r/643506

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.20; 2020-12-01).Nov 26 2020, 4:00 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 26 2020, 4:10 PM
Lucas_Werkmeister_WMDE moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 26 2020, 4:32 PM
WMDE-leszek closed this task as Resolved.Jan 6 2021, 12:24 PM
WMDE-leszek moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

I have failed to find any notices recorded in the logstash (errors before the change were also pretty rare, it seems), but no errors are reported either. I conclude this task is done. thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits