Page MenuHomePhabricator
Create Task
Maniphest T317343

Eventgate error: '' should have required property 'database', '' should have required property 'performer'
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

single document: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-default-1-7.0.0-1-2022.09.08?id=S17uHYMBzGlbejpUIo0n

normalized_message
'' should have required property 'database', '' should have required property 'performer'
exception.trace





Impact



~400-500 in past 12 hours at the time of writing



More than 32,000 in the last 24 hrs as of 9/9/23 UTC 15:33



~30,000 in the last 24 hrs as of 9/21/22 UTC 15:28 - https://logstash.wikimedia.org/goto/8f8b69836e510160c98319d22ed825b7



Notes



https://logstash.wikimedia.org/goto/bf44e9cea5784f98847e98cfbff283d4



Guessing error is related to this instrument that got deployed recently T306018



errored_stream_name: mediawiki.accountcreation_block
Details
SubjectRepoBranchLines +/-
 Block metrics: Add 'database' and 'performer' fieldsmediawiki/extensions/WikimediaEventsmaster+106 -4
Add EventFactory::createMediaWikiCommonAttrs()mediawiki/extensions/EventBusmaster+12 -0
analytics/mediawiki/accountcreation/block: Re-add required flagsschemas/event/secondarymaster+481 -4
Block metrics: Bump schema to un-require some fieldsmediawiki/extensions/WikimediaEventswmf/1.40.0-wmf.2+1 -1
Block metrics: Bump schema to un-require some fieldsmediawiki/extensions/WikimediaEventswmf/1.40.0-wmf.1+1 -1
Block metrics: Bump schema to un-require some fieldsmediawiki/extensions/WikimediaEventsmaster+1 -1
Un-require some analytics/mediawiki/accountcreation/block fieldsschemas/event/secondarymaster+460 -6
Customize query in gerrit
Related Objects
Event Timeline
bwang created this task.Sep 8 2022, 4:29 PM
Restricted Application added a subscriber: Aklapper.  ·  View Herald TranscriptSep 8 2022, 4:29 PM
Aklapper added a project: Event-Platform.Sep 9 2022, 11:09 AM
+Event-Platform (please add codebase project tags so such tasks show up on the appropriate workboards - thanks a lot!)
Restricted Application added a project: Data-Engineering.  ·  View Herald TranscriptSep 9 2022, 11:09 AM
cjming updated the task description. (Show Details)Sep 9 2022, 3:33 PM
cjming updated the task description. (Show Details)Sep 9 2022, 3:48 PM
cjming updated the task description. (Show Details)Sep 9 2022, 3:55 PM
cjming updated the task description. (Show Details)Sep 21 2022, 3:28 PM
cjming added projects: Product-Analytics, Growth-Team.
cjming updated the task description. (Show Details)Sep 21 2022, 3:32 PM
cjming updated the task description. (Show Details)
cjming subscribed.
Looks like these errors are still firing quite a bit -- added the Product Analytics + Growth Team tags because I think this is related to T306018 << recent patches should be on train (1.40.0-wmf.1)
kostajh edited projects, added Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team.Sep 21 2022, 3:39 PM
kostajh moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.
Tgr claimed this task.Sep 21 2022, 4:06 PM
Tgr moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.
Tgr subscribed.
The patches were on wmf.29 (and the errors have been around for a while - I forgot to update our analytics error dashboard to include errors about this schema so we missed it) but otherwise yes, this is caused by the block metrics work. I guess I just assumed that fragment data would be added automagically, which would not make much sense in retrospect.
KStoller-WMF subscribed.Sep 21 2022, 5:32 PM
gerritbot added a comment.Sep 21 2022, 7:16 PM
Change 833845 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):



[schemas/event/secondary@master] Un-require some analytics/mediawiki/accountcreation/block fields



https://gerrit.wikimedia.org/r/833845
gerritbot added a project: Patch-For-Review.Sep 21 2022, 7:16 PM
gerritbot added a comment.Sep 21 2022, 7:18 PM
Change 833846 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):



[mediawiki/extensions/WikimediaEvents@master] Block metrics: Bump schema to un-require some fields



https://gerrit.wikimedia.org/r/833846
gerritbot added a comment.Sep 21 2022, 8:09 PM
Change 833845 merged by jenkins-bot:



[schemas/event/secondary@master] Un-require some analytics/mediawiki/accountcreation/block fields



https://gerrit.wikimedia.org/r/833845
gerritbot added a comment.Sep 21 2022, 8:16 PM
Change 833846 merged by jenkins-bot:



[mediawiki/extensions/WikimediaEvents@master] Block metrics: Bump schema to un-require some fields



https://gerrit.wikimedia.org/r/833846
gerritbot added a comment.Sep 21 2022, 8:17 PM
Change 833809 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):



[mediawiki/extensions/WikimediaEvents@wmf/1.40.0-wmf.1] Block metrics: Bump schema to un-require some fields



https://gerrit.wikimedia.org/r/833809
gerritbot added a comment.Sep 21 2022, 8:18 PM
Change 833810 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):



[mediawiki/extensions/WikimediaEvents@wmf/1.40.0-wmf.2] Block metrics: Bump schema to un-require some fields



https://gerrit.wikimedia.org/r/833810
gerritbot added a comment.Sep 21 2022, 8:30 PM
Change 833809 merged by jenkins-bot:



[mediawiki/extensions/WikimediaEvents@wmf/1.40.0-wmf.1] Block metrics: Bump schema to un-require some fields



https://gerrit.wikimedia.org/r/833809
gerritbot added a comment.Sep 21 2022, 8:30 PM
Change 833810 merged by jenkins-bot:



[mediawiki/extensions/WikimediaEvents@wmf/1.40.0-wmf.2] Block metrics: Bump schema to un-require some fields



https://gerrit.wikimedia.org/r/833810
Stashbot added a comment.Sep 21 2022, 8:36 PM
Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:36:53Z] <tgr@deploy1002> Synchronized php-1.40.0-wmf.1/extensions/WikimediaEvents/includes/BlockMetrics/BlockMetricsHooks.php: Backport: [[gerrit:833809|Block metrics: Bump schema to un-require some fields (T317343)]] (duration: 03m 55s)
Stashbot added a comment.Sep 21 2022, 8:44 PM
Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:44:58Z] <tgr@deploy1002> Synchronized php-1.40.0-wmf.2/extensions/WikimediaEvents/includes/BlockMetrics/BlockMetricsHooks.php: Backport: [[gerrit:833810|Block metrics: Bump schema to un-require some fields (T317343)]] (duration: 03m 42s)
ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.1; 2022-09-12).Sep 21 2022, 9:00 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 9:30 PM
Tgr moved this task from In Progress to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Sep 21 2022, 9:37 PM
Fixed & added the stream to the Growth eventlogging error dashboard.
gerritbot added a comment.Sep 21 2022, 9:48 PM
Change 833857 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):



[schemas/event/secondary@master] analytics/mediawiki/accountcreation/block: Re-add required flags



https://gerrit.wikimedia.org/r/833857
gerritbot added a project: Patch-For-Review.Sep 21 2022, 9:48 PM
gerritbot added a comment.Sep 22 2022, 2:28 PM
Change 834338 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):



[mediawiki/extensions/WikimediaEvents@master] Block metrics: Add 'database' and 'performer' fields



https://gerrit.wikimedia.org/r/834338
gerritbot added a comment.Sep 22 2022, 2:29 PM
Change 834339 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):



[mediawiki/extensions/EventBus@master] Add EventFactory::createMediaWikiCommonAttrs()



https://gerrit.wikimedia.org/r/834339
Tgr moved this task from QA to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.Sep 22 2022, 2:31 PM
gerritbot added a comment.Sep 22 2022, 5:28 PM
Change 833857 merged by jenkins-bot:



[schemas/event/secondary@master] analytics/mediawiki/accountcreation/block: Re-add required flags



https://gerrit.wikimedia.org/r/833857
kostajh moved this task from Code Review to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Sep 22 2022, 5:29 PM
gerritbot added a comment.Sep 22 2022, 5:43 PM
Change 834339 merged by jenkins-bot:



[mediawiki/extensions/EventBus@master] Add EventFactory::createMediaWikiCommonAttrs()



https://gerrit.wikimedia.org/r/834339
ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.3; 2022-09-26); removed MW-1.40-notes (1.40.0-wmf.1; 2022-09-12).Sep 22 2022, 6:00 PM
gerritbot added a comment.Sep 22 2022, 7:41 PM
Change 834338 merged by jenkins-bot:



[mediawiki/extensions/WikimediaEvents@master] Block metrics: Add 'database' and 'performer' fields



https://gerrit.wikimedia.org/r/834338
Etonkovidova moved this task from QA to Test in Production | Watching on the Growth-Team (Sprint 0 (Growth Team)) board.Sep 23 2022, 10:29 PM
• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Sep 27 2022, 2:23 PM
mpopov moved this task from Triage to Tracking on the Product-Analytics board.Sep 27 2022, 5:15 PM
mpopov added a subscriber: nettrom_WMF.
• EChetty moved this task from Backlog to Event Platform on the Data-Engineering-Planning board.Oct 14 2022, 8:34 AM
Etonkovidova closed this task as Resolved.Oct 18 2022, 11:06 PM
Etonkovidova subscribed.
The error stopped at Sep 21, 2022 @ 20:43:33.068( logstash link)
Krinkle moved this task from Untriaged to Oct 2022 on the Wikimedia-production-error board.Nov 17 2022, 4:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL