Page MenuHomePhabricator
Create Task
Maniphest T255597

NewcomerTask EventLogging schema has invalid array items type specification
Closed, ResolvedPublic
Actions

Description

Has

"maintenance_templates": {
    "type": "array",
    "items": [
        {
            "type": "string"
        }
    ],

But should be

"maintenance_templates": {
    "type": "array",
    "items": 
        {
            "type": "string"
        }

This is causing the data to fail the Hive ingestion (Refine) step:

Failure(org.wikimedia.analytics.refinery.job.refine.RefineTargetException: Failed refinement of hdfs://analytics-hadoop/wmf/data/raw/eventlogging/eventlogging_NewcomerTask/hourly/2020/06/16/15 -> `event`.`NewcomerTask` (year=2020,month=6,day=16,hour=15). Original exception: java.lang.IllegalArgumentException: `maintenance_templates` array schema must specify the items type field)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Fix NewcomerTask schemamediawiki/extensions/GrowthExperimentswmf/1.35.0-wmf.37+1 -1
Fix NewcomerTask schemamediawiki/extensions/GrowthExperimentswmf/1.35.0-wmf.36+1 -1
Fix NewcomerTask schemamediawiki/extensions/GrowthExperimentsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Jun 16 2020, 6:52 PM
Restricted Application added a project: Analytics. · View Herald TranscriptJun 16 2020, 6:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr added a parent task: T251526: Newcomer tasks: create separate EventLogging schema for newcomer task impressions.Jun 16 2020, 8:06 PM
Tgr added projects: NewcomerTasks 1.2, Product-Analytics.
Tgr added a comment.EditedJun 16 2020, 8:11 PM

This is actually valid syntax, although incorrect for the specific schema as they mean different things.

gerritbot added a comment.Jun 16 2020, 8:33 PM

Change 606008 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Fix NewcomerTask schema

https://gerrit.wikimedia.org/r/606008

gerritbot added a project: Patch-For-Review.Jun 16 2020, 8:33 PM
Tgr added a project: Growth-Team (Sprint 0 (Growth Team)).Jun 16 2020, 8:34 PM
Tgr moved this task from Incoming to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.
gerritbot added a comment.Jun 17 2020, 8:10 AM

Change 606121 had a related patch set uploaded (by Kosta Harlan; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.35.0-wmf.37] Fix NewcomerTask schema

https://gerrit.wikimedia.org/r/606121

gerritbot added a comment.Jun 17 2020, 8:11 AM

Change 606122 had a related patch set uploaded (by Kosta Harlan; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.35.0-wmf.36] Fix NewcomerTask schema

https://gerrit.wikimedia.org/r/606122

gerritbot added a comment.Jun 17 2020, 8:18 AM

Change 606008 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Fix NewcomerTask schema

https://gerrit.wikimedia.org/r/606008

kostajh moved this task from Code Review to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Jun 17 2020, 8:38 AM
ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.38; 2020-06-23).Jun 17 2020, 9:00 AM
gerritbot added a comment.Jun 17 2020, 11:14 AM

Change 606122 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.35.0-wmf.36] Fix NewcomerTask schema

https://gerrit.wikimedia.org/r/606122

gerritbot added a comment.Jun 17 2020, 11:17 AM

Change 606121 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.35.0-wmf.37] Fix NewcomerTask schema

https://gerrit.wikimedia.org/r/606121

Stashbot added a comment.Jun 17 2020, 11:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-17T11:18:32Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.36/extensions/GrowthExperiments/extension.json: [[gerrit:606121|Fix NewcomerTask schema (T255597)]] (duration: 01m 06s)

Stashbot added a comment.Jun 17 2020, 11:23 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-17T11:23:04Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.37/extensions/GrowthExperiments/extension.json: [[gerrit:606122|Fix NewcomerTask schema (T255597)]] (duration: 01m 04s)

ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.36; 2020-06-09); removed MW-1.35-notes (1.35.0-wmf.38; 2020-06-23).Jun 17 2020, 12:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 17 2020, 12:10 PM
Milimetric subscribed.Jun 17 2020, 1:15 PM

New events are being refined correctly with this schema, the hour that failed has been scheduled to re-run.

MNeisler moved this task from Triage to Tracking on the Product-Analytics board.Jun 17 2020, 3:21 PM
MNeisler added a subscriber: nettrom_WMF.Jun 17 2020, 3:29 PM
Ottomata added a comment.Jun 17 2020, 3:34 PM

This is actually valid syntax, although incorrect for the specific schema as they mean different things.

Ya valid JSONSchema but not valid for what we allow. All array items types must be specific.

Ottomata added a comment.Jun 17 2020, 3:35 PM

Thanks BTW :)

• fdans edited projects, added Analytics-Radar; removed Analytics.Jun 18 2020, 4:01 PM
• fdans subscribed.

@Tgr can you confirm the correct data is there?

Tgr added a subscriber: Etonkovidova.Jun 20 2020, 9:55 PM
In T255597#6232016, @Ottomata wrote:

Ya valid JSONSchema but not valid for what we allow. All array items types must be specific.

The items tuple notation could still achieve that together with additionalItems: false, so that documentation should probably be more specific about what is accepted.

In T255597#6235302, @fdans wrote:

@Tgr can you confirm the correct data is there?

@nettrom_WMF or @Etonkovidova would you be able to help out with that?

Ottomata added a comment.Jun 22 2020, 1:01 PM

that documentation should probably be more specific about what is accepted.

Ok, updated https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#arrays, does that sound better?

Tgr added a comment.Jun 22 2020, 1:20 PM
In T255597#6244375, @Ottomata wrote:

Ok, updated https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#arrays, does that sound better?

Yeah, thanks!

Etonkovidova added a comment.EditedJun 22 2020, 4:37 PM

@Tgr There are no validation errors in wmf.37 - I checked it on testwiki, but the error is still present in betalabs.

 EventLogging Validation: [NewcomerTask] Value null is the wrong type for property "maintenance_templates" (array expected)
EventLogging Validation: [NewcomerTask] Value null is the wrong type for property "revision_id" (integer expected) 

EventLogging Validation: [NewcomerTask] Value null is the wrong type for property "page_id" (integer expected)

Should it be corrected in betalabs?

Tgr added a comment.Jun 22 2020, 5:01 PM

That only happens because the page doesn't exist, which won't happen in production.

Etonkovidova closed this task as Resolved.Jun 22 2020, 6:01 PM
In T255597#6245490, @Tgr wrote:

That only happens because the page doesn't exist, which won't happen in production.

Right! Thanks.

Ottomata added a comment.Jun 22 2020, 7:15 PM

FYI, this was not a JSONSchema validation error, it was a Hive ingestion error. The JSONSchema did not fully specify the strict types of all elements of the array, so Hive could not declare a SQL type for the field. I think Fdans was asking to make sure that the data that Dan backfilled in Hive is in place correctly. If you aren't worried about that data because this instrumentation isn't fully in production yet, then no need to check. :)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits