jsn opened https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/37

Draft: Add MP fragment schema for MassDelete worflows

Note that I have some WIP on this, but I had to ask some clarifying questions (which were promptly answered) and then set it down to work on other things; I should finish up my patches for this task next week.

moving to review column to reflect that I'm waiting on external review

Change #1123435 had a related patch set uploaded (by Jsn.sherman; author: Jsn.sherman):

[operations/mediawiki-config@master] [WIP] Add MP event stream for MassDelete workflows

https://gerrit.wikimedia.org/r/1123435

@Samwalton9-WMF Question:

Now that that the form is publicly available, should we also collect more info about the user, eg. user groups or edit bucket?

we discussed today; we'll only collect data for those who can perform mass deletes

Change #1123435 merged by jenkins-bot:

[operations/mediawiki-config@master] Add MP event stream for MassDelete workflows

https://gerrit.wikimedia.org/r/1123435

Mentioned in SAL (#wikimedia-operations) [2025-03-12T14:15:26Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1123435|Add MP event stream for MassDelete workflows (T382147)]], [[gerrit:1126648|Enable SUL3 signup for 50% of group 2 users (T384218)]], [[gerrit:rMW11269562e8f6|[enwiki] Throttle exemption for event (T388637)]]

Mentioned in SAL (#wikimedia-operations) [2025-03-12T14:18:48Z] <tgr@deploy2002> jsn, tgr, superpes: Backport for [[gerrit:1123435|Add MP event stream for MassDelete workflows (T382147)]], [[gerrit:1126648|Enable SUL3 signup for 50% of group 2 users (T384218)]], [[gerrit:rMW11269562e8f6|[enwiki] Throttle exemption for event (T388637)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-03-12T14:26:30Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1123435|Add MP event stream for MassDelete workflows (T382147)]], [[gerrit:1126648|Enable SUL3 signup for 50% of group 2 users (T384218)]], [[gerrit:rMW11269562e8f6|[enwiki] Throttle exemption for event (T388637)]] (duration: 11m 04s)

Hi @jsn.sherman !

By default, streams have canary_events_enabled.
https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Hadoop_Event_Ingestion_Lifecycle#Canary_Events

This mean that (unless you disable canary events) when you declare a stream, artificial canary events will automatically start being produced.
If the stream does not have a schema, these canary events will fail.

Add MP schema for MassDelete workflows has not yet been merged, but Add MP event stream for MassDelete workflows (1123435) has.

Generally, schemas be deployed before any usages of that schema (a stream) is declared.

This is currently causing canary event failures and alerts.

ASAP, we should do one of:

• Rollback Add MP event stream for MassDelete workflows (1123435)
• set canary_events_enabled: false for the 'mediawiki.product_metrics.ext_massdelete' stream.
• Merge the ext_massdelete schema MR.

jsn merged https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/37

Add MP schema for MassDelete workflows

In T382147#10633968, @CodeReviewBot wrote:

jsn merged https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/37

Add MP schema for MassDelete workflows

This seems to be causing unrelated test failures on other patches...

https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/jobs/466271 for https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/49

1) Schema Repository Structure in ./jsonschema/
     Schema with title analytics/mediawiki/product_metrics/ext_massdelete
       current version 1.0.0 must be equal to materialized latest version 1.0.0:
    AssertionError [ERR_ASSERTION]: Dereferenced current schema does not equal latest schema version 1.0.0
    + expected - actual
             "status_code": -6305039478318694
           }
           "instrument_name": "dolor"
           "massdelete_filter": {
    -        "date_from": "2013-01-22"
    -        "date_to": "2013-01-22"
    +        "date_from": "2013-03-19"
    +        "date_to": "2013-03-19"
             "limit": -6305039478318694
             "namespaces": [
               -6305039478318694
             ]
    
    at Context.<anonymous> (node_modules/@wikimedia/jsonschema-tools/lib/tests/structure.js:120:28)

It looks like the issue is the expected date in the date properties in the examples section; is there a proper placeholder value I should be using in the schema? it seems to expect the date to be the current date when the schema has an arbitrary old date. Is this me doing something wrong in the schema? Or should the CI be able to accept a past date in an example block?

It's the current date... But in 2013? Which feels even more odd

Very strange.

This test passed in Jason's MR:
https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/jobs/431225#L3359

...

If a schema does not have examples, and shouldGenerateExamples is enabled (it is), then https://github.com/json-schema-faker/json-schema-faker is used to generate the examples.

This test is a sanity check to make sure that whatever current.yaml has it exactly what the materialized versioned schema file has.

Something must be going wrong here.

yes, I was just poking around json-schema-tools and wondering where the options were set for the test run; this smells like a ci code issue rather than a schema problem. Having said that, I'm not discounting the possibility that I messed up somehow.

I think I see it. I'm not exactly sure why this is happening, but:

fragment/analytics/product_metrics/ext_massdelete/current.yaml examples is a bit weird:

examples:
  - massdelete_filter:
      target: dolor
      pattern: dolor
      limit: 0
      namespaces:
        - 0
      date_from: '2025-01-01'
      date_to: '2025-01-01'
  - massdelete_response_time_ms: 500

- massdelete_response_time_ms is its own example entry.

I think you've got a misplaced '-' in the yaml!

This probably should have failed the examples must validate against schema test, but the schema doesn't have any required properties, and massdelete_response_time_ms is an actual field, so it is valid!

I don't know why an example is being generated during this test at all though...

Maybe there is a subtle race condition in the test when it dereferences and materializes current.yaml to compare against the materialized file. Somehow maybe it is checking the current.yaml file for examples, not seeing any, generating them, THEN dereferencing and merging schemas together?

Reading the code I don't see it.

• https://github.com/wikimedia/jsonschema-tools/blob/master/lib/tests/structure.js#L116
• https://github.com/wikimedia/jsonschema-tools/blob/master/lib/jsonschema-tools.js#L821-L842

@Ottomata Can I just update the existing schema version or do I need to increment? We haven't deployed any instruments against this yet.

In T382147#10653298, @Ottomata wrote:

I think I see it. I'm not exactly sure why this is happening, but:

fragment/analytics/product_metrics/ext_massdelete/current.yaml examples is a bit weird:

examples:
  - massdelete_filter:
      target: dolor
      pattern: dolor
      limit: 0
      namespaces:
        - 0
      date_from: '2025-01-01'
      date_to: '2025-01-01'
  - massdelete_response_time_ms: 500

- massdelete_response_time_ms is its own example entry.

I think you've got a misplaced '-' in the yaml!

This probably should have failed the examples must validate against schema test, but the schema doesn't have any required properties, and massdelete_response_time_ms is an actual field, so it is valid!

I don't know why an example is being generated during this test at all though...

I updated that example, but I'm now seeing this error too:
https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/compare/master...Jsn.sherman%2FT382147-followup

7904 passing (7s)
302 pending
1 failing

1) Schema Repository Structure in ./jsonschema/
     Schema with title analytics/mediawiki/product_metrics/ext_massdelete
       current version 1.0.0 must be equal to materialized latest version 1.0.0:

    AssertionError [ERR_ASSERTION]: Dereferenced current schema does not equal latest schema version 1.0.0
    + expected - actual

             "status_code": -6305039478318694
           }
           "instrument_name": "dolor"
           "massdelete_filter": {
    -        "date_from": "2013-01-22"
    -        "date_to": "2013-01-22"
    +        "date_from": "2013-03-20"
    +        "date_to": "2013-03-20"
             "limit": -6305039478318694
             "namespaces": [
               -6305039478318694
             ]

    at Context.<anonymous> (node_modules/@wikimedia/jsonschema-tools/lib/tests/structure.js:120:28)

In T382147#10657176, @jsn.sherman wrote:

@Ottomata Can I just update the existing schema version or do I need to increment? We haven't deployed any instruments against this yet.

I answered my own question; incrementing the version resolves the issue with the updated example; thanks @Ottomata!

jsn opened https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/50

MP fragment schema for MassDelete: fix example

Hm, technically you can just update, this only changes the example, so as long as rematerialize all the right files (the fragment and the dependent concrete schemas).

Usually this may require a restart of eventgate, as eventgate assumes schema versions are immutable. In this case, the examples are not used by eventgate, so I think it will be fine!

Oh! incrementing the version is fine too.

@Ottomata @jsn.sherman this issue is blocking us from merging a schema update (https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/49), would it be possible to prioritize fixing this early this week?

In T382147#10667434, @kostajh wrote:

@Ottomata @jsn.sherman this issue is blocking us from merging a schema update (https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/49), would it be possible to prioritize fixing this early this week?

We'll get it merged today; I'm so used to the Gerrit +2 workflow that I forgot that approved != merged in gitlab.

In T382147#10667722, @jsn.sherman wrote:

In T382147#10667434, @kostajh wrote:

@Ottomata @jsn.sherman this issue is blocking us from merging a schema update (https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/49), would it be possible to prioritize fixing this early this week?

We'll get it merged today; I'm so used to the Gerrit +2 workflow that I forgot that approved != merged in gitlab.

So, I went to add changelogs since I incremented the schema versions and found that the build was failing again. Once again, the issue seemed to be that CI is expecting today's date, but in 2013

1) Schema Repository Structure in ./jsonschema/
     Schema with title analytics/mediawiki/product_metrics/ext_massdelete
       current version 1.0.1 must be equal to materialized latest version 1.0.1:
    AssertionError [ERR_ASSERTION]: Dereferenced current schema does not equal latest schema version 1.0.1
    + expected - actual
             "status_code": -6305039478318694
           }
           "instrument_name": "dolor"
           "massdelete_filter": {
    -        "date_from": "2013-03-20"
    -        "date_to": "2013-03-20"
    +        "date_from": "2013-03-24"
    +        "date_to": "2013-03-24"
             "limit": -6305039478318694
             "namespaces": [
               -6305039478318694
             ]

https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/jobs/470075

Updating the date to match today once again resolved it:
https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/jobs/470076

This makes me doubt that the example was the source of the issue. Any thoughts @Reedy or @Ottomata?

Odd.. The example was obviously definitely wrong, but beyond that...

Very strange.

Updating the date to match today once again resolved it:

What does this mean? What did you change?

What is the difference between b55d14dede745df1f007c86fe8c6c870c2af462b and eb02a28694e2f88503866165e77db07d4ff260d8?

Hm, you could consider manually editing the 1.0.0 versions and fixing their examples too. Just in case?

I don't see how that could affect this test though...

literally just the example dates in
https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/50/diffs#55ef7678e8de301e759e79f8a3476bc9489eac40_0_597
https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/50/diffs#55ef7678e8de301e759e79f8a3476bc9489eac40_0_598

those dates were generated as 2013-03-20 when I ran the scripts last week and passed. The same npm run test command failed locally and in CI today. I updated both to 2013-03-24 by hand and now npm run test is passing locally and in CI.

^ sounds like it will pass until tomorrow!

I wonder if the issue might have something to do with the fragment examples generation vs the concrete event schema's example generation!

I'll see if I can find something...

Sorry yall, this is quite a strange issue that we have not encountered before.

Ah ha! I found it.

https://github.com/wikimedia/jsonschema-tools/blob/master/lib/jsonschema-tools.js#L166-L171

We have a static value used for format: date-time example generation, but not for format: date. .massdelete_filter is format: date.

Creating a task and will submit a patch.

otto opened https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/51

jsonschema-tools upgrade to 1.5.0 to fix non deterministic CI test

otto closed https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/51

jsonschema-tools upgrade to 1.5.0 to fix non deterministic CI test

otto merged https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/50

MP fragment schema for MassDelete: fix example

otto opened https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/13

jsonschema-tools upgrade to 1.5.0 to fix non deterministic CI test

otto merged https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/13

jsonschema-tools upgrade to 1.5.0 to fix non deterministic CI test

Okay, T389881: Bug: jsonschema-tools generates non deterministic examples for date format fields is done. I also fixed and merged @jsn.sherman's MR 50 in schemas-event-secondary.

@kostajh @Reedy please rebase your MRs and confirm the fix! Thank you!