Page MenuHomePhabricator

Migrate legacy metawiki schemas to Event Platform
Open, Needs TriagePublic

Description

This task will track the migration of EventLogging schemas & stream so to Event Platform schemas.

Tracking and planning of what schemas to migrate is being done in the EventLogging Schema Migration Audit spreadsheet.

Explanation of what this means for legacy EventLogging schema owners:
https://wikitech.wikimedia.org/wiki/Event_Platform/EventLogging_legacy

We will do our best to migrate schemas in groups associated with teams.

Schemas produced by EventLogging extension

Schemas produced by other software


Migration plan for a schema:

1. Pick a schema to migrate
Schemas to migrate are listed in EventLogging Schema Migration Audit spreadsheet.

2. Create a new task to track this schema's migration

# This should work on MacOS to open a new Phab Task form in a browser with some fields already field out.
function new_el_migration_phab_task() {
    schema_name="$1"  
    open "https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?title=$schema_name Event Platform Migration&description=See: https://wikitech.wikimedia.org/wiki/Event_Platform/EventLogging_legacy

Unless otherwise notified, client IP and consequently geocoded data will no longer be collected for this event data after this migration. Please let us know if this should continue to be captured.  See also T262626.& &parent=259163&tags=Event-Platform&subscribers=Ottomata,Mforns"
}

new_el_migration_phab_task SearchSatisfaction

Link this task in the EventLogging Schema Migration Audit spreadsheet.

On the task, contact the owner of the schema and ask if they need client IP and/or geocoded data in the Hive table.

3. Create /analytics/legacy/<schemaname>/current.yaml schema

Using eventlogging_legacy_schema_convert script) in the schemas/event/secondary repository:

old_schema_name=SearchSatisfaction
new_schema_name=$(echo $old_schema_name | tr '[:upper:]' '[:lower:]')
mkdir ./jsonschema/analytics/legacy/$new_schema_name
node ./scripts/eventlogging_legacy_schema_convert.js $old_schema_name > ./jsonschema/analytics/legacy/$new_schema_name/current.yaml

You'll need to edit at least the JSONSchema examples in current.yaml. Easiest thing to do is get an event out of Kafka and use that as as staring point.

# Get the last event out of Kafka
kafkacat -C -b kafka-jumbo1001.eqiad.wmnet -o -1 -c 1 -t eventlogging_SearchSatisfaction

If the schema owner indicated that they need client IP and/or geocoded data in Hive, you'll need to add a $ref to the fragment/http/client_ip schema. Example here.

(!) Please, make sure that npm test is not thowing any errors other than snake_case inconsistencies (those are allowed for legacy schemas).
To do that, you need to first comment out L21 in test/jsonschema/repository.test.js, and then run npm test for your new schema to be checked.
When done, remember to revert test/jsonschema/repository.test.js to its original state before commiting!

4. Edit-protect the metawiki Schema page at https://meta.wikimedia.org/wiki/Schema:$old_schema_name

Use this as the edit-protect log message:

This schema has been moved to https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/legacy. See also https://wikitech.wikimedia.org/wiki/Event_Platform/EventLogging_legacy

5. Manually evolve the Hive table to use new schema
Once the above schema is merged (you might have to wait 10 minutes after merge for Spark to be able to retrieve it):

old_schema_name=SearchSatisfaction
new_schema_name=$(echo $old_schema_name | tr '[:upper:]' '[:lower:]')
table="event.${new_schema_name}"
schema_uri="/analytics/legacy/${new_schema_name}/latest"

# First run in dry-run mode (the default) to see what EvolveHiveTable will do.
spark2-submit --conf spark.driver.extraClassPath=/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/srv/deployment/analytics/refinery/artifacts/hive-jdbc-1.1.0-cdh5.10.0.jar:/srv/deployment/analytics/refinery/artifacts/hive-service-1.1.0-cdh5.10.0.jar --driver-java-options='-Dhttp.proxyHost=webproxy.eqiad.wmnet -Dhttp.proxyPort=8080 -Dhttps.proxyHost=webproxy.eqiad.wmnet -Dhttps.proxyPort=8080' --class org.wikimedia.analytics.refinery.job.refine.tool.EvolveHiveTable  /srv/deployment/analytics/refinery/artifacts/refinery-job.jar --table="${table}" --schema_uri="${schema_uri}"

# If that looks good, evolve the table:
spark2-submit --conf spark.driver.extraClassPath=/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/srv/deployment/analytics/refinery/artifacts/hive-jdbc-1.1.0-cdh5.10.0.jar:/srv/deployment/analytics/refinery/artifacts/hive-service-1.1.0-cdh5.10.0.jar --driver-java-options='-Dhttp.proxyHost=webproxy.eqiad.wmnet -Dhttp.proxyPort=8080 -Dhttps.proxyHost=webproxy.eqiad.wmnet -Dhttps.proxyPort=8080' --class org.wikimedia.analytics.refinery.job.refine.tool.EvolveHiveTable  /srv/deployment/analytics/refinery/artifacts/refinery-job.jar --table="${table}" --schema_uri="${schema_uri}" --dry_run=false

6. Add entry to wgEventStreams, wgEventLoggingStreamNames and wgEventLoggingSchemas in operations/mediwiki-config
Rolling deploy changes to make EventLogging extension produce data to EventGate. Example: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/607333/2/wmf-config/InitialiseSettings.php

To test that events make it through the pipeline:
Check that events for your stream are still flowing through in this Grafana dashboard, or by consuming the eventlogging_$old_schema_name topic from Kafka.

You can submit an event via a browser developer console:

old_schema_name=Test;
event = { "OtherMessage" => "Hello from JS" } // example event here;
mw.eventLog.logEvent(old_schema_name, event);

Or, if a server side PHP EventLogging event, you can emit a test event with mwscript shell.php on deployment.eqiad.wmnet:

cd /srv/mediawiki-staging
mwscript shell.php --wiki testwiki

>>> $old_schema_name = 'Test';
>>> $event = [ "OtherMessage" => "Hello from PHP" ];
>>> EventLogging::logEvent( $old_schema_name, -1, $event );

7. Once the legacy stream's data is fully produced through EventGate, switch to using Refine job that uses schema repo instead of meta.wm.org

Also add the SchemaName to the eventlogging-processor disabled schemas list in puppet in modules/eventlogging/files/plugins.py**

Example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/644259

This will prevent eventlogging-processor from producing what are now invalid legacy events from clients that are running old code.

Restart eventlogging-processor on eventlog1002:

sudo puppet agent -t  # make sure the change has been applied
sudo service eventlogging-processor@client-side-* restart

8. Edit the producer extension.json and set EventLoggingSchemas to the new schema URI
Example: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentTranslation/+/639578

9. Once the producer extension.json is fully deployed, edit wgEventLoggingSchemas in operations/mediawiki-config InitialiseSettings.php and remove the schema's entry.
Example: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/639579

10. Mark the schema as migrated in the EventLogging Schema Migration Audit spreadsheet

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+0 -2
operations/mediawiki-configmaster+0 -0
operations/mediawiki-configmaster+1 -4
mediawiki/extensions/EventLoggingwmf/1.36.0-wmf.18+7 -7
mediawiki/extensions/EventLoggingwmf/1.36.0-wmf.16+7 -7
mediawiki/extensions/EventLoggingwmf/1.36.0-wmf.16+7 -7
mediawiki/extensions/EventLoggingmaster+7 -7
operations/deployment-chartsmaster+231 -231
eventgate-wikimediamaster+106 -5
operations/mediawiki-configmaster+11 -0
mediawiki/extensions/ContentTranslationmaster+1 -1
operations/puppetproduction+1 -0
schemas/event/secondarymaster+64 -51
schemas/event/secondarymaster+0 -3
schemas/event/secondarymaster+505 -0
schemas/event/secondarymaster+3 -3
operations/puppetproduction+1 -0
operations/mediawiki-configmaster+1 -3
operations/mediawiki-configmaster+11 -0
schemas/event/secondarymaster+487 -1
operations/mediawiki-configmaster+0 -1
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenOttomata
Openjlinehan
OpenOttomata
ResolvedGilles
Openmforns
DeclinedOttomata
ResolvedOttomata
Openmforns
ResolvedMholloway
OpenOttomata
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
ResolvedMholloway
DuplicateNone
OpenOttomata
OpenNone
OpenNone
OpenNone
Openmforns
OpenOttomata
Openmforns
ResolvedOttomata
OpenOttomata
DeclinedNone
DeclinedNone
Openbmansurov
Openmforns
ResolvedOttomata
ResolvedOttomata
OpenOttomata
DuplicateNone
ResolvedOttomata
OpenMNeisler
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 639578 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/ContentTranslation@master] Set default ContentTranslationAbuseFilter version to Event Platform schema URI

https://gerrit.wikimedia.org/r/639578

Change 639579 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Remove wgEventLoggingSchemas ContentTranslationAbuseFilter override

https://gerrit.wikimedia.org/r/639579

Change 639578 merged by jenkins-bot:
[mediawiki/extensions/ContentTranslation@master] Set default ContentTranslationAbuseFilter version to Event Platform schema URI

https://gerrit.wikimedia.org/r/639578

Ottomata updated the task description. (Show Details)

I've added Product Analytics so the team's aware of this, we have our board refinement coming up today. I also see that our team members get subscribed to the child tasks as they're created so they're individually aware of them, thanks for doing that!

Change 639621 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Default dt to now

https://gerrit.wikimedia.org/r/639621

Change 639622 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventLogging@master] Allow EventGate to default dt to server side receive time

https://gerrit.wikimedia.org/r/639622

Change 639621 merged by Ottomata:
[eventgate-wikimedia@master] Default dt to server side receive time for legacy EventLogging events

https://gerrit.wikimedia.org/r/639621

Change 641266 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-* - Bump eventgate-wikimedia version to 2020-11-16-212345-production

https://gerrit.wikimedia.org/r/641266

Change 641266 merged by Ottomata:
[operations/deployment-charts@master] eventgate-* - Bump eventgate-wikimedia version to 2020-11-16-212345-production

https://gerrit.wikimedia.org/r/641266

Change 639622 merged by Ottomata:
[mediawiki/extensions/EventLogging@master] Allow EventGate to default dt to server side receive time

https://gerrit.wikimedia.org/r/639622

Change 641799 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.16] Allow EventGate to default dt to server side receive time

https://gerrit.wikimedia.org/r/641799

Change 641799 abandoned by Ottomata:
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.16] Allow EventGate to default dt to server side receive time

Reason:
bad change-id

https://gerrit.wikimedia.org/r/641799

Change 641801 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.16] Allow EventGate to default dt to server side receive time

https://gerrit.wikimedia.org/r/641801

Change 641768 had a related patch set uploaded (by Urbanecm; owner: Ottomata):
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.18] Allow EventGate to default dt to server side receive time

https://gerrit.wikimedia.org/r/641768

Change 641801 merged by jenkins-bot:
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.16] Allow EventGate to default dt to server side receive time

https://gerrit.wikimedia.org/r/641801

Change 641768 merged by jenkins-bot:
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.18] Allow EventGate to default dt to server side receive time

https://gerrit.wikimedia.org/r/641768

Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)

Change 654425 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Remove overrides from wgEventLoggingSchemas

https://gerrit.wikimedia.org/r/654425

Change 654425 merged by Ottomata:
[operations/mediawiki-config@master] Remove overrides from wgEventLoggingSchemas

https://gerrit.wikimedia.org/r/654425

@sdkim @mforns @jlinehan FYI I've added a few extra sheets in our EventLogging Audit Spreadsheet: To Do, Done, To Decomission and Unknown. Marcel and I can work from the To Do sheet more easily now I think.

Change 639579 abandoned by Ottomata:
[operations/mediawiki-config@master] Remove wgEventLoggingSchemas ContentTranslationAbuseFilter override

Reason:
Done in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /654425

https://gerrit.wikimedia.org/r/639579

Change 657688 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Remove migrated EventLoggingSchemas overrides

https://gerrit.wikimedia.org/r/657688

Change 657688 merged by Ottomata:
[operations/mediawiki-config@master] Remove migrated EventLoggingSchemas overrides

https://gerrit.wikimedia.org/r/657688

Mentioned in SAL (#wikimedia-operations) [2021-01-25T14:37:46Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Remove 2 Remove migrated EventLoggingSchemas overrides - T259163, T267352 (duration: 00m 56s)

Ottomata updated the task description. (Show Details)