Page MenuHomePhabricator
Create Task
Maniphest T226051

Fix mediawiki_page_restrictions_change table after migration to eventgate-main and schema version 1.0.0
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

In the 1.0.0 page-restrictions-change schema created for the migration to EventGate, I changed the page_restrictions field the unsupported patternProperties to our (newly) supported additionalProperties { type: string} AKA map type.

This works, except for the mediawiki_page_restrictions_change Hive table already has the page_restrictions field as a struct. I think this table probably should have previously been blacklisted from Refine. It will work fine now though.

We either need to drop all existing Hive data and start from scratch (fine with me, as AFAIK no one uses this table, and it wasn't really compatible with Refine in the first place...), OR convert all the existing data to the new schema. I'm not entirely sure how to do the conversion from struct to map in a Hive query, so we'll probably need to do a transform in Spark.

Related Objects
Search...

Event Timeline

Ottomata created this task.Jun 18 2019, 8:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 18 2019, 8:31 PM
Ottomata added a comment.Jun 19 2019, 1:37 PM

I'm pretty sure no one is using this table, so I'm going to make a decision to move the old data out of the way and start fresh. I'll keep the old data in a renamed table for a while, until we all discuss. If we decide to backfill, we can still do it.

Stashbot added a comment.Jun 19 2019, 1:41 PM

Mentioned in SAL (#wikimedia-analytics) [2019-06-19T13:41:07Z] <ottomata> renaming event.mediawiki_page_restrictions_change to event.mediawiki_page_restrictions_change_T226051 - T226051

Ottomata added a comment.Jun 19 2019, 1:44 PM
sudo -u analytics hdfs dfs -mv /wmf/data/event/mediawiki_page_restrictions_change /wmf/data/event/mediawiki_page_restrictions_change_T226051
ALTER TABLE event.mediawiki_page_restrictions_change RENAME TO event.mediawiki_page_restrictions_change_T226051;
ALTER TABLE event.mediawiki_page_restrictions_change_T226051 SET LOCATION "hdfs://analytics-hadoop/wmf/data/event/mediawiki_page_restrictions_change_T226051";
Ottomata triaged this task as High priority.Jun 19 2019, 1:44 PM
Ottomata set the point value for this task to 3.
Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.
Ottomata moved this task from Backlog to To be Estimated/To be discussed on the Event-Platform board.Jun 19 2019, 1:47 PM

Refine: Successfully refined 14 of 14 dataset partitions into table event.mediawiki_page_restrictions_change (total # refined records: 175)

Ottomata moved this task from Done to In Code Review on the Analytics-Kanban board.Jun 19 2019, 1:59 PM
Milimetric closed this task as Resolved.Jun 20 2019, 4:41 PM
Milimetric moved this task from In Code Review to Done on the Analytics-Kanban board.
Milimetric subscribed.

for the record: decided to drop old table, event.mediawiki_page_restrictions_change_T226051.

Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.Jun 20 2019, 4:42 PM
Milimetric moved this task from Operational Excellence to Event Platform on the Analytics board.
Ottomata added a comment.Jun 20 2019, 4:43 PM
hive (event)> drop table event.mediawiki_page_restrictions_change_t226051;

[@stat1004:/mnt/hdfs/wmf/data/event] $ sudo -u analytics hdfs dfs -rm -R /wmf/data/event/mediawiki_page_restrictions_change_T226051
19/06/20 16:43:29 INFO fs.TrashPolicyDefault: Moved: 'hdfs://analytics-hadoop/wmf/data/event/mediawiki_page_restrictions_change_T226051' to trash at: hdfs://analytics-hadoop/user/analytics/.Trash/Current/wmf/data/event/mediawiki_page_restrictions_change_T226051
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits