Change Details

Sqoop https://www.mediawiki.org/wiki/Extension:CheckUser/cu_changes_table This sqoop is a bit different from the "grab everything and overwrite" approach of our current sqoop script. We need to make sure we have a full month of raw cu_changes data, process it, and then delete it. According to Asaf, the filter we need is: * no bots (in previous geowiki this was done with both user_groups and a static list of bots) * include anonymous users * only editing activity (no admin activities, codes 0 and 1 - see https://www.mediawiki.org/wiki/Manual:Recentchanges_table#rc_type) * include all projects (geowiki only included wikipedia)

Sqoop https://www.mediawiki.org/wiki/Extension:CheckUser/cu_changes_table This sqoop is a bit different from the "grab everything and overwrite" approach of our current sqoop script. We need to make sure we have a full month of raw cu_changes data, process it, and then delete it. According to Asaf, the filter we need is: * no bots (in previous geowiki this was done with both user_groups and a static list of bots) * include anonymous users * only editing activity (no admin activities, codes 0 and 1 - see https://www.mediawiki.org/wiki/Manual:Recentchanges_table#rc_type) * include all projects (geowiki only included wikipedia) (scope creep: the patch that resolves this ended up fixing the sqoop script to handle logging and error reporting better, and do more robust success flag setting)

Sqoop https://www.mediawiki.org/wiki/Extension:CheckUser/cu_changes_table This sqoop is a bit different from the "grab everything and overwrite" approach of our current sqoop script. We need to make sure we have a full month of raw cu_changes data, process it, and then delete it. According to Asaf, the filter we need is: * no bots (in previous geowiki this was done with both user_groups and a static list of bots) * include anonymous users * only editing activity (no admin activities, codes 0 and 1 - see https://www.mediawiki.org/wiki/Manual:Recentchanges_table#rc_type) * include all projects (geowiki only included wikipedia) (scope creep: the patch that resolves this ended up fixing the sqoop script to handle logging and error reporting better, and do more robust success flag setting)