Page MenuHomePhabricator
Create Task
Maniphest T212625

Prepare and check storage layer for hywwiki
Closed, ResolvedPublic
Actions

Description

  • Project: Wikipedia.
  • Wiki type: public.

Thank you.

Related Objects
Search...

Event Timeline

MarcoAurelio created this task.Dec 26 2018, 3:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 26 2018, 3:19 PM
• Marostegui triaged this task as Medium priority.Dec 26 2018, 3:28 PM
• Marostegui edited projects, added Data-Services; removed Cloud-Services.
• Marostegui subscribed.

Let us know when this is created so we can sanitize it and give the green light to Cloud Team!
Thanks

• Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board.Dec 26 2018, 3:28 PM
bd808 moved this task from Backlog to Wiki replicas on the Data-Services board.Dec 27 2018, 1:09 AM
• Marostegui mentioned this in T212781: Strategy for how to change the "cu_changes" DB schema.Jan 2 2019, 2:27 PM
Liuxinyu970226 subscribed.Jan 15 2019, 1:08 AM
DavidSaroyan subscribed.Jan 17 2019, 1:03 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:57 PM
Liuxinyu970226 unsubscribed.Feb 15 2019, 1:47 PM
MarcoAurelio added a comment.Mar 27 2019, 2:03 PM

@Marostegui Although the wiki config had to be reverted the addWiki script was run so I guess the tables are now in place?

• Marostegui added a comment.Mar 27 2019, 2:05 PM

So those tables will not be dropped + created again as part of any other process and will remain there until the wiki config is in place?

Stashbot added a comment.Mar 27 2019, 2:12 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-27T14:12:01Z] <marostegui> Sanitize hywwiki on db1124:3313 T212625

Ladsgroup subscribed.Mar 27 2019, 2:17 PM
In T212625#5061983, @Marostegui wrote:

So those tables will not be dropped + created again as part of any other process and will remain there until the wiki config is in place?

The tables are created already and it's very very likely that they won't dropped unless the reason for the issue that prevented us from serving traffic was some sort of data corruption which is pretty unlikely. I will keep you posted.

• Marostegui claimed this task.Mar 27 2019, 2:20 PM

I have sanitized hywwiki on db1124:3313 and triggers are now in place and filtered tables deleted.

mysql.py -h db1124:3313 hywwiki -e "show triggers\G"
*************************** 1. row ***************************
             Trigger: abuse_filter_log_insert
               Event: INSERT
               Table: abuse_filter_log
           Statement: SET NEW.afl_ip = ''
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 2. row ***************************
             Trigger: abuse_filter_log_update
               Event: UPDATE
               Table: abuse_filter_log
           Statement: SET NEW.afl_ip = ''
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 3. row ***************************
             Trigger: archive_insert
               Event: INSERT
               Table: archive
           Statement: SET NEW.ar_comment_id = 0
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 4. row ***************************
             Trigger: archive_update
               Event: UPDATE
               Table: archive
           Statement: SET NEW.ar_comment_id = 0
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 5. row ***************************
             Trigger: recentchanges_insert
               Event: INSERT
               Table: recentchanges
           Statement: SET NEW.rc_ip = ''
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 6. row ***************************
             Trigger: recentchanges_update
               Event: UPDATE
               Table: recentchanges
           Statement: SET NEW.rc_ip = ''
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 7. row ***************************
             Trigger: revision_insert
               Event: INSERT
               Table: revision
           Statement: SET NEW.rev_text_id = 0
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 8. row ***************************
             Trigger: revision_update
               Event: UPDATE
               Table: revision
           Statement: SET NEW.rev_text_id = 0
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 9. row ***************************
             Trigger: user_insert
               Event: INSERT
               Table: user
           Statement: SET NEW.user_password = '', NEW.user_newpassword = '', NEW.user_email = '', NEW.user_token = '', NEW.user_email_authenticated = '', NEW.user_email_token = '', NEW.user_email_token_expires = '', NEW.user_newpass_time = ''
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary
*************************** 10. row ***************************
             Trigger: user_update
               Event: UPDATE
               Table: user
           Statement: SET NEW.user_password = '', NEW.user_newpassword = '', NEW.user_email = '', NEW.user_token = '', NEW.user_email_authenticated = '', NEW.user_email_token = '', NEW.user_email_token_expires = '', NEW.user_newpass_time = ''
              Timing: BEFORE
             Created: NULL
            sql_mode: IGNORE_BAD_TABLE_OPTIONS
             Definer: root@localhost
character_set_client: utf8
collation_connection: utf8_general_ci
  Database Collation: binary

@Ladsgroup please ping me once the config is live, so I can actually test and check that the private data is being filtered before handling this for the views creation.

MarcoAurelio added a comment.EditedMar 28 2019, 11:39 AM

https://hyw.wikipedia.org is now up. However there's on https://hyw.wikipedia.org/wiki/%D4%B3%D5%AC%D5%AD%D5%A1%D6%82%D5%B8%D6%80_%D5%A7%D5%BB (only page affected so far - it's the Main Page)

[XJyyFQpAAD0AAEF-ZM8AAAAD] 2019-03-28 11:37:57: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"

I guess this will get fixed once everything is finished? Someone can look it up for the trace in logstash to know what's missing :)

• Marostegui added a comment.Mar 28 2019, 11:39 AM

We are investigating on -databases

• Marostegui added a comment.Mar 28 2019, 6:03 PM

This is no longer throwing errors from what I can see.
@Ladsgroup you fixed it in the end?

I can also see 3 users there already created and well filtered on labs.

Ladsgroup added a comment.Mar 28 2019, 6:05 PM

No it got reverted. It was sending fatal errors.

• Marostegui added a comment.Mar 28 2019, 6:12 PM

Ok - let me know when it is attempted again. Thank you!

• Marostegui added a comment.Apr 1 2019, 5:58 AM

This wiki is triggering some false positives on our labs private data checking methods, even if it is correctly sanitized (T212625#5062038) it is still not appearing on s3.dblist etc...can we either fully finish this process or revert it (as in, dropping the database until we are ready for this to be done)?

• Marostegui added a comment.Apr 1 2019, 3:17 PM
In T212625#5072845, @Marostegui wrote:

This wiki is triggering some false positives on our labs private data checking methods, even if it is correctly sanitized (T212625#5062038) it is still not appearing on s3.dblist etc...can we either fully finish this process or revert it (as in, dropping the database until we are ready for this to be done)?

This is now fixed after the wiki has been correctly created.
I have re-run the check_private_data script on sanitarium and it looks clean. I have also created a user on the wiki and it has arrived correctly filtered to labs hosts.

• Marostegui moved this task from Blocked external/Not db team to Done on the DBA board.Apr 1 2019, 3:24 PM
• Marostegui added subscribers: • Bstorm, bd808, • GTirloni.

cloud-services-team this is ready for the views creation.
I have added the usual GRANT so the views can be created for this wiki:

Please remember that labsdb1012 is the new host and also needs the views.

bd808 edited projects, added cloud-services-team (Kanban); removed cloud-services-team.Apr 1 2019, 3:25 PM
bd808 moved this task from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
• Marostegui removed • Marostegui as the assignee of this task.Apr 1 2019, 3:26 PM
• GTirloni unsubscribed.Apr 3 2019, 10:21 AM
• Bstorm claimed this task.Apr 10 2019, 8:53 AM
• Bstorm closed this task as Resolved.Apr 10 2019, 6:04 PM

All set from WMCS end. Confirmed I can run queries from Toolforge.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL