Page MenuHomePhabricator
Create Task
Maniphest T276293

Test silverpop export with utf8mb4 tables
Closed, ResolvedPublic
Actions

Description

As we are converting the database tables to charset=utf8mb4, we want to test and rule out any issues in processes that use the DB. We need to test the export of the silverpop data to see if there are any differences in the export after charset conversion. None are anticipated, but it should be confirmed.

Current testing idea:

  • Copy the silverpop db/tables to a second instance.
  • Convert the db/tables in the second instance to utf8mb4
  • Change view silverpop export uses to point at converted data
  • Run silverpop export
  • Compare output CSV from converted data to that of unconverted
  • Change view back to original

Details

SubjectRepoBranchLines +/-
Switch general to unicode.wikimedia/fundraising/toolsmaster+1 -1
Fix a few more utf8s in silverpopwikimedia/fundraising/toolsmaster+5 -5
Shift from utf8 to utf8mb4 encoding for the DB tableswikimedia/fundraising/toolsmaster+68 -68
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dwisehaupt created this task.Mar 3 2021, 12:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2021, 12:14 AM
Dwisehaupt updated the task description. (Show Details)Mar 3 2021, 12:27 AM
Dwisehaupt added a comment.Mar 3 2021, 10:15 PM

The silverpop DB has been backed up and then restored as dev_silverpop on frdb1001. dev_silverpop and all of it's tables were then converted to utf8mb4. It is ready for testing the export script.

Dwisehaupt added a comment.Mar 3 2021, 10:27 PM

Looking at the table space, everything has been converted to utf8mb4 except the two views: dev_silverpop.silverpop_export_view and dev_silverpop.silverpop_export_view_full

That's kinda what we are testing here and they may need to be recreated to pick up the altered charsets as the conversion script didn't address views specifically.

Dwisehaupt moved this task from Triage to Up Next on the fundraising-tech-ops board.Mar 4 2021, 5:50 PM
• DStrine moved this task from Triage to Current Sprint on the Fundraising-Backlog board.Mar 8 2021, 9:26 PM
• DStrine added a project: Fundraising Sprint Esperantoland.
gerritbot added a comment.Mar 8 2021, 11:11 PM

Change 670000 had a related patch set uploaded (by Dwisehaupt; owner: Dwisehaupt):
[wikimedia/fundraising/tools@master] Shift from utf8 to utf8mb4 encoding for the DB tables

https://gerrit.wikimedia.org/r/670000

gerritbot added a project: Patch-For-Review.Mar 8 2021, 11:11 PM
• DStrine triaged this task as High priority.Mar 8 2021, 11:17 PM
gerritbot added a comment.Mar 11 2021, 1:02 AM

Change 670000 merged by Eileen:
[wikimedia/fundraising/tools@master] Shift from utf8 to utf8mb4 encoding for the DB tables

https://gerrit.wikimedia.org/r/670000

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2021, 1:10 AM
gerritbot added a comment.Mar 11 2021, 3:01 AM

Change 670632 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/tools@master] Fix a few more utf8s in silverpop

https://gerrit.wikimedia.org/r/670632

gerritbot added a project: Patch-For-Review.Mar 11 2021, 3:01 AM
Eileenmcnaughton subscribed.Mar 11 2021, 3:05 AM

I just did some testing and found just a few rows that were not switched to utf8mb4 - presumably the inconsistent caching caused them to slip through - I put up a patch for that (& hence moving this into review)

Regarding the view - that is constantly recreated with slightly different date parameters

Eileenmcnaughton moved this task from Backlog to Review on the Fundraising Sprint Esperantoland board.Mar 11 2021, 3:06 AM
Jgreen moved this task from Up Next to In Progress on the fundraising-tech-ops board.Mar 11 2021, 2:43 PM
gerritbot added a comment.Mar 11 2021, 4:47 PM

Change 670632 merged by jenkins-bot:
[wikimedia/fundraising/tools@master] Fix a few more utf8s in silverpop

https://gerrit.wikimedia.org/r/670632

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2021, 5:11 PM
gerritbot added a comment.Mar 12 2021, 6:48 PM

Change 671217 had a related patch set uploaded (by Cstone; owner: Cstone):
[wikimedia/fundraising/tools@master] Switch general to unicode.

https://gerrit.wikimedia.org/r/671217

gerritbot added a project: Patch-For-Review.Mar 12 2021, 6:48 PM
gerritbot added a comment.Mar 12 2021, 6:58 PM

Change 671217 merged by Cstone:
[wikimedia/fundraising/tools@master] Switch general to unicode.

https://gerrit.wikimedia.org/r/671217

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2021, 7:10 PM
Jgreen added a parent task: T268056: FY2020-21 Q3 maintenance window.Mar 12 2021, 7:42 PM
Dwisehaupt moved this task from In Progress to Watching on the fundraising-tech-ops board.Mar 15 2021, 9:51 PM

The work has been completed and silverpop runs are occurring. We are tracking an accuracy task in T277366 but there is currently nothing left to do on the ops side for the utf8mb4 conversion at the database or code for base functionality.

Eileenmcnaughton closed this task as Resolved.Mar 16 2021, 7:49 PM
Eileenmcnaughton claimed this task.
Eileenmcnaughton moved this task from Review to Done on the Fundraising Sprint Esperantoland board.
Jgreen moved this task from Watching to Done on the fundraising-tech-ops board.May 17 2021, 2:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL