Page MenuHomePhabricator

Japanese names not all rendering correctly in eoy receipt
Closed, ResolvedPublic

Description

Donor civicrm/contact/view?reset=1&cid=46153568 has been in touch with DS about their name being incorrectly transposed to question marks

cids
15965100 Kanji text
46153568 Hiragana text

bad string looks like ???様

(the last bit is the 'san' that we have in our text)

@KHancock99 has investigated and found at least one other instance. I have turned jobs off to investigate

Event Timeline

Eileenmcnaughton triaged this task as Unbreak Now! priority.Jan 5 2021, 3:44 AM
Eileenmcnaughton created this task.

I've been writing a test to catch this but the issue is the table

CREATE TABLE `wmf_eoy_receipt_donor` (
  `job_id` int(10) unsigned DEFAULT NULL,
  `email` varchar(254) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL,
  `preferred_language` varchar(16) DEFAULT NULL,
  `name` varchar(255) DEFAULT NULL,
  `status` varchar(255) DEFAULT NULL,
  `contributions_rollup` text,
  UNIQUE KEY `wmf_eoy_receipt_donor_job_id_email` (`job_id`,`email`),
  KEY `job_id` (`job_id`),
  KEY `email` (`email`),
  KEY `status` (`status`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |

that's no kind of utf8

Change 654375 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Boilerplate update

https://gerrit.wikimedia.org/r/654375

Change 654376 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Add unit test for Japanese language fix

https://gerrit.wikimedia.org/r/654376

DStrine lowered the priority of this task from Unbreak Now! to Medium.Jan 5 2021, 10:10 PM

Change 654375 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Boilerplate update

https://gerrit.wikimedia.org/r/654375

Change 654376 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Add unit test for Japanese language fix

https://gerrit.wikimedia.org/r/654376

Change 654703 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Update ja eoy email to remove the salutation

https://gerrit.wikimedia.org/r/654703

Just to summarise where we are at. At the moment the job is running but Japanese names are excluded. We sent out 818 emails with question marks replacing some or all of their names. These are mostly but not all Japanese. I think this is resolved now as the Japanese names are excluded for T271189 not this issue

@MBeat33 FYI

count(*)preferred_language

+----------+--------------------+

26en_CN
2en_IL
4en_JP
1en_NL
1en_PL
2en_RO
1en_TH
1en_TW
4en_UA
1en_US
10he_IL
1ja_CN
756ja_JP
1pl_PL
1ru_LU
3ru_UA
1uk_UA
1zh_HANS
1zh_HANT

Query to find sent emails with mangled names

SELECT utffail.* FROM wmf_eoy_receipt_donor donor
  INNER JOIN (
  SELECT DISTINCT name, status, eoy.preferred_language, eoy.email, names.first_name
  FROM wmf_eoy_receipt_donor eoy
       LEFT JOIN (
    SELECT first_name, e.email
    FROM wmf_eoy_receipt_donor dd
         LEFT JOIN civicrm.civicrm_email e ON e.email = dd.email
         LEFT JOIN civicrm.civicrm_contact c ON c.id = e.contact_id
      AND first_name = name
  ) AS mails ON mails.email = eoy.email AND first_name = name
       LEFT JOIN (
    SELECT first_name, e.email
    FROM wmf_eoy_receipt_donor dd
         LEFT JOIN civicrm.civicrm_email e ON e.email = dd.email
         LEFT JOIN civicrm.civicrm_contact c ON c.id = e.contact_id
  ) AS names ON names.email = eoy.email

  WHERE mails.first_name IS NULL
    AND names.first_name IS NOT NULL
) as utffail ON utffail.email = donor.email
AND donor.status = 'Sent'
 ;