Page MenuHomePhabricator
Create Task
Maniphest T291234

Locale dependence of strtolower() and strtoupper()
Closed, ResolvedPublic
Actions

Description

MediaWiki often uses strtolower() and strtoupper() as if they do ASCII case folding, but in fact they are locale dependent.

PHP uses ASCII case folding for comparison, for example in strcasecmp(), but unfortunately does not expose an ASCII equivalent of strtolower().

strtolower() is not aware of UTF-8. It feeds the string into the glibc tolower() function one byte at a time.

I installed all glibc locales and compiled the results of strtolower() and strtoupper() for all byte values.

P17289 strtolower/strtoupper glibc data dump
1php > $locales = explode("\n", `locale -a`);
2php > $all = ''; for ( $i = 0; $i < 256; $i++ ) { $all .= chr($i); }
3php > $lcResults = [];
4php > $ucResults = [];
5php > foreach ( $locales as $locale ) { setlocale(LC_ALL, $locale); $lcResults[strtolower($all)][] = $locale; $ucResults[strtoupper($all)][] = $locale; }
6
7
8
9php > foreach ( $lcResults as $result => $names ) { print bin2hex($result) . "\t" . implode(',', $names) . "\n"; }
10000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff C,C.UTF-8,POSIX,aa_DJ.utf8,aa_ER,aa_ER@saaho,aa_ET,af_ZA.utf8,agr_PE,ak_GH,am_ET,an_ES.utf8,anp_IN,ar_AE,ar_AE.utf8,ar_BH,ar_BH.utf8,ar_DZ,ar_DZ.utf8,ar_EG,ar_EG.utf8,ar_IN,ar_IQ,ar_IQ.utf8,ar_JO,ar_JO.utf8,ar_KW,ar_KW.utf8,ar_LB,ar_LB.utf8,ar_LY,ar_LY.utf8,ar_MA,ar_MA.utf8,ar_OM,ar_OM.utf8,ar_QA,ar_QA.utf8,ar_SA,ar_SA.utf8,ar_SD,ar_SD.utf8,ar_SS,ar_SY,ar_SY.utf8,ar_TN,ar_TN.utf8,ar_YE,ar_YE.utf8,as_IN,ast_ES.utf8,ayc_PE,az_IR,be_BY.utf8,be_BY@latin,bem_ZM,ber_DZ,ber_MA,bg_BG.utf8,bhb_IN.utf8,bho_IN,bho_NP,bi_VU,bn_BD,bn_IN,bo_CN,bo_IN,br_FR.utf8,brx_IN,bs_BA.utf8,byn_ER,ca_AD.utf8,ca_ES.utf8,ca_ES@valencia,ca_FR.utf8,ca_IT.utf8,ce_RU,chr_US,ckb_IQ,cmn_TW,cs_CZ.utf8,csb_PL,cv_RU,cy_GB.utf8,da_DK.utf8,de_AT.utf8,de_BE.utf8,de_CH.utf8,de_DE.utf8,de_IT.utf8,de_LI.utf8,de_LU.utf8,doi_IN,dsb_DE,dv_MV,dz_BT,el_CY.utf8,el_GR.utf8,en_AG,en_AU.utf8,en_BW.utf8,en_CA.utf8,en_DK.utf8,en_GB.utf8,en_HK.utf8,en_IE.utf8,en_IL,en_IN,en_NG,en_NZ.utf8,en_PH.utf8,en_SC.utf8,en_SG.utf8,en_US.utf8,en_ZA.utf8,en_ZM,en_ZW.utf8,eo,eo_US.utf8,es_AR.utf8,es_BO.utf8,es_CL.utf8,es_CO.utf8,es_CR.utf8,es_CU,es_DO.utf8,es_EC.utf8,es_ES.utf8,es_GT.utf8,es_HN.utf8,es_MX.utf8,es_NI.utf8,es_PA.utf8,es_PE.utf8,es_PR.utf8,es_PY.utf8,es_SV.utf8,es_US.utf8,es_UY.utf8,es_VE.utf8,et_EE.utf8,eu_ES.utf8,eu_FR.utf8,fa_IR,ff_SN,fi_FI.utf8,fil_PH,fo_FO.utf8,fr_BE.utf8,fr_CA.utf8,fr_CH.utf8,fr_FR.utf8,fr_LU.utf8,fur_IT,fy_DE,fy_NL,ga_IE.utf8,gd_GB.utf8,gez_ER,gez_ER@abegede,gez_ET,gez_ET@abegede,gl_ES.utf8,gu_IN,gv_GB.utf8,ha_NG,hak_TW,he_IL,he_IL.utf8,hi_IN,hif_FJ,hne_IN,hr_HR.utf8,hsb_DE.utf8,ht_HT,hu_HU.utf8,hy_AM,ia_FR,id_ID.utf8,ig_NG,ik_CA,is_IS.utf8,it_CH.utf8,it_IT.utf8,iu_CA,ja_JP.eucjp,ja_JP.utf8,ka_GE.utf8,kab_DZ,kk_KZ.utf8,kl_GL.utf8,km_KH,kn_IN,ko_KR.euckr,ko_KR.utf8,kok_IN,ks_IN,ks_IN@devanagari,kw_GB.utf8,ky_KG,lb_LU,lg_UG.utf8,li_BE,li_NL,lij_IT,ln_CD,lo_LA,lt_LT.utf8,lv_LV.utf8,lzh_TW,mag_IN,mai_IN,mai_NP,mfe_MU,mg_MG.utf8,mhr_RU,mi_NZ.utf8,miq_NI,mjw_IN,mk_MK.utf8,ml_IN,mn_MN,mni_IN,mnw_MM,mr_IN,ms_MY.utf8,mt_MT.utf8,my_MM,nan_TW,nan_TW@latin,nb_NO.utf8,nds_DE,nds_NL,ne_NP,nhn_MX,niu_NU,niu_NZ,nl_AW,nl_BE.utf8,nl_NL.utf8,nn_NO.utf8,nr_ZA,nso_ZA,oc_FR.utf8,om_ET,om_KE.utf8,or_IN,os_RU,pa_IN,pa_PK,pap_AW,pap_CW,pl_PL.utf8,ps_AF,pt_BR.utf8,pt_PT.utf8,quz_PE,raj_IN,ro_RO.utf8,ru_RU.utf8,ru_UA.utf8,rw_RW,sa_IN,sah_RU,sat_IN,sc_IT,sd_IN,sd_IN@devanagari,sd_PK,se_NO,sgs_LT,shn_MM,shs_CA,si_LK,sid_ET,sk_SK.utf8,sl_SI.utf8,sm_WS,so_DJ.utf8,so_ET,so_KE.utf8,so_SO.utf8,sq_AL.utf8,sq_MK,sr_ME,sr_RS,sr_RS@latin,ss_ZA,st_ZA.utf8,sv_FI.utf8,sv_SE.utf8,sw_KE,sw_TZ,szl_PL,ta_IN,ta_LK,tcy_IN.utf8,te_IN,tg_TJ.utf8,th_TH,th_TH.utf8,the_NP,ti_ER,ti_ET,tig_ER,tk_TM,tl_PH.utf8,tn_ZA,to_TO,tpi_PG,ts_ZA,tt_RU,ug_CN,ug_CN@latin,uk_UA.utf8,unm_US,ur_IN,ur_PK,uz_UZ.utf8,uz_UZ@cyrillic,ve_ZA,vi_VN,wa_BE.utf8,wae_CH,wal_ET,wo_SN,xh_ZA.utf8,yi_US,yi_US.utf8,yo_NG,yue_HK,yuw_PG,zh_CN,zh_CN.gb18030,zh_CN.gbk,zh_CN.utf8,zh_HK,zh_HK.utf8,zh_SG,zh_SG.gbk,zh_SG.utf8,zh_TW,zh_TW.euctw,zh_TW.utf8,zu_ZA.utf8,
11000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6d7f8f9fafbfcfdfedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff aa_DJ,af_ZA,br_FR,ca_ES,da_DK,de_AT,de_BE,de_CH,de_DE,de_IT,de_LU,en_AU,en_BW,en_CA,en_DK,en_GB,en_HK,en_IE,en_NZ,en_PH,en_SG,en_US,en_ZA,en_ZW,es_AR,es_BO,es_CL,es_CO,es_CR,es_DO,es_EC,es_ES,es_GT,es_HN,es_MX,es_NI,es_PA,es_PE,es_PR,es_PY,es_SV,es_US,es_UY,es_VE,et_EE,eu_ES,eu_FR,fi_FI,fo_FO,fr_BE,fr_CA,fr_CH,fr_FR,fr_LU,ga_IE,gl_ES,gv_GB,id_ID,is_IS,it_CH,it_IT,kl_GL,kw_GB,ms_MY,nb_NO,nl_BE,nl_NL,nn_NO,oc_FR,om_KE,pt_BR,pt_PT,so_DJ,so_KE,so_SO,sq_AL,st_ZA,sv_FI,sv_SE,tl_PH,uz_UZ,wa_BE,xh_ZA,zu_ZA
12000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a8a7a8a9aaabacadaeafb0b1b2b3b8b5b6b7b8b9babbbdbdffbfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6d7f8f9fafbfcfdfedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff an_ES,ast_ES,br_FR@euro,ca_AD,ca_ES@euro,ca_FR,ca_IT,de_AT@euro,de_BE@euro,de_DE@euro,de_LU@euro,en_DK.iso885915,en_GB.iso885915,en_IE@euro,en_US.iso885915,es_ES@euro,et_EE.iso885915,eu_ES@euro,eu_FR@euro,fi_FI@euro,fr_BE@euro,fr_FR@euro,fr_LU@euro,ga_IE@euro,gd_GB,gl_ES@euro,it_IT@euro,mg_MG,nl_BE@euro,nl_NL@euro,pt_PT@euro,sv_FI@euro,sv_SE.iso885915,wa_BE@euro
13000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768496a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff az_AZ,crh_UA,ku_TR.utf8,tr_CY.utf8,tr_TR.utf8,tt_RU@iqtelif
14000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f908382838485868788899a8b9c9d9e9f909192939495969798999a9b9c9d9e9fa0a2a2bca4b4a6a7b8a9baabacadaebfb0b1b3b3b4b5b6b7b8b9babbbcbebebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeffe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff be_BY,bg_BG,kk_KZ.rk1048,ru_RU.cp1251
15000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0b1a2b3a4b5b6a7a8b9babbbcadbebfb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6d7f8f9fafbfcfdfedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff bs_BA,cs_CZ,hr_HR,hsb_DE,hu_HU,pl_PL,ro_RO,sk_SK,sl_SI
16000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a2a2a3a5a5aba7b8a9baabbcadaeffb1b1b3b3b5b5b6b9b8b9babfbcbebebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff cy_GB
17000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5dcb7dddedfbbfcbdfdfec0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1d2f3f4f5f6f7f8f9fafbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff el_CY,el_GR,el_GR@euro
18000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b3b3b5b5b7b7b9b9bbbbbdbdbfbfc1c1c3c3c5c5c7c7c9c9cbcbcdcdcfcfd1d1d3d3d5d5d7d7d9d9dbdbdddddfdfe1e1e3e3e5e5e7e7e9e9ebebededefeff1f1f3f3f5f5f7f7f9f9fbfbfdfdfeff hy_AM.armscii8
19000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788899a8b9c8d8e8f909192939495969798999a9b9c9d9effa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff ka_GE
20000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f9083ad838485998998899a9b9c9d9e9f909192939495969798999a9b9c9d9e9fa0a2a2bcb5b4b1a7b8a9baabacadaebfb0b1b3b3b4b5b6b7b8b9babbbcbebebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeffe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff kk_KZ
21000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768fd6a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6d7f8f9fafbfc69fedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff ku_TR,tr_CY,tr_TR
22000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0b1b2b3b4b5b6a7b8b9babbbcadbebfb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff lg_UG
23000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7b8a9baabacadaebfb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6d7f8f9fafbfcfdfedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff lt_LT,lv_LV,mi_NZ
24000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0f1f2f3f4f5f6f7f8f9fafbfcadfeffd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeefd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff mk_MK,ru_RU
25000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0b1a2a3a4a5b6a7a869babbbcadaebfb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2c3e4e5e6e7e8e9eaebecedeeefd0f1f2f3f4f5f6d7f8f9fafbfcfdfedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff mt_MT
26000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2a3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf ru_RU.koi8r
27000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2a3a4b5a6a7b8b9babbbcadbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf ru_UA,uk_UA
28000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f406162636465666768696a6b6c6d6e6f707172737475767778797a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182818485868788898a8b8a8d8d8f809192939495969798999a9b9c9d9e9fa0a1a1a3a4a5a6a7a8a9aaabacadaeafb0b1b2a3b4a5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf tg_TJ
29
30
31
32php > foreach ( $ucResults as $result => $names ) { print bin2hex($result) . "\t" . implode(',', $names) . "\n"; }
33000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff C,C.UTF-8,POSIX,aa_DJ.utf8,aa_ER,aa_ER@saaho,aa_ET,af_ZA.utf8,agr_PE,ak_GH,am_ET,an_ES.utf8,anp_IN,ar_AE,ar_AE.utf8,ar_BH,ar_BH.utf8,ar_DZ,ar_DZ.utf8,ar_EG,ar_EG.utf8,ar_IN,ar_IQ,ar_IQ.utf8,ar_JO,ar_JO.utf8,ar_KW,ar_KW.utf8,ar_LB,ar_LB.utf8,ar_LY,ar_LY.utf8,ar_MA,ar_MA.utf8,ar_OM,ar_OM.utf8,ar_QA,ar_QA.utf8,ar_SA,ar_SA.utf8,ar_SD,ar_SD.utf8,ar_SS,ar_SY,ar_SY.utf8,ar_TN,ar_TN.utf8,ar_YE,ar_YE.utf8,as_IN,ast_ES.utf8,ayc_PE,az_IR,be_BY.utf8,be_BY@latin,bem_ZM,ber_DZ,ber_MA,bg_BG.utf8,bhb_IN.utf8,bho_IN,bho_NP,bi_VU,bn_BD,bn_IN,bo_CN,bo_IN,br_FR.utf8,brx_IN,bs_BA.utf8,byn_ER,ca_AD.utf8,ca_ES.utf8,ca_ES@valencia,ca_FR.utf8,ca_IT.utf8,ce_RU,chr_US,ckb_IQ,cmn_TW,cs_CZ.utf8,csb_PL,cv_RU,cy_GB.utf8,da_DK.utf8,de_AT.utf8,de_BE.utf8,de_CH.utf8,de_DE.utf8,de_IT.utf8,de_LI.utf8,de_LU.utf8,doi_IN,dsb_DE,dv_MV,dz_BT,el_CY.utf8,el_GR.utf8,en_AG,en_AU.utf8,en_BW.utf8,en_CA.utf8,en_DK.utf8,en_GB.utf8,en_HK.utf8,en_IE.utf8,en_IL,en_IN,en_NG,en_NZ.utf8,en_PH.utf8,en_SC.utf8,en_SG.utf8,en_US.utf8,en_ZA.utf8,en_ZM,en_ZW.utf8,eo,eo_US.utf8,es_AR.utf8,es_BO.utf8,es_CL.utf8,es_CO.utf8,es_CR.utf8,es_CU,es_DO.utf8,es_EC.utf8,es_ES.utf8,es_GT.utf8,es_HN.utf8,es_MX.utf8,es_NI.utf8,es_PA.utf8,es_PE.utf8,es_PR.utf8,es_PY.utf8,es_SV.utf8,es_US.utf8,es_UY.utf8,es_VE.utf8,et_EE.utf8,eu_ES.utf8,eu_FR.utf8,fa_IR,ff_SN,fi_FI.utf8,fil_PH,fo_FO.utf8,fr_BE.utf8,fr_CA.utf8,fr_CH.utf8,fr_FR.utf8,fr_LU.utf8,fur_IT,fy_DE,fy_NL,ga_IE.utf8,gd_GB.utf8,gez_ER,gez_ER@abegede,gez_ET,gez_ET@abegede,gl_ES.utf8,gu_IN,gv_GB.utf8,ha_NG,hak_TW,he_IL,he_IL.utf8,hi_IN,hif_FJ,hne_IN,hr_HR.utf8,hsb_DE.utf8,ht_HT,hu_HU.utf8,hy_AM,ia_FR,id_ID.utf8,ig_NG,ik_CA,is_IS.utf8,it_CH.utf8,it_IT.utf8,iu_CA,ja_JP.eucjp,ja_JP.utf8,ka_GE.utf8,kab_DZ,kk_KZ.utf8,kl_GL.utf8,km_KH,kn_IN,ko_KR.euckr,ko_KR.utf8,kok_IN,ks_IN,ks_IN@devanagari,kw_GB.utf8,ky_KG,lb_LU,lg_UG.utf8,li_BE,li_NL,lij_IT,ln_CD,lo_LA,lt_LT.utf8,lv_LV.utf8,lzh_TW,mag_IN,mai_IN,mai_NP,mfe_MU,mg_MG.utf8,mhr_RU,mi_NZ.utf8,miq_NI,mjw_IN,mk_MK.utf8,ml_IN,mn_MN,mni_IN,mnw_MM,mr_IN,ms_MY.utf8,mt_MT.utf8,my_MM,nan_TW,nan_TW@latin,nb_NO.utf8,nds_DE,nds_NL,ne_NP,nhn_MX,niu_NU,niu_NZ,nl_AW,nl_BE.utf8,nl_NL.utf8,nn_NO.utf8,nr_ZA,nso_ZA,oc_FR.utf8,om_ET,om_KE.utf8,or_IN,os_RU,pa_IN,pa_PK,pap_AW,pap_CW,pl_PL.utf8,ps_AF,pt_BR.utf8,pt_PT.utf8,quz_PE,raj_IN,ro_RO.utf8,ru_RU.utf8,ru_UA.utf8,rw_RW,sa_IN,sah_RU,sat_IN,sc_IT,sd_IN,sd_IN@devanagari,sd_PK,se_NO,sgs_LT,shn_MM,shs_CA,si_LK,sid_ET,sk_SK.utf8,sl_SI.utf8,sm_WS,so_DJ.utf8,so_ET,so_KE.utf8,so_SO.utf8,sq_AL.utf8,sq_MK,sr_ME,sr_RS,sr_RS@latin,ss_ZA,st_ZA.utf8,sv_FI.utf8,sv_SE.utf8,sw_KE,sw_TZ,szl_PL,ta_IN,ta_LK,tcy_IN.utf8,te_IN,tg_TJ.utf8,th_TH,th_TH.utf8,the_NP,ti_ER,ti_ET,tig_ER,tk_TM,tl_PH.utf8,tn_ZA,to_TO,tpi_PG,ts_ZA,tt_RU,ug_CN,ug_CN@latin,uk_UA.utf8,unm_US,ur_IN,ur_PK,uz_UZ.utf8,uz_UZ@cyrillic,ve_ZA,vi_VN,wa_BE.utf8,wae_CH,wal_ET,wo_SN,xh_ZA.utf8,yi_US,yi_US.utf8,yo_NG,yue_HK,yuw_PG,zh_CN,zh_CN.gb18030,zh_CN.gbk,zh_CN.utf8,zh_HK,zh_HK.utf8,zh_SG,zh_SG.gbk,zh_SG.utf8,zh_TW,zh_TW.euctw,zh_TW.utf8,zu_ZA.utf8,
34000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6f7d8d9dadbdcdddeff aa_DJ,af_ZA,br_FR,ca_ES,da_DK,de_AT,de_BE,de_CH,de_DE,de_IT,de_LU,en_AU,en_BW,en_CA,en_DK,en_GB,en_HK,en_IE,en_NZ,en_PH,en_SG,en_US,en_ZA,en_ZW,es_AR,es_BO,es_CL,es_CO,es_CR,es_DO,es_EC,es_ES,es_GT,es_HN,es_MX,es_NI,es_PA,es_PE,es_PR,es_PY,es_SV,es_US,es_UY,es_VE,et_EE,eu_ES,eu_FR,fi_FI,fo_FO,fr_BE,fr_CA,fr_CH,fr_FR,fr_LU,ga_IE,gl_ES,gv_GB,id_ID,is_IS,it_CH,it_IT,kl_GL,kw_GB,ms_MY,nb_NO,nl_BE,nl_NL,nn_NO,oc_FR,om_KE,pt_BR,pt_PT,so_DJ,so_KE,so_SO,sq_AL,st_ZA,sv_FI,sv_SE,tl_PH,uz_UZ,wa_BE,xh_ZA,zu_ZA
35000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a6a9aaabacadaeafb0b1b2b3b4b5b6b7b4b9babbbcbcbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6f7d8d9dadbdcdddebe an_ES,ast_ES,br_FR@euro,ca_AD,ca_ES@euro,ca_FR,ca_IT,de_AT@euro,de_BE@euro,de_DE@euro,de_LU@euro,en_DK.iso885915,en_GB.iso885915,en_IE@euro,en_US.iso885915,es_ES@euro,et_EE.iso885915,eu_ES@euro,eu_FR@euro,fi_FI@euro,fr_BE@euro,fr_FR@euro,fr_LU@euro,ga_IE@euro,gd_GB,gl_ES@euro,it_IT@euro,mg_MG,nl_BE@euro,nl_NL@euro,pt_PT@euro,sv_FI@euro,sv_SE.iso885915,wa_BE@euro
36000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748694a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff az_AZ,crh_UA,ku_TR.utf8,tr_CY.utf8,tr_TR.utf8,tt_RU@iqtelif
37000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182818485868788898a8b8c8d8e8f809192939495969798998a9b8c8d8e8fa0a1a1a3a4a5a6a7a8a9aaabacadaeafb0b1b2b2a5b5b6b7a8b9aabba3bdbdafc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf be_BY,bg_BG,kk_KZ.rk1048,ru_RU.cp1251
38000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0a1b2a3b4a5a6b7b8a9aaabacbdaeafc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6f7d8d9dadbdcdddeff bs_BA,cs_CZ,hr_HR,hsb_DE,hu_HU,pl_PL,ro_RO,sk_SK,sl_SI
39000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a1a3a4a4a6a7a8a9aaa6acadaeafb0b0b2b2b4b4b6b7a8b7aabbacbdbdbbc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddeaf cy_GB
40000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbb6b8b9bae0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d3d3d4d5d6d7d8d9dadbbcbebfff el_CY,el_GR,el_GR@euro
41000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b2b4b4b6b6b8b8bababcbcbebec0c0c2c2c4c4c6c6c8c8cacaccccceced0d0d2d2d4d4d6d6d8d8dadadcdcdedee0e0e2e2e4e4e6e6e8e8eaeaececeeeef0f0f2f2f4f4f6f6f8f8fafafcfcfeff hy_AM.armscii8
42000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798998a9b8c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfe9f ka_GE
43000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182818485868788878a8b8c8d8e8f809192939495969788868a8b8c8d8e8fa0a1a1a3a4a5a6a7a8a9aaabac82aeafb0a6b2b2a5a4b6b7a8b9aabba3bdbdafc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf kk_KZ
44000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748dd4a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6f7d8d9dadbdc49deff ku_TR,tr_CY,tr_TR
45000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0a1a2a3a4a5a6b7a8a9aaabacbdaeafc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddeff lg_UG
46000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7a8b9aabbbcbdbeafc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6f7d8d9dadbdcdddeff lt_LT,lv_LV,mi_NZ
47000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecff0a1a2a3a4a5a6a7a8a9aaabacfdaeaf mk_MK,ru_RU
48000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0a1b2b3b4b5a6b7b849aaabacbdbeafc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfc0c1c2e3c4c5c6c7c8c9cacbcccdcecff0d1d2d3d4d5d6f7d8d9dadbdcdddeff mt_MT
49000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2b3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeffe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff ru_RU.koi8r
50000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2b3b4a5b6b7a8a9aaabacbdaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeffe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff ru_UA,uk_UA
51000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f604142434445464748494a4b4c4d4e4f505152535455565758595a7b7c7d7e7f908382838485868788898c8b8c8e8e8f909192939495969798999a9b9c9d9e9fa0a2a2b3a4b5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeffe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff tg_TJ

The UTF-8 variant locales are better behaved than the legacy locales. For example, fr-FR converts \xc0 to \xe0, whereas fr-FR.utf8 leaves all non-ASCII characters alone. The only exceptions are the Turkish/Kurdish locales, which mangle ASCII strings due to their handling of dotted and dotless i, even in UTF-8 locales.

I think this is an upstream bug, and I'll post about it upstream. I think strtolower() and strtoupper() should be ASCII since locale-dependence is almost always a bug. Without UTF-8 support, there is not much hope of successfully case folding non-English strings with these functions.

Encouragingly, at https://bugs.php.net/bug.php?id=67815 , nikic wrote "My general recommendation is to avoid locales and locale-dependent functions, as locales are a fundamentally broken concept." I agree with that and I would like PHP to provide locale-independent functions.

In the meantime, MediaWiki should not allow installation with $wgShellLocale set to Turkish/Kurdish or a value that causes non-ASCII characters to be mangled.

Details

SubjectRepoBranchLines +/-
Bump parsoid to 0.15.0-a1mediawiki/vendormaster+401 -440
Remove $wgShellLocale, always use Cmediawiki/coremaster+27 -230
Roll our own *nix shell escaping functionmediawiki/libs/Shellboxmaster+87 -5
Set an appropriate locale in maintenance scriptsmediawiki/services/parsoidmaster+4 -0
Bump parsoid to 0.14.0-a19mediawiki/vendormaster+1 K -1 K
Replace preg_replace_callback() with strtolower()mediawiki/services/parsoidmaster+6 -9
Verify that strtolower() works for all byte valuesmediawiki/services/parsoidmaster+13 -6
Customize query in gerrit

Related Objects

Event Timeline

• tstarling created this task.Sep 17 2021, 1:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2021, 1:44 AM
gerritbot added a comment.Sep 17 2021, 3:22 AM

Change 721650 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/services/parsoid@master] Verify that strtolower() works for all byte values

https://gerrit.wikimedia.org/r/721650

gerritbot added a project: Patch-For-Review.Sep 17 2021, 3:22 AM
• tstarling added a comment.Sep 17 2021, 3:24 AM

I posted to php internals: https://news-web.php.net/php.internals/116051

gerritbot added a comment.Sep 17 2021, 3:35 AM

Change 721651 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/services/parsoid@master] Replace preg_replace_callback() with strtolower()

https://gerrit.wikimedia.org/r/721651

gerritbot added a comment.Sep 17 2021, 3:42 AM

Change 721650 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Verify that strtolower() works for all byte values

https://gerrit.wikimedia.org/r/721650

gerritbot added a comment.Sep 17 2021, 1:41 PM

Change 721651 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Replace preg_replace_callback() with strtolower()

https://gerrit.wikimedia.org/r/721651

Maintenance_bot removed a project: Patch-For-Review.Sep 17 2021, 2:10 PM
gerritbot added a comment.Sep 20 2021, 7:03 PM

Change 722416 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/vendor@master] Bump parsoid to 0.14.0-a19

https://gerrit.wikimedia.org/r/722416

gerritbot added a project: Patch-For-Review.Sep 20 2021, 7:03 PM
Arlolra edited projects, added Parsoid (Tracking); removed Parsoid.Sep 20 2021, 7:34 PM
gerritbot added a comment.Sep 20 2021, 9:31 PM

Change 722416 merged by jenkins-bot:

[mediawiki/vendor@master] Bump parsoid to 0.14.0-a19

https://gerrit.wikimedia.org/r/722416

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2021, 10:10 PM
• tstarling added a comment.Sep 21 2021, 2:30 AM

The history of locale dependence in MediaWiki:

  • 2002-2008: The golden age of blissful ignorance
  • August 2008: PHP 5.2.6 introduced locale-dependence to escapeshellarg(). With the locale set to "C", the typical situation, the result was that non-ASCII characters would be silently stripped from the return value of escapeshellarg(). This is https://bugs.php.net/bug.php?id=45132 . We talked with the maintainers who considered it to not be a bug. The reasoning was that locale-sensitivity in escapeshellarg() is the correct way to interface with a locale-sensitive shell. Our response (6e0d253f6d1ec93732f14f5bbe04f67bc7524725) was to unconditionally set the locale to en_US.UTF-8 in Setup.php.
  • September 2008: We had a report (T16944) that a certain server did not have English locales installed. It only had German. So we added code to the installer to detect a suitable installed locale, with the LANG environment variable being given high priority. The detected locale was $wgShellLocale, and setlocale() was called on demand on the first call to wfEscapeShellArg(). (c69d5d46e82030139af4d2785c1cfcb6041a0d3e)
  • May 2017: In T107128 it was reported that, with the locale set to a natural language like English or German, Lua was sorting strings alphabetically instead of asciibetically, i.e. with a<B. For Lua to do string processing in the expected way, like every other Lua installation, it is necessary to use a C-like locale. Our response was to "recommend" C.UTF-8, and to move the setlocale() call back to Setup.php, but unfortunately the LANG environment variable was still given high priority in the installer. fb6f4d35e757c502fb9358226ef366328b73d6b9

I think we should remove $wgShellLocale and just do:

if ( !setlocale( LC_ALL, 'C.UTF-8' ) ) {
	setlocale( LC_ALL, 'C' );
}

The Shell layer will be responsible for forwarding the locale in the environment and escaping shell arguments correctly. The difficulty of forwarding arguments from MediaWiki to a shell in an arbitrary locale can be resolved by forcing the shell's locale to be C or C.UTF-8.

• tstarling added a comment.Sep 21 2021, 5:32 AM

Note that PHP in release 8.0 stopped respecting the LC_CTYPE set by the environment. It is now "C" by default instead of inheriting from the environment.

The security issue that PHP 5.2.6 was fixing was as follows: if the locale is set to zh_CN.gbk, a byte in the range \x81-\xFE can hide the following backslash from the shell. It was stated that the attacker may be in control of the environment locale. There is a risk of reintroducing this security vulnerability if we call setlocale() without setting the environment variable.

I was able to reproduce this. The issue does not affect dash, the default /bin/sh on Debian-like distros. By replacing /bin/sh with bash, I obtained:

$ LANG=zh_CN.gbk php7.4 -a
Interactive mode enabled

php > passthru('echo ' . escapeshellcmd("\xe1\x81\x81\\;id"));
၁\;id
php > setlocale(LC_CTYPE,'C.UTF-8');
php > passthru('echo ' . escapeshellcmd("\xe1\x81\x81\\;id"));
၁\\
uid=33(www-data) gid=33(www-data) groups=33(www-data)

escapeshellarg() is not affected, because the character to be eaten must be in the range \x40 - \xFE and it must immediately follow controllable user input. Backslash is in that range, but single quote is not.

gerritbot added a comment.Sep 21 2021, 7:46 AM

Change 722548 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/libs/Shellbox@master] Roll our own *nix shell escaping function

https://gerrit.wikimedia.org/r/722548

gerritbot added a project: Patch-For-Review.Sep 21 2021, 7:46 AM
gerritbot added a comment.Sep 22 2021, 2:34 AM

Change 722705 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] Remove $wgShellLocale, always use C

https://gerrit.wikimedia.org/r/722705

Arlolra mentioned this in T234049: "Error loading data from server: apierror-visualeditor-docserver-http-error: (curl error: 7) Couldn't connect to server.".Sep 23 2021, 7:15 PM
gerritbot added a comment.Sep 23 2021, 10:52 PM

Change 723327 had a related patch set uploaded (by Arlolra; author: Arlolra):

[mediawiki/services/parsoid@master] Set an appropriate locale in maintenance scripts

https://gerrit.wikimedia.org/r/723327

gerritbot added a comment.Sep 24 2021, 4:25 AM

Change 723327 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Set an appropriate locale in maintenance scripts

https://gerrit.wikimedia.org/r/723327

gerritbot added a comment.Sep 24 2021, 4:56 PM

Change 722548 merged by jenkins-bot:

[mediawiki/libs/Shellbox@master] Roll our own *nix shell escaping function

https://gerrit.wikimedia.org/r/722548

gerritbot added a comment.Sep 25 2021, 12:48 AM

Change 722705 merged by jenkins-bot:

[mediawiki/core@master] Remove $wgShellLocale, always use C

https://gerrit.wikimedia.org/r/722705

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.2; 2021-09-28).Sep 25 2021, 1:00 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 25 2021, 1:10 AM
gerritbot added a comment.Sep 27 2021, 6:18 PM

Change 724142 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/vendor@master] Bump parsoid to 0.15.0-a1

https://gerrit.wikimedia.org/r/724142

gerritbot added a project: Patch-For-Review.Sep 27 2021, 6:18 PM
gerritbot added a comment.Sep 27 2021, 9:53 PM

Change 724142 merged by jenkins-bot:

[mediawiki/vendor@master] Bump parsoid to 0.15.0-a1

https://gerrit.wikimedia.org/r/724142

Maintenance_bot removed a project: Patch-For-Review.Sep 27 2021, 10:10 PM
• tstarling closed this task as Resolved.Oct 19 2023, 6:16 AM
• tstarling claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL