Page MenuHomePhabricator
Paste P73601

Page load time regression on mobile page views since May 2024 (Jan 2023 - Feb 2025)
ActivePublic
Actions

Authored by Krinkle on Feb 25 2025, 11:33 PM.
Tags
None
Referenced Files
F67092193: Page load time regression on mobile page views since May 2024 (Jan 2023 - Feb 2025)
Oct 25 2025, 1:35 AM
F66831154: Page load time regression on mobile page views since May 2024 (Jan 2023 - Feb 2025)
Oct 24 2025, 2:29 AM
F66761784: Page load time regression on mobile page views since May 2024 (Jan 2023 - Feb 2025)
Oct 19 2025, 9:27 AM
F58495628: Page load time regression on mobile page views since May 2024 (Jan 2023 - Feb 2025)
Feb 26 2025, 12:16 AM
F58495244: Page load time regression on mobile page views since May 2024 (Jan 2023 - Feb 2025)
Feb 25 2025, 11:33 PM
Subscribers
None
# Context: https://phabricator.wikimedia.org/T214998
#
# These SQL queries were run in Hadoop.
# p50 fetchStart from mobile pageviews
_fetc _resp
2023-01 7975400 14.0 275.0
2023-02 6832976 15.0 288.0
2023-03 7493825 14.0 277.0
2023-04 7270219 13.0 268.0
2023-05 7550402 13.0 268.0
2023-06 7181867 13.0 271.0
2023-07 7659863 13.0 260.0
2023-08 7712761 13.0 270.0
2023-09 7310588 14.0 282.0
2023-10 7867392 14.0 276.0
2023-11 7551501 14.0 269.0
2023-12 7897229 13.0 260.0
2024-01 8683274 13.0 253.0
2024-02 7769574 14.0 266.0
2024-03 8239057 14.0 276.0
2024-04 7829965 15.0 287.0
2024-05 8347648 48.0 326.0 # <-- 3X increase in fetchStart
2024-06 7946155 75.0 337.0
2024-07 8274818 68.0 328.0
2024-08 8456235 67.0 332.0
2024-09 7941604 71.0 335.0
2024-10 8368568 71.0 329.0
2024-11 8190066 68.0 323.0
2024-12 8418583 65.0 317.0
2025-01 9007119 65.0 314.0
2025-02 5614294 65.0 314.0
SELECT CONCAT(year, '-', LPAD(month, 2, '0')) _dt, COUNT(*), ROUND(PERCENTILE(event['fetchStart'],0.5)) _fetchStart, ROUND(PERCENTILE(event['responseStart'],0.5)) _responseStart FROM event_sanitized.NavigationTiming WHERE (year=2023 OR year=2024 OR year=2025) AND event['action']='view' AND event['isAnon']=true AND event['isOversample']=false AND event['mobileMode']='stable' GROUP BY year, month;
# p75 fetchStart from mobile pageviews
_fetc _resp
2023-01 7975400 49.0 555.0
2023-02 6832976 58.0 578.0
2023-03 7493825 53.0 553.0
2023-04 7270219 46.0 539.0
2023-05 7550402 49.0 541.0
2023-06 7181867 53.0 548.0
2023-07 7659863 52.0 535.0
2023-08 7712761 57.0 542.0
2023-09 7310588 61.0 549.0
2023-10 7867392 61.0 544.0
2023-11 7551501 63.0 530.0
2023-12 7897229 62.0 515.0
2024-01 8683274 62.0 506.0
2024-02 7769574 66.0 525.0
2024-03 8239057 73.0 545.0
2024-04 7829965 83.0 559.0
2024-05 8347648 240.0 617.0 # <-- 3X increase in fetchStart
2024-06 7946155 279.0 634.0
2024-07 8274818 265.0 616.0
2024-08 8456235 262.0 619.0
2024-09 7941604 267.0 627.0
2024-10 8368568 264.0 614.0
2024-11 8190066 256.0 604.0
2024-12 8418583 246.0 593.0
2025-01 9007119 244.0 587.0
2025-02 6823968 245.0 588.0
# p80 fetchStart from mobile pageviews
_fetc _resp
2023-01 7975400 79.0 646.0
2023-02 6832976 94.0 673.0
2023-03 7493825 87.0 642.0
2023-04 7270219 77.0 627.0
2023-05 7550402 81.0 628.0
2023-06 7181867 87.0 637.0
2023-07 7659863 85.0 623.0
2023-08 7712761 93.0 629.0
2023-09 7310588 98.0 635.0
2023-10 7867392 97.0 632.0
2023-11 7551501 97.0 616.0
2023-12 7897229 95.0 597.0
2024-01 8683274 95.0 586.0
2024-02 7769574 100.0 609.0
2024-03 8239057 109.0 631.0
2024-04 7829965 122.0 645.0
2024-05 8347648 307.0 708.0 # <-- 3X increase in fetchStart
2024-06 7946155 346.0 726.0
2024-07 8274818 329.0 706.0
2024-08 8456235 324.0 708.0
2024-09 7941604 330.0 717.0
2024-10 8368568 327.0 707.0
2024-11 8190066 318.0 695.0
2024-12 8418583 307.0 682.0
2025-01 9007119 304.0 676.0
2025-02 5626270 306.0 677.0
# Focus: de.wikipedia.org and id.wikipedia.org
#
# Why German and Indonesian Wikipedia?
# - Rule out other factors that may influence the global average.
# - Not super close to our datacenters (RTT is more pronounced),
# - Nearest datacenter has not changed between 2023-01 and 2025-02.
# Eqsin/Singapore rollout was earlier in 2017.
# Drmrs/France rollout was earlier in 2022.
# Unaffected by Magru/Brazil rollout in 2024.
# - Germany and Indonesia have fairly large audiences (enough sample data).
# - Germany and Indonesia have a strongly correlated wiki. That is,
# -- Germany has its own language wiki.
# -- A majority of visits from Germany are to dewiki.
# -- A majority of visits to dewiki are from Germany.
# -- Idem for Indonesia and idwiki.
#
# E.g. there aren't other popular languages/wikis in the country,
# and there aren't many speakers outside this one country.
# This would e.g. not apply to English or Spanish Wikipedia
# (speakers in many countries), or India (multiple languages
# in common use, including many visiting English Wikipedia).
#
# Why by wiki instead of country?
# - Our long-term sanitized data does not preserve event['geoCountry'].
# This would increase entropy too much and risk de-anonymization.
#
# Why p75?
# - When picking a single country, we have much less sample data,
# so to reduce influence from month-to-month noise we use p75 instead
# of p80 or p90.
# - The Performance Team dashboards and alerts for this dataset have
# standardised on the p75 since at least 2015.
#
# p75 fetchStart from mobile pageviews to dewiki
_fetch _resp
2023-01 391544 43.0 339.0 dewiki
2023-02 319487 46.0 352.0 dewiki
2023-03 346836 43.0 340.0 dewiki
2023-04 347635 39.0 344.0 dewiki
2023-05 342303 40.0 354.0 dewiki
2023-06 323037 45.0 372.0 dewiki
2023-07 356045 43.0 371.0 dewiki
2023-08 363428 44.0 392.0 dewiki
2023-09 327851 44.0 410.0 dewiki
2023-10 367309 45.0 385.0 dewiki
2023-11 345900 47.0 370.0 dewiki
2023-12 383224 47.0 369.0 dewiki
2024-01 416993 47.0 358.0 dewiki
2024-02 362904 50.0 381.0 dewiki
2024-03 390202 57.0 396.0 dewiki
2024-04 362993 65.0 423.0 dewiki
2024-05 387999 167.0 455.0 dewiki # <-- 3X increase in fetchStart
2024-06 379729 190.0 446.0 dewiki
2024-07 387702 185.0 457.0 dewiki
2024-08 391007 188.0 454.0 dewiki
2024-09 366258 184.0 473.0 dewiki
2024-10 398222 180.0 436.0 dewiki
2024-11 369194 177.0 429.0 dewiki
2024-12 416490 171.0 412.0 dewiki
2025-01 440976 173.0 412.0 dewiki
2025-02 327002 176.0 420.0 dewiki
# p75 fetchStart from mobile pageviews to idwiki
_fetc _resp
2023-01 54967 90.0 801.0 idwiki
2023-02 50572 96.0 805.0 idwiki
2023-03 55267 85.0 736.0 idwiki
2023-04 49057 66.0 710.0 idwiki
2023-05 56054 81.0 698.0 idwiki
2023-06 47876 84.0 709.0 idwiki
2023-07 50765 90.0 702.0 idwiki
2023-08 57704 102.0 696.0 idwiki
2023-09 54319 100.0 691.0 idwiki
2023-10 59540 99.0 672.0 idwiki
2023-11 59514 105.0 657.0 idwiki
2023-12 53417 97.0 638.0 idwiki
2024-01 64088 109.0 650.0 idwiki
2024-02 64117 110.0 652.0 idwiki
2024-03 60226 134.0 691.0 idwiki # <-- 40% increase in fetchStart
2024-04 55597 142.0 684.0 idwiki # <-- 50% increase in fetchStart
2024-05 65118 355.0 787.0 idwiki # <-- 3X increase in fetchStart
2024-06 54498 376.0 795.0 idwiki
2024-07 59667 375.0 771.0 idwiki
2024-08 67497 380.0 756.0 idwiki
2024-09 63182 364.0 755.0 idwiki
2024-10 71721 350.0 741.0 idwiki
2024-11 63920 355.0 756.0 idwiki
2024-12 59001 333.0 738.0 idwiki
2025-01 65489 354.0 750.0 idwiki
2025-02 50085 352.0 742.0 idwiki
SELECT CONCAT(year, '-', LPAD(month, 2, '0')) _dt, COUNT(*), ROUND(PERCENTILE(event['fetchStart'],0.75)) _fetchStart, ROUND(PERCENTILE(event['responseStart'],0.75)) _responseStart, wiki FROM event_sanitized.NavigationTiming WHERE (year=2023 OR year=2024 OR year=2025) AND event['action']='view' AND event['isAnon']=true AND event['isOversample']=false AND (wiki='idwiki') AND event['mobileMode']='stable' GROUP BY year, month, wiki;
# p75 fetchStart from mobile pageviews to fawiki
_dt _count _fetchStart _responseStart
2023-01 166466 39.0 836.0
2023-02 133366 40.0 864.0
2023-03 143519 36.0 838.0
2023-04 146179 35.0 825.0
2023-05 142420 35.0 801.0
2023-06 138544 36.0 823.0
2023-07 133418 37.0 826.0
2023-08 133354 37.0 842.0
2023-09 131242 37.0 854.0
2023-10 146877 39.0 861.0
2023-11 140575 40.0 858.0
2023-12 159943 41.0 851.0
2024-01 166683 41.0 849.0
2024-02 160433 41.0 833.0
2024-03 166731 42.0 840.0
2024-04 166525 45.0 859.0
2024-05 186389 509.0 1009.0 <!-- increase in fetchStart
2024-06 168650 558.0 1005.0
2024-07 173290 551.0 1002.0
2024-08 173831 553.0 1020.0
2024-09 152917 538.0 991.0
2024-10 166421 565.0 1006.0
2024-11 171813 550.0 972.0
2024-12 193567 540.0 944.0
2025-01 196203 544.0 953.0
2025-02 165783 534.0 946.0
Time taken: 47.965 seconds, Fetched 26 row(s)
SELECT
CONCAT(year, '-', LPAD(month, 2, '0')) _dt,
COUNT(*) AS _count,
ROUND(PERCENTILE(event['fetchStart'],0.75)) _fetchStart,
ROUND(PERCENTILE(event['responseStart'],0.75)) _responseStart
FROM event_sanitized.NavigationTiming
WHERE (year=2023 OR year=2024 OR (year=2025 AND month IN (1,2)))
AND event['action']='view' AND event['isAnon']=true AND event['isOversample']=false
AND event['mobileMode']='stable'
AND wiki='fawiki'
GROUP BY year, month, wiki
ORDER BY _dt ASC;
# p75 from mobile pageviews to itwiki
_dt _count _fetchStart _responseStart
2023-01 310534 38.0 391.0
2023-02 261750 40.0 388.0
2023-03 272530 37.0 394.0
2023-04 273986 34.0 388.0
2023-05 283206 36.0 388.0
2023-06 255110 41.0 409.0
2023-07 258395 40.0 409.0
2023-08 298729 39.0 409.0
2023-09 260276 39.0 381.0
2023-10 261766 42.0 388.0
2023-11 256257 42.0 373.0
2023-12 266718 42.0 351.0
2024-01 323337 43.0 351.0
2024-02 290281 47.0 353.0
2024-03 295565 53.0 372.0
2024-04 280376 62.0 377.0
2024-05 299440 188.0 448.0 <!-- increase in fetchStart
2024-06 272893 227.0 461.0
2024-07 274133 221.0 456.0
2024-08 316949 229.0 470.0
2024-09 278212 215.0 450.0
2024-10 281733 215.0 454.0
2024-11 278978 213.0 444.0
2024-12 286975 207.0 442.0
2025-01 341939 204.0 432.0
2025-02 288496 212.0 442.0
Time taken: 65.798 seconds, Fetched 26 row(s)
SELECT
CONCAT(year, '-', LPAD(month, 2, '0')) _dt,
COUNT(*) AS _count,
ROUND(PERCENTILE(event['fetchStart'],0.75)) _fetchStart,
ROUND(PERCENTILE(event['responseStart'],0.75)) _responseStart
FROM event_sanitized.NavigationTiming
WHERE (year=2023 OR year=2024 OR (year=2025 AND month IN (1,2)))
AND event['action']='view' AND event['isAnon']=true AND event['isOversample']=false
AND event['mobileMode']='stable'
AND wiki='itwiki'
GROUP BY year, month, wiki
ORDER BY _dt ASC;
# p75 from mobile pageviews to commonswiki
_dt _count _fetchStart _responseStart
2018-02 13244 528.0 1407.0
2018-03 20533 509.0 1393.0
2018-04 17882 782.0 1336.0
2018-05 15406 934.0 1417.0
2018-06 14863 758.0 1474.0
2018-07 15036 623.0 1502.0
2018-08 15053 567.0 1455.0
2018-09 23850 566.0 1289.0
2018-10 15405 483.0 1396.0
2018-11 14741 455.0 1355.0
2018-12 14394 458.0 1362.0
2019-01 16408 442.0 1319.0
2019-02 16784 460.0 1277.0
2019-03 14655 442.0 1317.0
2019-04 14047 383.0 1282.0
2019-05 15887 345.0 1174.0
2019-06 16315 338.0 1153.0
2019-07 15614 306.0 1157.0
2019-08 16617 318.0 1241.0
2019-09 22053 375.0 1144.0
2019-10 16284 322.0 1126.0
2019-11 15025 336.0 1354.0
2019-12 14840 322.0 1228.0
2020-01 16710 329.0 1261.0
2020-02 19208 368.0 1245.0
2020-03 17799 322.0 1188.0
2020-04 19000 321.0 1236.0
2020-05 18940 287.0 1201.0
2020-06 18189 284.0 1185.0
2020-07 18237 278.0 1271.0
2020-08 17464 279.0 1321.0
2020-09 17573 293.0 1198.0
2020-10 19814 260.0 1186.0
2020-11 16219 249.0 1244.0
2020-12 16043 245.0 1252.0
2021-01 17613 243.0 1216.0
2021-02 15419 253.0 1265.0
2021-03 18148 272.0 1336.0
2021-04 17097 277.0 1241.0
2021-05 16698 258.0 1179.0
2021-06 16049 277.0 1177.0
2021-07 16250 276.0 1210.0
2021-08 16771 264.0 1180.0
2021-09 18164 287.0 1077.0
2021-10 18692 270.0 1041.0
2021-11 15905 278.0 1094.0
2021-12 16280 280.0 1056.0
2022-01 18226 283.0 1049.0
2022-02 16590 294.0 1615.0
2022-03 18009 305.0 1506.0
2022-04 16798 300.0 1527.0
2022-05 17359 289.0 1487.0
2022-06 18144 300.0 1403.0
2022-07 18485 289.0 1430.0
2022-08 19236 279.0 1294.0
2022-09 19257 279.0 1040.0
2022-10 19933 265.0 1015.0
2022-11 18849 250.0 1006.0
2022-12 18975 234.0 1006.0
2023-01 20379 241.0 1024.0
2023-02 24996 306.0 1519.0
2023-03 27068 265.0 1398.0
2023-04 20937 254.0 1038.0
2023-05 21710 265.0 1022.0
2023-06 21206 269.0 1039.0
2023-07 22408 271.0 1032.0
2023-08 22033 266.0 1053.0
2023-09 20365 266.0 1049.0
2023-10 21395 260.0 977.0
2023-11 19088 237.0 867.0
2023-12 19688 232.0 873.0
2024-01 21518 225.0 857.0
2024-02 31228 233.0 849.0
2024-03 32462 227.0 885.0
2024-04 21352 228.0 913.0
2024-05 23065 256.0 931.0
2024-06 23140 282.0 970.0
2024-07 24081 258.0 948.0
2024-08 24001 254.0 964.0
2024-09 23764 264.0 918.0
2024-10 25222 241.0 841.0
2024-11 20574 219.0 848.0
2024-12 21032 212.0 850.0
2025-01 22383 224.0 837.0
2025-02 30197 225.0 819.0
Fetched 85 row(s)
SELECT
CONCAT(year, '-', LPAD(month, 2, '0')) _dt,
COUNT(*) AS _count,
ROUND(PERCENTILE(event['fetchStart'],0.75)) _fetchStart,
ROUND(PERCENTILE(event['responseStart'],0.75)) _responseStart
FROM event_sanitized.NavigationTiming
WHERE (year IN (2016,2017,2018,2019,2020,2021,2022,2023,2024) OR (year=2025 AND month IN (1,2)))
AND event['action']='view' AND event['isAnon']=true AND event['isOversample']=false
AND event['mobileMode']='stable'
AND wiki='commonswiki'
GROUP BY year, month, wiki
ORDER BY _dt ASC;
# p50 fetchStart from desktop pageviews
_fetc _resp
2023-01 2921886 6.0 217.0
2023-02 2639165 7.0 232.0
2023-03 2905086 7.0 216.0
2023-04 2605929 6.0 205.0
2023-05 2785293 7.0 185.0
2023-06 2510253 6.0 178.0
2023-07 2430522 6.0 174.0
2023-08 2496376 6.0 189.0
2023-09 2493268 7.0 214.0
2023-10 2750344 7.0 215.0
2023-11 2632519 7.0 215.0
2023-12 2406875 7.0 217.0
2024-01 2797839 7.0 194.0
2024-02 2598703 7.0 211.0
2024-03 2705376 7.0 219.0
2024-04 2708189 7.0 217.0
2024-05 2808166 7.0 217.0 <!-- fetchStart unchanged on desktop
2024-06 2468459 7.0 223.0
2024-07 2469832 7.0 219.0
2024-08 2452046 7.0 226.0
2024-09 2532347 7.0 224.0
2024-10 2751229 8.0 213.0
2024-11 2683650 8.0 215.0
2024-12 2506360 8.0 211.0
2025-01 2799589 8.0 201.0
2025-02 1846376 8.0 195.0
SELECT CONCAT(year, '-', LPAD(month, 2, '0')) _dt, COUNT(*), ROUND(PERCENTILE(event['fetchStart'],0.5)) _fetchStart, ROUND(PERCENTILE(event['responseStart'],0.5)) _responseStart, FROM event_sanitized.NavigationTiming WHERE (year=2023 OR year=2024 OR year=2025) AND event['action']='view' AND event['isAnon']=true AND event['isOversample']=false AND isnull(event['mobileMode']) GROUP BY year, month;
# p75 fetchStart from desktop pageviews
_fetch _resp
2023-01 2921886 13.0 440.0
2023-02 2639165 14.0 466.0
2023-03 2905086 13.0 438.0
2023-04 2605929 13.0 423.0
2023-05 2785293 14.0 394.0
2023-06 2510253 13.0 385.0
2023-07 2430522 12.0 382.0
2023-08 2496376 12.0 405.0
2023-09 2493268 14.0 430.0
2023-10 2750344 15.0 430.0
2023-11 2632519 16.0 433.0
2023-12 2406875 15.0 431.0
2024-01 2797839 15.0 404.0
2024-02 2598703 16.0 426.0
2024-03 2705376 15.0 443.0
2024-04 2708189 15.0 441.0
2024-05 2808166 15.0 444.0 <!-- fetchStart unchanged on desktop
2024-06 2468459 14.0 457.0
2024-07 2469832 14.0 448.0
2024-08 2452046 15.0 457.0
2024-09 2532347 15.0 458.0
2024-10 2751229 15.0 437.0
2024-11 2683650 15.0 439.0
2024-12 2506360 15.0 427.0
2025-01 2799589 15.0 408.0
2025-02 2217441 15.0 400.0
# p75 fetchStart from desktop pageviews to dewiki
_fetch _resp
2023-01 225789 12.0 289.0 dewiki
2023-02 190253 12.0 297.0 dewiki
2023-03 205819 12.0 285.0 dewiki
2023-04 175712 12.0 285.0 dewiki
2023-05 183388 13.0 271.0 dewiki
2023-06 175753 13.0 270.0 dewiki
2023-07 175051 12.0 266.0 dewiki
2023-08 176390 12.0 301.0 dewiki
2023-09 156514 15.0 340.0 dewiki
2023-10 168430 17.0 330.0 dewiki
2023-11 173727 18.0 331.0 dewiki
2023-12 155475 17.0 340.0 dewiki
2024-01 191824 17.0 327.0 dewiki
2024-02 172457 17.0 346.0 dewiki
2024-03 172418 17.0 353.0 dewiki
2024-04 178846 16.0 358.0 dewiki
2024-05 173452 15.0 351.0 dewiki <!-- fetchStart unchanged on desktop
2024-06 169622 15.0 341.0 dewiki
2024-07 167223 15.0 347.0 dewiki
2024-08 158685 15.0 342.0 dewiki
2024-09 162590 15.0 365.0 dewiki
2024-10 171852 15.0 329.0 dewiki
2024-11 173090 15.0 326.0 dewiki
2024-12 160459 15.0 316.0 dewiki
2025-01 195099 16.0 312.0 dewiki
2025-02 149871 15.0 312.0 dewiki
# p75 fetchStart from desktop pageviews to idwiki
_fetch _resp
2023-01 8790 17.0 855.0 idwiki
2023-02 8609 19.0 896.0 idwiki
2023-03 9309 17.0 792.0 idwiki
2023-04 6892 17.0 837.0 idwiki
2023-05 9272 17.0 640.0 idwiki
2023-06 7152 16.0 654.0 idwiki
2023-07 7373 16.0 654.0 idwiki
2023-08 8731 17.0 669.0 idwiki
2023-09 8689 19.0 712.0 idwiki
2023-10 9601 20.0 703.0 idwiki
2023-11 9639 20.0 687.0 idwiki
2023-12 6909 19.0 651.0 idwiki
2024-01 8813 20.0 615.0 idwiki
2024-02 8568 21.0 662.0 idwiki
2024-03 7937 21.0 704.0 idwiki
2024-04 7029 20.0 689.0 idwiki
2024-05 9182 20.0 683.0 idwiki <!-- fetchStart unchanged on desktop
2024-06 7486 21.0 699.0 idwiki
2024-07 9343 19.0 631.0 idwiki
2024-08 9519 22.0 639.0 idwiki
2024-09 9427 22.0 643.0 idwiki
2024-10 11627 22.0 645.0 idwiki
2024-11 10058 21.0 651.0 idwiki
2024-12 8187 20.0 663.0 idwiki
2025-01 8785 21.0 634.0 idwiki
2025-02 7649 21.0 625.0 idwiki
SELECT CONCAT(year, '-', LPAD(month, 2, '0')) _dt, COUNT(*), ROUND(PERCENTILE(event['fetchStart'],0.75)) _fetchStart, ROUND(PERCENTILE(event['responseStart'],0.75)) _responseStart, wiki FROM event_sanitized.NavigationTiming WHERE (year=2023 OR year=2024 OR year=2025) AND event['action']='view' AND event['isAnon']=true AND event['isOversample']=false AND (wiki='idwiki') AND isnull(event['mobileMode']) GROUP BY year, month, wiki;
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits