Change Details

This graph indicates that mean MediaWiki backend latency has increased by ~60% between October and December 2025: https://grafana.wikimedia.org/goto/AOTG3FNDg?orgId=1 The one-year view suggests this is not due to seasonality. This was previously noted in T302623: >>! In T302623#11499381, @Krinkle wrote: > Looks like we have a major regression in latency, starting in November. > > [Grafana: Backend Pageview Timing](https://grafana.wikimedia.org/d/QLtC93rMz/backend-pageview-timing?orgId=1&from=now-90d&to=now&timezone=utc) dashboard: > > {F71460975 height=300} >>! In T302623#11500585, @dr0ptp4kt wrote: > Going by DC is this isolated on desktop in a given DC? That's maybe an obvious and expected thing based upon the most active (or inactive) DC. I suspect the stuff around the 30th of December is related to some natural (or automated) spike, but more looking at the general trends here. > > > {F71462799 size=full} > > Any way to drill down on the wiki DB to see if it's confined? >>! In T302623#11502725, @Krinkle wrote: >>>! In T302623#11499381, @Krinkle wrote: >> Looks like we have a major regression in latency, starting in November. >> >> [Grafana: Backend Pageview Timing](https://grafana.wikimedia.org/d/QLtC93rMz/backend-pageview-timing?orgId=1&from=now-90d&to=now&timezone=utc) dashboard: >> >> {F71460975 height=150} > > [Grafana: Appserver RED](https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&from=now-90d&to=now&timezone=utc&var-site=$__all&var-deployment=mw-web&var-method=GET&var-code=200&var-handler=php&var-service=mediawiki&refresh=1m) dashboard (mw-web, any DC, GET, HTTP 200): > > {F71467827 height=400} > > There does not appear to be a significant change in the overall rate or distribution of requests during this time (e.g. by DC, or HTTP method): > > {F71467902 height=200}

This graph indicates that mean MediaWiki backend latency has increased by ~60% between October and December 2025: https://grafana.wikimedia.org/goto/AOTG3FNDg?orgId=1 {F71690594} The one-year view suggests this is not due to seasonality. As mentioned in T302623, looking at [Grafana: Backend Pageview Timing](https://grafana.wikimedia.org/d/QLtC93rMz/backend-pageview-timing?orgId=1&from=now-90d&to=now&timezone=utc) dashboard: {F71690615} And the `mw-web` [[ https://grafana-rw.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&from=2025-10-25T19:54:32.975Z&to=2025-11-02T22:19:20.000Z&timezone=utc&var-site=codfw&var-deployment=mw-web&var-method=GET&var-code=200&var-service=mediawiki&viewPanel=panel-9 | RED dashboard ]] {F71690671} **Potential Factors (to be investigated)** * parsoid rollout * PHP 8.3 rollout * scraper activity * other ideas?

This graph indicates that mean MediaWiki backend latency has increased by ~60% between October and December 2025: https://grafana.wikimedia.org/goto/AOTG3FNDg?orgId=1 {F71690594} The one-year view suggests this is not due to seasonality. This was previously noted in T302623As mentioned in T302623, looking at [Grafana: Backend Pageview Timing](https://grafana.wikimedia.org/d/QLtC93rMz/backend-pageview-timing?orgId=1&from=now-90d&to=now&timezone=utc) dashboard: >>! In T302623#11499381, @Krinkle wrote: > Looks like we have a major regression in latency, starting in November. > > [Grafana: Backend Pageview Timing](https://grafana{F71690615} And the `mw-web` [[ https://grafana-rw.wikimedia.org/d/QLtC93rMz/backend-pageview-timing?orgId=1&from=now-90d&to=now&timezone=utc) dashboard: > > {F71460975 height=30035WSHOjVk/application-servers-red-k8s?orgId=1&from=2025-10-25T19:54:32.975Z&to=2025-11-02T22:19:20.000Z&timezone=utc&var-site=codfw&var-deployment=mw-web&var-method=GET&var-code=200&var-service=mediawiki&viewPanel=panel-9 | RED dashboard ]] {F71690671} >>! In T302623#11500585, @dr0ptp4kt wrote:**Potential Factors (to be investigated)** > Going by DC is this isolated on desktop in a given DC? That's maybe an obvious and expected thing based upon the most active (or inactive) DC. I suspect the stuff around the 30th of December is related to some natural (or automated) spike, but more looking at the general trends here.* parsoid rollout >* PHP 8.3 rollout > > {F71462799 size=full} > > Any way to drill down on the wiki DB to see if it's confined? >>! In T302623#11502725, @Krinkle wrote: >>>! In T302623#11499381, @Krinkle wrote: >> Looks like we have a major regression in latency, starting in November. >>* scraper activity >> [Grafana: Backend Pageview Timing](https://grafana.wikimedia.org/d/QLtC93rMz/backend-pageview-timing?orgId=1&from=now-90d&to=now&timezone=utc) dashboard: >> >> {F71460975 height=150} > > [Grafana: Appserver RED](https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&from=now-90d&to=now&timezone=utc&var-site=$__all&var-deployment=mw-web&var-method=GET&var-code=200&var-handler=php&var-service=mediawiki&refresh=1m) dashboard (mw-web, any DC, GET, HTTP 200): > > {F71467827 height=400} > > There does not appear to be a significant change in the overall rate or distribution of requests during this time (e.g. by DC, or HTTP method): > > {F71467902 height=200} * other ideas?