From https://grafana.wikimedia.org/dashboard/db/resourceloader
- p99 and upper: 200ms (min in last 24h), 1min (max in last 24h)
This means that every day, every minute, at least one module build takes 200ms. Note that we often a dozen or more modules in a single HTTP request. Not to mention the startup module, which has to compute many more (though for the common case of file modules we have optimisations in place that bypass the module build).
The highest point is even over a minute.
- p95: 1.9ms (highest point in last 3 months)
On the other hand, our p95 has not gone above 2ms in over 3 months. This is quite an accomplishment and largely due to our content-based caching for minification and Less compilation. There are many different urls to the same resource essentially (N wikis * N user languages * N skins), and most of these don't vary, so once computed, they're re-used until the resource changes.
However this hasn't changed the fact that Less compilation and other code involved in module build still seems pretty darn slow in case of cache miss.
Over a 24 hour period:
module max avg ext_cx_header 5.3s 949.3ms ext_guidedTour_styles 4.2s 2.4s mobile_editor_ve 2.2s 1.5s mobile_notifications_overlay 2.2s 1.4s skins_minerva_special_watchlist_styles 2.0s 1.1s mobile_special_uploads_styles 1.9s 990.9ms mobile_backtotop 1.9s 952.1ms mobile_app_pagestyles_android 1.8s 911.2ms ext_flow_visualEditor 1.7s 1.2s ext_wikimediaBadges 1.4s 1.0s
Plenty of modules that are consistently slow (not just peaks).
From: https://grafana.wikimedia.org/dashboard/db/resourceloadermodule
Graph: Build times for skins_vector_styles
