Initial sketch for the warmup of Memcached (cluster-wide) and APC (per-server).

• 3-5 urls for each of the 750 public wikis (not private, fishbowl or closed) - page views and load.php (not sure there is anything for api or thumb): ~3,800 urls in total.
• Orchestrate across all servers in the cluster.
• Configure concurrency of urls to request simultaneously on one server.
• Configure concurrency of servers to hit simultaneously. (all?)
• Configure global max concurrency of requests in transit (to avoid overloading common infra beyond natural traffic such as External store)

Requirements:

• Where to fetch a list of current app servers in the cluster? (conf.d?)
• Where to fetch a list of canonical server names for all public wikis, including the mobile variant? (sitematrix? + maybe hardcode wgMobileUrlTemplate from wmf-config/InitialiseSettings.php)
• Typical request rate for an app server? (cursory research shows 15-50 req/s per P4863)
• Typical cluster-wide request rate for mw backends? (2000-6000req/s)

URLs:

# Purpose: Root redirect
https://%server/
# Purpose: Main Page, Skin sidebar, Localisation cache
https://%server/wiki/Main_Page
# Purpose: MobileFrontend, Main Page
https://%mobileServer/wiki/Main_Page
# Purpose: Login page
https://%server/wiki/Special:UserLogin
# Purpose: API, Recent changes
https://%server/w/api.php?format=json&action=query&list=recentchanges

# Purpose: APC for ResourceLoader
https://%server/w/load.php?debug=false&modules=startup&only=scripts
https://%server/w/load.php?debug=false&modules=jquery%2Cmediawiki&only=scripts
https://%server/w/load.php?debug=false&modules=site%7Csite.styles

The total run should take less than 3 minutes. Estimates:

• global (warm-urls-cluster.txt): 750 public wikis * 5 urls / 500 rate = 7.5s
  • A local test against text-lb shows latency will average between 500ms and 7s. Runtime varied from 30s to 45s.
• per-server (warm-urls-server.txt): 750 public wikis * 3 urls / 50 rate = 45s = 1min
  • A local test against mwdebug1001 shows latency will average between 150ms and 14s. Runtime varied from 48s to 55s.
  • Using a concurrency of 45 per server, means global concurrency may peak around 5500 req/s.

Source code: https://gist.github.com/Krinkle/dfdcecb094570f1df520c76fc4630e56

For the requirements it might be helpful to shuffle the list of wiki/urls for each server to not call the same URL from different servers at the same time. My 2 cents.

thanks @Krinkle !
I have some questions mostly due to my ignorance of what mw does with memcache, if we were to wipe the caches in codfw say today without touching mw config and run the warmup script against codfw, would that be a realistic test of what would happen during the switchover in terms of performance?
Also if I understand correctly the warmup is one of the things that bounds our read-only time during the switchover, in that case we could start warming up wikis sorted by e.g. their pageviews to further shorten the acceptable read-only time.

Warning: I am swapping all the mc2* codfw hosts with new hardware in T155755, I should complete the work in a couple of days.

Correct me if I'm wrong, but I think the Main page call can be skipped for all non-standard-wiki-serving machines, so API, image/video scalers; also: do we really need to warm up APC for all of the wikis? Or could we target only the ones doing 99% of the traffic (which I guess are way less than that?).

In T156922#2992486, @fgiunchedi wrote:

warmup is one of the things that bounds our read-only time during the switchover, in that case we could start warming up wikis sorted by e.g. their pageviews to further shorten the acceptable read-only time.

That would significantly complicate the script as well as the actual switchover process. You'd have to deploy many changes to mw-config during the switchover to gradually read-only more and more wikis. The warmup script, meanwhile, takes less than a minute to run. I doubt we'd be reasonably saving any time considering the gradual read-only switching would have to be done manually and is about saving a subset of 50 seconds time.

In T156922#2992780, @Joe wrote:

Correct me if I'm wrong, but I think the Main page call can be skipped for all non-standard-wiki-serving machines, so API, image/video scalers; also: do we really need to warm up APC for all of the wikis? Or could we target only the ones doing 99% of the traffic (which I guess are way less than that?).

For video scalers, yes, maybe. But API and app servers both should have the main page query and the RC query. They are catch-all entry points that warm up a lot of shared resources that are not limited to page views or API queries.

Media scalers can be excluded from the entire process as none of the urls apply to those right now. I'll make sure to exclude those from the confd query if feasible. either way, though, the script is much better than last year and should run in under a minute regardless.

In T156922#2992765, @elukey wrote:

Warning: I am swapping all the mc2* codfw hosts with new hardware in T155755, I should complete the work in a couple of days.

Completed today, restarted all the nutcrackers in codfw to pick up the change. Please note: after https://gerrit.wikimedia.org/r/#/c/335780 Nutcracker is not restarted anymore on config change (the problem arose when a change in the codfw pool triggered a restart in all the eqiad nutcrackers due to how the config is laid out).

Another interesting possibility we might want to explore:

HHVM allows you to define a script (and headers!) you want to execute as a warmup procedure. It could be interesting to integrate the APC warmup into such a script so that any HHVM server will start up in the future with APC properly populated.

Probably not matter for this quarter though?

In T156922#3003542, @Krinkle wrote:

In T156922#2992486, @fgiunchedi wrote:

warmup is one of the things that bounds our read-only time during the switchover, in that case we could start warming up wikis sorted by e.g. their pageviews to further shorten the acceptable read-only time.

That would significantly complicate the script as well as the actual switchover process. You'd have to deploy many changes to mw-config during the switchover to gradually read-only more and more wikis. The warmup script, meanwhile, takes less than a minute to run. I doubt we'd be reasonably saving any time considering the gradual read-only switching would have to be done manually and is about saving a subset of 50 seconds time.

Indeed, it seems a whole lot of effort for small gains over 50s. Do you know if we could simulate a warmup (and a wipe beforehand) in codfw given how it is configured today in mediawiki?

In T156922#3012431, @fgiunchedi wrote:

In T156922#3003542, @Krinkle wrote:

In T156922#2992486, @fgiunchedi wrote:

warmup is one of the things that bounds our read-only time during the switchover, in that case we could start warming up wikis sorted by e.g. their pageviews to further shorten the acceptable read-only time.

That would significantly complicate the script as well as the actual switchover process. You'd have to deploy many changes to mw-config during the switchover to gradually read-only more and more wikis. The warmup script, meanwhile, takes less than a minute to run. I doubt we'd be reasonably saving any time considering the gradual read-only switching would have to be done manually and is about saving a subset of 50 seconds time.

Indeed, it seems a whole lot of effort for small gains over 50s. Do you know if we could simulate a warmup (and a wipe beforehand) in codfw given how it is configured today in mediawiki?

I think we should indeed do a few tests in codfw before the switchover. I guess we'll have to coordinate with the DBAs to be sure this won't harm codfw.

Next steps:

• Put node-warmup script in puppet.
• Have it ensured on at least one host in eqiad and one in codfw.
  • Decide where to host this. I'm proposing terbium/wasat (maintenance script hosts).
  • We'll also need to ensure nodejs is installed on these hosts (no dependencies, just core nodejs v4 or later).

In T156922#3033929, @Krinkle wrote:

• Decide where to host this. I'm proposing terbium/wasat (maintenance script hosts).

terbium is still on trusty, so the nodejs candidate is 0.10.25~dfsg2-2ubuntu1+wmf1, while wasat is a jessie hence 6.9.1~dfsg-1+wmf1 will be installed. I'd rather choose something more homogeneous unless terbium si due to be upgraded shortly. I'll ask for the status on T143536.

• We'll also need to ensure nodejs is installed on these hosts (no dependencies, just core nodejs v4 or later).

On jessie hosts it would be 6.9.1~dfsg-1+wmf1 or later, so I'd say test it with this version to ensure there are no v4->v6 issues.

Change 339802 had a related patch set uploaded (by Krinkle):
[WIP] mediawiki: Add cache-warmup to maintenance

https://gerrit.wikimedia.org/r/339802

I took what @Krinkle did in his patchset, fixed a couple of things in order to implement the "clone mode" and be able to simulate the full procedure:

• I wiped all memcacheds in codfw
• Restarted all HHVM appservers
• Ran the script in spread mode: it took about 1.5 minutes to execute
• Ran the script in clone mode only on the appservers in codfw; it took around 2 mintues to execute

This put some strain on the databases, but nothing really critical. For some reason that's not perfectly clear to me, the number of rows read from s3 is prominent, according to our database metrics.

Mentioned in SAL (#wikimedia-operations) [2017-02-28T21:05:50Z] <_joe_> manually installing nodejs on wasat T156922

In T156922#3061198, @Joe wrote:

• I wiped all memcacheds in codfw
• Restarted all HHVM appservers
• Ran the script in spread mode: it took about 1.5 minutes to execute
• Ran the script in clone mode only on the appservers in codfw; it took around 2 mintues to execute

This put some strain on the databases, but nothing really critical. For some reason that's not perfectly clear to me, the number of rows read from s3 is prominent, according to our database metrics.

I ran another test, without any Memcached wiping or HHVM restarting, though.

• Spread mode for cluster urls (sent 3657 urls to codfw appservers; concurrency: 500): Took 26s and 12s (ran twice)
• Clone mode for server urls (sent 2196 urls to 77 codfw appservers each; concurrency: 500/global, 100/server): Took 176s (2.9min) and 160s (2.6min)
• Clone mode for server urls (sent 2196 urls to 77 codfw appservers each; concurrency: 2000/global, 100/server): Took 173s (2.8min)

On https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?var-dc=codfw%20prometheus%2Fops, I noticed nothing out of the ordinary except, same as @Joe noticed, "Rows read (s3)" spiked quite prominently (from 800 rps to 1.1M rps). The other clusters remained under 100 rps. The spike caused by our few thousand urls makes sense considering we're comparing to an idle state with only health status requests. Comparing to Eqiad, most of the "Rows read" metrics are regularly between 3M and 8M.

As for s3 in particular, it's where most wikis are, so that'll reflect most of the traffic. During our warmup we're sending an atypical proportion of traffic to the servers and databases. Normally the bigger wikis get more traffic than the smaller wikis, which is why most wikis can be on one cluster (s3). Since we're sending the same 8 urls to all (public) wikis, s3 receives significantly more traffic than the others for the short duration of the test.

On https://grafana.wikimedia.org/dashboard/file/server-board.json, looking at various app servers. CPU spiked from 0% to 8% for most codfw app servers. Compared to an average of 25% of most eqiad app servers. Load average jumped from 1 to 4 for a minute, compared to load average of 12 on eqiad servers. No increase in RAM usage.

On https://grafana.wikimedia.org/dashboard/db/hhvm-apc-usage?var-cluster=2, APC value size (codfw/p95) went from ~2MB to ~1GB, which is expected given that previously it only got 1 url of 1 wiki (enwiki Main Page health check), and now 8 urls for 750 wikis.

Still only a quarter of the typical APC value size of Eqiad (~4 GB), presumably since most cache keys are specific to users or pages, most of which don't get warmed up here, but that's fine.

Change 339802 merged by Giuseppe Lavagetto:
mediawiki: Add cache-warmup to maintenance

https://gerrit.wikimedia.org/r/339802

I guess we'll have to coordinate with the DBAs to be sure this won't harm codfw.
@Joe noticed, "Rows read (s3)" spiked quite prominently (from 800 rps to 1.1M rps).

Please don't be worried about the main databases on codfw, like s3 -if they were fragile, they wouldn't withstand production queries :-). In fact, they throttle concurrency and put things in a queue if they cannot handle the thoughput. I checked the stats and the largest s3 server peak in connections was 95-79 concurrent connections, and we are ready for 5000-10000.

In fact, my only worry (why I mentioned before) was that the parsercache would affect eqiad because replication (writes on codfw->eqiad), but I haven't seen any issue with that.

As a reminder, main dbs did not have issues last time, without a proper warmup, only es* hosts did, because normally they have very, very little load due to caching, and they were briefly overloaded. The load there is much higher and there seems to create lag, which normally doesn't happen at all: https://grafana-admin.wikimedia.org/dashboard/db/mysql-aggregated?from=1488277850999&to=1488293698570&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All We want to warmup to avoid the es overload mainly. If those are happy, that should be enough.

Change 340539 had a related patch set uploaded (by Krinkle):
[operations/puppet] mediawiki-cache-warmup: Remove unused var, reduce concurrency, log slowest-5

https://gerrit.wikimedia.org/r/340539

Change 340539 merged by Giuseppe Lavagetto:
[operations/puppet] mediawiki-cache-warmup: Remove unused var, reduce concurrency, log slowest-5

https://gerrit.wikimedia.org/r/340539