We should run some load tests of our cross-project searching plan to ensure that it is performant enough for release.
Our plan is to take X% of searches (X to be determined) and run cross-project searches for those queries. The results presented to the user will not be changed during this test, as the test is meant purely to determine server load.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | debt | T145918 [EPIC] Backend: cross-wiki search results functionality | |||
| Resolved | EBernhardson | T149740 Run load tests of cross-project searching to verify its stability | |||
| Resolved | EBernhardson | T149741 Improve how search queries are built for cross-wiki searches to allow for better filtering |
A couple things that have to be taken into account:
My idea:
Caveats:
I suppose to be able to do this, it also means we will need to come up with the map of wikipedia -> sister projects + prefixes to fill wgCirrusSearchInterwikiSources. Or have to delay the test to work together with @dcausse work to refactor the core InterwikiLookup such that we can lookup the relevant sister projects.
Change 319485 had a related patch set uploaded (by EBernhardson):
[WIP] Add configuration value to run interwiki load test
Change 319498 had a related patch set uploaded (by EBernhardson):
[WIP] Script to generate wmgCirrusSearchInterwikiSources
I pulled some numbers from hive to get an idea of the level of impact. This data is for Oct 30 19:00-20:00 UTC which is a weekly peak on our grafana graphs. It is restricted to queries that have at least one full_text query in the log.
| type | avg qps |
| all | 498 |
| web | 120 |
| wikipedia web | 117 |
So, something like 24% of full text searches, at peak load, would start also doing interwiki search. If we wanted to expand the interwiki search (at some later point) such that we were doing queries from say itwikisource to all it's sisters, the additional load would probably be un-noticable.
Change 320220 had a related patch set uploaded (by EBernhardson):
Setup CirrusSearch interwiki load test
Change 319485 merged by jenkins-bot:
Add configuration value to run interwiki load test
Change 321492 had a related patch set uploaded (by EBernhardson):
Revert "Revert "Setup CirrusSearch interwiki load test""
There were some log spamming issues, so we had to revert. We're looking at trying this again within a day or two.
Change 321492 merged by jenkins-bot:
Revert "Revert "Setup CirrusSearch interwiki load test""
Mentioned in SAL (#wikimedia-operations) [2016-11-15T19:51:49Z] <thcipriani@tin> Synchronized wmf-config/CirrusSearch-interwikiSources.php: SWAT: [[gerrit:321492|Revert "Revert "Setup CirrusSearch interwiki load test"" (T149740)]] PART I (duration: 01m 43s)
Mentioned in SAL (#wikimedia-operations) [2016-11-15T19:53:39Z] <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:321492|Revert "Revert "Setup CirrusSearch interwiki load test"" (T149740)]] PART II (duration: 00m 48s)
Mentioned in SAL (#wikimedia-operations) [2016-11-15T19:54:58Z] <thcipriani@tin> Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:321492|Revert "Revert "Setup CirrusSearch interwiki load test"" (T149740)]] PART III (duration: 00m 48s)
Change 321724 had a related patch set uploaded (by EBernhardson):
Increase CirrusSearch interwiki load test to 25%
Initial 5% deployment had no noticable change to our load or latency metrics. will increase to 25% in this evenings SWAT deployment. Will also work up how to query this information out of hive, so we have data on just these requests and not the aggregate api+web that our standard metrics report.
Change 321724 merged by jenkins-bot:
Increase CirrusSearch interwiki load test to 25%
Mentioned in SAL (#wikimedia-operations) [2016-11-16T00:26:08Z] <dereckson@tin> Synchronized wmf-config/CirrusSearch-production.php: Increase CirrusSearch interwiki load test to 25% (T149740) (duration: 00m 58s)
Initial latency changes look pretty minor. They are there, but minor.
| isloadtest | count | % req | p50 | p75 | p95 |
| False | 456005 | 95.8% | 159.0 | 243.0 | 764.0 |
| True | 19683 | 4.1% | 182.0 | 258.0 | 765.0 |
| change: | +23ms / 14% | +15ms / 6% | +1ms / 0.1% | ||
The load test was set for 5%, so not sure why this is showing 4.1%. Perhaps my selection of wikis (a lazy len(wikiid) == 6) wasn't selective enough,
hive query for future reference:
select payload['interwikiLoadTest'] as isLoadTest, count(1), percentile(cast(tookms as int), 0.50) as p50, percentile(cast(tookms as int), 0.75) as p75, percentile(cast(tookms as int), 0.95) as p95 from cirrussearchrequestset where year=2016 and month=11 and day=15 and hour=23 and length(wikiid) = 6 and source = 'web' group by payload['interwikiLoadTest'];
Traffic today looks to have peaked between 17:00 and 18:00 UTC. Stats for today:
| interwiki | reqs | % reqs | p50 | p75 | p95 |
| false | 417707 | 76.5% | 152.0 | 219.0 | 685.0 |
| true | 128539 | 23.5% | 189.0 | 270.0 | 742.0 |
| change | 37ms / 24% | 51ms / 23% | 57ms / 8% | ||
The difference is a little larger and more noticable than at 5%. That could though be an effect of time, 5% was deployed after the daily peak where this is taken from the peak hour. I'm not seeing any noticeable changes in cluster cpu or iowait load. IO usage does appear to be up. It's a little hard to see, because the weekly hadoop dump ran from the 14th at 13:00 until the 15th at 22:00. Es looks to have also finished most of it's segment merges by 22:00. We deployed 5% interwiki on the 15th around 20:00, and 25% on the 16th at 00:20.
Some data about disk usage, values are kinda guestimates by looking at the graphite data between 17:00 and 18:00 UTC and choosing a reasonable value. Peaks are taken by looking at 5 servers with the highest values, and taking the highest peak that isn't an obvious outlier.
| iops/server | bytes/s/server | peak iops | peak bytes/s | ||
| last week | ~470 | 4MB/s | 1.2k | 18 MB/s | |
| current | ~470 | 4MB/s | 1.7k | 24 MB/s | |
So we arn't necessarily pulling more data on average, but some servers might be hot spotting a bit more than before. Overall though i think we are good to increase to 50%.
Change 321925 had a related patch set uploaded (by EBernhardson):
Increase cirrus interwiki loadtest to 50%
Mentioned in SAL (#wikimedia-operations) [2016-11-17T00:35:15Z] <dereckson@tin> Synchronized wmf-config/CirrusSearch-production.php: Increase cirrus interwiki loadtest to 50% (T149740) (duration: 00m 48s)
There's a question outstanding; was this load test done with all requests, including API requests, at 50%? If so, we're already past what we'd handle if we just did it for desktop, so the load test is successful already! @dcausse will check, and if it's just desktop traffic, increase it to 100%. We expect to deploy that change next week, since this week is a holiday week.
If I'm reading the code correctly the load test should only test desktop searches, the interwiki flag is on by default but the API will set it to false to default. This is basically everything except API requests. I'll try to extract some numbers to compare with what Erik already analyzed. I'll prepare a patch to increase it to 75% accordingly.
Some data for Nov 22 at 12 UTC:
| interwiki | #queries | p50 | p75 | p95 |
| NULL (assuming false) | 284462 | 149.0 | 228.0 | 840.0 |
| true | 230105 | 168.0 | 245.0 | 704.0 |
Numbers seem stable, I'm not sure to understand why p95 is better when interwiki is enabled...
I'll take another snapshot today at 23h UTC so we can directly compare with the previous numbers but overall this looks promising, I'll prepare a patch to increase to 75%, I'll swat it tomorrow.
Change 322900 had a related patch set uploaded (by DCausse):
[cirrus] Increase interwiki loadtest to 75%
Mentioned in SAL (#wikimedia-operations) [2016-11-22T19:17:08Z] <thcipriani@tin> Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:322900|[cirrus] Increase interwiki loadtest to 75%]] (T149740) (duration: 00m 55s)
Another snapshot taken on Nov 24 at 23h UTC:
| interwiki | #queries | p50 | p75 | p95 |
| NULL (assuming false) | 136117 | 171.0 | 281.0 | 982.0 |
| true | 235698 | 160.0 | 236.0 | 680.0 |
I still don't know how to interpret that interwiki times are better...
That is a bit odd that interwiki searches are coming back faster than the standard ones. Overall though I don't think it's a big deal, and the more important question is if overall cluster health seems worse, and if the timings are noticably worse than before we started the test. Overall the numbers don't seem to have too much variation from before, so this generally looks like a success. Will push a patch to make it 100% today, but not expecting anything particularly bad.
Change 323868 had a related patch set uploaded (by EBernhardson):
Increase Cirrus interwiki load test to 100%
Mentioned in SAL (#wikimedia-operations) [2016-11-28T19:34:49Z] <thcipriani@tin> Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:323868|Increase Cirrus interwiki load test to 100%]] (T149740) (duration: 00m 46s)
Change 325329 had a related patch set uploaded (by EBernhardson):
Turn off CirrusSearch interwiki load test
Final results against the busiest hour since 100% was enabled (dec 4, 19:00 UTC)
| isloadtest | count | p50 | p75 | p95 |
| NULL | 91853 | 239.0 | 479.0 | 1143.0 |
| true | 396219 | 175.0 | 249.0 | 616.0 |
Same conclusions as before, the interwiki searches are not seeing any significant degredation of performance vs pre-test numbers.
Mentioned in SAL (#wikimedia-operations) [2016-12-05T19:41:20Z] <thcipriani@tin> Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:325329|Turn off CirrusSearch interwiki load test]] T149740 PART I (duration: 00m 44s)
Mentioned in SAL (#wikimedia-operations) [2016-12-05T19:44:48Z] <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:325329|Turn off CirrusSearch interwiki load test]] T149740 PART II (duration: 00m 47s)
Mentioned in SAL (#wikimedia-operations) [2016-12-05T19:45:53Z] <thcipriani@tin> Synchronized wmf-config: SWAT: [[gerrit:325329|Turn off CirrusSearch interwiki load test]] T149740 PART III (duration: 00m 46s)
Change 319498 abandoned by EBernhardson:
[WIP] Script to generate wmgCirrusSearchInterwikiSources
Reason:
was used for interwiki load testing, but will not be necessary longer term. I6aebfa22c05 allows deprecation of the wgCirrusSearchInterwikiSources config