Page MenuHomePhabricator

Set up A/B testing mechanism for PHP7
Closed, ResolvedPublic

Description

What we want to happen is that upon making a request to our sites, the following happens:
If the (anonymous) user doesn't have the PHP_ENGINE cookie set, set it. There is a wgPHP7percent chance that the cookie will route all subsequent requests for the same user to be rendered by php7 for a fixed amount of time (7 days seems sensible?)

An additional possible measure is:
If the request is uncachable and has no PHP_ENGINE cookie set, make the chance extraction in vcl and send the request to the appropriate backend. Add the cookie in the backend response to try to persist the choice

The first part of this is analogous to what we've done for the HHVM transition in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/NavigationTiming/+/164984 - with possibly two differences:

  1. I'd like to include logged-in users in the extraction (or at least allow it)
  2. We'll add the code to WikimediaEvents, which looks like the right place for this.

Related Objects

Event Timeline

Joe triaged this task as Medium priority.Feb 21 2019, 6:42 AM
Joe created this task.
Joe updated the task description. (Show Details)

Change 492305 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Make the PHP_ENGINE cookie non-httpOnly

https://gerrit.wikimedia.org/r/492305

Change 492306 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Allow directing a sample of users to PHP 7

https://gerrit.wikimedia.org/r/492306

Change 492305 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Make the PHP_ENGINE cookie non-httpOnly

https://gerrit.wikimedia.org/r/492305

@Joe I understand the choice between VCL in Varnish and client-side JS favouring the latter. While I'm not sure which exact issue you encountered, I can imagine several (e.g. difficult to produce rands in VCL, or wanting to avoid the cost server-side for bots/scrapers.)

However, what I'm not understanding is why we want to sample anonymous users and why we'd want to do it from client-side JS, when the decision only affects backend requests.

Is there a conflict with the beta feature that I'm not seeing? Is logged-in traffic not enough? Do we explicitly want to test out edits by logged-out users (why)? Or is it about logged-out non-editor cache-misses that populate Varnish?

Assuming we don't fragment the cache, the current patch will result in php7-users being served hhvm ones, and vice versa. As such, there isn't anything sticky for users that don't have a session and/or don't edit. So if it's about populating cache from PHP7 slowly, we might as well per-request randomise from VCL without any tokens or cookies. If it's about logged-out editors, then presumably we could use the same logic we have already on the PHP side.

@Joe I understand the choice between VCL in Varnish and client-side JS favouring the latter. While I'm not sure which exact issue you encountered, I can imagine several (e.g. difficult to produce rands in VCL, or wanting to avoid the cost server-side for bots/scrapers.)

The reason not to do this in vcl is that it's impossible to offer a consistent view to the user without setting more cookies, and we want to progressively move over the traffic, probably starting as low as 0.1% of the global users.
Our approach has an issue obviously - it doesn't cover the API traffic. For that, I think it will be inevitable to choose a VCL-based approach.

However, what I'm not understanding is why we want to sample anonymous users and why we'd want to do it from client-side JS, when the decision only affects backend requests.

Is there a conflict with the beta feature that I'm not seeing? Is logged-in traffic not enough? Do we explicitly want to test out edits by logged-out users (why)? Or is it about logged-out non-editor cache-misses that populate Varnish?

We're starting with anonymous users first, as a courtesy to logged-in users who could choose to opt-in to the beta or not. We want to respect that choice until we're confident there are no blockers for the transition. The beta uses the exact same approach (setting the same cookie) as the random extraction, so the two would conflict. We could work around it, but I prefer to decide at some point that the beta is over and we just start treating anons and logged-in users consistently.

Assuming we don't fragment the cache, the current patch will result in php7-users being served hhvm ones, and vice versa. As such, there isn't anything sticky for users that don't have a session and/or don't edit. So if it's about populating cache from PHP7 slowly, we might as well per-request randomise from VCL without any tokens or cookies. If it's about logged-out editors, then presumably we could use the same logic we have already on the PHP side.

This would be correct if we didn't separate the cache, but we're separating it for the time being. So what will happen is that the first request to a user will always be served by HHVM. After that, we store a random seed in their local storage and, depending on sampling, we set the cookie that directs them to php7. From that moment on, they'll be consistently only seeing the sites served from php7. This should simplify enormously tracking down problems and reproducing them. Of course there is a limitation to this - the parsercache and memcached are not separated - but it's what I think is a good compromise.

This would be correct if we didn't separate the cache, but we're separating it for the time being. [..]

Ah, so Varnish objects on cache_text vary by PHP engine cookie? I didn't realise that.

Change 492975 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Use a cookie to persist the seed for php7's a/b test

https://gerrit.wikimedia.org/r/492975

Change 492306 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Allow directing a sample of users to PHP 7

https://gerrit.wikimedia.org/r/492306

Change 494478 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.19] Allow directing a sample of users to PHP 7

https://gerrit.wikimedia.org/r/494478

Change 494478 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.19] Allow directing a sample of users to PHP 7

https://gerrit.wikimedia.org/r/494478

Mentioned in SAL (#wikimedia-operations) [2019-03-06T12:32:10Z] <oblivian@deploy1001> Synchronized php-1.33.0-wmf.19/extensions/WikimediaEvents: SWAT: Allow directing a sample of users to PHP 7 backport to wmf.19 T216676 (duration: 00m 57s)

Change 494756 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Direct 0.1% of anonymous users to php7

https://gerrit.wikimedia.org/r/494756

Change 494756 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Direct 0.1% of anonymous users to php7

https://gerrit.wikimedia.org/r/494756

Change 492975 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Use a cookie to persist the seed for php7's a/b test

https://gerrit.wikimedia.org/r/492975

Change 494756 merged by jenkins-bot:
[operations/mediawiki-config@master] Direct 0.1% of anonymous users to php7

https://gerrit.wikimedia.org/r/494756

Change 499238 had a related patch set uploaded (by Krinkle; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.22] Use a cookie to persist the seed for php7's a/b test

https://gerrit.wikimedia.org/r/499238

Change 492975 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Use a cookie to persist the seed for php7's a/b test

https://gerrit.wikimedia.org/r/492975

Change 499238 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.22] Use a cookie to persist the seed for php7's a/b test

https://gerrit.wikimedia.org/r/499238

Mentioned in SAL (#wikimedia-operations) [2019-03-27T11:24:43Z] <oblivian@deploy1001> Synchronized php-1.33.0-wmf.22/extensions/WikimediaEvents: SWAT: Backport Use a cookie to persist the seed for php7 a/b test to .22 T216676 (duration: 00m 58s)

Krinkle renamed this task from Set up A/B testing mechanism for PHP7, to Set up A/B testing mechanism for PHP7.Aug 10 2022, 1:06 AM