We would like to perform an A/B test to determine the success of the new suggester / recommender using a service deployed on Cloud VPS.
The service would be called from an existing and deployed MediaWiki extension (PropertySuggester). Effectively replacing a manually maintained SQL database there with calls to this service.
If the test is successful and the A/B test shows that we should use the new recommender we would then stop calling the Cloud VPS instance, and go through the full process of a production deploy of the service.
There is a precedent for calling external (not hosted in WMF production) services from WMF production / MediaWiki in specific cases.
Per https://wikitech.wikimedia.org/wiki/Cross-Realm_traffic_guidelines#Case_3:_generic_network_access_prod_--%3E_cloud in principle this network access is allowed, and this page outlines how it should be done.
However this may still need some other signoff by someone?
Request & Data
Neither payload, request or response, contain any remotely worrying / possible user focused data.
Responses are generated from publicly dumped data, and requests are essentially lists of property and item IDs.
The request from the MediaWiki extension to the service looks something like this:
$response = MediaWikiServices::getInstance()->getHttpRequestFactory()->post( $this->schemaTreeSuggesterUrl, [ 'postData' => json_encode( [ 'Properties' => $properties, 'Types' => $types ] ), 'timeout' => 1 ], __METHOD__ );
And the data would look something like this:
{
"properties": [
"P1"
"P2452",
"P3513551"
],
"types": [
"Q13",
"Q245125",
"Q1314"
]
}Stability & Reliability considerations
If the request to the service times out (1 second) or fails, the old behaviour will be used as a fallback (using the SQL DB)
So even if cloud VPS etc is inaccessible users should not be impacted.
Calls to the service can easily be turned off with a single MediaWiki config variable change.
Generally requests from the new services are handled in < 200 ms
- Traffic & Scope**
The A/B test covers some of the functionality provided by the wbsgetsuggestions API
Generally in Wikidata production there are 0.5 -> 1.5 requests per second to the wbsgetsuggestions API
This service would be receiving roughly 50% of those, so 0.25-0.75 requests per second
The API traffic involved in the test can be seen at https://grafana.wikimedia.org/d/000000559/api-requests-breakdown?orgId=1&refresh=5m&var-metric=p50&var-module=wbsgetsuggestions
Checklists
From that page:
- the VM instance uses a web proxy created in horizon.
-
if only production is intended to consume the service, neutron security groups should be used to firewall the access.
This is meant to serve as a mostly complete checklist for getting this A/B test to production via the agreed path, & will be completed jointly by marta & WMDE.
Steps / Timeline / Order to Production:
- WMDE: Merge patch 1 of T284820 merged https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/689161 containing the code to read from the new service
- Automatic: Code will be deployed to beta every ~15 mins
- operations/mediawiki-config change to switch beta wikidata to read from the new service
- Marta write config patch https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/703205
- WMDE provide initial review
- Marta add patch to backport window
- Patch is deployed
- Marta test things work as expected on beta wikidata
- WMDE: Merge patch 2 of T284820 merged https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/694496/15 containing the EventLogging code for the A/B test
- WMDE: Merge patch 3 of T284820 (Eventlogging) https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/689152 ready for use on beta (This doesn't need to block things until wikidata.org)
- Marta test things work as expected on beta wikidata, and check the status of logged events with WMDE (beta event checking)
- Automatic: Code will be deployed to beta every ~15 mins
- Merge patch 4 https://gerrit.wikimedia.org/r/705713
- Merge patch 5 https://gerrit.wikimedia.org/r/706326
- Merge patch 7 https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/708714 (fix event logging JS name)
- Merge patch 8 https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/709209 (fix -1 request duration)
- Code patches are deployed to test & production wikidata in wmf/1.37.0-wmf.17, (2nd -> 6th August)
- operations/mediawiki-config change to switch test wikidata to read from the new service with A/B test
- Marta write config patch
- WMDE provide initial review
- Marta add patch to backport window
- Patch is deployed (after wmf/1.37.0-wmf.17 to group0)
- Marta test things work as expected on test wikidata
- WMDE Validate eventlogging data being recorded
- operations/mediawiki-config change to switch wikidata to read from the new service with A/B test
- Have posted / communicated to the community that this is going to be turned on
- Marta write config patch
- WMDE provide initial review
- Marta add patch to backport window
- Patch is deployed (after wmf/1.37.0-wmf.17 to group1)
- Marta test things work as expected on test wikidata
- WMDE Validate eventlogging data being recorded
Config
Default values from the extension:
$wgPropertySuggesterSchemaTreeUrl = ''; $wgPropertySuggesterABTestingState = false; $wgPropertySuggesterDefaultSuggester = 'PropertySuggester'; $wgPropertySuggesterTestingRatio = '2'; $wgPropertySuggesterPropertyBaseUrl = ''; $wgPropertySuggesterTypesBaseUrl = '';
When PropertySuggester is set to read from the new service but no EventLogging / A/B testing is to happen, the config should look like:
TBA
When PropertySuggester is set to perform the A/B test (once Event Logging is ready) the config should look like:
TBA