On 2017-08-02 around 19:00 UTC, Backend Save Timing (p75) rapidly started regressing by 40-60% from a stable ~450ms (±50ms) to 700-900ms.
<https://grafana.wikimedia.org/dashboard/db/save-timing?refresh=5m&orgId=1&from=1500879593680&to=1501791224192>
{F8942623 height=200}
{F8942635 height=200}
>
> **2017-08-02**
>
> 18:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part I (duration: 00m 47s)
> 18:13 thcipriani@tin: Synchronized wmf-config/jobqueue.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part II (duration: 00m 47s)
> 18:18 gehel: un-banning and repooling elastic102[4567] - T168816
> 18:18 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialUndelete.php: SWAT: Fix Special:Undelete search - use variable and not request param (duration: 00m 46s)
> 18:20 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(24|25|26|27).eqiad.wmnet
> 18:21 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: beta-only change Enable HTML5 sections in betalabs (duration: 00m 46s)
> 18:41 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialRecentchanges.php: SWAT: Follow-up 31be7d0: send tags list if experimental mode is disabled (duration: 00m 47s)
> 19:23 twentyafterfour: group1 wikis to 1.30.0-wmf.12 refs T168053
> 19:24 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs refs T168053
> 19:50 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.11 refs T168053 - rollback due to T172320
>
The roll-out of 1.30.0-wmf.12 seems like the obvious cause, but note that the regression hasn't reversed, despite the branch having been rolled back within 30 minutes.