Page MenuHomePhabricator

Analyze impact on unique devices of data center launches
Closed, ResolvedPublic

Description

Prior related work: T222078: Analyze readers' engagement in countries affected by Singapore Data Center's switch

We need to calibrate expectations for WE 2.4 key result:

This KR focuses on improving web-site performance in an under-served region. Research suggests that a significant and noticeable reduction in web site response time improves user engagement. As part of this KR, we will improve web site response times in South America, e.g. by deploying an additional cache site in the region. Although we anticipate the first significant impact to be measurable in Brazil by the end of the fiscal year, the entire region is expected to see significant performance benefits shortly after.

For that, we need to look at how unique devices (the core metric for Relevance) were affected regionally by the launch of:

... and use that to provide rough estimates of impact we might expect to see in Latin America.

Timeline: Maryana & Selena would need numbers by August

Event Timeline

mpopov triaged this task as High priority.Jul 6 2023, 11:07 PM
mpopov created this task.
nshahquinn-wmf added a subscriber: kzimmerman.

Initial results ready to be shared with @kzimmerman. We'll have to talk about next steps.

I'm scheduling time to follow up with Selena & others on landing on the relevance target. Needed ahead of meeting:

  • estimates for relevant South American countries lift in unique devices, given the observed lift from the addition of the Singapore data center
  • estimate of overall impact to global unique devices (which will likely be fairly small, given that LATAM proportion of unique devices is small)
kzimmerman renamed this task from Analyze impact on unique devices following launch of Marseille DC to Analyze impact on unique devices of data center launches.Sep 8 2023, 7:46 PM
kzimmerman updated the task description. (Show Details)

I'm working on a full report, but I'm already confident in the key finding:

I estimate that the South American data center will add 6.6 M monthly unique devices, which would be a 0.41% increase globally.

Details:
The CausalImpact models tell us that, over the year following the switch to the Singapore data center, tranche 1 countries saw a 6.2% increase in monthly unique devices, while tranche 2 countries saw an 8.3% increase. This translates to a 6.3% increase overall (because tranche 1 was far bigger than tranche 2).

Looking at the last year of data, South American monthly unique devices averaged 104 M. A 6.3% increase would be 6.6 M. Among 1.59 B average global unique devices, that is 0.41% increase.

@nshahquinn-wmf: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!

I've published my code at https://gitlab.wikimedia.org/nshahquinn-wmf/new-data-center-impact/ (previously it was in a private GitLab repo because it contained some sensitive data, so I've removed all the history before publishing). The readme has a detailed guide to the work and the results.

This task will stay open as I plan to produce a full report during the week of 23 October.

nshahquinn-wmf changed Due Date from Oct 27 2023, 7:00 AM to Nov 2 2023, 2:00 AM.Oct 30 2023, 5:23 AM
nshahquinn-wmf changed Due Date from Nov 2 2023, 2:00 AM to Nov 9 2023, 3:00 AM.
nshahquinn-wmf lowered the priority of this task from High to Medium.Oct 30 2023, 5:26 AM
nshahquinn-wmf moved this task from Next 2 weeks to Doing on the Movement-Insights board.

@nshahquinn-wmf: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!

I'm currently finishing up the final report; the current draft is in the repo.

nshahquinn-wmf added a subscriber: ssingh.

I made a bunch of edits based on good feedback from @nettrom_WMF and @ssingh. I shared the final report, which is in the same location, on Wednesday.

Before closing this out, I still need to:

  • Output an archival PDF version
  • Create a page on Meta-Wiki