In T378253#10381138, @nshahquinn-wmf wrote:

@fkaelin has prepared an MR implementing this, so I should review it relatively soon.

In T403437#11421301, @nshahquinn-wmf wrote:

Last week, I:

• Investigated the correlation between referral traffic from Google and from other external referrers
• Finished slides for the board meeting presentation

Barring any last minute requests for changes to the slides, this work is done and the hypothesis will be closed shortly.

Last week, I:

• Investigated the correlation between referral traffic from Google and from other external referrers
• Finished slides for the board meeting presentation

In T403437#11399293, @nshahquinn-wmf wrote:

Last week, I:

• Wrapped up small wiki investigation and chose proposed example wikis
• Started work on slides for the board meeting presentation

Last week, I:

• Wrapped up small wiki investigation and chose proposed example wikis
• Started work on slides for the board meeting presentation

One additional point I've thought of: if you look in MediaWiki history and find that a group of revisions have the same exact author, timestamp, and page title, this is (almost?) certainly a group with one original and the remainder imported duplicates of that original.

I just worked on a Wikipedia 25-related request from the WMF Communications department for:

• The monthly article count for each Wikipedia during its history
• The creation data for each Wikipedia
• The first article created at each Wikipedia

In T278467#8892637, @xcollazo wrote:

Agreed that the Hive tables can stay as they are, and the new Iceberg tables can do proper DATEs and TIMESTAMPs. When inserting into Iceberg, we can cast accordingly. See T335305 for an example conversion from year, month, day INTs to a DATE.

@Niharika what kind of input are you looking for?

@JAllemandou absolutely, I think the rule improvements are in great shape! By specifying TLDs for each search engine, you have already gone well above and beyond the requirements 😊

In T406531#11335567, @JAllemandou wrote:

I have copied some values where referer has changed grouped by referer-domain and ordered by number of hists desc here (note: this list doesn't include IP-referer that have now been categorized as unknown instead of external).

In T403437#11308848, @nshahquinn-wmf wrote:

Since my last update, I:

• Produced lots more visuals and analysis
• I'm on track to have this analysis and visualization work largely completed by the end of the day Tue, 28 Oct (since I'll be on vacation Wed-Fri)

Since my last update, I:

• Produced lots more visuals and analysis
• I'm on track to have this analysis and visualization work largely completed by the end of the day Tue, 28 Oct (since I'll be on vacation Wed-Fri)

In T403437#11288192, @nshahquinn-wmf wrote:

Since my last update, I:

• Did a ton of analysis and data visualization in preparation for public communications about recent declines in pageviews
• Dug into trends in referrers
• Tested and found support for the the hypothesis that iOS traffic declined less than Android traffic (suggesting that pageviews from people with high socio-economic status declined less)

Since my last update, I:

• Did a ton of analysis and data visualization in preparation for public communications about recent declines in pageviews
• Dug into trends in referrers
• Tested and found support for the the hypothesis that iOS traffic declined less than Android traffic (suggesting that pageviews from people with high socio-economic status declined less)

Random suggestions:

• Consider including Kagi as a search engine
• Consider addressing T383088 by dropping the requirement that the referrer start with "http:// or "https://"

In T403437#11243456, @nshahquinn-wmf wrote:

This week, I:

• Analyzed Comscore and unique device data
  • Lots of contradictory signals, plus noise due to the ongoing traffic data backfill, which should finish by Tue, Oct 7.
• Analyzed Google-reported clickthrough and Google-referred pageview data

This week, I:

• Analyzed Comscore and unique device data
  • Lots of contradictory signals, plus noise due to the ongoing traffic data backfill, which should finish by Tue, Oct 7.
• Analyzed Google-reported clickthrough and Google-referred pageview data

In T405533#11235980, @JAllemandou wrote:

I'm very sorry for the misunderstanding here. I made a vocabulary mistake with the French false-friend "demander". I was really not meaning that you were forcing for the change, I'll be careful when using the word "demand" in future.

In T402324#11237257, @Mayakp.wiki wrote:

speaking for our needs, I think that's totally fine! :)

In T405533#11213571, @JAllemandou wrote:

We have changed the source-field used for unique-devices computation when adapting to the .m subdomain removal. We now use the project field used in pageviews, which doesn't have the www for the projects you mention.

In T403437#11227488, @nshahquinn-wmf wrote:

Since my last update, I:

• Came up with a shortlist of instrumentation data streams I can use
• Started analysis template for Google Search Console data
• Evaluated a sample of SimilarWeb traffic data shared by Nino in Comms
• Got access to Google Search Console data in BigQuery and figured out it doesn’t go back far enough to be useful
• Picked 8 wikis to focus on and got access to them all in Google Search Console

Since my last update, I:

• Came up with a shortlist of instrumentation data streams I can use
• Started analysis template for Google Search Console data
• Evaluated a sample of SimilarWeb traffic data shared by Nino in Comms
• Got access to Google Search Console data in BigQuery and figured out it doesn’t go back far enough to be useful
• Picked 8 wikis to focus on and got access to them all in Google Search Console

@JAllemandou I know you've already merged the task, but FWIW, I just took a look and it makes sense to me! Thank you very much for smoothing out this little bump 😁

In T403863#11220407, @BTullis wrote:

While we are at it, I think that we should suppress this warning:

We are encrypting our connection over SSH, so this warning does not really apply in our setup.

FYI, assuming that the May backfill finished at the end of yesterday and using @mforns's estimate that a month takes about 3.5 days to backfill, I calculate that the hourly backfill will probably finish by the end of Monday, October 6.

In T401666#11211709, @JAllemandou wrote:

While I understand the not-canonical concern, it happens that we've been using the non-www domain since the beginning of unique devices for those domains, at least in cassandra.
The fix is not cheap as not only does it make us change and backfill unique-devices again, but also we'd need to change wikistats and reload cassandra, deleting the old rows (not easy).
I suggest we keep it as is, even if not canonical. Let me know if it feels ok for you :)

@JAllemandou I filed T405533 to continue the conversation.

@JAllemandou there's actually a bug in the new data: the domains names for Wikidata, Wikifunctions, and MediaWiki.org are missing the the leading "www." (e.g. the data has "wikidata.org" rather than "www.wikidata.org"). The versions with the "www." are canonical: they are what appear in the address bar, in the SiteMatrix, in canonical data, and so on.

I was actually coincidentally working on search console documentation, so I've gone ahead and made the necessary updates. The internal documentation is now at a new "search consoles" Office-Wiki page, and I've updated the Wikitech page to point there.

Based on the age of the task, I'm going to guess this is no longer an issue. If it still is and ITS still recommends an escalation to SRE, this should be reopened and tagged with SRE.

@NBaca-WMF I was actually coincidentally working on search console documentation, so I've gone ahead and made the necessary updates. The internal documentation is now at a new "search consoles" Office-Wiki page, and I've updated the Wikitech page to point there.

In T403437#11182901, @nshahquinn-wmf wrote:

Last week, I:

• Continued pursuing access to SimilarWeb data and Google Search Console data in BigQuery
• Started analyzing Comscore data
• Dug through instrumentation data streams in search of one that can proxy for mobile web pageviews
• Got a private GitLab repo to store the analysis (T404533)

Last week, I:

• Continued pursuing access to SimilarWeb data and Google Search Console data in BigQuery
• Started analyzing Comscore data
• Dug through instrumentation data streams in search of one that can proxy for mobile web pageviews
• Got a private GitLab repo to store the analysis (T404533)