Page MenuHomePhabricator

Windows 11 missing in analytics ?
Open, HighPublicBUG REPORT

Description

I was looking at the OS analytics, and was surprised by how fast Windows was falling in the metrics, much faster than another desktop, macOS.

Looking a bit further, it seems most likely to me that Windows 11 is simply not being detected and grouped incorrectly into the category other in the dashboard. 98% of Windows is supposedly Windows 10, which seems near impossible to me.

Screenshot 2023-09-20 at 14.09.08.png (536×539 px, 108 KB)
Screenshot 2023-09-20 at 14.09.45.png (615×1 px, 38 KB)

The community makes decisions on this information, it would be nice if the metrics were accurate

Event Timeline

@Milimetric Am I misunderstanding the graphs ? It just seems really strange to not have any Windows 11 listed there, when according to google, 20% of windows is 11, and just 70% is windows 10 ?

Looking a bit further, it seems most likely to me that Windows 11 is simply not being detected and grouped incorrectly into the category other in the dashboard.

https://meta.wikimedia.org/wiki/Config:Dashiki:SimpleRequestBreakdowns

This is a high level analysis of user agents browsing projects hosted by the Wikimedia Foundation (WMF). It was created to support decisions by our engineers and product owners, as well as to provide general information about our project to interested folks such as researchers. The WMF Analytics team is responsible for maintaining the data and dashboard. User agents that do not account for a significant percentage of our traffic are rolled up into an 'Other' group, for privacy reasons.

It's definitely unclear if it's going into Windows (as it should) or other...

If we filter to only Windows 10 (because it's an option, and more recent)...

Screenshot 2023-09-20 at 14.54.23.png (709×2 px, 144 KB)

And then add in Windows and some other versions...

Screenshot 2023-09-20 at 14.56.57.png (621×2 px, 169 KB)

So "Windows" got grouped in ~May 2018 into one big thing, so presumably we shouldn't be seeing Windows 11 separately?

And then for desktops..

Screenshot 2023-09-20 at 14.58.29.png (615×2 px, 261 KB)

But the version should still show in the major breakdown right ? Same as iOS family can go into major breakdown…

Milimetric added a subscriber: VirginiaPoundstone.

I vaguely remember this thing in 2018... Windows did get grouped up, but I agree with the DJ's points and that this data makes no sense without at least some kind of annotation.

Pinging @VirginiaPoundstone and putting on our board.

This feels like an effect of Chrome's UA reduction where in Phase 5, the device OS was replaced. See Rollout details here

image.png (587×1 px, 129 KB)

cc @mforns

This feels like an effect of Chrome's UA reduction where in Phase 5, the device OS was replaced. See Rollout details here
cc @mforns

That makes WAY more sense !

OK, the downward spiral of Windows is... impressive in that case. I really wonder if that is all due to mobile devices being _used_ more by individuals, or if the amount of individuals using Windows is stable and it's just the amount of individuals only using Mobile is growing this rapidly (new audience coming online?).

Ok, so the action here would be to label the data better, and add an annotation for Phase 5 and any other big changes.

Hi all! Adding some thoughts:

Continuous drop of Windows
I think the continuous drop of the Windows operating system (os family) is legitimate, and it indicates a true decline of Windows, which has gone from 33%* of access to Wikimedia sites to 20%* in a bit more than 2 years.
(*) These stats do not include the Windows traffic that fell into too small buckets to pass the privacy threshold, and thus was labeled as "Other". Probably, if all Windows traffic of the "Other" bucket was counted, we'd see a decline from around 36% to around 23% in the same period. I'll write some more on the "Other" bucket below.
My guess for the main reason of the decline is: Traffic shifts from desktop to mobile/tablet, and Windows Phone is much less popular than Windows Desktop.

Broken Windows chart?
Before May 13th 2018 the UA parser we use classified Window devices by putting both the OS family and OS major in the os_family field of our UserAgent map. That's why we see all versions of Windows in the chart above, while other OSs are only mentioned without the major version. This was wrong, and was fixed upstream on May 13th 2018. After that, all Windows traffic is labeled correctly: os_family = Windows and os_major = [7, 8, 9, 10, etc.]. Since the chart shows only the OS family, all Windows lines got collapsed into 1. I believe this chart is technically correct (from May 13th on...), but agree with @Milimetric that it should have an anotation.

Windows 11 missing
The Chrome UA reduction's phase 5, start 2023, would have affected this. But, earlier than that, something else produced the lack of Wikdows 11 traffic in the browser stats: Windows 11 was never represented as such in the UserAgent string, see: https://developers.whatismybrowser.com/learn/browser-detection/user-agents/detect-windows-11-user-agent
According to the link, there's no way to detect Windows 11 from the UserAgent string... Not sure if we can do it in another way? Until then, all Windows 11 traffic is labeled as Windows 10. :-/

The "Other" bucket
The link @Milimetric pasted explains it all. And we can reduce the "Other" bucket size with some optimizations, but I just wanted to add that I believe the naming is wrong and confusing... It should not be "Other". "Other" indicates that traffic belongs to none of the other mentioned OSs. But the "Other" metric contains the smallest buckets that didn't make it over the privacy threshold, so it can contain Windows traffic, Android, iOS, etc. A better name could be "Unknown" or "Hidden".

Conclusions

  • Agree to annotating the data better! (don't think Chrome UA reduction phase 5 is the culprit though)
  • Let's prioritize the optimization of the metric aggregation so we can reduce the size of the "Other" bucket.
  • Should we rename the "Other" bucket to "Unknown"?