In T401666#11211709, @JAllemandou wrote:

While I understand the not-canonical concern, it happens that we've been using the non-www domain since the beginning of unique devices for those domains, at least in cassandra.
The fix is not cheap as not only does it make us change and backfill unique-devices again, but also we'd need to change wikistats and reload cassandra, deleting the old rows (not easy).
I suggest we keep it as is, even if not canonical. Let me know if it feels ok for you :)

I'm not concerned about Cassandra at all: it's never used directly by analysts, nothing has changed, and there's no friction for AQS users as the API already accepts "wikidata", "wikidata.org", or "www.wikidata.org".

But I do think this is something that should be fixed eventually in the Data Lake tables, although I can understand if it needs to be deferred indefinitely (as so many bugs always are 😁) because the fix is expensive. I've tried to explain my reasoning more in the description.

How difficult would it be simply to alter the Data Lake tables and change the Cassandra loading to remove the "www." in the future? We must have already been doing something like that as previously the Data Lake data did have the "www." for the main domain, but it never did in Cassandra.

In T405533#11212441, @nshahquinn-wmf wrote:

How difficult would it be simply to alter the Data Lake tables and change the Cassandra loading to remove the "www." in the future? We must have already been doing something like that as previously the Data Lake data did have the "www." for the main domain, but it never did in Cassandra.

We have changed the source-field used for unique-devices computation when adapting to the .m subdomain removal. We now use the project field used in pageviews, which doesn't have the www for the projects you mention. The fix you're demanding would impact not only unique-devices but also pageviews and related datasets. We could hard-code adding www. as a prefix for the domain you mentioned when loading unique-devices, but this would not be future-proof for new domains formatted this way.
Let me know what you think, but to me the cost of change is too high in this case.

In T405533#11213571, @JAllemandou wrote:

We have changed the source-field used for unique-devices computation when adapting to the .m subdomain removal. We now use the project field used in pageviews, which doesn't have the www for the projects you mention.

Thank you, this is helpful context!

The fix you're demanding

Just to be clear, I'm not demanding anything. I'm requesting a fix I think would be useful, and as I understood it, you were kindly taking the time to discuss it and explain your reasoning, even though you have the authority to make the final decision and are already very inclined to say no.

Please let me know if you feel I have overstepped a bound here.

would impact not only unique-devices but also pageviews and related datasets.

I certainly agree we should not change the pageview datasets; their use of a unique code scheme is not ideal, but it has been that way for a long time and the costs of the change would be greater than the benefits. That is different from the case here, where I'm suggesting staying consistent with the previous identification scheme (although it may be, as you say, too expensive to do so).

We could hard-code adding www. as a prefix for the domain you mentioned when loading unique-devices, but this would not be future-proof for new domains formatted this way.

Yes, hard-coding does not feel like a great solution. I can offer the alternative that canonical_data.wikis (which my team is committed to maintaining) has both the pageview code and the canonical domain name, so it could be used to get the correct domain even for wikis which are added in the future.

Let me know what you think, but to me the cost of change is too high in this case.

I would like to see this be done for the reasons I mentioned, but I agree your time is valuable and you are the expert on how much work this would require.

To my naive eye, it seems like it might be relatively simple to (1) leave pageview data unchanged, (2) use canonical data to translate between the pageview code and the canonical domain, (3) take advantage of Iceberg to alter the existing unique device data relatively easily, and (4) strip any leading www. when loading to Cassandra. But, obviously, I could be wildly wrong about that.

I don't have any other potentially-useful input I can provide, so please feel free to make your decision now. I won't be outraged if you decline this.

In T405533#11235615, @nshahquinn-wmf wrote:

In T405533#11213571, @JAllemandou wrote:

The fix you're demanding

Just to be clear, I'm not demanding anything. I'm requesting a fix I think would be useful, and as I understood it, you were kindly taking the time to discuss it and explain your reasoning, even though you have the authority to make the final decision and are already very inclined to say no.

I'm very sorry for the misunderstanding here. I made a vocabulary mistake with the French false-friend "demander". I was really not meaning that you were forcing for the change, I'll be careful when using the word "demand" in future.

Please let me know if you feel I have overstepped a bound here.

Absolutely no overstepping a boundary here. It's great you ask for changes, and that we can talk about them :)

To my naive eye, it seems like it might be relatively simple to (1) leave pageview data unchanged, (2) use canonical data to translate between the pageview code and the canonical domain, (3) take advantage of Iceberg to alter the existing unique device data relatively easily, and (4) strip any leading www. when loading to Cassandra. But, obviously, I could be wildly wrong about that.

With the use of the canonical table it's way better (no hard-coding). I'll apply the plan you're mentioning just above :)

In T405533#11235980, @JAllemandou wrote:

I'm very sorry for the misunderstanding here. I made a vocabulary mistake with the French false-friend "demander". I was really not meaning that you were forcing for the change, I'll be careful when using the word "demand" in future.

Now I'm very sorry too! I should have checked with you before assuming so much based on a single word, especially when I've known you for a long time and you've been nothing but kind 😊

To my naive eye, it seems like it might be relatively simple to (1) leave pageview data unchanged, (2) use canonical data to translate between the pageview code and the canonical domain, (3) take advantage of Iceberg to alter the existing unique device data relatively easily, and (4) strip any leading www. when loading to Cassandra. But, obviously, I could be wildly wrong about that.

With the use of the canonical table it's way better (no hard-coding). I'll apply the plan you're mentioning just above :)

Oh, wonderful! I'm so glad it's helpful.

FWIW, I would implement this as taking as the domain from canonical data when found, but falling back to just adding ".org" otherwise. The main reason I saw is that we have unique device data for wikimediafoundation.org (in this case the canonical version in the address bar is without the "www."), which is not found in canonical data because it's not a wiki. There are also a handful of weird domains with a tiny bit of data that seem like a mix of real domains that we don't want and invalid domains that slipped through validation. Nothing that really needs to be worried about (although it wouldn't hurt to just delete that stuff), but adding the ".org" would also do the most reasonable thing in any future cases like that. Here's the full list:

Structured lists of the domains:

('uz.wikimedia.org', 'mo.wikipedia.org', 'wikipedia.org', 'als.wiktionary.org', 'za.wikimedia.org', 'als.wikiquote.org', 'als.wikibooks.org', 'mo.wiktionary.org', 'slo.wikimedia.org', 'wikimedia.org', 'yue.wikibooks.org', 'noc.wikimedia.org', 'download.mediawiki.org', 'zh.wikidata.org', 'yue.wikiquote.org', 'login.wikipedia.org', 'en.wikidata.org', 'zh.wikifunctions.org', 'pk.wikimedia.org', 'ru.wikidata.org', 'meta.wikidata.org', 'fr.wikidata.org', 'd1j57rhr48eenml3mfb0.wikidata.org')

Change #1194885 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Fix unique-devices per domain domains

https://gerrit.wikimedia.org/r/1194885

joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1743

Fix unique_devices canonical domain

Change #1194885 merged by Joal:

[analytics/refinery@master] Fix unique-devices per domain domains

https://gerrit.wikimedia.org/r/1194885

Mentioned in SAL (#wikimedia-analytics) [2025-10-15T15:18:31Z] <mforns> Deploying Refinery at 94efa6e8221602a331c19c39ea909eeaa90d98b4 for T405533 unique devices domains

Mentioned in SAL (#wikimedia-analytics) [2025-10-15T16:05:56Z] <mforns> Finished deploying Refinery at 94efa6e8221602a331c19c39ea909eeaa90d98b4 for T405533 unique devices domains

mforns merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1743

Fix unique_devices canonical domain

Patch has been deployed, last 2 days of backfill ongoing, calling this done :)