Page MenuHomePhabricator

Console domain and property access request
Open, Needs TriagePublic

Description

Hello -
Recently a request came through (specifically https://phabricator.wikimedia.org/T381204 ) that required us to submit a form to ensure that we have opted out of a Google feature which has the potential to impact all domains we own within the Search Console. For my work with the Web Team and others, as well as the work of my manager Suman, we often need to make SEO-related changes to properties within the search console.

This leads to two questions:

  1. For the specific request, is there a way to get a list of all domains and properties that we own as an organization, so I can be sure to put them all down?
  2. For future requests in this area: Currently my manager, scherukuwada@wikimedia.org , owns many of these domains, but not all, thus our current inability to get the full, comprehensive list. I also currently own only a subset. He and I discussed this, and decided it made sense to ensure we have ownership over these domains for the sake of both this particular request as well as requests in the future. Suman suggested that SRE, as the group that controls access to the service account that underlies the search console, could grant such access for future reference and use.

Please advise if this request is better directed elsewhere, if it would be beneficial to have a followup conversation about this, or if there are other questions or concerns I can work through. Thank you!

Event Timeline

Thanks for the summary @NBaca-WMF.

  1. For the specific request, is there a way to get a list of all domains and properties that we own as an organization, so I can be sure to put them all down?

I know that I can check, e.g., for which domains we publish DNS TXT records to authenticate ownership, but I'm not sure off hand whether that's exhaustive and will need to confirm if there's a better way.

Edit: Alright, I now see that these correspond to our (validated) domain properties, while the question here might instead be focused on enumerating the path properties that should exist under them.

  1. For future requests in this area: [...]

Aligning ownership between you and @SCherukuwada, and ensuring that the set of owned domains matches #1, sounds reasonable to me. Also, we (SRE) seem like the right folks to do this, as it seems we manage domain addition and delegation of per-domain management.

I'll look into how to proceed with this.

@NBaca-WMF - When you get a chance could you please confirm the following:

  • That my (edited) interpretation for question #1 in T381904#10395283 is correct - i.e., you're interested in enumerating all the path properties that should exist in the console, corresponding to all extant subdomains of our domain properties.
  • That the request to align access with @SCherukuwada is specifically focused on (delegated) ownership of the domain properties (which, IIUC, should give you the ability to manage the path properties they contain).

Thanks!

Hi @Scott_French - Thank you for taking a look at this!

  1. Yes - this is a good summary. We can currently only see a subset of domains and properties, but I know there are many more out there, both for projects and language variants. If it's possible to generate such a comprehensive list of the full set of available properties and then add that full set to the console we would appreciate it!
  2. Delegated ownership should be fine, thanks for checking on this. Though with the caveat that at present even Suman does not seem to have access to the full conceptual set of properties and domains.

Thanks again for helping out with this, happy to answer any further followups!

Great, thank you @NBaca-WMF.

Alright, it seems like there are two different issues intertwined here:

Page annotations opt-outs

For T381204, a list of all relevant property permutations is needed, seemingly in the form of path-less "URL-prefix properties" (e.g., with all subdomain and scheme permutations). In addition, the requester must have Owner or Full permissions on the property.

What's unclear to me is whether the requester must have permissions in the console on a property that exactly matches vs. whether transitive permissions (e.g., from a "parent" domain property) are sufficient. @NBaca-WMF - Do you happen to know off hand?

In any case, for a given set of domains of interest (e.g., wikipedia.org) we should be able to generate this programmatically.

Access control model and permissions

Reviewing the properties that exist in the search console, there's inconsistency in structure (e.g., domains that do or do not have domain properties) and ownership.

At the very least, we need to identify exactly which domains both @NBaca-WMF and @SCherukuwada should have ownership over and correct that.

My possibly flawed understanding of the access model in the console is that if we consistently configure domain properties (e.g., like we have for wikipedia.org, in addition to the more-granular URL-prefix properties used to provide limited access) and provide delegated ownership, then we should be in the desired state (i.e., they should be able to add/remove/manage URL-prefix properties for those domains).

However, that's something we'll need more folks' thoughts on before proceeding.

I'm going to pull in Infrastructure-Foundations to consult on this, as the SRE subteam best suited to answer questions about managing access controls. In particular, input on both the set of domains for which we want to delegate ownership and a desirable structure by which we should do it.

I'll stay subscribed but unassign myself for the time being. Please feel free to assign back to actuate changes etc.

Hi @Scott_French - thanks for looking into this and the summary!

Alright, it seems like there are two different issues intertwined here:

Yes! thanks for separating these out.

What's unclear to me is whether the requester must have permissions in the console on a property that exactly matches vs. whether transitive permissions (e.g., from a "parent" domain property) are sufficient. @NBaca-WMF - Do you happen to know off hand?

I do not. We have separately started conversations with Google about this which are ongoing, we can ask about this when we next meet.

In any case, for a given set of domains of interest (e.g., wikipedia.org) we should be able to generate this programmatically.

Great, thanks!

I'm going to pull in Infrastructure-Foundations to consult on this, as the SRE subteam best suited to answer questions about managing access controls. In particular, input on both the set of domains for which we want to delegate ownership and a desirable structure by which we should do it.

Thanks for this! Infrastructure-Foundations , please let us know if you have any questions / happy to meet about this to talk further.

Alright, after discussion yesterday with @NBaca-WMF and @LSobanski, I believe the next steps in order to facilitate this involve building a couple of lists of domains:

  1. The full set of "content" domains under our control, which should in turn be in scope for opt-outs in T381204.

Specifically, this would be the set of domains where our organization directly serves content, which is a subset of those for which we publish DNS records more generally - e.g., domains served by ncredir and persistently redirected to some "authoritative" location would be out of scope.

  1. The subset of domains from #1 for which we publish google-site-verification TXT records for the purposes of Search Console verification.
  1. The subset of #2 that exist in the Search Console to our admin user as verified properties.

All these together should clarify what properties are missing and the work to be done to address that.

I suspect it should be straightforward to infer #1 and #2 from operations/dns, which should give us a sense of what which domains are not verified, but should be.

#3 is interesting, as it seems that there are at least a few domains with google-site-verification TXT records that are not present, which I don't have a good answer for (e.g., may be owned by yet another account).

@BBlack - My understanding is that you have been talking to Lukasz about some of the history with the verification TXT records. Could you (or someone else from Traffic) also take a look at the above? Specifically, whether the definition in #1 sounds right, and how best to proceed with generating that list?

@Scott_French I was about to have a look at what we have in the console but I see you already progressed on this. From I/F point of view it's totally fine to give access to all the domains needed. Let me know if I can be of any assistance.

Tagging Traffic in hopes that someone (especially with expertise in our DNS configuration) may be able to help advance the request in T381904#10464573, particularly items #1 and #2 (the correlation with Search Console properties in #3 can happen later). Thanks!

Tagging Traffic in hopes that someone (especially with expertise in our DNS configuration) may be able to help advance the request in T381904#10464573, particularly items #1 and #2 (the correlation with Search Console properties in #3 can happen later). Thanks!

At the risk of being hopelessly out of my depth these seem to be the zones matching #2, very simply from searching the zone files that have the goolg-site-verification TXT record.

cmooney@wikilap:~/repos/dns/templates$ grep "google-site-verification" * | awk -F":" '{print $1}'
mediawiki.org
wikibooks.org
wikidata.org
wikifunctions.org
wikimedia.org
wikimedia.org
wikimedia.org
wikimedia.org
wikimedia.org
wikinews.org
wikipedia.org
wikiquote.org
wikisource.org
wikiversity.org
wikivoyage.org
wiktionary.org
w.wiki

Number 1 is less clear to me tbh. A list of hostnames we direct to 'dyna.wikimedia.org', and thus serve content from, is here: P72657, and that list minus the first label is in P72658. Someone from traffic may be able to be more precise here or know of edge cases I'm not considering.

Thanks for doing that research, @cmooney! The lists look good. Note that our list of canonical domains is documented at https://wikitech.wikimedia.org/wiki/HTTPS#Foundation_canonical_domain_names. As far as search console access is concerned, I don't believe that's something Traffic has access to (ITS?)

Thank you both!

Great, the list in T381904#10502098 is consistent with what I have from when I did the correlation with Search Console properties in T381904#10464573:

#3 is interesting, as it seems that there are at least a few domains with google-site-verification TXT records that are not present, which I don't have a good answer for (e.g., may be owned by yet another account).

Also, TIL that we publish the of canonical domains on Wikitech! (https://wikitech.wikimedia.org/wiki/Domains#Canonical_domain_names)

Alright, so that means every one of the canonical domains has a google-site-verification TXT record with the exception of wikimediafoundation.org and wmfusercontent.org.

@NBaca-WMF - I think this covers #1 and #2, and #3 should be straightforward (albeit manual) to get from the Search Console (I've already done a cursory version of this and can share the result out of band).

@LSobanski - From our discussion on the 14th, I believe the next step is a decision on access control (I'm not quite sure where that discussion has landed).

BCornwall subscribed.