Page MenuHomePhabricator

Google Webmaster Tools - 1000 domain limit
Closed, ResolvedPublic

Description

Google Webmaster Tools apparently have a limit of adding 1000 "domains"/URLs and we did reach that limit and can't add any new ones. (as mentioned in T98283 and other places).

This is a ticket to discuss that issue separately from access requests to existing domains.

Options seem to be:

  • delete existing ones that are not needed or used
  • add an additional account , like a "noc2@"
  • contact Google and ask if the limit is set in stone or we can have an exception because Wikipedia

Event Timeline

Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn added projects: GoogleLogin, acl*sre-team.
Dzahn subscribed.

@dr0ptp4kt has said he'll look into options 1 and 3.

What do we use Google Webmaster Tools for?

One warning about #1 though it is still good to look into it since I haven't actually tried.

My understanding from the documentation is that we can't actually 'delete' a site from our count until we 'un verify' the flight and since we verify at the 'top level domain and many of the sites are at the lower levels that could be incredibly annoying or difficult without blowing the thing up and trying to start new.

What do we use Google Webmaster Tools for?

My main use case is communicating with google through the automated means. It's how we get reports of RTBF takedowns for example (or the occasional complaint) and it makes it much easier to force the google purge when we have a problematic page stuck on a bad version.

I know others in product and tech have used it for the statistics, especially with mobile, and I think it is tied in somehow to the google play account for apps as well (and so that there are links in search to our app for articles) but I'm not the right person to go into depth about other use cases.

We also use it for fixing some erroneous results in the Google Search; such as when they're marked as being hacked or similar by Google

ArielGlenn triaged this task as Medium priority.

I've reached out to a contact for help with bulk site add/remove as well as bulk site show on a per-user basis (e.g., for the small set of restricted users who may need it for their job responsibilities).

Update: I've received some guidance. May be a bit until I can take action on it. In a nutshell, though:

  • Not possible to increase the sites limit
  • We could consolidate the multitude of distinct sitemaps.wikimedia.org subfolders that each have a site entry down into one distinct entry to free up space

As hinted earlier in the ticket we could create another username and add a bunch of sites to it. We may end up wanting to do this anyway in order to add mdot subdomains as well. But I'd like to ponder on this a bit over the coming week and review https://developers.google.com/site-verification/v1/webResource and https://developers.google.com/webmaster-tools/ before making any changes. Okay?

I've thought about this some more. I'm not too keen on creation of another @wikimedia.org account to hold more sites in it.

I have asked the contact at Google if a feature request could be put in to increase the sites limit and to also support the notion of granting a particular user read-only access across all sites within an owner's Google Webmaster Tools (now called Search Console) profile automatically.

As for adding a bunch of additional sites, I think we should probably defer on that.

At the moment, it seems that the main thing that would be beneficial is to add https://www.wikipedia.org/ (which seems to be what's indexed) plus any strictly https:// .wikipedia.org subdomains, deferring any further site changes until a later date.

Looking on an iOS 8 device, it seems that searches are yielding URLs that are on m.wikipedia.org subdomains on http:// (e.g., http://en.m.wikipedia.org/wiki/Barack_Obama). So it may make sense to add, say, the top 15 mdot Wikipedia langwikis there on http:// (plus any currently strictly https:// ones).

I have asked the contact at Google if a feature request could be put in to increase the sites limit and to also support the notion of granting a particular user read-only access across all sites within an owner's Google Webmaster Tools (now called Search Console) profile automatically.

@dr0ptp4kt: Did you get any feedback?

@Aklapper, I'm pinging on the thread.

@dr0ptp4kt: Was there any outcome?

@Aklapper, thanks for reminder. Yes, the feedback was suggestive of there not being a means of modifying the system's behavior. It was suggested people need to concoct their own custom bulk addition/removal tools using the existing APIs (it's doable, at first glance, but requires careful attention to details!) and, for the case of orgs like ours where stuff literally can span thousands of domain names, may need to split stuff into separate accounts if they care to have coverage across all of them as hinted at in the Description (of course, being judicious in which sites to have in one profile is another approach).

Another aspect that both makes this more timely and possibly presents another downside of splitting the account:

It's now possible to associate an Android app to the Search Console, see
https://support.google.com/webmasters/answer/6178045 and
https://googlewebmastercentral.blogspot.com/2015/05/rolling-out-red-carpet-for-app-owners.html

We got advised to use this to investigate and fix a significant drop in the Wikipedia app's install numbers. Plus it seems that connecting the app is a good idea anyway, for example to enable the "fetch as Google" feature. IIRC the app is already connected (in Google's mind) in other ways to the Wikipedia domains at least, I'm not sure how it would work out if it were to be added to a separate Search Console account.

Update: @Dbrant was able to successfully add the Wikipedia app to the Webmaster Tools of the "android@wikimedia.org" group. (It's not yet clear what this means for Search Console features that rely on the app being connected to our main domains.)

I am going to close this as resolved as the one who created it back in 2015. Since we haven't had updates here since 2016 and people are using the search console all the time it seems safe to assume there is no discussion needed anymore/right now.

If this was a current problem for people I am sure we would have gotten a newer ticket meanwhile.

Dzahn claimed this task.