Folks, here's a plan of action.

Seddon and I are meeting on Friday. We'll have a concrete action plan (or
the beginnings of one) to share on Monday.

We're meeting with them in the next couple of weeks to troubleshoot our scraping problems. Will report back once we learn more.

Manager here: I approve.

This has been reported to Google. We're waiting for them to get back.

@Jdlrobson Did the frontend standards group meeting come to a conclusion, or did it bring up any new insights not already shared in this ticket?

Thanks for all the inputs.

@elukey Do we have another idea on the table aside from asking a team of Android devs (3) to maintain a recommendations service?

This has been granted for a period of one year.

@Soda I need a gmail account to provide access to.

OK, having talked to some folks in the Enterprise org and other teams and having eliminated a few possible problems, the one I'm investigating now is the possibility that for some reason Google's bot is ratelimiting itself. I'll continue to post any findings here.

Thank you for the supporting links.

Here are some unindexed articles (confirmed from search console and from Google). I came upon them by simply hitting "Zufällige Seite" (Random Page) on de Wikisource and checking if the resulting page is indexed at all.

@Xover That's very useful to know.

@Soda Yeah Navboxes would indeed have helped.

@Krinkle Could you please answer the question I had above when you have a chance?

As far as I can tell, a lot of the pages that aren't appearing in the index are simply not linked to from within the Wiki. There are no sitemaps any more, and Special:Lonelypages is uncrawlable because there's a robots.txt rule blocking all Special: pages from being crawled.

I think we can close this task for now.

If the volunteer is given access to Search Console data, they will be able
to examine the following information:

• which queries lead to search results on Google that have Wikis in them
• which results get clicked on by users
• the total query volume that has search results from the Wikis
• Whether a given page from a Wiki property has been indexed or has

problems preventing it from being indexed

• a breakdown of the above information by device platform, country, and

date.

Manager approval if needed: approved

I've responded on the ticket with the volunteer. I'll handle it once they get the NDA and C-level approval out of the way.

Please assign this to me once C-level approval and NDA have been taken care of.

No strong opinions either way, but it seems to me that the examples that @pmiazga provides aren't all user types. "admin", "group:something", or "en_wiki:admin" are not really user types. They're memberships or privileges, which can grow and change as needed. Isn't there another way to represent those? I would imagine that a user type is something that's largely permanent, while privileges (through memberships) are expected to change.

Silly question: @Krinkle Given that robots.txt has a Disallow: /wiki/Special: rule, how do search engines read the LonelyPages or RecentChanges pages? As far as I can tell these pages aren't being indexed. Am I missing something?

Apologies for the ridiculous delay. I have Wikisource search console access now and am looking at it.

Manager approves.

WMF Staff who will need this on an ongoing basis. No expiry.

Perhaps @Dbrant knows? I remember Jdlrobson mentioning that they did some work together on this.

Rather than speculate on this one feature in isolation, I recommend letting @ovasileva decide and prioritize based on what else needs doing. She likely to have a more holistic view and more insights from the community.

Just had a conversation with @Catrope . I'm comfortable owning the risk and understand what that entails.

That is not the case. Quoting from T326816 (please read for details):

Please note that any changes to this extension might first need to be
discussed in T326956.

I've mostly focused on search performance and stats for the Wikipedias and haven't had a chance to set up and build an understanding of where we are with Wikisource yet. That's why I don't have an immediate answer for this problem.

The vendor replied with numbers over the last two days. Here are some numbers that might help decide on a strategy. There is a new dump of IPs every day. Between the last two successive dumps (today and yesterday) there is:

• 10-12% daily change
• 1.7M New entries
• 2M Aged off entries (i.e. dropped/removed)

Thanks for replying, @Ladsgroup

Are DBA and SRE folks aware that this entire database will essentially be wiped every one or two days and recreated from a dump? Does that complicate things in any way?

Apologies, I don't have access to wikisource. @mpopov does probably.

Trying to summarize the discussion above, please point out where incorrect:

Also granted access to ptwiki and hiwiki (desktop + mobile) as requested by @ovasileva

English has also been added. Thanks for the action, @jcrespo. Noted, will notify the on-duty next time. Modified the wiki accordingly.

I've granted access. @jcrespo could you please add the expiration date to maint-announce and close the ticket?

Access has now been revoked.

Manager is OOO.
Skip-level Manager here, approved (if needed).

@phuedx Was the exclusion of is_bot intentional?

Just checked, these are safe to delete.

Option 1: "Inline after the lead section" has some advantages.

1. It resembles the previous state and allows the menu to go back to its previous narrower form (now that it doesn't have to match widths with the ToC which needs to be wider). This will probably leave people that like narrow windows largely unaffected.
2. Yes, in that case, there's no floating ToC any more, but then if the window is that narrow, there's no room for one anyway.

I've given the above-mentioned e-mail address access to the two English
Wikipedia domains (en.wikipedia.org and en.m.wikpedia.org).

@jcrespo I've been involved in this discussion so I know what's going on here. I've updated the ticket to reflect what they need. I can take care of providing access; how do I put an entry in the maint-announce calendar to revoke access on the expiry date?

That's _very_ interesting. Thanks for doing this! Would it make sense to just test this theory a bit more?

Works as expected, thank you.

Update: @AndyRussG already has access to Bing, as of yesterday. I'm working on a process to follow if/when more people request access.

Apologies, this fell through the cracks somewhat. I'll reply conclusively
before the end of this week.

@JMeybohm Given that we didn't actually add each language domain one by one, there should be, a "wikipedia.org" entry listed as a "Domain Property" along with all of the per-language Wikipedias on Search Console. When you select that one and add me as an "Owner" that should do it. There isn't an actual "Administrator" ACL level as far as I can tell - it's just "Owner", "Restricted", or "Full".

It got a bit trickier.

I confirm that Bing.com verification has worked properly. However, for Yandex it seems they need the TXT entry to be under www.wikipedia.org and not wikipedia.org. Sent out patch https://gerrit.wikimedia.org/r/c/operations/dns/+/768664 to that effect.

Just had a discussion with @jcrespo. To understand what each of these webmaster consoles provides and what ACLs and such they support (so that we can have a process around giving access to it), we want to first add a few domains to each and learn more about the platforms. Stand by for a DNS patch.

You're absolutely right to be concerned about traffic from search engines. That said, I'm familiar enough with how this works to be comfortable owning it, and my PM counterpart and I (and a half dozen or so other people at the Foundation including @AndyRussG) gaze at this data often enough that we'd know if something were amiss, and we'll obviously be watchful if we decide make any changes at all.

Just filed https://phabricator.wikimedia.org/T302617 to start discussing domain ownership verification on various consoles with SRE. I'll be unavailable for a few days starting now, but I've already had a discussion with @AndyRussG about this.

In case someone's wondering, DuckDuckGo doesn't actually have a webmaster console. Strange.

I'm working on this, please stay tuned on this ticket; ETA tomorrow.

Just did. All good, thank you and sorry for the trouble!

My new public key is here: