Well, how can we check this? (I honestly wonder.)

Tangentially, why was old-bugzilla disallowing everything? https://old-bugzilla.wikimedia.org/robots.txt (which led to wikibugs-l mirrors being the only google search results... >.< )
I assume something to do with security - and does this need to be tested/tasked for phab?

Phabricator is incredibly search-engine-friendly. The number of tasks that they index is more a factor of how important the search engines think our content is. Google will likely index everything, though they might detect a lot of it as duplicates due to the mailing list mirrors who have the original text from the old setup where bug reports went bugzilla->mailing list->list archive->search index.

Search engines want to index everything of value on the internet. duplicates of the same content are not considered to be very valuable.

Do we need a sitemap?

Search engines should eventually re-crawl all the web archives of wikibugs-l. Since even http://bugzilla.wikimedia.org/robots.txt redirects to phabricator.w.o, they should following the redirects from bugzilla to the good Phabricator "TNNN" URLs.

In T76991#834238, @Quiddity wrote:

Tangentially, why was old-bugzilla disallowing everything?

That's T15881, which was fixed then regressed (and now with Phabricator it's correctly Declined). I couldn't find a security rationale for it; if a search engine can crawl to a security risk then telling the world "don't do that" isn't addressing the problem :)

Search engines should eventually re-crawl all the web archives of wikibugs-l

wikibugs-l is not archived and additionally lists.wikimedia.org prevents all crawling (which btw is catastrophically bad, but off topic here). Which "mirror" archive do you have in mind, and what makes you think search engines follow its links?

That's T15881, which was fixed then regressed (and now with Phabricator it's correctly Declined). I couldn't find a security rationale for it;

Merely a performance issue, IIRC. At some point kaulen was overloaded and changing robots.txt felt easier than actually looking into the cause.

Using Google as a reference:

• Searching for "Ensure that all phabricator tasks are crawled by search engines": "No results found for "Ensure that all phabricator tasks are crawled by search engines". Results for Ensure that all phabricator tasks are crawled by search engines (without quotes):"
• Searching for "Re-juggle shard and replica counts in Elasticsearch" (an older task, not migrated from Bugzilla): correct result found.
• Searching for "optionWidget with check icon should not appear for invalid entries in the category input field" (a Bugzilla report migrated): Bugzilla URL appears first, Phabricator task right after. Good.

Haven't tried others, but your help running these tests is welcome.

All in all it looks like they are catching up.

just search google for the task id: T102920 for example, is indexed.

Thanks a lot for clarifying! I can reproduce this, indeed.

But searching for phabricator T842 works for me

In T76991#1532842, @mmodell wrote:

But searching for phabricator T842 works for me

It's not about T842 but T842** like T84213

@Aklapper: T84213 is protected, of course it isn't indexed.

Every example I've seen so far is either intentionally hidden by security policy or not reproducible.

The funny thing with Google is that searching for "phabricator T105000" works for any valid task number, whereas searching for "phabricator wikimedia T105000" brings a lot more problems and many times the same tasks are not found. "Wikimedia" is in the title of the pages and you can see it transcribed in the results, so it is really strange.

Google doesn't treat wikimedia as wikimedia. They conflate wikimedia and wikipedia, essentially this makes wikimedia almost a generic term for the purposes of google's algorithm.

In T76991#1535014, @mmodell wrote:

T84213 is protected, of course it isn't indexed.

Ah true that. I realize that range of task IDs are the tickets imported from RT (by default with restricted access).

Which might make this task invalid.

Will presumably get worse with T110710. What avenues of access are left for crawlers to find all tasks?

Crawlers should find all public tasks (as in https://phabricator.wikimedia.org/T987654321 ).
If they do not, please provide a testcase.

@Nemo_bis: only workboards are blocked, the crawlers can still find all of the projects and from there they can find all tasks via the projects' task lists. Additionally, There is the master maniphest task list at https://phabricator.wikimedia.org/maniphest/