https://gerrit.wikimedia.org/robots.txt : gerrit is currently well crawled, e.g. https://www.google.it/search?q=site%3Agerrit.wikimedia.org .
It seems most instances don't allow crawling of the code review portion, would need to confirm it is possible.
https://secure.phabricator.com/robots.txt
https://developer.blender.org/robots.txt
http://reviews.llvm.org/robots.txt
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Call git submodule sync before git submodule update | repos/releng/scap!130 | dancy | master-c5d7 | master |
| Status | Subtype | Assigned | Task | |
|---|---|---|---|---|
| · · · | ||||
| Resolved | Aklapper | T22 Identify features Bugzilla users would miss in Phabricator | ||
| Resolved | Qgil | T23 Identify features Gerrit users would miss in Phabricator | ||
| Invalid | Qgil | T157 Phabricator should sustain crawling by internet search engines | ||
| Resolved | Qgil | T161 Set Applications -> Differential -> Default View Policy to "Public (No Login Required)" | ||
| · · · |
qgil wrote on 2014-04-29 23:12:15 (UTC)
I guess in a worst case scenario we can patch the robots.txt of our instance, applying the same policy as Gerrit, Bugzilla, etc?
mattflaschen wrote on 2014-04-30 02:10:30 (UTC)
I think this is actually just a consequence of T262: Review holders of commit rights in WMF deployed extensions (if anonymous users can't view, neither can GoogleBot)..
The current robots.txt excludes only /diffusion/ , the repository browser (e.g. http://fab.wmflabs.org/diffusion/MW/repository/master/ ). It does not exclude Differential reviews, e.g. http://fab.wmflabs.org/D1 .
We're currently forbidding crawling the code at https://git.wikimedia.org/robots.txt too . It might be nice to allow crawling the code somewhere (if it can be done performantly), but since our other repository viewer does the same there's no regression I see here.
Differential (code review) entries D1 should show up in search, though. They do not because of T262.
I just searched for "Phabricator should sustain crawling by internet search engines" in Google, and this task appeared as the first result. Closing as Invalid. Please reopen if there is any content in this instance that should be crawled and it is not.