Keywords: shell

Bulk-assigning open BZ issues to Fred.

fvassard wrote:

Added the following:
Allow: /show_bug.cgi
to robots.txt.

There is a slight chance that this might cause additional load, so I will be monitoring the webserver to make sure that there is no noticeable performance hit.

I believe the change above should do the trick, but time will tell.

This isn't fixed. When I search for MediaWiki error messages in Google or Bing, I *never* get results from bugzilla.wikimedia.org. For example, search for

notice Uncommitted DB writes transaction from DatabaseBase::query MessageBlobStore::clear

All I get is useless osdir.com and gmane rehashes of mail threads involving bug 56269. The bug should be the first result.

I'm pretty sure it's because in https://bugzilla.wikimedia.org/robots.txt, the line

Disallow: /*.cgi

blocks any .cgi URL, including https://bugzilla.wikimedia.org/show_bug.cgi?id=56269 . Even though

Allow: /*show_bug.cgi

comes later, "The first match found is used." The fix is to move the Allow line first, compare with Mozilla's https://bugzilla.mozilla.org/robots.txt.

Is this file [1] the one needing to be changed? i.e. is it the file used to generate https://bugzilla.mozilla.org/robots.txt, or is it just a random copy sitting in Git?

[1] https://git.wikimedia.org/blob/wikimedia%2Fbugzilla%2Fmodifications.git/master/extensions%2FSitemap%2Frobots.txt

No that's not the file. /extensions/Sitemap will be killed soon (bug 33406).
No idea if robots.txt is in Git. If it is, it's somewhere in operations/puppet/modules/bugzilla

It doesn't seem to be in that repo. Per "you still have to copy upstream bugzilla itself to the bugzilla path and clone our modifications from the wikimedia/bugzilla/modifcations repo" [1] I guess it is not in Git, similar to the rest of the BZ server-side files.

This needs shell/ops then.

[1] http://git.wikimedia.org/blob/operations%2Fpuppet.git/18f755cfecf9abdd23a0678e82f278188e059379/modules%2Fbugzilla%2FREADME.md

(In reply to spage from comment #4)

This isn't fixed.

It worked for a while, it's a recent regression. No idea when the robots.txt block was reintroduced.

We could add this file to puppet but it's low priority.