Yeah, we should announce before it happens.

We also need to remove https://github.com/wikimedia/operations-puppet/blob/bdd27ef834044a25c5a4e6057da940cab4c5941c/modules/role/manifests/gerrit/production.pp#L57

@greg do you know who would send the email out?

One of me, Mukunda, or Chad, probably.

Ok Thanks for replying.

@greg Hi, I spoke with @Dzahn about this as he will be in the same timezone as me I think he said we can most likely do this on wedsday but I'm not 100% sure but you can send the email out without say the actual date since I'm not sure if we will do it on Wednesday but we can say in the next 2-3 days git.wikimedia.org will be redirected. Then send a follow up email saying that git.wikimedia.org has now been redirected.

please.

Please coordinate with releng first.

@greg oh, how, what do we say.

I'll write it this afternoon Pacific time.

@greg ok, thanks.

Just to be clear, an email with one day notice is not enough :) we'll give a week's notice.

(I hope I cleared up the task ordering confusion with the title/description edits.)

@greg yep thanks.

@Dzahn Can you commit to a window next week to do this (merge all the needed puppet changes to redirect git.wikimedia.org urls to Phabricator)? This should happen at a pre-determined time. One of us (RelEng) will be around if needed of course.

Proposed date/time: Wednesday June 29th at 9am Pacific (after morning SWAT). Good? (I think you'll be back from Wikimania and working normal-ish hours then...)

Thanks!

email (pending date confirmation): https://etherpad.wikimedia.org/p/gitblit-deprecate-email

@greg i was just going to do it during Wikimania.. Honestly it seems almost overkill to me what we are doing on this task, given the historic stableness of gitblit that this actually improves. of course up to you though. if you want to wait then June 29th it is

Thanks @Dzahn. Just trying to set a good example :)

@Dzahn we can create the patch's then merge them on a date that the releng team decides on which they like the date 29 june 2016.

@greg I think we can do it on the date you like which is the 29 june 2016, you can now add the date to the mock email please. :)

In T137224#2395743, @Paladox wrote:

@greg I think we can do it on the date you like which is the 29 june 2016, you can now add the date to the mock email please. :)

Of course, that was the obvious next thing based on Daniel's response. No need to repeat :)

Scheduled: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160629T1600

I'll send the email tomorrow morning, it's almost midnight here.

@greg thanks and sorry.

email sent. The countdown begins :)

@greg thanks :)

Change 294867 merged by Dzahn:
Rewrite rules for git.wikimedia.org

https://gerrit.wikimedia.org/r/294867

Change 293747 abandoned by Dzahn:
switch git.wikimedia.org from misc to text cluster

Reason:
we are putting the rules on the phabricator host instead

https://gerrit.wikimedia.org/r/293747

Change 296138 had a related patch set uploaded (by Paladox):
Rewrite rules for git.wikimedia.org

https://gerrit.wikimedia.org/r/296138

The announced date should be put on hold until we had a real test in labs.

In T137224#2407860, @Dzahn wrote:

I merged the change that puts the rewrite rules on iridium (WITHOUT also switching varnish over, which was of course still waiting until the 29th). But they did NOT work. Actually they broke all the old Bugzilla redirects, everything got redirected to diffusion home page. The logging part in it adds an access log called "git.wikimedia.org" but has bugzilla.wm.org hits in it. Apparently this was not tested on something that is actually setup like the Phabricator instance. We need a real test on labs on one of the phab-0x instances where Apache is really like in production. I had to revert the production change.

The Gitblit -> Diffusion rules themselves did not break anything IMHO. They have been thoroughly tested and work perfectly. The issue will be somewhere else.

Idea: If the Bugzilla redirects handler (most likely not just rules, since something needs to add 2000 to the task number) are on the same machine, they may collide due incorrect settings / spearation of Bugzilla and Gitblit redirectings (ie. Bugzilla requests are now handled by the newly added file with rules instead ofthe original one). Just guessing the cause, but definitely not rules themselves standalone.

If I knew where the old Bugzilla -> Phabricator redirector handler is located, I could take a look on it...

Feel free to assign me again, should you need some more rewrite rules...

I guess the cause may be the asterisk in <VirtualHost> - https://httpd.apache.org/docs/current/vhosts/examples.html because then depends on order and if everything is on the same machine...

Yes, the problem appears when these rules are added to the existing Apache config in production, as you say they conflict with other rules for Bugzilla that are already on it. That is why things need to be tested on something that is actually like production and a test on a laptop is not sufficient. phab-01 or one of the other phab-0x instances should be used. And these should use the same roles that are used in production. Maybe they use a separate "labs" role though which is not ideal because it means we never have something that is really 100% like in prod.

The redirect Bugzilla looks like https://phabricator.wikimedia.org/rOPUP61e2fe1f38deb37889e9a339dba6d0e24e912da1

In T137224#2412573, @Dzahn wrote:

Yes, the problem appears when these rules are added to the existing Apache config in production, as you say they conflict with other rules for Bugzilla that are already on it. That is why things need to be tested on something that is actually like production and a test on a laptop is not sufficient. phab-01 or one of the other phab-0x instances should be used. And these should use the same roles that are used in production. Maybe they use a separate "labs" role though which is not ideal because it means we never have something that is really 100% like in prod.

I'm probably missing something...
If there is <virtualhost> for Bugzilla redirs using the ServerName directive to match bugzilla.wikimedia.org and bugs.wikimedia.org and we add the <virtualhost> with ServerName git.wikimedia.org and its redirects to Diffusion after it, it should work properly.
Another thing is, I can't really imagine, how Bugzilla->Phabricator rules can interact with Gitblit->Phabricator as a) they come from different domain, b) have different patterns, c) redirect to different places.
Can anybody please point me to what I missed? Thanks.

Those server aliases where missing, I amended the patch to add them.

Change 293221 abandoned by Dzahn:
git.wikimedia.org -> Diffusion redirects

https://gerrit.wikimedia.org/r/293221

Change 296138 merged by Dzahn:
Rewrite rules for git.wikimedia.org

https://gerrit.wikimedia.org/r/296138

URLs tested: P3315

tested with apache-fast-test from tin on iridium

redirects before merging 296138: P3317

redirects after merging 296138: P3318

@Danny_B It was because the Bugzilla redirects are in templates/redirect_config.json.erb directly in Phabricator while the git.wm and phabricator.wm URLs are setup as Apache virtual hosts using apache::site. /etc/apache2/sites-available/ only has git.wm and phab.wm but not bugzilla.wm

now:

[iridium:/etc/apache2/sites-enabled] $ apache2ctl -S | grep alias

alias phab.wmfusercontent.org
alias bugzilla.wikimedia.org
alias bugs.wikimedia.org

Thanks to @JanZerebecki for the fix!

Change 293789 merged by Dzahn:
varnish: git.wm.org to iridium, remove related config/tests/monitoring

https://gerrit.wikimedia.org/r/293789

Mentioned in SAL [2016-06-29T18:06:09Z] <mutante> we stopped using gitblit. git.wikimedia.org URLs P3318 T137224

\o/ Thanks all for the work here. Lovely working with you all :)

Not sure if this has been notified by Developer-Advocacy / Community-Relations-Support teams or whoever is responsible for Developer-notice / Notice
Same with T123718: Phase out antimony.wikimedia.org (git.wikimedia.org / gitblit)
@Quiddity Could you please check (and possibly assure about the notice)? Thanks!

It has (though referring to the email rather than the Phab ticket).