On Fri Apr 04 16:05:09 2014, glavagetto wrote:

Since about Apr 3 2014, 18:00 UTC git.wikimedia.org is highly unstable
and
reaches an high cpu usage (and a full heap) about 4 hours after being
restarted.

When this happens, the site is mostly unresponsive and basically
useless.

Access logs do not show a lot of activity, but they may miss
something.

More investigation is needed and maybe an alternative software can be
an option

The alternative is moving to Phabricator. Another stop-gap between then
and now would be a waste of time imho.
-Chad

Status changed from 'new' to 'open' by RT_System

for completeness, here was the gerrit link that tried to tune it a bit. it
seemed to help at first for a couple hours, then the issue came back
https://gerrit.wikimedia.org/r/#/c/123848/
should we at least/just add a cron job in puppet that restarts the service?
shrug

should we at least/just add a cron job in puppet that restarts the service?

Is this something actionable? Shall this be moved to public Wikimedia-Git-or-Gerrit project if this should remain open?

see https://phabricator.wikimedia.org/tag/gitblit-deprecate/

I think this ticket should stay open, to testify how much harm and inertia some abandonware might cause. We should have had the courage to admint that

1. Gitblit doesn't fit our needs in terms of stability
2. We had no time in a year to really work on a transition
3. We refused to surrender to the evidence that this dinosaur was left alive for no good reason, and has eaten a lot of ops man-hours in the process.

I'd like us all, the next time we say something is unstable and someone else says "we won't work on it, there is a plan for X" to ask for a time of decommission, that will happen at that date independently of someone preparing the promised alternative (X).

I personally feel ashamed we've left in production an application that probably has a 75% uptime without anyone caring (besides the occasional restart from ops) for more than a year.

also see T104356

also see T109004

It's down again. Perhaps some kind of monitoring could be implemented to detect this until the migration to a stable successor system is implemented.

In T83702#1542247, @Kghbln wrote:

Perhaps some kind of monitoring could be implemented to detect this until the migration to a stable successor system is implemented.

icinga does report it to #wikimedia-operations... but can't you use diffusion or github instead?

Ah, nobody there at operations. Was not aware of this. :p Yeah, since Diffusion is kinda useless since it does not allow downloads and also has a pretty confusing interface I will use GitHub from now on. Should have done so much earlier. However, this is another story.

Anybody aware of the fact that about every extension's page points to git.wikimedia.org?

@Glaisher Thanks for your suggestion!

In T83702#1542278, @Kghbln wrote:

Anybody aware of the fact that about every extension's page points to git.wikimedia.org?

Yes. See T108864: Update mediawiki.org templates to link to Diffusion, not gitblit.

Cool, it's in the making. :)

Why did you remove that project? It seems the right one to me. This is about not using gitblit anymore, that would solve the unstable-ness.

In T83702#1603716, @Dzahn wrote:

Why did you remove that project? It seems the right one to me. This is about not using gitblit anymore, that would solve the unstable-ness.

Because this isn't a blocker to doing the deprecation. There's lots of things that would be benefited from not using gitblit, but tracking them in that project doesn't help.

Isn't blocker done by the "blocked-by" feature rather than the project tag?

This is offtopic (sorry everyone else) but:

1. what is being lost by not having the gitblit deprecate project? Nothing, is my contention. This (the wikimedia-git-or-gerrit) is the proper project for this task.
2. the gitblit deprecate project says "Project for tracking the deprecation of Gitblit and use of Phabricator's Diffusion as its replacement". This task does not help us deprecate gitblit any faster. It'd like adding every issue that we have with Gerrit to the Gerrit-deprecation project; not helpful.