I am a Release Engineer on the Wikimedia Release-Engineering-Team.
Sat, Feb 17
@jcrespo: just let me know when you are available, I'll be glad to get to work with you on any phabricator tasks.
Definitely needs testing, however, it's a good idea :)
Fri, Feb 16
The default query in maniphest doesn't use a search keyword so relevance doesn't come into play at all, that's why it's grouped and ordered.
This is now the most frequent error in fatal monitor by a full order of magnitude. It probably should have blocked the train, however, it slipped through the cracks yesterday during the train rollout. I'm going to make this block wmf.21 for now, for the sake of visibility at least.
If it's really as simple as importing the packages from buster and deploying them, then that seems like a fairly ideal solution to me. We would need to do some testing beyond simply installing them on a vps instance and declaring it works!!, however, I don't think there will be any huge issues with compatibility, phabricator has supported 7.1 for a full year now.
Thu, Feb 15
Interesting: I think I may have found a relevant (and long standing) bug report for the deadlock: https://bugs.php.net/bug.php?id=31749, it's even got comments from our own @tstarling who points to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/103433#140
Yeah it's especially odd that it's happening on the exact same json string every time. I'll see if upstream has heard of anything like it
Looks like we still have workers in the "G" state:
Paladin: how many packages from buster were required to get it working?
Mail stamps are available now.
I'm now convinced that the problem is a bug in php internals and our best bet is probably to switch to just about any newer version of php. That said, it is probably triggered by something sortof unusual that phabricator is doing in recent versions. Fixing it from the Phabricator side would really just be working around a php bug, however, I really don't know where to look for the trigger.
@elukey: last night, before the upgrade, I had @Dzahn merge a few opcache related config changes: rOPUP9ddf42a8267a: Phabricator: Increase zend opcache limits in php.ini
can this be closed, then?
@Legoktm: So although I think we should still do some things to make phabricator update faster, given that the original need is now met, should we close this?
Ok so the upstream code is deployed, now we just wait and see if the situation improves.
This looks much better now:
Wed, Feb 14
Phabricator doesn't only exist as an accessory for gerrit. Having repositories in Phabricator, for one thing, provides a mirror we control. Phabricator currently provides us with a multi-datacenter redundant repository hosting infrastructure. Soon we will be able to have deployments fetch from a git repo hosted in the same datacenter as the target server. If phabricator is a sunk cost then we can close a lot of open tasks and I might as well start looking for a new job.
I disagree. I still think all repos should be sync'd to phabricator.
@Marostegui: that's one of the largest maniphest_task tables in teh world, despite it's size in raw bytes ;)
@jcrespo: it's possible but I'd have to hack around some detection scheme, phabricator normally refuses to even serve pages when the migrations are out of date. I could probably just modify the migration script to make it a noop, that way it'll get recorded as if it was applied. Then it's simple enough to run the exact same code manually after the upgrade.
The slow one is backfilling the data, the alter table is in a separate migration which should be much quicker than the population script. There is a similar pair of migrations in there for some differential tables, however, we don't have as much data in differential so I don't expect that one to be slow.
Here is the one that I expect to be slow: https://secure.phabricator.com/source/phabricator/browse/master/resources/sql/autopatches/20180208.maniphest.02.populate.php;215b8b4727aa96dc5cbd2a53120f97d5f4d4ce3b
Mail Stamps will soon be exposed in the body of messages, rather than only in the X-Phabricator-Mail-Stamps header. This should make it possible for gmail users to create a filter which matches any notification which @mentions their username. See https://secure.phabricator.com/w/changelog/2018.06/
Tue, Feb 13
Yeah, I could try to rush the update, there isn't necessarily a reason to wait until Wednesday, other than that's the scheduled deployment window
Looks good. Tests pass. Good work @thcipriani!
Although PHP 7.1 was declined in T160714, I believe we should reconsider. There is a backport of PHP 7.1 to jessie and stretch, maintained by the same package maintainer as official debian packaging for PHP. It should be fairly trivial to upload that package to our apt repo and keep it in sync from upstream, or alternatively, to run the build ourself using the same source package. If there is any budget for it, we could even consider tipping the maintainer to help encourage the continued support for this package.
Confirmed: all of the news entries on https://tracker.debian.org/pkg/php7.1 are by Ondřej Surý.
I think those packages are the same - sury.org is owned by the debian maintainer for php - so those are semi-official backports, as far as I can tell.
Mon, Feb 12
@MoritzMuehlenhoff: There is at least one 3rd party PHP 7.1 package available [[[ https://packages.sury.org/php/ | 1 ]]]. Could we not use their source packages to build our own binaries? I realize it is indeed a large amount of maintenance, however, it seems preferable to being stuck on a buggy and obsolete version of PHP for the next couple of years.