The long term fix should still be T178663

The Debian mirror sync uses ftpsync, unlike the Ubuntu mirror sync. So you won't find a puppetized cron and rsync like you do for Ubuntu.

In T223393#5223143, @Joe wrote:

Once you've done that, I'd say let's be bold and just change the proxy/rewrite rules from

basically a duplicate of T180641

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=miscweb2001

In T202367#5219604, @jcrespo wrote:

CCing @Dzahn as he expressed interest for this to happen in the past, and finally the hardware is here (no actionable needed from you at the moment).

I just ran into this when reinstalling phab2001 from jessie to stretch.

In T222308#5154875, @Tgr wrote:

Not sure what counts as consensus for something like this

The user name "darthmon" cannot be found anywhere in the admin module. Please add accounts there when adding them to LDAP groups.

transitioning the ~1400 existing tasks currently in "backlog" on the workboard to "acknowledged" without loads of manual work and triggering notifications?

removed again since we are not seeing the leaks anymore since our recent upgrade to stretch and phab1003

I have removed a bunch of aliases where people responded they were not aware of having them or that they don't need them anymore.

There are new failures:

moving public open tasks from Backlog to Acknowledged

Current Status: CRITICAL
(for 0d 6h 0m 16s)
Status Information: 121 gt 2

fyi db2035 is shown again as having unhealthy disks (https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=db2035&service=Device+not+healthy+-SMART-)

In T224247#5209664, @MoritzMuehlenhoff wrote:

miscweb1001/2001?

using PHP 7.2 was declined in T224247#5209664

In T223496#5209502, @Volans wrote:

@Dzahn I guess @georgina would be more appropriate, the expiration contact should be related to the contractors point of contact, not the group's owner.

Thanks for removing the number of aliases on our side in general. Note i had just opened a ticket on Zendesk for OIT as well to add all the remaining aliases to legal@ so we can remove all of them on our side.

I will bring this up in my next subteam discussion meeting which should be in a week. Until then i will hold on to phab1001. Maybe we wait a few days to keep it as is.. and then install stretch.

In T125357#4804491, @Paladox wrote:

We increased the max_execution_time to 30s in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477595/ for php-fpm.

So once we switch to stretch we can switch on php-fpm and then it should work.

I am thinking now we could make the process easier and just keep phab1003 as the prod server and just discuss whether we want to keep phab1001 as a permanent stand-by in the same DC or give it back to the spares pool.

Phabricator has been switched to phab1003 as the prod server now and that meant:

phab is on phab1003 now which runs PHP 7.2

23:21 <+icinga-wm> RECOVERY - Router interfaces on cr3-ulsfo is OK: OK: host 198.35.26.192, interfaces up: 68, down: 0, dormant: 0, excluded: 0, unused: 0

• Maintenance window:

Start Date and Time: 2019-May-23 03:00 UTC
End Date and Time: 2019-May-23 07:00 UTC

and..it is DOWN again

02:35 mutante: phabricator - going read-write again

02:24 twentyafterfour: manually started aphlict on phab1003
02:06 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet
02:04 mutante: puppetmaster1001 - sudo -i conftool-merge
01:52 twentyafterfour: phabricator is now served by phab1003 though still in read-only mode for a bit longer
01:52 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet
01:49 mutante: puppetmaster1001 - conftool-merge
01:37 mutante: depooled phab1001-vcs from git-ssh via conftool
01:36 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet
01:33 mutante: run puppet on mx1001/mx2001 - switch mail route for phab to phab1003
01:30 mutante: switched from phab1001 to phab1003 - applied on cp1008 varnish canary first
01:28 twentyafterfour: stopping phd on phab1001
01:18 mutante: phabricator going readonly momentarily
01:09 twentyafterfour: extended phab downtime in icinga, actual downtime hasn't started yet, prep work taking longer than expected
00:45 mutante: phab1003 - rsyncing /srv/repos from phab1001

</pre>

grafana is not on this host anymore meanwhile. unlinking subtask , not blocking this anymore

test

Just checked on this again and i notice that meanwhile somebody has done this.

@Xqt What happened? Was it reverted?

@aborrero I am referring to the "cgred" module. I am now suggesting to delete it at https://gerrit.wikimedia.org/r/c/operations/puppet/+/511791

I am interested if this is still being used nowadays. If it is we will need a systemd initscript for it and we'd want to convert it to use systemd::service, also to make it possible to upgrade to stretch. If not i would not bother or suggest to remove code.

The comments on T215042#4977385 sounded like this wasn't going to be done, for the temporary evaluation that it is. ?

Updating patch to include expiry_date May 31, 2020. Who should be expiry_contact? Nuria?

Thanks @Iflorez for attention to detail. So the full story is that's all one LDAP user but there are different fields:

@greg So this is approved by you?

Thanks @bd808 . Merged your change. Ran puppet, i saw it add the new locale. Looks like i just didn't use the "extended" version of the list?

In T179126#5198383, @Aklapper wrote:

Indeed, there is a one byte difference in the string now so strings don't match.