Page MenuHomePhabricator

Phabricator dump frozen since Aug 28
Closed, ResolvedPublic

Description

Similar problem as T108587: Nightly Phabricator dump hasn't run since Aug 2/3

Date: Thu, 10 Sep 2015 17:42:49 GMT
Content-Type: application/octet-stream
Content-Length: 320980224
Last-Modified: Fri, 28 Aug 2015 04:09:19 GMT

Event Timeline

JAufrecht assigned this task to ArielGlenn.
JAufrecht raised the priority of this task from to Needs Triage.
JAufrecht updated the task description. (Show Details)
JAufrecht added a project: Phabricator.

checked where this is on the iridium (phabricator) side and this time it seems to be a local issue here, as opposed to last time when it was about uploading it to the dumps server. because i see the last one was written on Aug 28, confirmed:

307M -rw-r--r-- 1 root root 307M Aug 28 04:09 phabricator_public.dump
(/srv/dumps)

Ah, the reason is that the cron job to run this has been disabled on purpose:

43         cron { $dump_script:
44             ensure  => absent,
45             command => $dump_script,

I suspect this may have been due to T109279.

yes, it is:

https://gerrit.wikimedia.org/r/#/c/234563/

commit 405a54eeef1e529a074b94fc98ee5087a9c5a43f
Author: cpettet <rush@wikimedia.org>
Date:   Fri Aug 28 11:26:04 2015 -0500

    phab: disable tools crons
    
    This is temporary as we are seeing large mysql
    connection spikes.  I am suspcious of these jobs
    and I want to go a few days with them disabled to
    guage a baseline.  There is no long term negative
    outcome here.
    
    Change-Id: I23cf4f30769e1bf03314ae88f8cbb5743725ba5d

@jcrespo said there that we should change the dump script to use mysql slaves instead of the master.

Don't make it a blocker, but it should be a great opportunity to do it :-)

And next week I should be more available to help with the patch.

Change 237513 had a related patch set uploaded (by Dzahn):
Revert "phab: disable tools crons"

https://gerrit.wikimedia.org/r/237513

Change 237514 had a related patch set uploaded (by Dzahn):
phab: re-enable dump script

https://gerrit.wikimedia.org/r/237514

Change 237514 merged by Dzahn:
phab: re-enable dump script

https://gerrit.wikimedia.org/r/237514

Ok, @JAufrecht i re-enabled it since @jcrespo said don't make it a blocker, so yes, should show up again tomorrow.

What i did _not_ do is re-enable the 2 other disabled cron jobs (bz_header_update, bz_comments_update) that @chasemp had disabled. One by one so we can see if it causes any issues and they were not related to this task anyways. That would be https://gerrit.wikimedia.org/r/#/c/237513/4

Notice: /Stage[main]/Phabricator::Tools/Cron[/srv/phab/tools/public_task_dump.py]/ensure: created

confirmed on iridium

I'll claim it will just work again because all we did is switch off/on the existing cron with the identical command.

If unexpectedly it does not, please reopen the ticket.

@JAufrecht: just to confirm, a new dump is there meanwhile

https://dumps.wikimedia.org/other/misc/

phabricator_public.dump 11-Sep-2015 03:51 338303315

Check - I downloaded it successfully.

*--Joel Aufrecht*
Team Practices Group
Wikimedia Foundation

Change 237513 merged by Dzahn:
Revert "phab: disable tools crons"

https://gerrit.wikimedia.org/r/237513

re-enabled the 2 Bugzilla related crons as well:

Notice: /Stage[main]/Phabricator::Tools/Cron[bz_comment_update]/ensure: created
Notice: /Stage[main]/Phabricator::Tools/Cron[bz_header_update]/ensure: created