Page MenuHomePhabricator

Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade.
Closed, ResolvedPublic

Description

Date: Fri, 19 Feb 2016 19:36:37 GMT
Content-Length: 465969478
Last-Modified: Thu, 18 Feb 2016 03:04:32 GMT

Presumably, Last-Modified should be Fri 3:04 GMT. So the last time it ran was 7 pm PT Wednesday, which I thought was after the upgrade, so I thought it survived the upgrade.

(Previously: T108587, T112135)

Event Timeline

JAufrecht renamed this task from Phabricator data dump didn't run Thursday night to Phabricator data dump didn't run Thursday night (Feb 18 2016).Feb 19 2016, 7:41 PM
JAufrecht added a subscriber: mmodell.

Dump script broken by the removal of some columns from maniphest_task. See https://github.com/phacility/phabricator/commit/32225d1dd0f8274ae9323650cbf20ea0571424de

I'm running it again manually after removing those columns from the query. Hopefully soon I will find out if that's the only problem.

Dump script ran successfully.

Change 272320 had a related patch set uploaded (by 20after4):
Remove 2 obsolete columns from select query fixes T127524

https://gerrit.wikimedia.org/r/272320

The automated Sunday night run (actually around 7 pm PT, since that's after the dump normally goes) seems to have worked fine and has fresh data up through Sunday. Thanks.

ArielGlenn added a subscriber: ArielGlenn.

Cron Daemon root@iridium.eqiad.wmnet via wikimedia.org
4:59 AM (10 hours ago)

to root

Traceback (most recent call last):

File "/srv/phab/tools/wmfphablib/phabdb.py", line 1474, in sql_x
  x.execute(statement, arguments)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  raise errorclass, errorvalue

OperationalError: (1054, "Unknown column 'attached' in 'field list'")
(1054, "Unknown column 'attached' in 'field list'")
rollback!
Traceback (most recent call last):

File "/srv/phab/tools/public_task_dump.py", line 31, in <module>
  task_phids = [id[1] for id in tasks]

TypeError: 'NoneType' object is not iterable

So that would be the Monday night run I guess. Any ideas?

I guess the patch still needs merging, pulling to the puppetmaster, deploying

Was it run as a live hack, and puppet not disabled on the machine? So, hence, it ran fine manually, then puppet reverted it?

@Reedy is right. I'm still waiting on a review of the patch.

Change 272320 merged by Rush:
Remove 2 obsolete columns from select query fixes T127524

https://gerrit.wikimedia.org/r/272320

Not sure if that was the final commit to production, and if it was in time to catch the nightly job, but the current dump is still last Sunday's.

That commit still hasn't been deployed.

JAufrecht renamed this task from Phabricator data dump didn't run Thursday night (Feb 18 2016) to Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade..Feb 26 2016, 10:26 PM

Is this blocked by needing code review or something, or is it waiting for a scheduled release? This is increasingly urgent day by day since five teams are using this data. Can I do anything constructive to facilitate this?

@JAufrecht: the fix is now deployed, hopefully the next automatic dump will work. I can run it manually now if that would help.

A fresh manual dump would help; when will the automatic dump kick in? Tomorrow evening? I think it goes around 1 am UT? Thanks

I'm not sure when the cron is scheduled but I'll go ahead and run it manually.

Nevermind, it looks like the dump was just updated:

-rw-r--r-- 1 root root 450M Feb 27 03:21 phabricator_public.dump

@JAufrecht I'm closing this, reopen if it's still not updating but as far as I can tell it updated automatically on schedule last night.

looks like it ran automatically on schedule Monday night as well. thanks.

I see cron output today:
Cron <root@iridium> /srv/phab/tools/public_task_dump.py

NOTICE: rtppl not found!

Is this an issue? If not, please go ahead and close this again.

Thanks for keeping an eye out. It appears that last night's dump ran successfully, because http://phlogiston.wmflabs.org/ shows data as recent as Thu 2016-Mar-03 07:00 PM (PT).

And the dump ran:

Date: Fri, 04 Mar 2016 17:43:31 GMT
Content-Type: application/octet-stream
Content-Length: 476345151
Last-Modified: Fri, 04 Mar 2016 03:23:00 GMT

I see cron output today:
NOTICE: rtppl not found!

Is this an issue? If not, please go ahead and close this again.

We are seeing these again. cron mail with the "NOTICE: rtppl not found!" message. I don't know why that is or if it's an issue yet.

We are seeing these again. cron mail with the "NOTICE: rtppl not found!" message. I don't know why that is or if it's an issue yet.

(Also see T172558: Phlogiston data not updated since Aug 2 for recent issues with the dump file.)

(Also see T172558: Phlogiston data not updated since Aug 2 for recent issues with the dump file.)

Yea, but there it says the reports are fine now as if that NOTICE: rtppl not found doesn't really matter / keep it from creating dumps.

@chasemp ^ In https://github.com/wikimedia/phabricator-tools/blob/master/wmfphablib/rtlib.py can't find rtppl in "from rtppl import ppl as users" . Do you know where rtppl is?

@chasemp ^ In https://github.com/wikimedia/phabricator-tools/blob/master/wmfphablib/rtlib.py can't find rtppl in "from rtppl import ppl as users" . Do you know where rtppl is?

rtppl was a one-time for migration sensitive file that had emails and associations from rt that was on demand generated. That isn't really a warning so much as a notice that w/o that file all those mappings are null, and throwing it here is a product of wmfphablib/__init__.py importing rtlib.

I'll verify and hopefully remove to clarify

Thank you very much chasemp. The goal is just to silence the cron job one way or another then.

Thank you very much chasemp. The goal is just to silence the cron job one way or another then.

yep, seems like that phab/tools repo is now a mirror in gerrit and needs differential for the change. I don't have arcanist setup at all anymore. We could alternatively silence stdout for the cron :) Commenting out that import or redirecting stdout should work.

Change 372413 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: silence public_task_dump cron mails

https://gerrit.wikimedia.org/r/372413

Change 372413 merged by Dzahn:
[operations/puppet@production] phabricator: silence stdout of public_task_dump cron

https://gerrit.wikimedia.org/r/372413