Page MenuHomePhabricator
Create Task
Maniphest T232883

Make PHD run on the backup phabricator server (phab2001, currently)
Closed, DeclinedPublic
Actions

Description

What needs to happen:

  • make sure that the database connection works
  • make sure outgoing email works
  • test run phd
  • make puppet run phd (currently disabled by check for active server via hiera setting)

https://secure.phabricator.com/book/phabricator/article/cluster_daemons/

Details

SubjectRepoBranchLines +/-
Revert "phabricator: enable phd service on phab2001"operations/puppetproduction+1 -1
phabricator: enable phd service on phab2001operations/puppetproduction+1 -1
phabricator: monitor the PHD process if PHD is set to running in Hieraoperations/puppetproduction+9 -5
Phabricator: Make a separate hiera option to ensure phd stopped/runningoperations/puppetproduction+11 -14
Customize query in gerrit

Related Objects
Search...

Event Timeline

• mmodell created this task.Sep 13 2019, 6:15 PM
gerritbot added a comment.Sep 13 2019, 7:50 PM

Change 536669 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Phabricator: Make a separate hiera option to ensure phd stopped/running

https://gerrit.wikimedia.org/r/536669

gerritbot added a project: Patch-For-Review.Sep 13 2019, 7:50 PM
gerritbot added a comment.Sep 13 2019, 10:20 PM

Change 536669 merged by Dzahn:
[operations/puppet@production] Phabricator: Make a separate hiera option to ensure phd stopped/running

https://gerrit.wikimedia.org/r/536669

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2019, 11:10 PM
Dzahn awarded a token.Sep 14 2019, 7:53 AM
Marostegui removed a project: DBA.Sep 16 2019, 4:58 AM

I am removing the DBA tag from here as there is nothing for us to do. I will remain subscribed to the task though just in case you need me to check the database connection.

thcipriani triaged this task as Medium priority.Sep 18 2019, 2:28 PM
thcipriani edited projects, added Release-Engineering-Team-TODO (201909); removed Release-Engineering-Team-TODO.
thcipriani subscribed.

Patches are moving, looks like this is currently in progress, doing the workboard fiddling accordingly

thcipriani moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201909) board.Sep 18 2019, 2:29 PM
• mmodell moved this task from Ready to Doing on the Release-Engineering-Team-TODO (201909) board.Sep 26 2019, 7:41 AM
• mmodell edited projects, added Release-Engineering-Team-TODO (201910); removed Release-Engineering-Team-TODO (201909).Oct 2 2019, 10:05 PM
• mmodell moved this task from INBOX to Doing on the Release-Engineering-Team-TODO (201910) board.
Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (201911); removed Release-Engineering-Team-TODO (201910).Nov 6 2019, 4:21 PM
Jdforrester-WMF moved this task from INBOX to Doing on the Release-Engineering-Team-TODO (201911) board.
• mmodell closed this task as Resolved.Nov 6 2019, 6:05 PM
• mmodell reopened this task as Open.
• mmodell updated the task description. (Show Details)
gerritbot added a comment.Nov 8 2019, 6:29 PM

Change 549906 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: enable phd service on phab2001 and monitor it

https://gerrit.wikimedia.org/r/549906

gerritbot added a project: Patch-For-Review.Nov 8 2019, 6:29 PM
Dzahn added a comment.Nov 8 2019, 6:31 PM

Whether phd is running is actually not directly controlled by the "active_server" setting. Monitoring for it is though.

The former is controlled by "profile::phabricator::main::phd_service_ensure" in separate hosts files, one each for phab1001, phab1003 and phab2001.

Then the monitoring part says "don't monitor if not active_server" though. See change above. So that would enable it on 2001 but also cause an alert on 1001.

Dzahn added a comment.Nov 8 2019, 6:38 PM

Let's change the monitoring to also use "profile::phabricator::main::phd_service_ensure" to decide whether to monitor the process and forget the active_server setting.

Dzahn added a project: serviceops.Nov 8 2019, 6:38 PM
gerritbot added a comment.Nov 8 2019, 6:48 PM

Change 549913 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: monitor the PHD process if PHD is set to running in Hiera

https://gerrit.wikimedia.org/r/549913

Dzahn added a comment.Nov 8 2019, 7:41 PM

https://gerrit.wikimedia.org/r/c/operations/puppet/+/549906

gerritbot added a comment.Nov 8 2019, 10:04 PM

Change 549913 merged by Dzahn:
[operations/puppet@production] phabricator: monitor the PHD process if PHD is set to running in Hiera

https://gerrit.wikimedia.org/r/549913

gerritbot added a comment.Nov 19 2019, 10:53 PM

Change 549906 merged by Dzahn:
[operations/puppet@production] phabricator: enable phd service on phab2001

https://gerrit.wikimedia.org/r/549906

gerritbot added a comment.Nov 19 2019, 10:59 PM

Change 551943 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "phabricator: enable phd service on phab2001"

https://gerrit.wikimedia.org/r/551943

gerritbot added a comment.Nov 19 2019, 11:04 PM

Change 551943 merged by Dzahn:
[operations/puppet@production] Revert "phabricator: enable phd service on phab2001"

https://gerrit.wikimedia.org/r/551943

Dzahn added a comment.Nov 19 2019, 11:06 PM
  • It's now possible to start the phd service with a a change like this (separate from the "active_server" setting mentioned above.
  • When i merged and puppet tried to start phd it got this error below with "The MariaDB server is running with the --read-only option ... so I reverted.

P9678 phd on phab2001 fails due to readonly DB
1Nov 19 22:54:38 phab2001 systemd[1]: Starting phabricator-phd...
2Nov 19 22:54:38 phab2001 phd[10326]: Freeing active task leases...
3Nov 19 22:54:38 phab2001 phd[10326]: [2019-11-19 22:54:38] EXCEPTION: (AphrontQueryException) #1290: The MariaDB server is running with the --read-only option so it cannot execute this statement at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:386]
4Nov 19 22:54:38 phab2001 phd[10326]: arcanist(), ava(), phabricator(), phutil(), security(), sprint(), translations(), wmf-ext-misc()
5Nov 19 22:54:38 phab2001 phd[10326]: #0 AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:320]
6Nov 19 22:54:38 phab2001 phd[10326]: #1 AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:216]
7Nov 19 22:54:38 phab2001 phd[10326]: #2 AphrontBaseMySQLDatabaseConnection::executeQuery(PhutilQueryString) called at [<phabricator>/src/infrastructure/storage/xsprintf/queryfx.php:8]
8Nov 19 22:54:38 phab2001 phd[10326]: #3 queryfx(AphrontMySQLiDatabaseConnection, string, string) called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:522]
9Nov 19 22:54:38 phab2001 phd[10326]: #4 PhabricatorDaemonManagementWorkflow::freeActiveLeases() called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:306]
10Nov 19 22:54:38 phab2001 phd[10326]: #5 PhabricatorDaemonManagementWorkflow::executeStartCommand(array) called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementStartWorkflow.php:37]
11Nov 19 22:54:38 phab2001 phd[10326]: #6 PhabricatorDaemonManagementStartWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:457]
12Nov 19 22:54:38 phab2001 phd[10326]: #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:349]
13Nov 19 22:54:38 phab2001 phd[10326]: #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/daemon/manage_daemons.php:24]
14Nov 19 22:54:38 phab2001 systemd[1]: phd.service: Control process exited, code=exited, status=255/EXCEPTION
15Nov 19 22:54:38 phab2001 systemd[1]: phd.service: Failed with result 'exit-code'.
16Nov 19 22:54:38 phab2001 systemd[1]: Failed to start phabricator-phd.
17
18Error: /Stage[main]/Phabricator/Systemd::Service[phd]/Service[phd]/ensure: change from 'stopped' to 'running' failed: Systemd start for phd failed!
19journalctl log for phd:

Maintenance_bot removed a project: Patch-For-Review.Nov 19 2019, 11:10 PM
Dzahn updated the task description. (Show Details)Nov 19 2019, 11:13 PM
Dzahn mentioned this in T137928: Deploy phabricator to phab2001.codfw.wmnet.Nov 19 2019, 11:16 PM
Marostegui added a comment.Nov 20 2019, 6:02 AM
In T232883#5676791, @Dzahn wrote:
  • When i merged and puppet tried to start phd it got this error below with "The MariaDB server is running with the --read-only option ... so I reverted.

I mentioned in the patchset that the databases in codfw are on RO mode, so that is sort of expected.

Dzahn added a comment.Nov 20 2019, 1:21 PM

Yes, it was. But it was not expected that phd fails to start entirely with a readonly DB.

Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (201912); removed Release-Engineering-Team-TODO (201911).Dec 2 2019, 7:31 PM
Jdforrester-WMF moved this task from INBOX to Doing on the Release-Engineering-Team-TODO (201912) board.
Dzahn added a subscriber: 20after4.Dec 6 2019, 8:37 PM

@20after4 per recent talks we had.. is it still our goal?

Dzahn changed the task status from Open to Stalled.Dec 20 2019, 4:20 AM
• mmodell closed this task as Declined.Jan 4 2020, 5:44 AM

@Dzahn: As long as repositories are not clustered (using phabricator's clustering) I think that running daemons on the backup host will be problematic. It's been tested and it works, that's the important part for disaster recovery / high availability.

RLazarus mentioned this in T313879: Phabricator: Unable to view tasks in DB read-only mode.Jul 27 2022, 12:18 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL