Page MenuHomePhabricator
Create Task
Maniphest T377889

install a service on phab1005
Closed, ResolvedPublic
Actions

Description

phab1005 (previously gerrit1004 T372817) is new hardware that was requested in T354688, procured in T368918 and racked in T369671.

But it's still just using the "insetup" role and needs a real service (likely phabricator:)) installed on it.

Details

Other Assignee
Dzahn
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
mariadb: add grants for phab1005AOkothoperations/puppetproduction+47 -1
hieradata: reenable scap bootstrapping on phab1005AOkothoperations/puppetproduction+0 -1
phabricator: add multi-replica supportAOkothoperations/puppetproduction+14 -11
site: apply migration role to phab1005AOkothoperations/puppetproduction+1 -1
site: revert phab1005 to insetupAOkothoperations/puppetproduction+1 -1
scap: stop hardcoding scap user home to fix puppet breakageDzahnoperations/puppetproduction+1 -2
phabricator::migration: fix /srv/phab symlink, /srv/repos dirDzahnoperations/puppetproduction+3 -3
phabricator::migration: fix variable name used for testdb storage passDzahnoperations/puppetproduction+1 -1
phabricator::migration: puppetize password for testdb in script-varsDzahnoperations/puppetproduction+4 -1
add fake password for phab test db admin userDzahnlabs/privatemaster+4 -0
phabricator::migration: add parameters phabdir,storage_user,deploy_userDzahnoperations/puppetproduction+13 -5
phabricator::migration: ensure /srv/phab is the correct symlinkDzahnoperations/puppetproduction+2 -1
phabricator::migration: install PHPDzahnoperations/puppetproduction+85 -0
phabricator::migration: include phabricator::httpd, add /srv/phab dirDzahnoperations/puppetproduction+5 -0
phabricator::migration: add missing sudo_defaults file for scapDzahnoperations/puppetproduction+19 -0
phabricator::migration: add /etc/phabricator/script-vars for scapDzahnoperations/puppetproduction+8 -0
Revert "phabricator: comment out scap::target in migration class"Dzahnoperations/puppetproduction+7 -7
phabricator: comment out scap::target in migration classDzahnoperations/puppetproduction+7 -7
phabricator/scap: disable scap bootstrapping on phab1005Dzahnoperations/puppetproduction+1 -0
phabricator::migration: add scap::target, add deploy scripts, rm symlinkDzahnoperations/puppetproduction+41 -4
phabricator: apply phabricator::migration role on host phab1005Dzahnoperations/puppetproduction+1 -1
phabricator: apply a staging role/profile to host phab1005Dzahnoperations/puppetproduction+38 -1
phabricator::migration: use nftables as firewall providerDzahnoperations/puppetproduction+2 -0
hieradata: add phab shell user groups on phab1005Dzahnoperations/puppetproduction+4 -0
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
phabricator-targets: add phab1005repos/phabricator/deployment!115aokothT377889wmf/stable
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedFeatureAklapperT367500 Explore displaying additional info in comment field for open Outreach related tasks
 ResolvedFeatureAklapperT369982 Display info for new technical contributors in task comment field for new accounts
 ResolvedFeatureAklapperT354769 Display warning about commenting on a task already closed as duplicate
Restricted Task
 OpenNoneT84 Make sure anti-vandalism features are up to snuff
Restricted Task
 Resolvedvalerio.bozzolanT356384 Phorge (Phabricator) Code Sprint 2024
 ResolvedBUG REPORTvalerio.bozzolanT217984 Listing tickets "Closed By: Current Viewer" while logged out causes PhabricatorTypeaheadInvalidTokenException: This datasource ("PhabricatorPeopleUserFunctionDatasource") can not evaluate the function "viewer(...)".
 ResolvedAklapperT91428 tooltip mentions non-existent "Wishlist priority" in Open Tasks by Project and Priority report
 ResolvedAklapperT310543 Unable to "manage user" on mobile
 ResolvedAklapperT266079 Error when searching Maniphest task IDs with improper format could be less cryptic: "Expected a numeric scalar or null for %Ld conversion"
Restricted Task
 ResolvedAklapperT132335 After "Log In to Comment", go back to previous page instead of Phab home page
 ResolvedFeatureAklapperT208434 Allow contracting (collapsing and expanding) of columns on project workboard in mobile view
 ResolvedAklapperT338306 When I delete a comment, I don't want to be added as a subscriber
 ResolvedFeatureAklapperT288117 Add Open Graph support to Phabricator Maniphest Tasks to have link preview on Telegram, Slack, and other messaging apps
 ResolvedBUG REPORTAklapperT233541 Unhandled Exception when clicking "Award" without choosing a badge; No badge type-ahead proposals if no permission to award badges
 ResolvedbrennenT370266 Update to Phorge upstream 2024.35 release
 ResolvedNoneT147980 Diffusion uses insufficient color coding for Less files (fixed in Pygments 2.8.0 version in Debian Bookworm)
 ResolvedDzahnT384595 Upgrade Collab hosts to Bookworm
 ResolvedDzahnT372619 Upgrade phab (phorge) hosts to bookworm
 ResolvedJclark-ctrT372817 reimage gerrit1004.wikimedia.org as phab1005.eqiad.wmnet
 ResolvedArnoldokothT377889 install a service on phab1005
 ResolvedDzahnT392386 PuppetFailure - phab1005
 ResolvedMarosteguiT390034 Prepare a database test for m3
 ResolvedABran-WMFT397886 PuppetDisabled - phab1005:9100
 ResolvedDzahnT399480 Can't update scap on phab1005.eqiad.wmnet
 ResolvedArnoldokothT423727 replace phab2002 with phab2003
 ResolvedArnoldokothT427308 PuppetFailure - phab2003
 ResolvedDzahnT429641 SystemdUnitFailed - rsync phab2002
 ResolvedArnoldokothT435087 Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4)
 ResolvedNoneT435435 SystemdUnitFailed - rsync-phabricator-repos.service on phab2003
 ResolvedBUG REPORTMarosteguiT435442 phstats user lacks database permissions on phab1005
 ResolvedRequestJclark-ctrT436635 decommission phab1004

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Dzahn added a comment.EditedJun 17 2025, 5:24 PM
Info: Applying configuration version '(53672bf49c) Dzahn - Revert "phabricator: comment out scap::target in migration class"'
Error: Execution of '/usr/bin/scap deploy-local --repo phabricator/deployment -D log_json:False' returned 1: 17:14:53 Fetch from: http://deploy1003.eqiad.wmnet/phabricator/deployment/.git
17:14:54 Update submodules
17:14:54 Updating .gitmodule: /srv/deployment/phabricator/deployment-cache/cache
17:15:12 Checkout rev: f8d7b38df39896e5e2792d70fa04b81f3b6450e3
17:15:12 Updating .gitmodule: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/local.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/local.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/mail.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/mail.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/phd.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/phd.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/vcs.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/vcs.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/www.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/www.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/preamble.php using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/preamble.php: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/redirect_config.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/redirect_config.json: [rendered]
17:15:31 Executing check 'config_deploy'
17:15:31 Check 'config_deploy' failed: /usr/local/sbin/phab_deploy_config_deploy: 8: .: cannot open /etc/phabricator/script-vars: No such file


Error: /Stage[main]/Profile::Phabricator::Migration/Scap::Target[phabricator/deployment]/Package[phabricator/deployment]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/scap deploy-local --repo phabricator/deployment -D log_json:False' returned 1: 17:14:53 Fetch from: http://deploy1003.eqiad.wmnet/phabricator/deployment/.git
17:14:54 Update submodules
17:14:54 Updating .gitmodule: /srv/deployment/phabricator/deployment-cache/cache
17:15:12 Checkout rev: f8d7b38df39896e5e2792d70fa04b81f3b6450e3
17:15:12 Updating .gitmodule: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/local.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/local.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/mail.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/mail.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/phd.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/phd.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/vcs.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/vcs.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/www.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/conf/local/www.json: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/preamble.php using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/preamble.php: [rendered]
17:15:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/redirect_config.json using /etc/phabricator/config.yaml
17:15:31 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/redirect_config.json: [rendered]
17:15:31 Executing check 'config_deploy'
17:15:31 Check 'config_deploy' failed: /usr/local/sbin/phab_deploy_config_deploy: 8: .: cannot open /etc/phabricator/script-vars: No such file

BUT also:

Notice: /Stage[main]/Profile::Phabricator::Migration/Scap::Target[phabricator/deployment]/Ssh::Userkey[phab-deploy]/File[/etc/ssh/userkeys/phab-deploy]/ensure: defined content as '{sha256}0965c405d7f4d3b0b470328a0d4cb9e8dc6b4440a6b11608a09caf10016c6d64'
Info: Scap::Target[phabricator/deployment]: Unscheduling all events on Scap::Target[phabricator/deployment]
Notice: /Stage[main]/Scap::Ferm/Firewall::Service[deployment-ssh]/Nftables::Service[deployment-ssh]/File[/etc/nftables/input/10_deployment-ssh.nft]/ensure: defined content as '{sha256}e686be75ea00532efa5deab346ddfb9f59905bac3c3cbdc2e0d84cc7d82ef475'
Info: /Stage[main]/Scap::Ferm/Firewall::Service[deployment-ssh]/Nftables::Service[deployment-ssh]/File[/etc/nftables/input/10_deployment-ssh.nft]: Scheduling refresh of Service[nftables]
Notice: /Stage[main]/Nftables/Systemd::Service[nftables]/Service[nftables]: Triggered 'refresh' from 1 event
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 59.19 seconds
Dzahn added a comment.Jun 17 2025, 5:27 PM

On second puppet run, after that scap::target was created and the phab-deploy SSH key was written.. and the firewall hole was opened.

Still have this though:

17:24:46 Check 'config_deploy' failed: /usr/local/sbin/phab_deploy_config_deploy: 8: .: cannot open /etc/phabricator/script-vars: No such file
Stashbot added a comment.Jun 17 2025, 5:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T17:27:48Z] <brennen@deploy1003> Started deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889

Stashbot added a comment.Jun 17 2025, 5:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T17:28:11Z] <brennen@deploy1003> Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (duration: 00m 23s)

Dzahn added a comment.Jun 17 2025, 5:29 PM

This is like T378769 and T257317 once again, or similar at least.

gerritbot added a comment.Jun 17 2025, 5:41 PM

Change #1160217 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: add /etc/phabricator/script-vars for scap

https://gerrit.wikimedia.org/r/1160217

gerritbot added a comment.Jun 17 2025, 6:09 PM

Change #1160217 merged by Dzahn:

[operations/puppet@production] phabricator::migration: add /etc/phabricator/script-vars for scap

https://gerrit.wikimedia.org/r/1160217

Dzahn added a comment.Jun 17 2025, 6:15 PM

After my latest merge, puppet created that missing /etc/phabricator/script-vars file now.

Now we are at:

fatal: destination path '/srv/deployment/phabricator/deployment-cache/cache' already exists and is not an empty directory.

18:14:40 deploy-local failed: <FailedCommand> Command 'git clone --jobs 46 --config lfs.url=https://gitlab.wikimedia.org/repos/phabricator/deployment.git/info/lfs http://deploy1003.eqiad.wmnet/phabricator/deployment/.git /srv/deployment/phabricator/deployment-cache/cache' failed with exit code 128;
Dzahn added a comment.EditedJun 17 2025, 6:17 PM

Did a manual rm -rf /srv/deployment/phabricator/deployment-cache/cache and trying again.

After this, the "fatal: destination path .. already exists" part is gone.

Now at:

Error: Execution of '/usr/bin/scap deploy-local --repo phabricator/deployment -D log_json:False' returned 1: 18:17:29 Fetch from: http://deploy1003.eqiad.wmnet/phabricator/deployment/.git

When trying to run this manually.. it's back to the previous issue before.

Stashbot added a comment.Jun 17 2025, 6:24 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T18:24:13Z] <brennen@deploy1003> Started deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling)

Stashbot added a comment.Jun 17 2025, 6:24 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T18:24:31Z] <brennen@deploy1003> Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling) (duration: 00m 18s)

Dzahn added a comment.Jun 17 2025, 6:24 PM

Deleted that .cache dir one more time. Ran the command manually as phab-deploy user, before puppet ran again..and:

[phab1005:~] $ sudo -u phab-deploy /usr/bin/scap deploy-local --repo phabricator/deployment -D log_json:False
18:23:14 Fetch from: http://deploy1003.eqiad.wmnet/phabricator/deployment/.git
18:23:14 Update submodules
18:23:14 Updating .gitmodule: /srv/deployment/phabricator/deployment-cache/cache
..
18:23:33 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/redirect_config.json using /etc/phabricator/config.yaml
18:23:33 /srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/phabricator/support/redirect_config.json: [unchanged]
18:23:33 Executing check 'config_deploy'
18:23:33 Check 'config_deploy' failed: $SCAP_REV_PATH is not defined.
Note: This script is only intended to run as a scap deploy check

Looking much greener now :)

gerritbot added a comment.Jun 17 2025, 6:39 PM

Change #1160231 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: add missing sudo_defaults file for scap

https://gerrit.wikimedia.org/r/1160231

Dzahn mentioned this in T397249: PuppetFailure - phab1005.Jun 17 2025, 6:44 PM
gerritbot added a comment.Jun 17 2025, 6:45 PM

Change #1160231 merged by Dzahn:

[operations/puppet@production] phabricator::migration: add missing sudo_defaults file for scap

https://gerrit.wikimedia.org/r/1160231

Dzahn added a comment.Jun 17 2025, 6:57 PM

The mssing scap_sudo_defaults file has been created:

Notice: /Stage[main]/Profile::Phabricator::Migration/File[/etc/sudoers.d/scap_sudo_defaults]/ensure

Now the puppet run finishes without any issues :)

@brennen Please try again now.

Stashbot added a comment.Jun 17 2025, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T19:12:36Z] <brennen@deploy1003> Started deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling)

Stashbot added a comment.Jun 17 2025, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T19:12:46Z] <brennen@deploy1003> Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling) (duration: 00m 10s)

brennen added a comment.Jun 17 2025, 7:15 PM

Much closer:

P78239 scap deploy -l phab1005.eqiad.wmnet
119:12:46 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'phabricator/deployment', '-g', 'default', 'finalize', '--refresh-config'] (ran as phab-deploy@phab1005.eqiad.wmnet) returned [1]: Registering scripts in directory '/srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/scap/scripts'
2registered script? '/srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/scap/scripts/deploy-perms.sh' True
3Registering scripts in directory '/srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/scap/scripts'
4registered script? '/srv/deployment/phabricator/deployment-cache/revs/f8d7b38df39896e5e2792d70fa04b81f3b6450e3/scap/scripts/deploy-perms.sh' True
5Executing check 'finalize'
6Check 'finalize' failed:
7 ->Running puppet...
8Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: 'phabricator deployment - phab-deploy');
9Use 'puppet agent --enable' to re-enable.
10
11 ->Applying storage migrations
12/usr/local/sbin/phab_deploy_finalize: line 24: /phabricator/bin/storage: No such file or directory
13
14 ->Restarting PHD
15Failed to start phd.service: Unit phd.service not found.
16
17 ->Reloading apache
18Failed to reload apache2.service: Unit apache2.service not found.
19
20 ->Enabling puppet agent
21
22 ->Verifying database status
23
24<13>Jun 17 19:12:46 root: >>>ERROR: Phabricator storage is in a bad state.
25
26
2719:12:46 phabricator/deployment: finalize stage(s): 100% (in-flight: 0; ok: 0; fail: 1; left: 0) |
2819:12:46 1 targets had deploy errors
2919:12:46 1 targets failed
3019:12:46 default deploy successful
3119:12:46 Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling) (duration: 00m 10s)
3219:12:46 Finished deploy [phabricator/deployment@f8d7b38] (duration: 00m 10s)

Stashbot added a comment.Jun 17 2025, 9:26 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T21:26:57Z] <brennen@deploy1003> Started deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling)

Stashbot added a comment.Jun 17 2025, 9:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T21:27:04Z] <brennen@deploy1003> Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling) (duration: 00m 07s)

Stashbot added a comment.Jun 17 2025, 9:29 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T21:29:29Z] <brennen@deploy1003> Started deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling)

Stashbot added a comment.Jun 17 2025, 9:29 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T21:29:36Z] <brennen@deploy1003> Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling) (duration: 00m 07s)

gerritbot added a comment.Jun 17 2025, 9:42 PM

Change #1160270 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: include phabricator::httpd, add /srv/phab dir

https://gerrit.wikimedia.org/r/1160270

gerritbot added a comment.Jun 17 2025, 9:45 PM

Change #1160270 merged by Dzahn:

[operations/puppet@production] phabricator::migration: include phabricator::httpd, add /srv/phab dir

https://gerrit.wikimedia.org/r/1160270

gerritbot added a comment.Jun 17 2025, 9:56 PM

Change #1160280 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: install PHP

https://gerrit.wikimedia.org/r/1160280

gerritbot added a comment.Jun 17 2025, 10:12 PM

Change #1160280 merged by Dzahn:

[operations/puppet@production] phabricator::migration: install PHP

https://gerrit.wikimedia.org/r/1160280

Stashbot added a comment.Jun 17 2025, 10:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T22:25:36Z] <brennen@deploy1003> Started deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling)

Stashbot added a comment.Jun 17 2025, 10:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T22:25:43Z] <brennen@deploy1003> Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling) (duration: 00m 07s)

Stashbot added a comment.Jun 17 2025, 10:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T22:26:59Z] <brennen@deploy1003> Started deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling)

gerritbot added a comment.Jun 17 2025, 11:05 PM

Change #1160310 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: ensure /srv/phab is the correct symlink

https://gerrit.wikimedia.org/r/1160310

Stashbot added a comment.Jun 17 2025, 11:30 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T23:30:03Z] <brennen@deploy1003> Finished deploy [phabricator/deployment@f8d7b38]: re-test deploy to phab1005 for T377889 (once more, with feeling) (duration: 63m 03s)

Dzahn mentioned this in T384595: Upgrade Collab hosts to Bookworm.Jun 18 2025, 4:18 PM
gerritbot added a comment.Jun 18 2025, 4:29 PM

Change #1160310 merged by Dzahn:

[operations/puppet@production] phabricator::migration: ensure /srv/phab is the correct symlink

https://gerrit.wikimedia.org/r/1160310

Dzahn updated Other Assignee, added: Dzahn.Jun 18 2025, 8:24 PM
Dzahn added a comment.Jun 18 2025, 8:29 PM

A scap phab deployment has been working now! :)

I re-enabled puppet after we had applied some manual hacks. (like commenting out apache restart before apache was installed, re-enabling puppet by deployment script, creating missing symlink,)..

The symlink is now puppetized; apache can be restarted, puppet can be enabled and disabled; just one thing is left: variable values in /etc/phabricator/script-vars are not expanded. The file itself is created by puppet from a template.

Dzahn added a comment.EditedJun 18 2025, 8:30 PM

Also: PHP8.2 is installed..with all the same modules a prod phab host has.. and if this works then Phorge is just fine with bookworm default version for T372619.

gerritbot added a comment.Jun 18 2025, 8:48 PM

Change #1161042 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: add parameters phabdir,storage_user,deploy_user

https://gerrit.wikimedia.org/r/1161042

gerritbot added a comment.Jun 18 2025, 9:05 PM

Change #1161042 merged by Dzahn:

[operations/puppet@production] phabricator::migration: add parameters phabdir,storage_user,deploy_user

https://gerrit.wikimedia.org/r/1161042

gerritbot added a comment.Jun 18 2025, 9:27 PM

Change #1161048 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: puppetize password for testdb in script-vars

https://gerrit.wikimedia.org/r/1161048

gerritbot added a comment.Jun 18 2025, 9:34 PM

Change #1161051 had a related patch set uploaded (by Dzahn; author: Dzahn):

[labs/private@master] add fake password for phab test db admin user

https://gerrit.wikimedia.org/r/1161051

gerritbot added a comment.Jun 18 2025, 9:35 PM

Change #1161051 merged by Dzahn:

[labs/private@master] add fake password for phab test db admin user

https://gerrit.wikimedia.org/r/1161051

gerritbot added a comment.Jun 18 2025, 9:36 PM

Change #1161048 merged by Dzahn:

[operations/puppet@production] phabricator::migration: puppetize password for testdb in script-vars

https://gerrit.wikimedia.org/r/1161048

gerritbot added a comment.Jun 18 2025, 9:45 PM

Change #1161053 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: fix variable name used for testdb storage pass

https://gerrit.wikimedia.org/r/1161053

gerritbot added a comment.Jun 18 2025, 9:46 PM

Change #1161053 merged by Dzahn:

[operations/puppet@production] phabricator::migration: fix variable name used for testdb storage pass

https://gerrit.wikimedia.org/r/1161053

gerritbot added a comment.Jun 18 2025, 9:55 PM

Change #1161059 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: fix /srv/phab symlink, /srv/repos dir

https://gerrit.wikimedia.org/r/1161059

gerritbot added a comment.Jun 18 2025, 9:57 PM

Change #1161059 merged by Dzahn:

[operations/puppet@production] phabricator::migration: fix /srv/phab symlink, /srv/repos dir

https://gerrit.wikimedia.org/r/1161059

Dzahn added a comment.Jun 18 2025, 10:02 PM
In T377889#10929728, @Dzahn wrote:

The symlink is now puppetized; apache can be restarted, puppet can be enabled and disabled; just one thing is left: variable values in /etc/phabricator/script-vars are not expanded. The file itself is created by puppet from a template.

This is also fixed now. The password for the test db is now properly puppetized in the private repo in it's own class.. without touching prod phab db passwords.

And puppet ensures /srv/phab is the proper symlink to the deployment dir.

brennen reopened subtask T390034: Prepare a database test for m3 as Open.Jun 18 2025, 10:35 PM
brennen closed subtask T390034: Prepare a database test for m3 as Resolved.Jun 19 2025, 7:48 PM
ABran-WMF added a subtask: T397886: PuppetDisabled - phab1005:9100.Jun 26 2025, 5:47 AM
ABran-WMF closed subtask T397886: PuppetDisabled - phab1005:9100 as Resolved.Jun 30 2025, 3:15 PM
Aklapper mentioned this in T399480: Can't update scap on phab1005.eqiad.wmnet.Jul 14 2025, 4:54 PM
Dzahn added a subtask: T399480: Can't update scap on phab1005.eqiad.wmnet.Jul 14 2025, 5:14 PM
dancy closed subtask T399480: Can't update scap on phab1005.eqiad.wmnet as Resolved.Jul 15 2025, 2:57 PM
gerritbot added a comment.Jul 17 2025, 8:16 PM

Change #1137818 merged by Dzahn:

[operations/puppet@production] scap: stop hardcoding scap user home to fix puppet breakage

https://gerrit.wikimedia.org/r/1137818

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2025, 8:32 PM
ABran-WMF subscribed.Sep 15 2025, 3:50 PM
brennen moved this task from Doing/Involved to Backlog on the User-brennen board.Mar 12 2026, 8:40 PM
Dzahn mentioned this in T423727: replace phab2002 with phab2003.Apr 17 2026, 4:58 PM

The same should happen with phab2003 replacing phab2002 over at T423727.

Dzahn mentioned this in T424055: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm.Apr 21 2026, 4:43 PM
A_smart_kitten subscribed.Apr 21 2026, 5:20 PM
Dzahn added a subtask: T423727: replace phab2002 with phab2003.Jun 5 2026, 7:04 PM
Arnoldokoth claimed this task.Jun 29 2026, 1:47 PM
gerritbot added a comment.Jun 29 2026, 1:48 PM

Change #1306283 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] phabricator: add multi-replica support

https://gerrit.wikimedia.org/r/1306283

gerritbot added a project: Patch-For-Review.Jun 29 2026, 1:48 PM
Arnoldokoth closed subtask T423727: replace phab2002 with phab2003 as Resolved.Jun 29 2026, 1:57 PM
Arnoldokoth moved this task from Backlog to Work in Progress on the Collaboration-Services board.Jul 2 2026, 6:13 AM
gerritbot added a comment.Jul 14 2026, 5:16 PM

Change #1310621 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] site: revert phab1005 to insetup

https://gerrit.wikimedia.org/r/1310621

ops-monitoring-bot added a comment.Jul 14 2026, 8:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by aokoth@cumin1003 for host phab1005.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jul 14 2026, 8:41 PM

Cookbook cookbooks.sre.hosts.reimage started by aokoth@cumin1003 for host phab1005.eqiad.wmnet with OS trixie completed:

  • phab1005 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607142023_aokoth_4137552_phab1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Aug 3 2026, 7:14 PM

Change #1320217 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] site: apply migration role to phab1005

https://gerrit.wikimedia.org/r/1320217

gerritbot added a comment.Aug 4 2026, 3:26 PM

Change #1320217 merged by AOkoth:

[operations/puppet@production] site: apply migration role to phab1005

https://gerrit.wikimedia.org/r/1320217

gerritbot added a comment.Aug 4 2026, 5:52 PM

Change #1306283 merged by AOkoth:

[operations/puppet@production] phabricator: add multi-replica support

https://gerrit.wikimedia.org/r/1306283

gerritbot added a comment.Aug 4 2026, 6:34 PM

Change #1321048 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] hieradata: reenable scap bootstrapping on phab1005

https://gerrit.wikimedia.org/r/1321048

gerritbot added a comment.Aug 4 2026, 6:44 PM

Change #1321048 merged by AOkoth:

[operations/puppet@production] hieradata: reenable scap bootstrapping on phab1005

https://gerrit.wikimedia.org/r/1321048

CodeReviewBot added a comment.Aug 5 2026, 12:36 PM

aokoth opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/115

phabricator-targets: add phab1005

CodeReviewBot added a comment.Aug 5 2026, 2:50 PM

brennen merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/115

phabricator-targets: add phab1005

Arnoldokoth mentioned this in rPHDEPe2ebca5bdd2e: phabricator-targets: add phab1005.Aug 5 2026, 2:53 PM
gerritbot added a comment.Aug 5 2026, 3:29 PM

Change #1321581 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] mariadb: add grants for phab1005

https://gerrit.wikimedia.org/r/1321581

Arnoldokoth created subtask T435087: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4).Aug 17 2026, 1:04 PM
gerritbot added a comment.Aug 19 2026, 3:31 PM

Change #1321581 merged by AOkoth:

[operations/puppet@production] mariadb: add grants for phab1005

https://gerrit.wikimedia.org/r/1321581

brennen moved this task from Backlog to Radar on the User-brennen board.Aug 20 2026, 6:33 AM
Arnoldokoth closed subtask T435087: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) as Resolved.Aug 20 2026, 3:46 PM
Aklapper added a comment.Aug 21 2026, 6:34 AM

@Arnoldokoth Can this task be resolved?

Arnoldokoth added a comment.Aug 21 2026, 6:04 PM

@Aklapper I was hoping to use this to track decommissioning phab1004. I think once we've done that we can consider this complete.

gerritbot added a comment.Aug 31 2026, 1:42 PM

Change #1332730 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] phabricator: decom phab1004 and add phab1006

https://gerrit.wikimedia.org/r/1332730

Jelto subscribed.Sep 1 2026, 9:22 AM
In T377889#12242438, @Arnoldokoth wrote:

@Aklapper I was hoping to use this to track decommissioning phab1004. I think once we've done that we can consider this complete.

There is a dedicated form for decoms: https://phabricator.wikimedia.org/maniphest/task/edit/form/52/.

Aklapper added a comment.Sep 1 2026, 10:13 AM

For the sake of finding tickets again I'd also prefer a separate ticket for phab1004 decommission. This ticket is called "install a service on phab1005". Different machine... :)

Arnoldokoth added a subtask: T436635: decommission phab1004.Sep 1 2026, 3:03 PM
Jclark-ctr closed subtask T436635: decommission phab1004 as Resolved.Sep 1 2026, 6:14 PM
Aklapper closed this task as Resolved.Sep 4 2026, 12:30 PM

I don't see anything left to do here?

Dzahn awarded a token.Sep 8 2026, 7:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits