Page MenuHomePhabricator
Create Task
Maniphest T163938

setup/install phab1001.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the setup and installation of phab1001.eqiad.wmnet. This was allocated via hardware-requests T156970, and approved to use system wmf4747.

  • - create sub-task for on-site update of hostname labels and visible label field in racktables
  • - update dns (mgmt and production) https://gerrit.wikimedia.org/r/#/c/350476/
  • - update switch port (description, enable, set internal vlan)
  • - operations/puppet repo updates (install_server, site.pp, basic replication of what exists for iridium/phab2001)
  • - resolution of hard disk failure on T163960
  • - os installation
  • - puppet/salt sign/accept/initial run
  • - handoff to release engineering for implementation

This plan was outdated:

Implementation plan:

Details

SubjectRepoBranchLines +/-
mariadb - phabricator: Remove public hashes from configuration filesoperations/puppetproduction+20 -14
mariadb/phabricator: update GRANTS from iridium to phab1001operations/puppetproduction+18 -17
phabricator: ensure /srv/dumps existsoperations/puppetproduction+7 -0
site/phabricator: remove phab role from iridium, make it a spareoperations/puppetproduction+9 -1
phabricator: remove iridium remnants, replace with phab1001operations/puppetproduction+4 -5
phabricator/dumps: remove iridium as allowed dumps hostoperations/puppetproduction+1 -1
phabricator: switch service IPs to phab1001operations/puppetproduction+2 -2
phab: phabricator-new to phab2001, phab1001 using normal domainoperations/puppetproduction+1 -4
cache::misc/phabricator: switch from iridium to phab1001 backendoperations/puppetproduction+1 -1
phabricator: set phab1001 to active phab serveroperations/puppetproduction+3 -3
exim/phabricator: send mail to phab1001, not iridiumoperations/puppetproduction+1 -1
datasets/phabricator: allow phab1001 as rsync host for dumpsoperations/puppetproduction+1 -1
phabricator: set phab_domain to phabricator-new for phab1001operations/puppetproduction+3 -0
cache::misc/phabricator: add director for phab-newoperations/puppetproduction+5 -0
cache::misc/phabricator: add director for phabricator-new, stagingoperations/puppetproduction+3 -0
phabricator: enable stats mail and dumps on phab1001operations/puppetproduction+0 -2
Revert "phabricator/admins: give phab admins access to phab1001"operations/puppetproduction+0 -4
phabricator/admins: give phab admins access to phab1001operations/puppetproduction+4 -0
phabricator: make phab1001 use role::phabricator_serveroperations/puppetproduction+1 -6
phabricator: make phab1001 use role::spare for nowoperations/puppetproduction+5 -1
phabricator: Allow listen_address to be empty for migration from iridiumoperations/puppetproduction+57 -42
add IPv6 records for phab1001.eqiad.wmnet.operations/dnsmaster+2 -0
phab1001: add interface::add_ip6_mappedoperations/puppetproduction+1 -0
phabricator: rsync /srv/repos on iridium to phab1001operations/puppetproduction+15 -0
setting all puppet parameters for phab1001.eqiad.wmnetoperations/puppetproduction+10 -2
phab1001 dns entriesoperations/dnsmaster+4 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
RobH reassigned this task from RobH to • Cmjohnson.Jul 19 2017, 5:53 PM

So this tries to load into the installer, and fails for sdb:

┌─────────────┤ [!!] Partition disks ├─────────────┐
│                                                  │
│ Input/output error during read on /dev/sdb       │
│                                                  │
│ ERROR!!!                                         │
│                                                  │
│                    Retry                         │
│                    Ignore                        │
│                    Cancel                        │
│                                                  │
│     <Go Back>                                    │

Additionally, when reading the installation debug logs, the following relevant errors show up:

Jul 19 17:47:28 kernel: [  133.714952] ata2.00: configured for UDMA/133
Jul 19 17:47:28 kernel: [  133.714966] sd 1:0:0:0: [sdb]  
Jul 19 17:47:28 kernel: [  133.714968] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 19 17:47:28 kernel: [  133.714971] sd 1:0:0:0: [sdb]  
Jul 19 17:47:28 kernel: [  133.714972] Sense Key : Aborted Command [current] [descriptor]
Jul 19 17:47:28 kernel: [  133.714976] Descriptor sense data with sense descriptors (in hex):
Jul 19 17:47:28 kernel: [  133.714978]         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
Jul 19 17:47:28 kernel: [  133.714988]         74 70 6d 00 
Jul 19 17:47:28 kernel: [  133.714993] sd 1:0:0:0: [sdb]  
Jul 19 17:47:28 kernel: [  133.714995] Add. Sense: No additional sense information
Jul 19 17:47:28 kernel: [  133.714997] sd 1:0:0:0: [sdb] CDB: 
Jul 19 17:47:28 kernel: [  133.714999] Read(10): 28 00 74 70 6d 00 00 00 08 00
Jul 19 17:47:28 kernel: [  133.715008] Buffer I/O error on device sdb, logical block 244190624
Jul 19 17:47:28 kernel: [  133.715021] ata2: EH complete


Jul 19 17:46:07 kernel: [   52.741935] sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Jul 19 17:46:07 kernel: [   52.741973] sd 1:0:0:0: [sdb] Write Protect is off
Jul 19 17:46:07 kernel: [   52.741976] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Jul 19 17:46:07 kernel: [   52.741993] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jul 19 17:46:07 kernel: [   52.742024] sd 0:0:0:0: [sda] Write Protect is off
Jul 19 17:46:07 kernel: [   52.742029] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Jul 19 17:46:07 kernel: [   52.742064] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jul 19 17:46:07 kernel: [   52.746292] random: nonblocking pool is initialized

Jul 19 17:46:08 kernel: [   52.836485] ata2.00: BMDMA stat 0x25
Jul 19 17:46:08 kernel: [   52.836489] ata2.00: failed command: READ DMA
Jul 19 17:46:08 kernel: [   52.836495] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
Jul 19 17:46:08 kernel: [   52.836495]          res 51/04:08:00:00:00/00:00:00:00:00/e0 Emask 0x1 (device error)
Jul 19 17:46:08 kernel: [   52.836498] ata2.00: status: { DRDY ERR }
Jul 19 17:46:08 kernel: [   52.836500] ata2.00: error: { ABRT }
Jul 19 17:46:08 kernel: [   52.860923] ata2.00: configured for UDMA/133
Jul 19 17:46:08 kernel: [   52.860927] ata2: EH complete
Jul 19 17:46:08 kernel: [   52.868454] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 19 17:46:08 kernel: [   52.868459] ata2.00: BMDMA stat 0x25
Jul 19 17:46:08 kernel: [   52.868462] ata2.00: failed command: READ DMA
Jul 19 17:46:08 kernel: [   52.868468] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
Jul 19 17:46:08 kernel: [   52.868468]          res 51/04:08:00:00:00/00:00:00:00:00/e0 Emask 0x1 (device error)
Jul 19 17:46:08 kernel: [   52.868472] ata2.00: status: { DRDY ERR }
Jul 19 17:46:08 kernel: [   52.868474] ata2.00: error: { ABRT }
Jul 19 17:46:08 kernel: [   52.892898] ata2.00: configured for UDMA/133
Jul 19 17:46:08 kernel: [   52.892903] ata2: EH complete
Jul 19 17:46:08 kernel: [   52.900432] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 19 17:46:08 kernel: [   52.900437] ata2.00: BMDMA stat 0x25
Jul 19 17:46:08 kernel: [   52.900440] ata2.00: failed command: READ DMA
Jul 19 17:46:08 kernel: [   52.900447] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
Jul 19 17:46:08 kernel: [   52.900447]          res 51/04:08:00:00:00/00:00:00:00:00/e0 Emask 0x1 (device error)
Jul 19 17:46:08 kernel: [   52.900450] ata2.00: status: { DRDY ERR }
Jul 19 17:46:08 kernel: [   52.900452] ata2.00: error: { ABRT }
Jul 19 17:46:08 kernel: [   52.924870] ata2.00: configured for UDMA/133
Jul 19 17:46:08 kernel: [   52.924876] sd 1:0:0:0: [sdb]  
Jul 19 17:46:08 kernel: [   52.924878] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 19 17:46:08 kernel: [   52.924880] sd 1:0:0:0: [sdb]  
Jul 19 17:46:08 kernel: [   52.924882] Sense Key : Aborted Command [current] [descriptor]
Jul 19 17:46:08 kernel: [   52.924885] Descriptor sense data with sense descriptors (in hex):
Jul 19 17:46:08 kernel: [   52.924886]         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
Jul 19 17:46:08 kernel: [   52.924893]         00 00 00 00 
Jul 19 17:46:08 kernel: [   52.924896] sd 1:0:0:0: [sdb]  
Jul 19 17:46:08 kernel: [   52.924897] Add. Sense: No additional sense information
Jul 19 17:46:08 kernel: [   52.924899] sd 1:0:0:0: [sdb] CDB: 
Jul 19 17:46:08 kernel: [   52.924901] Read(10): 28 00 00 00 00 00 00 00 08 00
Jul 19 17:46:08 kernel: [   52.924906] end_request: I/O error, dev sdb, sector 0
Jul 19 17:46:08 kernel: [   52.924909] Buffer I/O error on device sdb, logical block 0
Jul 19 17:46:08 kernel: [   52.924916] ata2: EH complete
Jul 19 17:46:08 kernel: [   52.932406] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 19 17:46:08 kernel: [   52.932411] ata2.00: BMDMA stat 0x25
Jul 19 17:46:08 kernel: [   52.932415] ata2.00: failed command: READ DMA
Jul 19 17:46:08 kernel: [   52.932421] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
Jul 19 17:46:08 kernel: [   52.932421]          res 51/04:08:00:00:00/00:00:00:00:00/e0 Emask 0x1 (device error)
Jul 19 17:46:08 kernel: [   52.932424] ata2.00: status: { DRDY ERR }
Jul 19 17:46:08 kernel: [   52.932426] ata2.00: error: { ABRT }
Jul 19 17:46:08 kernel: [   52.956848] ata2.00: configured for UDMA/133
Jul 19 17:46:08 kernel: [   52.956852] ata2: EH complete

It appears that the sdb has gone bad, so the re-image cannot continue. I'm escalating this to @Cmjohnson for him to have a warranty disk replacement done for sdb.

Once that is done, this system will have to be fully re-imaged, as it appears it was somehow was booted into the installer and wiped the mbr of the original installation. Once the disk replacement is done, please assign back to me for followup.

Thanks!

RobH renamed this task from setup/install phab1001.eqiad.wmnet to replace sdb and then setup/install phab1001.eqiad.wmnet.Jul 19 2017, 5:53 PM

This could also simply be a bad or lose cable on the drive bay! Basic troubleshooting of swapping sda and sdb to see if the error follows is recommended.

RobH added a comment.Jul 19 2017, 6:40 PM

dell dispatch SR951188562 for the replacement hard disk, shipping to eqiad.

• Cmjohnson reassigned this task from • Cmjohnson to RobH.Jul 20 2017, 6:02 PM

Disk has been replaced:

Return shipping info is

USPS 9202 3946 5301 2436 1520 81
FEDEX 9611918 2393026 72902102

RobH updated the task description. (Show Details)Jul 20 2017, 8:34 PM
Dzahn added a comment.Jul 21 2017, 5:23 PM

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=16&hoststatustypes=3&serviceprops=2097162

gerritbot added a comment.Jul 21 2017, 6:09 PM

Change 366885 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: make phab1001 use role::spare for now

https://gerrit.wikimedia.org/r/366885

gerritbot added a project: Patch-For-Review.Jul 21 2017, 6:09 PM

Change 366885 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: make phab1001 use role::spare for now

https://gerrit.wikimedia.org/r/366885

gerritbot added a comment.Jul 21 2017, 6:15 PM

Change 366885 merged by Dzahn:
[operations/puppet@production] phabricator: make phab1001 use role::spare for now

https://gerrit.wikimedia.org/r/366885

RobH added a comment.Jul 21 2017, 6:24 PM

So this had an issue when the new role assigned an IP to it that was in use in iridium. So we've put it back to spare, and reimaged to remove the bad role info. Reimage is still in progress.

RobH reassigned this task from RobH to Dzahn.Jul 21 2017, 7:58 PM
RobH updated the task description. (Show Details)

back to @Dzahn for service implementation

RobH removed a project: Patch-For-Review.Jul 21 2017, 7:58 PM
• mmodell added a comment.Jul 31 2017, 6:04 PM

@Dzahn: anything I can do to help get this one moving? I tried to log in to phab1001 so that I could verify that puppet has set things up correctly but I am not able to log in. I assume it hasn't had a puppet run yet?

Dzahn added a comment.Jul 31 2017, 6:18 PM

@mmodell It doesn't have the puppet role for phab on it because we had to remove it. The role just isn't ready for being used on multiple hosts and it never worked before. It hardcodes an additional IP address for the second SSH server. So if you apply it you will get the same IP on multiple servers. We need to get rid of the hardcoded IP in there.

• mmodell added a comment.Jul 31 2017, 6:20 PM

@Dzahn: Ok, I can fix that. Thanks! I think there is a lot of room for improvement in the way we handle IP addresses.

• mmodell added a comment.Jul 31 2017, 6:28 PM

So what should we do instead of having host-specific IPs in hieradata/role/[datacenter]/phabricator_server.yaml? Should puppet look up the info from DNS or should we have a host-specific hiera level?

Paladox added a comment.Jul 31 2017, 6:41 PM

@mmodell maybe a host specific hiera level

• mmodell removed a subtask: T168699: Verify that the codfw lvs is configured correctly for Phabricator.Jul 31 2017, 6:43 PM
• mmodell mentioned this in T137928: Deploy phabricator to phab2001.codfw.wmnet.Jul 31 2017, 6:46 PM
Dzahn added a comment.EditedJul 31 2017, 6:50 PM

role/[datacenter]/ seems actually correct and better than host names.

(well of course until iridium is gone we need a special solution and so yea, hieradata/hosts/ until we have one per datacenter again)

• mmodell added a comment.Jul 31 2017, 6:59 PM

@Dzahn: Do we have an IP assigned for git-ssh on phab1001?

Dzahn added a comment.Jul 31 2017, 7:20 PM

@mmodell Here's the thing.

There is the git-ssh IP for eqiad 208.80.154.250 and git-ssh for codfw 208.80.153.250. This IP is on the "lo" loopback interface. The one for eqiad is on iridium.

Then there is the "phab1001-vcs" 10.64.32.186 and "phab2001-vcs" 10.192.32.149. This IP is on the eth0 interface. Also here the "1001" one is on iridium.

When we added phab1001-vcs on _iridium_ we did not name it iridium-vcs because we knew iridium was supposed to be reinstalled as phab1001.

Then plans changed from "reinstall iridum as phab1001" to "setup separate phab1001 in parallel" and now we have "phab1001-vcs" on iridium but NOT on phab1001.

So now we have 2 options:

a) schedule downtime, take down iridium, remove the additional IPs, add them on phab1001. use phab1001, shut down iridium. done

or

b) add additional IPs with some random name that isn't phab1001-vcs even though it's on phab1001, keep the existing names/IPs on iridium., be able to test, switch to phab1001, then rename things again.. hmm.. this seems like we would barely get less downtime out of it but would have more work.

• mmodell added a comment.Jul 31 2017, 7:21 PM

Yeah I think scheduled downtime to switch the IP is reasonable. I'll make a patch and we can do it this week if you're up for it.

gerritbot added a comment.Jul 31 2017, 10:46 PM

Change 368947 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator/admins: give phab admins access to phab1001

https://gerrit.wikimedia.org/r/368947

gerritbot added a project: Patch-For-Review.Jul 31 2017, 10:46 PM
gerritbot added a comment.Jul 31 2017, 10:51 PM

Change 368947 merged by Dzahn:
[operations/puppet@production] phabricator/admins: give phab admins access to phab1001

https://gerrit.wikimedia.org/r/368947

gerritbot added a comment.Jul 31 2017, 10:57 PM

Change 368841 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] phabricator: rsync /srv/repos on iridium to phab1001

https://gerrit.wikimedia.org/r/368841

gerritbot added a comment.Jul 31 2017, 11:39 PM

Change 368841 merged by Dzahn:
[operations/puppet@production] phabricator: rsync /srv/repos on iridium to phab1001

https://gerrit.wikimedia.org/r/368841

gerritbot added a comment.Aug 1 2017, 12:06 AM

Change 368957 had a related patch set uploaded (by Paladox; owner: Dzahn):
[operations/puppet@production] phab1001: add interface::add_ip6_mapped

https://gerrit.wikimedia.org/r/368957

gerritbot added a comment.Aug 1 2017, 1:38 AM

Change 368957 merged by Dzahn:
[operations/puppet@production] phab1001: add interface::add_ip6_mapped

https://gerrit.wikimedia.org/r/368957

gerritbot added a comment.Aug 1 2017, 2:07 AM

Change 368969 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add IPv6 records for phab1001.eqiad.wmnet.

https://gerrit.wikimedia.org/r/368969

gerritbot added a comment.Aug 1 2017, 2:19 AM

Change 368969 merged by Dzahn:
[operations/dns@master] add IPv6 records for phab1001.eqiad.wmnet.

https://gerrit.wikimedia.org/r/368969

gerritbot added a comment.Aug 1 2017, 6:14 PM

Change 369001 had a related patch set uploaded (by Dzahn; owner: 20after4):
[operations/puppet@production] phabricator: Allow listen_address to be empty for migration from iridium

https://gerrit.wikimedia.org/r/369001

gerritbot added a comment.Aug 1 2017, 6:16 PM

Change 369001 merged by Dzahn:
[operations/puppet@production] phabricator: Allow listen_address to be empty for migration from iridium

https://gerrit.wikimedia.org/r/369001

gerritbot added a comment.Aug 1 2017, 6:21 PM

Change 369444 had a related patch set uploaded (by Dzahn; owner: Paladox):
[operations/puppet@production] Revert "phabricator: make phab1001 use role::spare for now"

https://gerrit.wikimedia.org/r/369444

gerritbot added a comment.Aug 1 2017, 6:23 PM

Change 369444 merged by Dzahn:
[operations/puppet@production] phabricator: make phab1001 use role::phabricator_server

https://gerrit.wikimedia.org/r/369444

gerritbot added a comment.Aug 1 2017, 6:32 PM

Change 369445 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "phabricator/admins: give phab admins access to phab1001"

https://gerrit.wikimedia.org/r/369445

gerritbot added a comment.Aug 1 2017, 6:33 PM

Change 369447 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: enable stats mail and dumps on phab1001

https://gerrit.wikimedia.org/r/369447

gerritbot added a comment.Aug 1 2017, 6:37 PM

Change 369445 merged by Dzahn:
[operations/puppet@production] Revert "phabricator/admins: give phab admins access to phab1001"

https://gerrit.wikimedia.org/r/369445

gerritbot added a comment.Aug 1 2017, 6:52 PM

Change 369447 merged by Dzahn:
[operations/puppet@production] phabricator: enable stats mail and dumps on phab1001

https://gerrit.wikimedia.org/r/369447

Stashbot subscribed.Aug 3 2017, 12:05 AM

Mentioned in SAL (#wikimedia-operations) [2017-08-03T00:05:31Z] <mutante> rsyncing /srv/repos from iridium to phab1001 (T163938)

gerritbot added a comment.Aug 3 2017, 1:43 AM

Change 369820 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cache::misc/phabricator: switch from iridium to phab1001 backend

https://gerrit.wikimedia.org/r/369820

• mmodell renamed this task from replace sdb and then setup/install phab1001.eqiad.wmnet to setup/install phab1001.eqiad.wmnet.Aug 3 2017, 1:48 AM
gerritbot added a comment.Aug 3 2017, 1:49 AM

Change 369821 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cache::misc/phabricator: add director for phabricator-new, staging

https://gerrit.wikimedia.org/r/369821

gerritbot added a comment.Aug 3 2017, 2:11 AM

Change 369823 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: set phab_domain to phabricator-new for phab1001

https://gerrit.wikimedia.org/r/369823

gerritbot added a comment.Aug 3 2017, 2:14 AM

Change 369821 merged by Dzahn:
[operations/puppet@production] cache::misc/phabricator: add director for phabricator-new, staging

https://gerrit.wikimedia.org/r/369821

gerritbot added a comment.Aug 3 2017, 2:45 AM

Change 369829 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cache::misc/phabricator: add director for phab-new

https://gerrit.wikimedia.org/r/369829

gerritbot added a comment.Aug 3 2017, 2:53 AM

Change 369829 merged by Dzahn:
[operations/puppet@production] cache::misc/phabricator: add director for phab-new

https://gerrit.wikimedia.org/r/369829

gerritbot added a comment.Aug 3 2017, 3:06 AM

Change 369823 merged by Dzahn:
[operations/puppet@production] phabricator: set phab_domain to phabricator-new for phab1001

https://gerrit.wikimedia.org/r/369823

gerritbot added a comment.Aug 3 2017, 3:21 AM

Change 369831 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] datasets/phabricator: allow phab1001 as rsync host for dumps

https://gerrit.wikimedia.org/r/369831

gerritbot added a comment.Aug 3 2017, 3:23 AM

Change 369831 merged by Dzahn:
[operations/puppet@production] datasets/phabricator: allow phab1001 as rsync host for dumps

https://gerrit.wikimedia.org/r/369831

gerritbot added a comment.Aug 3 2017, 3:28 AM

Change 369832 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb/phabricator: update GRANTS from iridium to phab1001

https://gerrit.wikimedia.org/r/369832

• mmodell added a comment.Aug 3 2017, 3:41 AM

I'm making this comment from phab1001.eqiad.wmnet :)

gerritbot added a comment.Aug 3 2017, 4:03 AM

Change 369834 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] exim/phabricator: send mail to phab1001, not iridium

https://gerrit.wikimedia.org/r/369834

gerritbot added a comment.Aug 3 2017, 4:24 AM

Change 369836 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: set phab1001 to active phab server

https://gerrit.wikimedia.org/r/369836

• Mostafa2018k subscribed.Aug 3 2017, 4:32 AM
Paladox added a comment.Aug 3 2017, 11:29 AM

I found https://phabricator-new.wikimedia.org/ continuously redirecting which then the browser short circuits it. But https://phabricator-new.wikimedia.org/diffusion/ works :)

Paladox removed a subscriber: • Mostafa2018k.Aug 3 2017, 11:29 AM
gerritbot added a comment.Aug 3 2017, 10:11 PM

Change 370104 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phab: phabricator-new to phab2001, phab1001 using normal domain

https://gerrit.wikimedia.org/r/370104

Dzahn added a comment.EditedAug 3 2017, 11:20 PM

steps for migration:

  • stop phd and puppet on iridium
  • rsync /srv/repos

/usr/bin/rsync -av rsync://iridium.eqiad.wmnet/srv-repos /srv/repos/

  • move IP addresses:

on iridium:

phab1001-vcs.eqiad.wmnet.
sudo ip addr del 10.64.32.186/32 dev eth0
sudo ip addr del 2620:0:861:103:10:64:32:186/128 dev eth0

git-ssh.eqiad.wikimedia.org.
sudo ip addr del 208.80.154.250/32 dev lo
sudo ip addr del 2620:0:861:ed1a::3:16/128 dev lo

on phab1001:

sudo ip addr add 10.64.32.186/32 dev eth0
sudo ip addr add 2620:0:861:103:10:64:32:186/128 dev eth0
(already done by puppet) # sudo ip addr add 208.80.154.250/32 dev lo
(already done by puppet) # sudo ip addr add 2620:0:861:ed1a::3:16/128 dev lo
...

gerritbot added a comment.Aug 3 2017, 11:33 PM

Change 370119 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: switch service IPs to phab1001

https://gerrit.wikimedia.org/r/370119

gerritbot added a comment.Aug 3 2017, 11:41 PM

Change 370122 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/phabricator: remove phab role from iridium, make it a spare

https://gerrit.wikimedia.org/r/370122

gerritbot added a comment.Aug 3 2017, 11:44 PM

Change 370123 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator/dumps: remove iridium as allowed dumps host

https://gerrit.wikimedia.org/r/370123

gerritbot added a comment.Aug 4 2017, 12:29 AM

Change 370119 had a related patch set uploaded (by Paladox; owner: Dzahn):
[operations/puppet@production] phabricator: switch service IPs to phab1001

https://gerrit.wikimedia.org/r/370119

gerritbot added a comment.Aug 4 2017, 12:30 AM

Change 370119 merged by Dzahn:
[operations/puppet@production] phabricator: switch service IPs to phab1001

https://gerrit.wikimedia.org/r/370119

Stashbot added a comment.Aug 4 2017, 2:15 AM

Mentioned in SAL (#wikimedia-operations) [2017-08-04T02:15:02Z] <mutante> phab1001 can't talk to mx servers via IPv6, but works via IPv4. iridium and other mailservers can also talk IPv6 to it. why? it did not change even when stopping ferm on client and on server it allows from anywhere. workaround for now was to hardcode IPv4 IP in phab config. (T163938)

gerritbot added a comment.Aug 4 2017, 2:40 AM

Change 370123 merged by Dzahn:
[operations/puppet@production] phabricator/dumps: remove iridium as allowed dumps host

https://gerrit.wikimedia.org/r/370123

gerritbot added a comment.Aug 4 2017, 2:49 AM

Change 370140 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: remove iridium remnants, replace with phab1001

https://gerrit.wikimedia.org/r/370140

Dzahn updated the task description. (Show Details)Aug 4 2017, 2:50 AM
gerritbot added a comment.Aug 4 2017, 2:53 AM

Change 370140 merged by Dzahn:
[operations/puppet@production] phabricator: remove iridium remnants, replace with phab1001

https://gerrit.wikimedia.org/r/370140

• mmodell added a subtask: T172478: git-ssh.wikimedia.org and IPv6 are broken after switch to phab1001.Aug 4 2017, 3:05 AM
ayounsi closed subtask T172478: git-ssh.wikimedia.org and IPv6 are broken after switch to phab1001 as Resolved.Aug 4 2017, 3:41 AM
• mmodell reopened subtask T172478: git-ssh.wikimedia.org and IPv6 are broken after switch to phab1001 as Open.Aug 4 2017, 4:23 AM
Dzahn closed subtask T172478: git-ssh.wikimedia.org and IPv6 are broken after switch to phab1001 as Resolved.Aug 4 2017, 7:23 AM
Dzahn updated the task description. (Show Details)Aug 4 2017, 7:28 AM

T152129 has been resolved

Dzahn closed this task as Resolved.Aug 4 2017, 7:45 AM
Dzahn updated the task description. (Show Details)
• mmodell awarded a token.Aug 4 2017, 7:59 AM
gerritbot added a comment.Aug 4 2017, 9:28 PM

Change 370122 merged by Dzahn:
[operations/puppet@production] site/phabricator: remove phab role from iridium, make it a spare

https://gerrit.wikimedia.org/r/370122

gerritbot added a comment.Aug 8 2017, 3:05 AM

Change 370607 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: ensure /srv/dumps exists

https://gerrit.wikimedia.org/r/370607

gerritbot added a comment.Aug 8 2017, 3:20 AM

Change 370607 merged by Dzahn:
[operations/puppet@production] phabricator: ensure /srv/dumps exists

https://gerrit.wikimedia.org/r/370607

Dzahn added a comment.Aug 8 2017, 3:52 AM

https://gerrit.wikimedia.org/r/370518 by paladox merged as well, to fix another cron that sends the stats mail to admins

Stashbot added a comment.Aug 8 2017, 3:55 AM

Mentioned in SAL (#wikimedia-operations) [2017-08-08T03:55:06Z] <mutante> phab1001 /usr/local/bin/community_metrics.sh | /usr/local/bin/project_changes.sh creating stats mails to admins (which failed before) (T163938)

Paladox added a comment.Aug 8 2017, 10:33 AM

we should probably only use @Aklapper email for /usr/local/bin/project_changes.sh on prod. I was testing /usr/local/bin/project_changes.sh and it sent it to him which then forwarded to the list without me realising.

• mmodell changed the status of subtask T164810: Switch phabricator production to codfw from Open to Stalled.Aug 28 2017, 7:12 AM
gerritbot added a comment.Sep 13 2017, 3:10 AM

Change 369832 merged by Jcrespo:
[operations/puppet@production] mariadb/phabricator: update GRANTS from iridium to phab1001

https://gerrit.wikimedia.org/r/369832

gerritbot added a comment.Sep 13 2017, 4:06 AM

Change 377703 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb - phabricator: Remove public hashes from configuration files

https://gerrit.wikimedia.org/r/377703

gerritbot added a comment.Sep 13 2017, 4:08 AM

Change 377703 merged by Jcrespo:
[operations/puppet@production] mariadb - phabricator: Remove public hashes from configuration files

https://gerrit.wikimedia.org/r/377703

• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:46 PM
• mmodell changed the status of subtask T164810: Switch phabricator production to codfw from Stalled to Open.Nov 7 2017, 10:15 PM
• mmodell removed a subtask: T164810: Switch phabricator production to codfw.Mar 23 2018, 8:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL