Page MenuHomePhabricator
Create Task
Maniphest T289112

decommission pc2007.codfw.wmnet
Closed, ResolvedPublicRequest
Actions

Description

This task will track the decommission-hardware of server pc2007.codfw.wmnet.

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.

pc2007

Steps for service owner:

https://wikitech.wikimedia.org/wiki/MariaDB/Decommissioning_a_DB_Host

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

Details

SubjectRepoBranchLines +/-
mariadb: Remove pc2007 from puppetoperations/puppetproduction+0 -11
ProductionServices.php: Remove pc2007operations/mediawiki-configmaster+0 -1
mariadb: Set pc2007 to spareoperations/puppetproduction+6 -3
pc[12]007-010: Disable notificationsoperations/puppetproduction+8 -0
Customize query in gerrit

Related Objects

Event Timeline

LSobanski renamed this task from decommission <FQDN of server> to decommission pc2007.codfw.wmnet.Aug 18 2021, 8:30 AM
LSobanski created this task.
LSobanski moved this task from Triage to Blocked on the DBA board.Aug 18 2021, 8:41 AM
gerritbot added a comment.Aug 30 2021, 6:46 AM

Change 715441 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc[12]007-010: Disable notifications

https://gerrit.wikimedia.org/r/715441

gerritbot added a project: Patch-For-Review.Aug 30 2021, 6:46 AM

Change 715441 merged by Marostegui:

[operations/puppet@production] pc[12]007-010: Disable notifications

https://gerrit.wikimedia.org/r/715441

• Marostegui subscribed.Aug 30 2021, 6:48 AM

I have disabled notifications to avoid pages from unused hosts. No rush in decommissioning, but I wanted to make sure we don't get paged for hosts that we don't use just in case.

• Marostegui updated the task description. (Show Details)Aug 30 2021, 6:49 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 30 2021, 7:12 AM
• Marostegui claimed this task.Sep 2 2021, 4:56 AM

I am going to decommission this host as it is lagging behind all the time and being annoying on orchestrator.

gerritbot added a comment.Sep 2 2021, 6:54 AM

Change 716205 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Set pc2007 to spare

https://gerrit.wikimedia.org/r/716205

gerritbot added a project: Patch-For-Review.Sep 2 2021, 6:54 AM

Change 716206 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] ProductionServices.php: Remove pc2007

https://gerrit.wikimedia.org/r/716206

gerritbot added a comment.Sep 2 2021, 6:57 AM

Change 716206 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: Remove pc2007

https://gerrit.wikimedia.org/r/716206

• Marostegui updated the task description. (Show Details)Sep 2 2021, 6:58 AM
gerritbot added a comment.Sep 2 2021, 6:58 AM

Change 716205 merged by Marostegui:

[operations/puppet@production] mariadb: Set pc2007 to spare

https://gerrit.wikimedia.org/r/716205

• Marostegui updated the task description. (Show Details)Sep 2 2021, 6:59 AM
Stashbot added a comment.Sep 2 2021, 6:59 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T06:59:18Z] <marostegui@deploy1002> Synchronized wmf-config/ProductionServices.php: Remove pc2007 T289112 (duration: 01m 06s)

Stashbot added a comment.Sep 2 2021, 7:00 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T07:00:22Z] <marostegui> Stop mariadb on pc2007 before decommissioning T289112

• Marostegui moved this task from Blocked to In progress on the DBA board.Sep 2 2021, 7:00 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 2 2021, 7:10 AM
gerritbot added a comment.Sep 3 2021, 5:19 AM

Change 716915 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Remove pc2007 from puppet

https://gerrit.wikimedia.org/r/716915

gerritbot added a project: Patch-For-Review.Sep 3 2021, 5:19 AM
gerritbot added a comment.Sep 3 2021, 5:21 AM

Change 716915 merged by Marostegui:

[operations/puppet@production] mariadb: Remove pc2007 from puppet

https://gerrit.wikimedia.org/r/716915

• Marostegui updated the task description. (Show Details)Sep 3 2021, 5:21 AM
ops-monitoring-bot added a comment.Sep 3 2021, 5:30 AM

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: pc2007.codfw.wmnet

  • pc2007.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

• Marostegui updated the task description. (Show Details)Sep 3 2021, 5:36 AM
• Marostegui added a subscriber: Volans.Sep 3 2021, 5:48 AM

@Volans the above decommissioning has failed this is the trace:

P17197 (An Untitled Masterwork)
1[05:20:26] marostegui@cumin1001:~$ sudo cookbook sre.hosts.decommission pc2007.codfw.wmnet -t T289112
2>>> ATTENTION: destructive action for 1 hosts: pc2007.codfw.wmnet
3Are you sure to proceed?
4Type "go" to proceed or "abort" to interrupt the execution
5> go
6START - Cookbook sre.hosts.decommission for hosts pc2007.codfw.wmnet
7Looking for matches in puppetmaster1001.eqiad.wmnet:/var/lib/git/operations/puppet
8----- OUTPUT of 'git -C '/var/lib...04[^0-9A-Za-z])'' -----
9manifests/site.pp:node 'pc2007.codfw.wmnet' {
10modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200: fixed-address pc2007.codfw.wmnet;
11================
12PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.12hosts/s]
13FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
14100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'git -C '/var/lib...04[^0-9A-Za-z])''.
15100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
16Looking for matches in puppetmaster1001.eqiad.wmnet:/srv/private
17----- OUTPUT of 'git -C '/srv/pri...04[^0-9A-Za-z])'' -----
18================
19PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.83hosts/s]
20FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
21100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'git -C '/srv/pri...04[^0-9A-Za-z])''.
22100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
23Looking for matches in deploy1002.eqiad.wmnet:/srv/mediawiki-staging
24----- OUTPUT of 'git -C '/srv/med...04[^0-9A-Za-z])'' -----
25================
26PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00, 1.10s/hosts]
27FAIL | | 0% (0/1) [00:01<?, ?hosts/s]
28100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'git -C '/srv/med...04[^0-9A-Za-z])''.
29100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
30Looking for matches in deploy1002.eqiad.wmnet:/srv/deployment-charts
31----- OUTPUT of 'git -C '/srv/dep...04[^0-9A-Za-z])'' -----
32================
33PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.78hosts/s]
34FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
35100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'git -C '/srv/dep...04[^0-9A-Za-z])''.
36100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
37>>> Found match(es) in the Puppet or mediawiki-config repositories (see above), proceed anyway?
38Type "go" to proceed or "abort" to interrupt the execution
39> go
40Looking for Kerberos credentials on KDC kadmin node.
41----- OUTPUT of 'find /srv/kerber...07.codfw.wmnet*"' -----
42================
43PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.73hosts/s]
44FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
45100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'find /srv/kerber...07.codfw.wmnet*"'.
46----- OUTPUT of '/usr/local/sbin/...07.codfw.wmnet*"' -----
47================
48PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.92hosts/s]
49FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
50100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/usr/local/sbin/...07.codfw.wmnet*"'.
51100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
52No Kerberos credentials found.
53----- OUTPUT of 'grep -P '\s*comm...cinga/icinga.cfg' -----
54command_file=/var/lib/icinga/rw/icinga.cmd
55================
56PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 2.68hosts/s]
57FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
58100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'grep -P '\s*comm...cinga/icinga.cfg'.
59100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
60----- OUTPUT of '/usr/local/bin/i...atus -j "pc2007"' -----
61{"pc2007": {"name": "pc2007", "state": "UP", "optimal": true, "downtimed": true, "notifications_enabled": true, "failed_services": []}}
62================
63PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.24hosts/s]
64FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
65100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/usr/local/bin/i...atus -j "pc2007"'.
66100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
67Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: pc2007
68----- OUTPUT of 'bash -c 'echo -n.../rw/icinga.cmd '' -----
69================
70PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 3.02hosts/s]
71FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
72100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'bash -c 'echo -n.../rw/icinga.cmd ''.
73100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
74----- OUTPUT of 'bash -c 'echo -n.../rw/icinga.cmd '' -----
75================
76PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 2.68hosts/s]
77FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
78100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'bash -c 'echo -n.../rw/icinga.cmd ''.
79100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
80Downtimed host on Icinga
81Management Password:
82Found physical host
83----- OUTPUT of '/usr/local/bin/i...atus -j "pc2007"' -----
84{"pc2007": {"name": "pc2007", "state": "UP", "optimal": true, "downtimed": true, "notifications_enabled": true, "failed_services": []}}
85================
86PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00, 1.07s/hosts]
87FAIL | | 0% (0/1) [00:01<?, ?hosts/s]
88100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/usr/local/bin/i...atus -j "pc2007"'.
89100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
90Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: pc2007
91----- OUTPUT of 'bash -c 'echo -n.../rw/icinga.cmd '' -----
92================
93PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 2.35hosts/s]
94FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
95100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'bash -c 'echo -n.../rw/icinga.cmd ''.
96100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
97----- OUTPUT of 'bash -c 'echo -n.../rw/icinga.cmd '' -----
98================
99PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 2.79hosts/s]
100FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
101100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'bash -c 'echo -n.../rw/icinga.cmd ''.
102100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
103Downtimed management interface on Icinga
104----- OUTPUT of 'true' -----
105================
106PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.10hosts/s]
107FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
108100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'true'.
109100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
110----- OUTPUT of 'lsblk --all --ou...-all --force %*'' -----
111/dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
112/dev/sda: 8 bytes were erased at offset 0x45d679ffe00 (gpt): 45 46 49 20 50 41 52 54
113/dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
114/dev/sda1: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
115/dev/sda2: 10 bytes were erased at offset 0x00000ff6 (swap): 53 57 41 50 53 50 41 43 45 32
116/dev/sda3: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 30 31
117================
118PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.20hosts/s]
119FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
120100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'lsblk --all --ou...-all --force %*''.
121100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
122Wiped all swraid, partition-table and filesystem signatures
123Running IPMI command: ipmitool -I lanplus -H pc2007.mgmt.codfw.wmnet -U root -E chassis power off
124Powered off
125Disable and reset vlan on asw-a1-codfw:ge-1/0/1 for local eno1
126Delete IP 10.192.0.104/22 on eno1
127Delete IP 2620:0:860:101:10:192:0:104/64 on eno1
128Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
129Sleeping for 20s to avoid race conditions...
130Removed host pc2007.codfw.wmnet from Debmonitor
131Removed from DebMonitor
132----- OUTPUT of 'puppet node clea...2007.codfw.wmnet' -----
133Notice: Revoked certificate with serial 6397
134Notice: Removing file Puppet::SSL::Certificate pc2007.codfw.wmnet at '/var/lib/puppet/server/ssl/ca/signed/pc2007.codfw.wmnet.pem'
135pc2007.codfw.wmnet
136================
137PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00, 1.85s/hosts]
138FAIL | | 0% (0/1) [00:01<?, ?hosts/s]
139100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'puppet node clea...2007.codfw.wmnet'.
140----- OUTPUT of 'puppet node deac...2007.codfw.wmnet' -----
141Submitted 'deactivate node' for pc2007.codfw.wmnet with UUID 49134376-d2f7-4a7f-987e-fa1039787304
142================
143PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00, 1.80s/hosts]
144FAIL | | 0% (0/1) [00:01<?, ?hosts/s]
145100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'puppet node deac...2007.codfw.wmnet'.
146100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
147Removed from Puppet master and PuppetDB
148Sleeping for 3 minutes to get Netbox caches in sync
149Generating the DNS records from Netbox data. It will take a couple of minutes.
150----- OUTPUT of 'cd /tmp && runus...e asset tag one"' -----
1512021-09-03 05:24:29,276 [INFO] Gathering devices, interfaces, addresses and prefixes from Netbox
1522021-09-03 05:27:52,361 [WARNING] Device frqueue1002 of IP 10.64.40.204/26 not in devices, skipping.
1532021-09-03 05:27:52,365 [WARNING] Device phab1003 of IP 10.65.1.16/16 not in devices, skipping.
1542021-09-03 05:27:52,405 [INFO] Gathered 2328 devices from Netbox
1552021-09-03 05:27:52,405 [INFO] Generating DNS records
1562021-09-03 05:28:02,216 [INFO] Generated 13396 direct and reverse records (6698 each) in 27 direct zones and 171 reverse zones
1572021-09-03 05:28:02,216 [INFO] Cloning /srv/netbox-exports/dns.git/ to /tmp/dns-c25pcHBldHM-a5evgy4f ...
1582021-09-03 05:28:02,455 [INFO] Generating zonefile snippets to directory /tmp/dns-c25pcHBldHM-a5evgy4f
1592021-09-03 05:28:03,237 [INFO] Committed changes: eb549766e7beb726034321e0e58762d43e157be5
1602021-09-03 05:28:03,264 [INFO] Validating generated data
1612021-09-03 05:28:03,264 [INFO] Commit details: {'insertions': 0, 'deletions': 4, 'lines': 4, 'files': 4}
162commit eb549766e7beb726034321e0e58762d43e157be5
163Author: generate-dns-snippets <noc@wikimedia.org>
164Date: Fri Sep 3 05:28:03 2021 +0000
165
166 marostegui@cumin1001: pc2007.codfw.wmnet decommissioned, removing all IPs except the asset tag one
167
168diff --git a/0.192.10.in-addr.arpa b/0.192.10.in-addr.arpa
169index 2f62d7d..5543f93 100644
170--- a/0.192.10.in-addr.arpa
171+++ b/0.192.10.in-addr.arpa
172@@ -99,7 +99,6 @@
173 101 1H IN PTR db2094.codfw.wmnet.
174 102 1H IN PTR graphite2003.codfw.wmnet.
175 103 1H IN PTR rpki2001.codfw.wmnet.
176-104 1H IN PTR pc2007.codfw.wmnet.
177 105 1H IN PTR ms-be2044.codfw.wmnet.
178 106 1H IN PTR ms-be2045.codfw.wmnet.
179 107 1H IN PTR elastic2037.codfw.wmnet.
180diff --git a/1.193.10.in-addr.arpa b/1.193.10.in-addr.arpa
181index f2d26e7..fbe0bbd 100644
182--- a/1.193.10.in-addr.arpa
183+++ b/1.193.10.in-addr.arpa
184@@ -273,7 +273,6 @@
185 206 1H IN PTR wmf6646.mgmt.codfw.wmnet.
186 207 1H IN PTR bast2002.mgmt.codfw.wmnet.
187 207 1H IN PTR wmf6649.mgmt.codfw.wmnet.
188-208 1H IN PTR pc2007.mgmt.codfw.wmnet.
189 208 1H IN PTR wmf6659.mgmt.codfw.wmnet.
190 209 1H IN PTR pc2008.mgmt.codfw.wmnet.
191 209 1H IN PTR wmf6660.mgmt.codfw.wmnet.
192diff --git a/codfw.wmnet b/codfw.wmnet
193index 9bdde11..b5315ae 100644
194--- a/codfw.wmnet
195+++ b/codfw.wmnet
196@@ -738,7 +738,6 @@ parse2017 1H IN A 10.192.48.150
197 parse2018 1H IN A 10.192.48.151
198 parse2019 1H IN A 10.192.48.152
199 parse2020 1H IN A 10.192.48.153
200-pc2007 1H IN A 10.192.0.104
201 pc2008 1H IN A 10.192.16.35
202 pc2009 1H IN A 10.192.32.10
203 pc2010 1H IN A 10.192.48.14
204diff --git a/mgmt.codfw.wmnet b/mgmt.codfw.wmnet
205index d2d167b..3cb199d 100644
206--- a/mgmt.codfw.wmnet
207+++ b/mgmt.codfw.wmnet
208@@ -537,7 +537,6 @@ parse2017 1H IN A 10.193.3.124
209 parse2018 1H IN A 10.193.3.125
210 parse2019 1H IN A 10.193.3.126
211 parse2020 1H IN A 10.193.3.127
212-pc2007 1H IN A 10.193.1.208
213 pc2008 1H IN A 10.193.1.209
214 pc2009 1H IN A 10.193.1.210
215 pc2010 1H IN A 10.193.1.211
216METADATA: {"path": "/tmp/dns-c25pcHBldHM-a5evgy4f", "sha1": "eb549766e7beb726034321e0e58762d43e157be5", "insertions": 0, "deletions": 4, "lines": 4, "files": 4}
217================
218PASS |██████████████████████████████████████████████████████████████████████████████| 100% (1/1) [03:35<00:00, 215.06s/hosts]
219FAIL | | 0% (0/1) [03:35<?, ?hosts/s]
220100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'cd /tmp && runus...e asset tag one"'.
221100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
222>>> Have you checked that the diff is OK?
223Type "go" to proceed or "abort" to interrupt the execution
224> go
225----- OUTPUT of 'cd /tmp && runus...58762d43e157be5"' -----
2262021-09-03 05:28:07,787 [INFO] Pushed with bitflags 256: 5ba1dcb..eb54976
2272021-09-03 05:28:07,878 [INFO] Temporary directory /tmp/dns-c25pcHBldHM-a5evgy4f removed.
228================
229PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 1.12hosts/s]
230FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
231100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'cd /tmp && runus...58762d43e157be5"'.
232100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
233Updating the Netbox passive copies of the repository on netbox2001.wikimedia.org
234----- OUTPUT of 'runuser -u netbo...rg master:master' -----
235From https://netbox1001.wikimedia.org/dns
236 5ba1dcb..eb54976 master -> master
237 5ba1dcb..eb54976 master -> netbox1001.wikimedia.org/master
238================
239PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00, 1.54s/hosts]
240FAIL | | 0% (0/1) [00:01<?, ?hosts/s]
241100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'runuser -u netbo...rg master:master'.
242100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
243Updating the authdns copies of the repository on authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
244===== NODE GROUP =====
245(12) authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
246----- OUTPUT of 'runuser -u netbo...e58762d43e157be5' -----
247From https://netbox-exports.wikimedia.org/dns
248 5ba1dcb..eb54976 master -> origin/master
249Updating 5ba1dcb..eb54976
250Fast-forward
251 0.192.10.in-addr.arpa | 1 -
252 1.193.10.in-addr.arpa | 1 -
253 codfw.wmnet | 1 -
254 mgmt.codfw.wmnet | 1 -
255 4 files changed, 4 deletions(-)
256================
257PASS |█████████████████████████████████████████████████████████████████████████████| 100% (12/12) [00:06<00:00, 1.29hosts/s]
258FAIL | | 0% (0/12) [00:06<?, ?hosts/s]
259100.0% (12/12) success ratio (>= 100.0% threshold) for command: 'runuser -u netbo...e58762d43e157be5'.
260100.0% (12/12) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
261Deploying the updated zonefiles on authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
262===== NODE GROUP =====
263(1) dns5002.wikimedia.org
264----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
265Assembling and testing data in /tmp/dns-check.mc0iy3_c
266 -- Generating zonefiles from zone templates
267 -- Processed 209 zones into directory /tmp/dns-check.mc0iy3_c/zones
268OK: No tabs
269Summary of violations:
270 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
271 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
272 W103|MISSING_MGMT_FOR_NAME: 64
273 W105|TOO_MANY_PUBLIC_NAMES: 3
274RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
275 -- Copying automatically generated zone files under target tree
276 -- Copying repo-driven real config files and admin_state
277 -- Copying puppetized config and GeoIP from /etc/gdnsd
278 -- Checking for illegal tabs in zonefiles
279 -- Running zone_validator to check WMF rules
280 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.mc0iy3_c
281Traceback (most recent call last):
282 File "utils/deploy-check.py", line 276, in <module>
283 main()
284 File "utils/deploy-check.py", line 268, in main
285 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
286 File "utils/deploy-check.py", line 214, in deploy_check
287 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
288 File "utils/deploy-check.py", line 88, in safe_cmd
289 p_err.decode('utf-8')))
290Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.mc0iy3_c checkconf failed with exit code 42, stderr:
291info: gdnsd version 3.8.0 @ pid 31160
292info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.154.238:53
293info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.153.231:53
294info: DNS listener threads (16 UDP + 16 TCP) configured for 91.198.174.239:53
295info: DNS listener threads (16 UDP + 16 TCP) configured for 198.35.27.27:53
296info: DNS listener threads (16 TCP PROXY) configured for 127.0.0.1:535
297info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
298info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
299info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.mc0iy3_c/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
300info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
301info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
302info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.mc0iy3_c/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
303info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
304info: admin_state: checking state file '/tmp/dns-check.mc0iy3_c/state/admin_state'...
305error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
306info: Zone wikibook.com. with serial 2019120223 loaded
307info: Zone wiktionary.eu. with serial 2019120223 loaded
308info: Zone wikidisclosure.com. with serial 2019120223 loaded
309info: Zone 0.8.c.e.2.0.a.2.ip6.arpa. with serial 2020100116 loaded
310info: Zone en-wp.org. with serial 2019120223 loaded
311info: Zone wikipedia.uk. with serial 2019120223 loaded
312info: Zone xn--b1aajamacm1dkmb.xn--p1ai. with serial 2019120223 loaded
313info: Zone wikiepdia.com. with serial 2019120223 loaded
314info: Zone wikivoyage.com. with serial 2019120223 loaded
315info: Zone wikibooks.org. with serial 2019120223 loaded
316info: Zone wikidpedia.org. with serial 2019120223 loaded
317info: Zone wikipedia.org.il. with serial 2019120223 loaded
318info: Zone wikipedia.net. with serial 2019120223 loaded
319info: Zone wikimediacommons.org. with serial 2019120223 loaded
320info: Zone wikimedia-dns.net. with serial 2021051815 loaded
321info: Zone wikiversity.com. with serial 2019120223 loaded
322info: Zone wikipedia.lol. with serial 2019120223 loaded
323info: Zone wikivoyage-old.org. with serial 2019120223 loaded
324info: Zone wikivoyage.eu. with serial 2019120223 loaded
325info: Zone wikispecies.com. with serial 2019120223 loaded
326info: Zone wikimediafoundation.net. with serial 2019120223 loaded
327info: Zone vikipedia.com.tr. with serial 2019120223 loaded
328info: Zone wikimedia-dns.com. with serial 2021051815 loaded
329info: Zone wikiversity.pt. with serial 2019120223 loaded
330info: Zone wikidata.es. with serial 2019120223 loaded
331info: Zone wmfusercontent.org. with serial 2020072316 loaded
332info: Zone wikidata.us. with serial 2019120223 loaded
333info: Zone wikisource.org. with serial 2019120223 loaded
334info: Zone wikipedia.lt. with serial 2019120223 loaded
335info: Zone wikipedia.bg. with serial 2019120223 loaded
336info: Zone wwwwikimedia.com. with serial 2019120223 loaded
337info: Zone wikipedia.es. with serial 2019120223 loaded
338info: Zone 57.15.185.in-addr.arpa. with serial 2020102310 loaded
339info: Zone wikimedia-dns.org. with serial 2021083014 loaded
340info: Zone en-wp.com. with serial 2019120223 loaded
341info: Zone wikipedoa.org. with serial 2019120223 loaded
342info: Zone 152.80.208.in-addr.arpa. with serial 2021072114 loaded
343info: Zone wiktionary.org. with serial 2019120223 loaded
344info: Zone wikidata.org. with serial 2021031301 loaded
345info: Zone wikjpedia.org. with serial 2019120223 loaded
346info: Zone 58.15.185.in-addr.arpa. with serial 2019120223 loaded
347info: Zone wikpedia.org. with serial 2019120223 loaded
348info: Zone 174.198.91.in-addr.arpa. with serial 2020100816 loaded
349info: Zone wikiipedia.org. with serial 2019120223 loaded
350info: Zone wiktionary.pl. with serial 2019120223 loaded
351info: Zone wikimediastories.net. with serial 2019120223 loaded
352info: Zone 3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. with serial 2021020109 loaded
353info: Zone wikipedia.info. with serial 2019120223 loaded
354info: Zone wikiquote.com. with serial 2019120223 loaded
355info: Zone wikimediafoundation.org. with serial 2020071716 loaded
356info: Zone wikitrends.org. with serial 2019120223 loaded
357info: Zone wilipedia.com. with serial 2019120223 loaded
358info: Zone wikimediacommons.net. with serial 2019120223 loaded
359info: Zone wikimediacommons.info. with serial 2019120223 loaded
360info: Zone wikipedia.in. with serial 2019120223 loaded
361info: Zone wikipediastats.org. with serial 2019120223 loaded
362info: Zone wikiartpedia.co. with serial 2019120223 loaded
363info: Zone wiktionary.pt. with serial 2019120223 loaded
364info: Zone wikiartpedia.mobi. with serial 2019120223 loaded
365info: Zone wikiquote.pl. with serial 2019120223 loaded
366info: Zone wikimaps.org. with serial 2019120223 loaded
367info: Zone wikipedial.org. with serial 2019120223 loaded
368info: Zone wikipediya.com. with serial 2019120223 loaded
369info: Zone wikispecies.net. with serial 2019120223 loaded
370info: Zone wikisource.pl. with serial 2019120223 loaded
371info: Zone indianwikimedia.com. with serial 2019120223 loaded
372info: Zone wikipedia.ee. with serial 2019120223 loaded
373info: Zone wikispecies.org. with serial 2019120223 loaded
374info: Zone wikiworkshop.org. with serial 2020022717 loaded
375info: Zone wikimediacloud.org. with serial 2021050615 loaded
376info: Zone wikkipedia.com. with serial 2019120223 loaded
377info: Zone wikinews.pt. with serial 2019120223 loaded
378info: Zone wikimediafoundation.info. with serial 2019120223 loaded
379info: Zone wikidisclosure.org. with serial 2019120223 loaded
380info: Zone vikipedia.com. with serial 2019120223 loaded
381info: Zone wikimaps.net. with serial 2019120223 loaded
382info: Zone indiawikipedia.com. with serial 2019120223 loaded
383info: Zone softwarewikipedia.net. with serial 2019120223 loaded
384info: Zone wikipdia.org. with serial 2019120223 loaded
385info: Zone donatetowikipedia.org. with serial 2019120223 loaded
386info: Zone webhostingwikipedia.com. with serial 2019120223 loaded
387info: Zone xn--wikirda-p0a62emr.pl. with serial 2020021417 loaded
388info: Zone wikepedia.org. with serial 2019120223 loaded
389info: Zone wikimedia.community. with serial 2020010209 loaded
390info: Zone wikimediacommons.co.uk. with serial 2019120223 loaded
391info: Zone wikimedia.com.pt. with serial 2019120223 loaded
392info: Zone wiikipedia.com. with serial 2019120223 loaded
393info: Zone wikipedia.com. with serial 2019120223 loaded
394info: Zone wikimedia.jp.net. with serial 2019120223 loaded
395info: Zone 59.15.185.in-addr.arpa. with serial 2020100116 loaded
396info: Zone wiki-pedia.org. with serial 2019120223 loaded
397info: Zone wekipedia.com. with serial 2019120223 loaded
398info: Zone wikinews.org. with serial 2019120223 loaded
399info: Zone wikivoyager.org. with serial 2019120223 loaded
400info: Zone wikipedia.co.za. with serial 2019120223 loaded
401info: Zone 0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. with serial 2021020109 loaded
402info: Zone mediawiki.org. with serial 2020043010 loaded
403info: Zone wikimediastories.org. with serial 2019120223 loaded
404info: Zone wmftest.org. with serial 2020011000 loaded
405info: Zone w.wiki. with serial 2019120223 loaded
406info: Zone 26.35.198.in-addr.arpa. with serial 2020100816 loaded
407info: Zone wikibase.org. with serial 2019120223 loaded
408info: Zone wikipedia.is. with serial 2019120223 loaded
409info: Zone wikisource.com. with serial 2019120223 loaded
410fatal: Initial load of zone data failed
411
412===== NODE GROUP =====
413(1) dns5001.wikimedia.org
414----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
415Assembling and testing data in /tmp/dns-check.gt1966ca
416 -- Generating zonefiles from zone templates
417 -- Processed 209 zones into directory /tmp/dns-check.gt1966ca/zones
418OK: No tabs
419Summary of violations:
420 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
421 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
422 W103|MISSING_MGMT_FOR_NAME: 64
423 W105|TOO_MANY_PUBLIC_NAMES: 3
424RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
425 -- Copying automatically generated zone files under target tree
426 -- Copying repo-driven real config files and admin_state
427 -- Copying puppetized config and GeoIP from /etc/gdnsd
428 -- Checking for illegal tabs in zonefiles
429 -- Running zone_validator to check WMF rules
430 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.gt1966ca
431Traceback (most recent call last):
432 File "utils/deploy-check.py", line 276, in <module>
433 main()
434 File "utils/deploy-check.py", line 268, in main
435 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
436 File "utils/deploy-check.py", line 214, in deploy_check
437 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
438 File "utils/deploy-check.py", line 88, in safe_cmd
439 p_err.decode('utf-8')))
440Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.gt1966ca checkconf failed with exit code 42, stderr:
441info: gdnsd version 3.8.0 @ pid 25464
442info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.154.238:53
443info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.153.231:53
444info: DNS listener threads (16 UDP + 16 TCP) configured for 91.198.174.239:53
445info: DNS listener threads (16 UDP + 16 TCP) configured for 198.35.27.27:53
446info: DNS listener threads (16 TCP PROXY) configured for 127.0.0.1:535
447info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
448info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
449info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.gt1966ca/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
450info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
451info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
452info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.gt1966ca/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
453info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
454info: admin_state: checking state file '/tmp/dns-check.gt1966ca/state/admin_state'...
455error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
456fatal: Initial load of zone data failed
457
458===== NODE GROUP =====
459(1) dns3001.wikimedia.org
460----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
461Assembling and testing data in /tmp/dns-check.trnfdvit
462 -- Generating zonefiles from zone templates
463 -- Processed 209 zones into directory /tmp/dns-check.trnfdvit/zones
464OK: No tabs
465Summary of violations:
466 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
467 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
468 W103|MISSING_MGMT_FOR_NAME: 64
469 W105|TOO_MANY_PUBLIC_NAMES: 3
470RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
471 -- Copying automatically generated zone files under target tree
472 -- Copying repo-driven real config files and admin_state
473 -- Copying puppetized config and GeoIP from /etc/gdnsd
474 -- Checking for illegal tabs in zonefiles
475 -- Running zone_validator to check WMF rules
476 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.trnfdvit
477Traceback (most recent call last):
478 File "utils/deploy-check.py", line 276, in <module>
479 main()
480 File "utils/deploy-check.py", line 268, in main
481 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
482 File "utils/deploy-check.py", line 214, in deploy_check
483 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
484 File "utils/deploy-check.py", line 88, in safe_cmd
485 p_err.decode('utf-8')))
486Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.trnfdvit checkconf failed with exit code 42, stderr:
487info: gdnsd version 3.8.0 @ pid 29502
488info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.154.238:53
489info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.153.231:53
490info: DNS listener threads (16 UDP + 16 TCP) configured for 91.198.174.239:53
491info: DNS listener threads (16 UDP + 16 TCP) configured for 198.35.27.27:53
492info: DNS listener threads (16 TCP PROXY) configured for 127.0.0.1:535
493info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
494info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
495info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.trnfdvit/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
496info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
497info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
498info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.trnfdvit/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
499info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
500info: admin_state: checking state file '/tmp/dns-check.trnfdvit/state/admin_state'...
501error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
502fatal: Initial load of zone data failed
503
504===== NODE GROUP =====
505(1) dns3002.wikimedia.org
506----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
507Assembling and testing data in /tmp/dns-check.stkawh1x
508 -- Generating zonefiles from zone templates
509 -- Processed 209 zones into directory /tmp/dns-check.stkawh1x/zones
510OK: No tabs
511Summary of violations:
512 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
513 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
514 W103|MISSING_MGMT_FOR_NAME: 64
515 W105|TOO_MANY_PUBLIC_NAMES: 3
516RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
517 -- Copying automatically generated zone files under target tree
518 -- Copying repo-driven real config files and admin_state
519 -- Copying puppetized config and GeoIP from /etc/gdnsd
520 -- Checking for illegal tabs in zonefiles
521 -- Running zone_validator to check WMF rules
522 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.stkawh1x
523Traceback (most recent call last):
524 File "utils/deploy-check.py", line 276, in <module>
525 main()
526 File "utils/deploy-check.py", line 268, in main
527 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
528 File "utils/deploy-check.py", line 214, in deploy_check
529 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
530 File "utils/deploy-check.py", line 88, in safe_cmd
531 p_err.decode('utf-8')))
532Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.stkawh1x checkconf failed with exit code 42, stderr:
533info: gdnsd version 3.8.0 @ pid 14851
534info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.154.238:53
535info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.153.231:53
536info: DNS listener threads (16 UDP + 16 TCP) configured for 91.198.174.239:53
537info: DNS listener threads (16 UDP + 16 TCP) configured for 198.35.27.27:53
538info: DNS listener threads (16 TCP PROXY) configured for 127.0.0.1:535
539info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
540info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
541info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.stkawh1x/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
542info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
543info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
544info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.stkawh1x/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
545info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
546info: admin_state: checking state file '/tmp/dns-check.stkawh1x/state/admin_state'...
547error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
548fatal: Initial load of zone data failed
549
550===== NODE GROUP =====
551(1) dns4002.wikimedia.org
552----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
553Assembling and testing data in /tmp/dns-check.e980frfq
554 -- Generating zonefiles from zone templates
555 -- Processed 209 zones into directory /tmp/dns-check.e980frfq/zones
556OK: No tabs
557Summary of violations:
558 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
559 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
560 W103|MISSING_MGMT_FOR_NAME: 64
561 W105|TOO_MANY_PUBLIC_NAMES: 3
562RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
563 -- Copying automatically generated zone files under target tree
564 -- Copying repo-driven real config files and admin_state
565 -- Copying puppetized config and GeoIP from /etc/gdnsd
566 -- Checking for illegal tabs in zonefiles
567 -- Running zone_validator to check WMF rules
568 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.e980frfq
569Traceback (most recent call last):
570 File "utils/deploy-check.py", line 276, in <module>
571 main()
572 File "utils/deploy-check.py", line 268, in main
573 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
574 File "utils/deploy-check.py", line 214, in deploy_check
575 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
576 File "utils/deploy-check.py", line 88, in safe_cmd
577 p_err.decode('utf-8')))
578Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.e980frfq checkconf failed with exit code 42, stderr:
579info: gdnsd version 3.8.0 @ pid 26744
580info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.154.238:53
581info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.153.231:53
582info: DNS listener threads (16 UDP + 16 TCP) configured for 91.198.174.239:53
583info: DNS listener threads (16 UDP + 16 TCP) configured for 198.35.27.27:53
584info: DNS listener threads (16 TCP PROXY) configured for 127.0.0.1:535
585info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
586info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
587info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.e980frfq/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
588info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
589info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
590info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.e980frfq/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
591info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
592info: admin_state: checking state file '/tmp/dns-check.e980frfq/state/admin_state'...
593error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
594info: Zone wikipedia.fi. with serial 2019120223 loaded
595info: Zone wikiipedia.org. with serial 2019120223 loaded
596info: Zone wikipedia.uk. with serial 2019120223 loaded
597info: Zone wicipedia.com. with serial 2019120223 loaded
598info: Zone wikimediastories.com. with serial 2019120223 loaded
599info: Zone 10.in-addr.arpa. with serial 2021082316 loaded
600info: Zone 152.80.208.in-addr.arpa. with serial 2021072114 loaded
601info: Zone wikipedia.id. with serial 2019120223 loaded
602info: Zone wikimedipedia.com. with serial 2019120223 loaded
603info: Zone visualwikipedia.com. with serial 2019120223 loaded
604info: Zone wikipdia.org. with serial 2019120223 loaded
605info: Zone mediawiki.com. with serial 2019120223 loaded
606info: Zone wikipedia.com. with serial 2019120223 loaded
607info: Zone 153.80.208.in-addr.arpa. with serial 2021051720 loaded
608info: Zone wikiquote.org. with serial 2019120223 loaded
609info: Zone vikipedia.com.tr. with serial 2019120223 loaded
610info: Zone wilkipedia.org. with serial 2019120223 loaded
611info: Zone wmfusercontent.org. with serial 2020072316 loaded
612info: Zone wikimedia.xyz. with serial 2019120223 loaded
613info: Zone wikipedia.is. with serial 2019120223 loaded
614info: Zone wikipedia.es. with serial 2019120223 loaded
615info: Zone 138.71.185.in-addr.arpa. with serial 2021083014 loaded
616info: Zone wikipediazero.org. with serial 2019120223 loaded
617info: Zone wikiquote.com. with serial 2019120223 loaded
618info: Zone wikimedia-dns.org. with serial 2021083014 loaded
619info: Zone wikipedoa.org. with serial 2019120223 loaded
620info: Zone webhostingwikipedia.com. with serial 2019120223 loaded
621info: Zone visualwikipedia.net. with serial 2019120223 loaded
622info: Zone wikimaps.com. with serial 2019120223 loaded
623info: Zone wikidata.org. with serial 2021031301 loaded
624info: Zone 0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. with serial 2021020109 loaded
625info: Zone wikipediafoundation.org. with serial 2019120223 loaded
626info: Zone parking. with serial 2019120223 loaded
627info: Zone wiki-pedia.org. with serial 2019120223 loaded
628info: Zone wikiworkshop.org. with serial 2020022717 loaded
629info: Zone 1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. with serial 2021020109 loaded
630info: Zone wikimemory.org. with serial 2019120223 loaded
631info: Zone wikivoyage.de. with serial 2019120223 loaded
632info: Zone wikijunior.com. with serial 2019120223 loaded
633info: Zone wikidata.us. with serial 2019120223 loaded
634info: Zone donatetowikipedia.com. with serial 2019120223 loaded
635info: Zone wikimedia.jp.net. with serial 2019120223 loaded
636info: Zone learn.wiki. with serial 2021083109 loaded
637info: Zone xn--b1aajamacm1dkmb.xn--p1ai. with serial 2019120223 loaded
638info: Zone wikiartpedia.mobi. with serial 2019120223 loaded
639info: Zone wikibase.org. with serial 2019120223 loaded
640info: Zone wikispecies.com. with serial 2019120223 loaded
641info: Zone wikipediya.com. with serial 2019120223 loaded
642info: Zone pywikibot.org. with serial 2020111908 loaded
643info: Zone wikiartpedia.info. with serial 2019120223 loaded
644info: Zone wikiartpedia.org. with serial 2019120223 loaded
645info: Zone wekipedia.com. with serial 2019120223 loaded
646info: Zone wikipedia.com.ar. with serial 2019120223 loaded
647info: Zone 174.198.91.in-addr.arpa. with serial 2020100816 loaded
648info: Zone en-wp.com. with serial 2019120223 loaded
649info: Zone vikipedia.com. with serial 2019120223 loaded
650info: Zone wikisource.org. with serial 2019120223 loaded
651info: Zone 57.15.185.in-addr.arpa. with serial 2020102310 loaded
652info: Zone wikidata.es. with serial 2019120223 loaded
653info: Zone wiikipedia.com. with serial 2019120223 loaded
654info: Zone wikkipedia.com. with serial 2019120223 loaded
655info: Zone wikinews.com. with serial 2019120223 loaded
656info: Zone wilipedia.com. with serial 2019120223 loaded
657info: Zone wikepedia.org. with serial 2019120223 loaded
658info: Zone wikimobipedia.net. with serial 2019120223 loaded
659info: Zone wikipedia.org. with serial 2020120918 loaded
660info: Zone wikimediacommons.org. with serial 2019120223 loaded
661info: Zone 2.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. with serial 2020100116 loaded
662info: Zone wikimediacommons.net. with serial 2019120223 loaded
663info: Zone 0.8.c.e.2.0.a.2.ip6.arpa. with serial 2020100116 loaded
664info: Zone wikiversity.com. with serial 2019120223 loaded
665info: Zone mediawiki.org. with serial 2020043010 loaded
666info: Zone wikipediavideo.com. with serial 2019120223 loaded
667info: Zone wikimania.asia. with serial 2019120223 loaded
668info: Zone wikipediastories.com. with serial 2019120223 loaded
669info: Zone wikimediafoundation.net. with serial 2019120223 loaded
670info: Zone wikibooks.org. with serial 2019120223 loaded
671info: Zone border-wikipedia.de. with serial 2019120223 loaded
672info: Zone wikimedia.com.pt. with serial 2019120223 loaded
673info: Zone wikibook.com. with serial 2019120223 loaded
674info: Zone wikipedia.bg. with serial 2019120223 loaded
675info: Zone wwwwikimedia.com. with serial 2019120223 loaded
676info: Zone wikipedia.org.il. with serial 2019120223 loaded
677info: Zone wikimediafoundation.info. with serial 2019120223 loaded
678info: Zone wikispecies.net. with serial 2019120223 loaded
679info: Zone wikimediacommons.co.uk. with serial 2019120223 loaded
680info: Zone wikisource.pt. with serial 2019120223 loaded
681info: Zone wikipedia.info. with serial 2019120223 loaded
682info: Zone wikimaps.org. with serial 2019120223 loaded
683info: Zone wikipediastats.com. with serial 2019120223 loaded
684info: Zone wikimania.org. with serial 2019120223 loaded
685info: Zone wikiepdia.org. with serial 2019120223 loaded
686info: Zone wikiversity.org. with serial 2019120223 loaded
687info: Zone wikipedia.lt. with serial 2019120223 loaded
688info: Zone wikimediafoundation.com. with serial 2019120223 loaded
689info: Zone voyagewiki.org. with serial 2019120223 loaded
690info: Zone wikiartpedia.biz. with serial 2019120223 loaded
691info: Zone wikidata.pt. with serial 2019120223 loaded
692info: Zone wikisource.com. with serial 2019120223 loaded
693info: Zone wikivoyager.org. with serial 2019120223 loaded
694info: Zone xn--wikisownik-e0b.pl. with serial 2020021417 loaded
695info: Zone wikjpedia.org. with serial 2019120223 loaded
696info: Zone 3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. with serial 2021020109 loaded
697info: Zone wikipedia.gr. with serial 2019120223 loaded
698fatal: Initial load of zone data failed
699
700===== NODE GROUP =====
701(1) dns4001.wikimedia.org
702----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
703Assembling and testing data in /tmp/dns-check.7lxfz9ay
704 -- Generating zonefiles from zone templates
705 -- Processed 209 zones into directory /tmp/dns-check.7lxfz9ay/zones
706OK: No tabs
707Summary of violations:
708 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
709 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
710 W103|MISSING_MGMT_FOR_NAME: 64
711 W105|TOO_MANY_PUBLIC_NAMES: 3
712RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
713 -- Copying automatically generated zone files under target tree
714 -- Copying repo-driven real config files and admin_state
715 -- Copying puppetized config and GeoIP from /etc/gdnsd
716 -- Checking for illegal tabs in zonefiles
717 -- Running zone_validator to check WMF rules
718 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.7lxfz9ay
719Traceback (most recent call last):
720 File "utils/deploy-check.py", line 276, in <module>
721 main()
722 File "utils/deploy-check.py", line 268, in main
723 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
724 File "utils/deploy-check.py", line 214, in deploy_check
725 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
726 File "utils/deploy-check.py", line 88, in safe_cmd
727 p_err.decode('utf-8')))
728Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.7lxfz9ay checkconf failed with exit code 42, stderr:
729info: gdnsd version 3.8.0 @ pid 4507
730info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.154.238:53
731info: DNS listener threads (16 UDP + 16 TCP) configured for 208.80.153.231:53
732info: DNS listener threads (16 UDP + 16 TCP) configured for 91.198.174.239:53
733info: DNS listener threads (16 UDP + 16 TCP) configured for 198.35.27.27:53
734info: DNS listener threads (16 TCP PROXY) configured for 127.0.0.1:535
735info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
736info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
737info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.7lxfz9ay/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
738info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
739info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
740info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.7lxfz9ay/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
741info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
742info: admin_state: checking state file '/tmp/dns-check.7lxfz9ay/state/admin_state'...
743error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
744fatal: Initial load of zone data failed
745
746===== NODE GROUP =====
747(1) authdns2001.wikimedia.org
748----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
749Assembling and testing data in /tmp/dns-check.kt4zn6ev
750 -- Generating zonefiles from zone templates
751 -- Processed 209 zones into directory /tmp/dns-check.kt4zn6ev/zones
752OK: No tabs
753Summary of violations:
754 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
755 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
756 W103|MISSING_MGMT_FOR_NAME: 64
757 W105|TOO_MANY_PUBLIC_NAMES: 3
758RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
759 -- Copying automatically generated zone files under target tree
760 -- Copying repo-driven real config files and admin_state
761 -- Copying puppetized config and GeoIP from /etc/gdnsd
762 -- Checking for illegal tabs in zonefiles
763 -- Running zone_validator to check WMF rules
764 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.kt4zn6ev
765Traceback (most recent call last):
766 File "utils/deploy-check.py", line 276, in <module>
767 main()
768 File "utils/deploy-check.py", line 268, in main
769 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
770 File "utils/deploy-check.py", line 214, in deploy_check
771 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
772 File "utils/deploy-check.py", line 88, in safe_cmd
773 p_err.decode('utf-8')))
774Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.kt4zn6ev checkconf failed with exit code 42, stderr:
775info: gdnsd version 3.8.0 @ pid 32720
776info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
777info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
778info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
779info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
780info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
781info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
782info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
783info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.kt4zn6ev/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
784info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
785info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
786info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.kt4zn6ev/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
787info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
788info: admin_state: checking state file '/tmp/dns-check.kt4zn6ev/state/admin_state'...
789error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
790fatal: Initial load of zone data failed
791
792===== NODE GROUP =====
793(1) dns2002.wikimedia.org
794----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
795Assembling and testing data in /tmp/dns-check.wuq5kk2l
796 -- Generating zonefiles from zone templates
797 -- Processed 209 zones into directory /tmp/dns-check.wuq5kk2l/zones
798OK: No tabs
799Summary of violations:
800 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
801 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
802 W103|MISSING_MGMT_FOR_NAME: 64
803 W105|TOO_MANY_PUBLIC_NAMES: 3
804RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
805 -- Copying automatically generated zone files under target tree
806 -- Copying repo-driven real config files and admin_state
807 -- Copying puppetized config and GeoIP from /etc/gdnsd
808 -- Checking for illegal tabs in zonefiles
809 -- Running zone_validator to check WMF rules
810 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.wuq5kk2l
811Traceback (most recent call last):
812 File "utils/deploy-check.py", line 276, in <module>
813 main()
814 File "utils/deploy-check.py", line 268, in main
815 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
816 File "utils/deploy-check.py", line 214, in deploy_check
817 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
818 File "utils/deploy-check.py", line 88, in safe_cmd
819 p_err.decode('utf-8')))
820Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.wuq5kk2l checkconf failed with exit code 42, stderr:
821info: gdnsd version 3.8.0 @ pid 11112
822info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
823info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
824info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
825info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
826info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
827info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
828info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
829info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.wuq5kk2l/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
830info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
831info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
832info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.wuq5kk2l/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
833info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
834info: admin_state: checking state file '/tmp/dns-check.wuq5kk2l/state/admin_state'...
835error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
836fatal: Initial load of zone data failed
837
838===== NODE GROUP =====
839(1) dns2001.wikimedia.org
840----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
841Assembling and testing data in /tmp/dns-check.kwc3e7kn
842 -- Generating zonefiles from zone templates
843 -- Processed 209 zones into directory /tmp/dns-check.kwc3e7kn/zones
844OK: No tabs
845Summary of violations:
846 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
847 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
848 W103|MISSING_MGMT_FOR_NAME: 64
849 W105|TOO_MANY_PUBLIC_NAMES: 3
850RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
851 -- Copying automatically generated zone files under target tree
852 -- Copying repo-driven real config files and admin_state
853 -- Copying puppetized config and GeoIP from /etc/gdnsd
854 -- Checking for illegal tabs in zonefiles
855 -- Running zone_validator to check WMF rules
856 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.kwc3e7kn
857Traceback (most recent call last):
858 File "utils/deploy-check.py", line 276, in <module>
859 main()
860 File "utils/deploy-check.py", line 268, in main
861 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
862 File "utils/deploy-check.py", line 214, in deploy_check
863 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
864 File "utils/deploy-check.py", line 88, in safe_cmd
865 p_err.decode('utf-8')))
866Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.kwc3e7kn checkconf failed with exit code 42, stderr:
867info: gdnsd version 3.8.0 @ pid 7640
868info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
869info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
870info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
871info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
872info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
873info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
874info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
875info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.kwc3e7kn/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
876info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
877info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
878info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.kwc3e7kn/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
879info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
880info: admin_state: checking state file '/tmp/dns-check.kwc3e7kn/state/admin_state'...
881error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
882fatal: Initial load of zone data failed
883
884===== NODE GROUP =====
885(1) authdns1001.wikimedia.org
886----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
887Assembling and testing data in /tmp/dns-check.5zr6cfw8
888 -- Generating zonefiles from zone templates
889 -- Processed 209 zones into directory /tmp/dns-check.5zr6cfw8/zones
890OK: No tabs
891Summary of violations:
892 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
893 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
894 W103|MISSING_MGMT_FOR_NAME: 64
895 W105|TOO_MANY_PUBLIC_NAMES: 3
896RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
897 -- Copying automatically generated zone files under target tree
898 -- Copying repo-driven real config files and admin_state
899 -- Copying puppetized config and GeoIP from /etc/gdnsd
900 -- Checking for illegal tabs in zonefiles
901 -- Running zone_validator to check WMF rules
902 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.5zr6cfw8
903Traceback (most recent call last):
904 File "utils/deploy-check.py", line 276, in <module>
905 main()
906 File "utils/deploy-check.py", line 268, in main
907 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
908 File "utils/deploy-check.py", line 214, in deploy_check
909 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
910 File "utils/deploy-check.py", line 88, in safe_cmd
911 p_err.decode('utf-8')))
912Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.5zr6cfw8 checkconf failed with exit code 42, stderr:
913info: gdnsd version 3.8.0 @ pid 31278
914info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
915info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
916info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
917info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
918info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
919info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
920info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
921info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.5zr6cfw8/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
922info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
923info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
924info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.5zr6cfw8/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
925info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
926info: admin_state: checking state file '/tmp/dns-check.5zr6cfw8/state/admin_state'...
927error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
928fatal: Initial load of zone data failed
929
930===== NODE GROUP =====
931(1) dns1001.wikimedia.org
932----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
933Assembling and testing data in /tmp/dns-check.sbj5ptsn
934 -- Generating zonefiles from zone templates
935 -- Processed 209 zones into directory /tmp/dns-check.sbj5ptsn/zones
936OK: No tabs
937Summary of violations:
938 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
939 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
940 W103|MISSING_MGMT_FOR_NAME: 64
941 W105|TOO_MANY_PUBLIC_NAMES: 3
942RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
943 -- Copying automatically generated zone files under target tree
944 -- Copying repo-driven real config files and admin_state
945 -- Copying puppetized config and GeoIP from /etc/gdnsd
946 -- Checking for illegal tabs in zonefiles
947 -- Running zone_validator to check WMF rules
948 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.sbj5ptsn
949Traceback (most recent call last):
950 File "utils/deploy-check.py", line 276, in <module>
951 main()
952 File "utils/deploy-check.py", line 268, in main
953 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
954 File "utils/deploy-check.py", line 214, in deploy_check
955 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
956 File "utils/deploy-check.py", line 88, in safe_cmd
957 p_err.decode('utf-8')))
958Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.sbj5ptsn checkconf failed with exit code 42, stderr:
959info: gdnsd version 3.8.0 @ pid 22791
960info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
961info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
962info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
963info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
964info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
965info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
966info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
967info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.sbj5ptsn/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
968info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
969info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
970info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.sbj5ptsn/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
971info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
972info: admin_state: checking state file '/tmp/dns-check.sbj5ptsn/state/admin_state'...
973error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
974info: Zone wikipediastats.net. with serial 2019120223 loaded
975info: Zone wikipedia.lol. with serial 2019120223 loaded
976info: Zone wikimemory.org. with serial 2019120223 loaded
977info: Zone wikipediavideo.com. with serial 2019120223 loaded
978info: Zone wilipedia.com. with serial 2019120223 loaded
979info: Zone 27.35.198.in-addr.arpa. with serial 2020051916 loaded
980info: Zone wikidata.pt. with serial 2019120223 loaded
981info: Zone wikipaedia.net. with serial 2019120223 loaded
982info: Zone wikipedia.es. with serial 2019120223 loaded
983info: Zone wikiquote.org. with serial 2019120223 loaded
984info: Zone wikpedia.org. with serial 2019120223 loaded
985info: Zone wikipdia.org. with serial 2019120223 loaded
986info: Zone wikivoyage.com. with serial 2019120223 loaded
987info: Zone xn--80adgfman1aa4l.xn--p1ai. with serial 2019120223 loaded
988info: Zone wikiartpedia.org. with serial 2019120223 loaded
989info: Zone wikinews.de. with serial 2019120223 loaded
990info: Zone wikifunctions.org. with serial 2021060120 loaded
991info: Zone wikimediafoundation.net. with serial 2019120223 loaded
992info: Zone donatetowikipedia.org. with serial 2019120223 loaded
993info: Zone wikimobipedia.net. with serial 2019120223 loaded
994info: Zone wikipedia.com. with serial 2019120223 loaded
995info: Zone wikimedia.is. with serial 2019120223 loaded
996info: Zone wikivoyage.de. with serial 2019120223 loaded
997info: Zone wikimaps.net. with serial 2019120223 loaded
998info: Zone wiktionary.pl. with serial 2019120223 loaded
999info: Zone wikijunior.net. with serial 2019120223 loaded
1000info: Zone wiktionary.com. with serial 2019120223 loaded
1001info: Zone wikimaps.org. with serial 2019120223 loaded
1002info: Zone xn--80adgdym4pbd.xn--j1amh. with serial 2019120223 loaded
1003info: Zone wikimediastories.net. with serial 2019120223 loaded
1004info: Zone voyagewiki.org. with serial 2019120223 loaded
1005info: Zone wikimania.org. with serial 2019120223 loaded
1006info: Zone xn--b1aajamacm1dkmb.xn--p1ai. with serial 2019120223 loaded
1007info: Zone wikinews.org. with serial 2019120223 loaded
1008info: Zone wikimediacommons.info. with serial 2019120223 loaded
1009info: Zone wikiquote.com. with serial 2019120223 loaded
1010info: Zone 58.15.185.in-addr.arpa. with serial 2019120223 loaded
1011info: Zone visualwikipedia.net. with serial 2019120223 loaded
1012info: Zone wikiversity.pt. with serial 2019120223 loaded
1013info: Zone wikimediadns.org. with serial 2021051815 loaded
1014info: Zone wmftest.com. with serial 2020011000 loaded
1015info: Zone wmfusercontent.org. with serial 2020072316 loaded
1016info: Zone wikidisclosure.org. with serial 2019120223 loaded
1017info: Zone wikimedipedia.com. with serial 2019120223 loaded
1018info: Zone 174.198.91.in-addr.arpa. with serial 2020100816 loaded
1019info: Zone wikimediacloud.org. with serial 2021050615 loaded
1020info: Zone wikipedia.bg. with serial 2019120223 loaded
1021info: Zone wikiquote.pl. with serial 2019120223 loaded
1022info: Zone voyagewiki.com. with serial 2019120223 loaded
1023info: Zone wikipedia.ee. with serial 2019120223 loaded
1024info: Zone wikipedia.is. with serial 2019120223 loaded
1025info: Zone wikinews.pt. with serial 2019120223 loaded
1026info: Zone wikijunior.com. with serial 2019120223 loaded
1027info: Zone wikipediastories.com. with serial 2019120223 loaded
1028info: Zone uiquipedia.org. with serial 2019120223 loaded
1029info: Zone wikimediacommons.org. with serial 2019120223 loaded
1030info: Zone wikjpedia.org. with serial 2019120223 loaded
1031info: Zone wikipediazero.org. with serial 2019120223 loaded
1032info: Zone wikisource.org. with serial 2019120223 loaded
1033info: Zone wikipedia.gr. with serial 2019120223 loaded
1034info: Zone wikidisclosure.com. with serial 2019120223 loaded
1035info: Zone wikipediafoundation.org. with serial 2019120223 loaded
1036info: Zone wikimedia.cloud. with serial 2020061109 loaded
1037info: Zone wikipedia.fi. with serial 2019120223 loaded
1038info: Zone wikimedia.us. with serial 2019120223 loaded
1039info: Zone wikimediafoundation.org. with serial 2020071716 loaded
1040info: Zone wikipediastories.net. with serial 2019120223 loaded
1041info: Zone wikidata.es. with serial 2019120223 loaded
1042info: Zone wikivoyage-old.org. with serial 2019120223 loaded
1043info: Zone wikisource.pt. with serial 2019120223 loaded
1044info: Zone wikimedia.jp.net. with serial 2019120223 loaded
1045info: Zone wikimobipedia.com. with serial 2019120223 loaded
1046info: Zone wikimedia.xyz. with serial 2019120223 loaded
1047info: Zone 153.80.208.in-addr.arpa. with serial 2021051720 loaded
1048info: Zone wikitrends.org. with serial 2019120223 loaded
1049info: Zone wikipedia.org.il. with serial 2019120223 loaded
1050info: Zone wikifamily.com. with serial 2019120223 loaded
1051info: Zone wikipedia.info. with serial 2019120223 loaded
1052info: Zone wiki.voyage. with serial 2019120223 loaded
1053info: Zone wikiipedia.org. with serial 2019120223 loaded
1054info: Zone wikimediafoundation.info. with serial 2019120223 loaded
1055info: Zone wikiquotes.info. with serial 2019120223 loaded
1056info: Zone wikipediya.com. with serial 2019120223 loaded
1057info: Zone wkipedia.org. with serial 2019120223 loaded
1058info: Zone en-wp.org. with serial 2019120223 loaded
1059info: Zone wikifamily.org. with serial 2019120223 loaded
1060info: Zone wicipedia.com. with serial 2019120223 loaded
1061info: Zone wikiartpedia.co. with serial 2019120223 loaded
1062info: Zone wikipeedia.org. with serial 2019120223 loaded
1063info: Zone wikiartpedia.info. with serial 2019120223 loaded
1064info: Zone wekipedia.com. with serial 2019120223 loaded
1065info: Zone xn--wikirda-p0a62emr.pl. with serial 2020021417 loaded
1066info: Zone 10.in-addr.arpa. with serial 2021082316 loaded
1067info: Zone wwwwikimedia.com. with serial 2019120223 loaded
1068info: Zone wikimemory.net. with serial 2019120223 loaded
1069info: Zone 2.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. with serial 2020100116 loaded
1070info: Zone 166.102.103.in-addr.arpa. with serial 2020100816 loaded
1071info: Zone 57.15.185.in-addr.arpa. with serial 2020102310 loaded
1072info: Zone donatetowikipedia.com. with serial 2019120223 loaded
1073info: Zone 59.15.185.in-addr.arpa. with serial 2020100116 loaded
1074info: Zone wikimediacommons.co.uk. with serial 2019120223 loaded
1075info: Zone vikipedia.com. with serial 2019120223 loaded
1076info: Zone w.wiki. with serial 2019120223 loaded
1077info: Zone wikimedia-dns.net. with serial 2021051815 loaded
1078fatal: Initial load of zone data failed
1079
1080===== NODE GROUP =====
1081(1) dns1002.wikimedia.org
1082----- OUTPUT of 'cd /srv/authdns/...nippets --deploy' -----
1083Assembling and testing data in /tmp/dns-check.u091n0za
1084 -- Generating zonefiles from zone templates
1085 -- Processed 209 zones into directory /tmp/dns-check.u091n0za/zones
1086OK: No tabs
1087Summary of violations:
1088 W001|MISSING_IP_FOR_NAME_AND_PTR: 37
1089 W002|MISSING_PTR_FOR_NAME_AND_IP: 26
1090 W103|MISSING_MGMT_FOR_NAME: 64
1091 W105|TOO_MANY_PUBLIC_NAMES: 3
1092RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
1093 -- Copying automatically generated zone files under target tree
1094 -- Copying repo-driven real config files and admin_state
1095 -- Copying puppetized config and GeoIP from /etc/gdnsd
1096 -- Checking for illegal tabs in zonefiles
1097 -- Running zone_validator to check WMF rules
1098 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.u091n0za
1099Traceback (most recent call last):
1100 File "utils/deploy-check.py", line 276, in <module>
1101 main()
1102 File "utils/deploy-check.py", line 268, in main
1103 deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), gdir)
1104 File "utils/deploy-check.py", line 214, in deploy_check
1105 safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
1106 File "utils/deploy-check.py", line 88, in safe_cmd
1107 p_err.decode('utf-8')))
1108Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.u091n0za checkconf failed with exit code 42, stderr:
1109info: gdnsd version 3.8.0 @ pid 15511
1110info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
1111info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
1112info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
1113info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
1114info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
1115info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
1116info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
1117info: plugin_geoip: map 'generic-map': Loading GeoIP2 database '/tmp/dns-check.u091n0za/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
1118info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1174886 dclists: 8
1119info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 dclists: 2
1120info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database '/tmp/dns-check.u091n0za/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-02 13:16:49 UTC
1121info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 dclists: 2
1122info: admin_state: checking state file '/tmp/dns-check.u091n0za/state/admin_state'...
1123error: CNAME 'pc1-master.eqiad.wmnet.' points to known same-zone NXDOMAIN 'pc2007.codfw.wmnet.'
1124fatal: Initial load of zone data failed
1125
1126================
1127PASS | | 0% (0/12) [00:36<?, ?hosts/s]
1128FAIL |█████████████████████████████████████████████████████████████████████████████| 100% (12/12) [00:36<00:00, 1.66s/hosts]
1129100.0% (12/12) of nodes failed to execute command 'cd /srv/authdns/...nippets --deploy': authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
11300.0% (0/12) success ratio (< 100.0% threshold) for command: 'cd /srv/authdns/...nippets --deploy'. Aborting.
11310.0% (0/12) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
1132Failed to run the sre.dns.netbox cookbook
1133Traceback (most recent call last):
1134 File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 409, in run
1135 dns_netbox_run(dns_netbox_args, self.spicerack)
1136 File "/srv/deployment/spicerack/cookbooks/sre/dns/netbox.py", line 129, in run
1137 git=AUTHDNS_DNS_CHECKOUT_PATH, netbox=AUTHDNS_NETBOX_CHECKOUT_PATH))
1138 File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 477, in run_sync
1139 is_safe=is_safe,
1140 File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 668, in _execute
1141 raise RemoteExecutionError(ret, "Cumin execution failed")
1142spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
1143**Failed to run the sre.dns.netbox cookbook**: Cumin execution failed (exit_code=2)
1144Running Homer on asw-a-codfw.mgmt.codfw.wmnet, it takes time ⏳ don't worry
1145INFO:homer.devices:Initialized 39 devices
1146INFO:homer:Committing config for query asw-a-codfw.mgmt.codfw.wmnet with message: Host decommission - marostegui@cumin1001 - T289112
1147INFO:homer:Gathering global Netbox data
1148INFO:homer.devices:Matched 1 device(s) for query 'asw-a-codfw.mgmt.codfw.wmnet'
1149INFO:homer:Generating configuration for asw-a-codfw.mgmt.codfw.wmnet
1150Configuration diff for asw-a-codfw.mgmt.codfw.wmnet:
1151
1152[edit interfaces interface-range disabled]
1153+ member ge-1/0/1;
1154 member ge-1/0/5 { ... }
1155[edit interfaces interface-range vlan-private1-a-codfw]
1156- member ge-1/0/1;
1157[edit interfaces]
1158- ge-1/0/1 {
1159- description pc2007;
1160- }
1161
1162Type "yes" to commit, "no" to abort.
1163> yes
1164INFO:homer.transports.junos:Committing the configuration on asw-a-codfw.mgmt.codfw.wmnet
1165INFO:homer:Homer run completed successfully on 1 devices: ['asw-a-codfw.mgmt.codfw.wmnet']
1166ERROR: some step failed, check the task updates.
1167Updated Phabricator task T289112
1168END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts pc2007.codfw.wmnet
1169[05:30:29] marostegui@cumin1001:~$
1170PASS | | 0% (0/12) [00:36<?, ?hosts/s]
1171FAIL |█████████████████████████████████████████████████████████████████████████████| 100% (12/12) [00:36<00:00, 1.66s/hosts]
1172100.0% (12/12) of nodes failed to execute command 'cd /srv/authdns/...nippets --deploy': authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
11730.0% (0/12) success ratio (< 100.0% threshold) for command: 'cd /srv/authdns/...nippets --deploy'. Aborting.
11740.0% (0/12) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
1175Failed to run the sre.dns.netbox cookbook
1176Traceback (most recent call last):
1177 File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 409, in run
1178 dns_netbox_run(dns_netbox_args, self.spicerack)
1179 File "/srv/deployment/spicerack/cookbooks/sre/dns/netbox.py", line 129, in run
1180 git=AUTHDNS_DNS_CHECKOUT_PATH, netbox=AUTHDNS_NETBOX_CHECKOUT_PATH))
1181 File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 477, in run_sync
1182 is_safe=is_safe,
1183 File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 668, in _execute
1184 raise RemoteExecutionError(ret, "Cumin execution failed")
1185spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
1186**Failed to run the sre.dns.netbox cookbook**: Cumin execution failed (exit_code=2)
1187Running Homer on asw-a-codfw.mgmt.codfw.wmnet, it takes time ⏳ don't worry
1188INFO:homer.devices:Initialized 39 devices
1189INFO:homer:Committing config for query asw-a-codfw.mgmt.codfw.wmnet with message: Host decommission - marostegui@cumin1001 - T289112
1190INFO:homer:Gathering global Netbox data
1191INFO:homer.devices:Matched 1 device(s) for query 'asw-a-codfw.mgmt.codfw.wmnet'
1192INFO:homer:Generating configuration for asw-a-codfw.mgmt.codfw.wmnet
1193Configuration diff for asw-a-codfw.mgmt.codfw.wmnet:
1194
1195[edit interfaces interface-range disabled]
1196+ member ge-1/0/1;
1197 member ge-1/0/5 { ... }
1198[edit interfaces interface-range vlan-private1-a-codfw]
1199- member ge-1/0/1;
1200[edit interfaces]
1201- ge-1/0/1 {
1202- description pc2007;
1203- }
1204
1205Type "yes" to commit, "no" to abort.
1206> yes
1207INFO:homer.transports.junos:Committing the configuration on asw-a-codfw.mgmt.codfw.wmnet
1208INFO:homer:Homer run completed successfully on 1 devices: ['asw-a-codfw.mgmt.codfw.wmnet']
1209ERROR: some step failed, check the task updates.
1210Updated Phabricator task T289112
1211END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts pc2007.codfw.wmnet

Netbox log does show the host's IP being removed:

Deleted IP Address 2620:0:860:101:10:192:0:104/64
sre_bot - 2021-09-03 05:21
Deleted IP Address 10.192.0.104/22
sre_bot - 2021-09-03 05:21
Modified Interface ge-1/0/1
sre_bot - 2021-09-03 05:21
Modified Device pc2007
sre_bot - 2021-09-03 05:21

If you could let us know what to do next, it would be appreciated!

• Marostegui updated the task description. (Show Details)Sep 3 2021, 7:04 AM

This is ready for DC-Ops

• Marostegui reassigned this task from • Marostegui to wiki_willy.Sep 3 2021, 7:04 AM
• Marostegui reassigned this task from wiki_willy to Papaul.
• Marostegui edited projects, added ops-eqiad, DC-Ops; removed Patch-For-Review, DBA.
• Marostegui edited projects, added ops-codfw; removed ops-eqiad.
• Marostegui added a subscriber: wiki_willy.
Maintenance_bot added a project: SRE.Sep 3 2021, 7:45 AM
Papaul updated the task description. (Show Details)Sep 3 2021, 2:38 PM
Papaul closed this task as Resolved.Sep 3 2021, 3:14 PM

Complete

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL