Page MenuHomePhabricator

tools-redis-1003 ended up with a duplicate record
Closed, ResolvedPublic

Description

The celery worker of refill-api was failing partly because it is a consumer of tools-redis.

On investigation, it appears to be caused by some kind of DNS issue.

bstorm@tools-sgebastion-08:~$ nslookup 172.16.1.166
Server:		208.80.154.143
Address:	208.80.154.143#53

Non-authoritative answer:
166.1.16.172.in-addr.arpa	name = tools-redis-1003.tools.eqiad1.wikimedia.cloud.
166.1.16.172.in-addr.arpa	name = canary1027-01.cloudvirt-canary.eqiad1.wikimedia.cloud

I believe tools-redis-1003 should just be 172.16.1.107
bstorm@canary1027-01:~$ nslookup tools-redis-1003.tools.eqiad1.wikimedia.cloud
Server: 208.80.154.143
Address: 208.80.154.143#53

Non-authoritative answer:
Name: tools-redis-1003.tools.eqiad1.wikimedia.cloud
Address: 172.16.1.166
Name: tools-redis-1003.tools.eqiad1.wikimedia.cloud
Address: 172.16.1.107

Event Timeline

Bstorm triaged this task as Unbreak Now! priority.Fri, May 15, 3:40 PM
Bstorm created this task.
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptFri, May 15, 3:40 PM
Bstorm updated the task description. (Show Details)Fri, May 15, 3:41 PM
Bstorm updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-cloud) [2020-05-15T15:47:10Z] <bd808> Manually running wmcs-novastats-dnsleaks from cloudcontrol1003 (T252889)

bd808 added a subscriber: bd808.Fri, May 15, 3:59 PM
$ OS_PROJECT_ID=cloudinfra openstack recordset show eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud.
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| action      | NONE                                           |
| created_at  | 2020-04-03T18:07:22.000000                     |
| description | None                                           |
| id          | 82f97423-595d-4bcd-90fa-81549b44a1c3           |
| name        | tools-redis-1003.tools.eqiad1.wikimedia.cloud. |
| project_id  | cloudinfra                                     |
| records     | 172.16.1.166                                   |
|             | 172.16.1.107                                   |
| status      | ACTIVE                                         |
| ttl         | None                                           |
| type        | A                                              |
| updated_at  | None                                           |
| version     | 1                                              |
| zone_id     | 67603ef4-3d64-40d6-90d3-5b7776a99034           |
| zone_name   | eqiad1.wikimedia.cloud.                        |
+-------------+------------------------------------------------+
$ OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud. --record 172.16.1.107
Managed records may not be updated

Ran:

root@cloudcontrol1003:~# OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud. --record 172.16.1.107 --edit-managed
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| action      | UPDATE                                         |
| created_at  | 2020-04-03T18:07:22.000000                     |
| description | None                                           |
| id          | 82f97423-595d-4bcd-90fa-81549b44a1c3           |
| name        | tools-redis-1003.tools.eqiad1.wikimedia.cloud. |
| project_id  | cloudinfra                                     |
| records     | 172.16.1.107                                   |
| status      | PENDING                                        |
| ttl         | None                                           |
| type        | A                                              |
| updated_at  | 2020-05-15T16:26:06.000000                     |
| version     | 2                                              |
| zone_id     | 67603ef4-3d64-40d6-90d3-5b7776a99034           |
| zone_name   | eqiad1.wikimedia.cloud.                        |
+-------------+------------------------------------------------+
Bstorm closed this task as Resolved.Fri, May 15, 6:14 PM
Bstorm claimed this task.

tools-redis is working again