tools-redis-1003 ended up with a duplicate record
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Bstorm
	May 15 2020, 3:40 PM

Description

The celery worker of refill-api was failing partly because it is a consumer of tools-redis.

On investigation, it appears to be caused by some kind of DNS issue.

bstorm@tools-sgebastion-08:~$ nslookup 172.16.1.166
Server:		208.80.154.143
Address:	208.80.154.143#53

Non-authoritative answer:
166.1.16.172.in-addr.arpa	name = tools-redis-1003.tools.eqiad1.wikimedia.cloud.
166.1.16.172.in-addr.arpa	name = canary1027-01.cloudvirt-canary.eqiad1.wikimedia.cloud

I believe tools-redis-1003 should just be 172.16.1.107
bstorm@canary1027-01:~$ nslookup tools-redis-1003.tools.eqiad1.wikimedia.cloud
Server: 208.80.154.143
Address: 208.80.154.143#53

Non-authoritative answer:
Name: tools-redis-1003.tools.eqiad1.wikimedia.cloud
Address: 172.16.1.166
Name: tools-redis-1003.tools.eqiad1.wikimedia.cloud
Address: 172.16.1.107

Event Timeline

• Bstorm triaged this task as Unbreak Now! priority.May 15 2020, 3:40 PM

• Bstorm created this task.

Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptMay 15 2020, 3:40 PM

• Bstorm updated the task description. (Show Details)May 15 2020, 3:41 PM

• Bstorm updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-cloud) [2020-05-15T15:47:10Z] <bd808> Manually running wmcs-novastats-dnsleaks from cloudcontrol1003 (T252889)

$ OS_PROJECT_ID=cloudinfra openstack recordset show eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud.
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| action      | NONE                                           |
| created_at  | 2020-04-03T18:07:22.000000                     |
| description | None                                           |
| id          | 82f97423-595d-4bcd-90fa-81549b44a1c3           |
| name        | tools-redis-1003.tools.eqiad1.wikimedia.cloud. |
| project_id  | cloudinfra                                     |
| records     | 172.16.1.166                                   |
|             | 172.16.1.107                                   |
| status      | ACTIVE                                         |
| ttl         | None                                           |
| type        | A                                              |
| updated_at  | None                                           |
| version     | 1                                              |
| zone_id     | 67603ef4-3d64-40d6-90d3-5b7776a99034           |
| zone_name   | eqiad1.wikimedia.cloud.                        |
+-------------+------------------------------------------------+
$ OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud. --record 172.16.1.107
Managed records may not be updated

Ran:

root@cloudcontrol1003:~# OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud. --record 172.16.1.107 --edit-managed
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| action      | UPDATE                                         |
| created_at  | 2020-04-03T18:07:22.000000                     |
| description | None                                           |
| id          | 82f97423-595d-4bcd-90fa-81549b44a1c3           |
| name        | tools-redis-1003.tools.eqiad1.wikimedia.cloud. |
| project_id  | cloudinfra                                     |
| records     | 172.16.1.107                                   |
| status      | PENDING                                        |
| ttl         | None                                           |
| type        | A                                              |
| updated_at  | 2020-05-15T16:26:06.000000                     |
| version     | 2                                              |
| zone_id     | 67603ef4-3d64-40d6-90d3-5b7776a99034           |
| zone_name   | eqiad1.wikimedia.cloud.                        |
+-------------+------------------------------------------------+

tools-redis is working again

tools-redis-1003 ended up with a duplicate recordClosed, ResolvedPublicActions

Description

Event Timeline

tools-redis-1003 ended up with a duplicate record
Closed, ResolvedPublic
Actions