From a page.
summary: 10 instances in the admin-monitoring project 4 minutes ago instance: cloudcontrol1003 team: wmcs @cluster: wikimedia.org
Had a quick look as there was a known puppet issue, and wanted to verify, this time it seems to be something related to DNS cleanup:
dcaro@cloudcontrol1003:~$ echo "$(sudo journalctl -n 5000 -u nova-fullstack.service | grep '"log.level": "ERROR"' | tail -n 10 | sed -e 's/^.*@cee: //')" | jq '."error.stack"'
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 748, in main\n vdns = verify_dns_cleanup(host, dnsd, timeout=60.0)\n File \"/usr/local/sbin/nova-fullstack\", line 272, in verify_dns_cleanup\n raise Exception(\"Failed to clean up dns for {}\".format(hostname))\nException: Failed to clean up dns for fullstackd-20211121050502.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 697, in main\n vdns = verify_dns(host, addr, dnsd, timeout=30)\n File \"/usr/local/sbin/nova-fullstack\", line 199, in verify_dns\n \"Timed out waiting for A record for {}\".format(hostname)\nException: Timed out waiting for A record for fullstackd-20211121051528.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 748, in main\n vdns = verify_dns_cleanup(host, dnsd, timeout=60.0)\n File \"/usr/local/sbin/nova-fullstack\", line 272, in verify_dns_cleanup\n raise Exception(\"Failed to clean up dns for {}\".format(hostname))\nException: Failed to clean up dns for fullstackd-20211121052122.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 748, in main\n vdns = verify_dns_cleanup(host, dnsd, timeout=60.0)\n File \"/usr/local/sbin/nova-fullstack\", line 272, in verify_dns_cleanup\n raise Exception(\"Failed to clean up dns for {}\".format(hostname))\nException: Failed to clean up dns for fullstackd-20211121053200.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 697, in main\n vdns = verify_dns(host, addr, dnsd, timeout=30)\n File \"/usr/local/sbin/nova-fullstack\", line 199, in verify_dns\n \"Timed out waiting for A record for {}\".format(hostname)\nException: Timed out waiting for A record for fullstackd-20211121054238.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 697, in main\n vdns = verify_dns(host, addr, dnsd, timeout=30)\n File \"/usr/local/sbin/nova-fullstack\", line 199, in verify_dns\n \"Timed out waiting for A record for {}\".format(hostname)\nException: Timed out waiting for A record for fullstackd-20211121054832.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 697, in main\n vdns = verify_dns(host, addr, dnsd, timeout=30)\n File \"/usr/local/sbin/nova-fullstack\", line 199, in verify_dns\n \"Timed out waiting for A record for {}\".format(hostname)\nException: Timed out waiting for A record for fullstackd-20211121055426.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 748, in main\n vdns = verify_dns_cleanup(host, dnsd, timeout=60.0)\n File \"/usr/local/sbin/nova-fullstack\", line 272, in verify_dns_cleanup\n raise Exception(\"Failed to clean up dns for {}\".format(hostname))\nException: Failed to clean up dns for fullstackd-20211121060020.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 748, in main\n vdns = verify_dns_cleanup(host, dnsd, timeout=60.0)\n File \"/usr/local/sbin/nova-fullstack\", line 272, in verify_dns_cleanup\n raise Exception(\"Failed to clean up dns for {}\".format(hostname))\nException: Failed to clean up dns for fullstackd-20211121061059.admin-monitoring.eqiad1.wikimedia.cloud"
"Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 748, in main\n vdns = verify_dns_cleanup(host, dnsd, timeout=60.0)\n File \"/usr/local/sbin/nova-fullstack\", line 272, in verify_dns_cleanup\n raise Exception(\"Failed to clean up dns for {}\".format(hostname))\nException: Failed to clean up dns for fullstackd-20211121062150.admin-monitoring.eqiad1.wikimedia.cloud"
# Just one of then with full info:
{
"ecs.version": "1.7.0",
"log.level": "ERROR",
"log.origin.file.line": 758,
"log.origin.file.name": "nova-fullstack",
"log.origin.file.path": "/usr/local/sbin/nova-fullstack",
"log.origin.function": "main",
"message": "fullstackd-20211121061059 failed, leaking",
"process.name": "MainProcess",
"process.thread.id": 21839,
"process.thread.name": "MainThread",
"timestamp": "2021-11-21T06:16:50.302484",
"error.stack": "Traceback (most recent call last):\n File \"/usr/local/sbin/nova-fullstack\", line 748, in main\n vdns = verify_dns_cleanup(host, dnsd, timeout=60.0)\n File \"/usr/local/sbin/nova-fullstack\", line 272, in verify_dns_cleanup\n raise Exception(\"Failed to clean up dns for {}\".format(hostname))\nException: Failed to clean up dns for fullstackd-20211121061059.admin-monitoring.eqiad1.wikimedia.cloud"
}Looking