We are provisioning new Prometheus hardware in parent task, and I'm running into permissions problems with generate-mysqld-exporter-targets:
Jan 25 10:39:34 prometheus2005 systemd[1]: Started generates prometheus-mysqld-exporter targets from zarcillo. Jan 25 10:39:34 prometheus2005 mysqld_exporter_config.py[2546241]: /usr/local/sbin/mysqld_exporter_config.py:48: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is uns> Jan 25 10:39:34 prometheus2005 mysqld_exporter_config.py[2546241]: config = yaml.load(open(DB_CONFIG_FILE)) Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: Traceback (most recent call last): Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/local/sbin/mysqld_exporter_config.py", line 215, in <module> Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: main() Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/local/sbin/mysqld_exporter_config.py", line 205, in main Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: data = get_data(host, port, database, user, password, dc) Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/local/sbin/mysqld_exporter_config.py", line 66, in get_data Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: db = pymysql.connect(host=host, port=port, database=database, Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: return Connection(*args, **kwargs) Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__ Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: self.connect() Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 599, in connect Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: self._request_authentication() Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 861, in _request_authentication Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: auth_packet = self._read_packet() Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: packet.check_error() Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: err.raise_mysql_exception(self._data) Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: raise errorclass(errno, errval) Jan 25 10:39:35 prometheus2005 mysqld_exporter_config.py[2546241]: pymysql.err.InternalError: (1698, "Access denied for user 'prometheus-mysqld-exporter'@'10.192.16.75'"
For reference the addresses of the new hosts are the following:
prometheus1005.eqiad.wmnet has address 10.64.0.82 prometheus1005.eqiad.wmnet has IPv6 address 2620:0:861:101:10:64:0:82 prometheus2005.codfw.wmnet has address 10.192.16.75 prometheus2005.codfw.wmnet has IPv6 address 2620:0:860:102:10:192:16:75 prometheus1006.eqiad.wmnet has address 10.64.16.62 prometheus1006.eqiad.wmnet has IPv6 address 2620:0:861:102:10:64:16:62 prometheus2006.codfw.wmnet has address 10.192.32.67 prometheus2006.codfw.wmnet has IPv6 address 2620:0:860:103:10:192:32:67
What's the process in this case? Also I take it we'd need to file tasks to remove the old grants once we have decom'd hosts ?
thanks!