Page MenuHomePhabricator
Create Task
Maniphest T424207

maintain-dbusers should gracefully handle missing kubeconfig
Open, In Progress, MediumPublic
Actions

Description

Sometimes maintain-dbusers tries to provision an account before maintain-kubeusers has had the chance to create the Kubernetes namespace and credentials. When that happens, the service on NFS nodes 500s with this:

Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]: [2026-04-23 07:15:37,122] ERROR in app: Exception on /v1/write-replica-cnf [POST]
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]: Traceback (most recent call last):
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 1511, in wsgi_app
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     response = self.full_dispatch_request()
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 919, in full_dispatch_request
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     rv = self.handle_user_exception(e)
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 917, in full_dispatch_request
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     rv = self.dispatch_request()
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 902, in dispatch_request
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:   File "/home/www-data/replica_cnf_api_service/replica_cnf_api_service/views.py", line 220, in write_replica_cnf
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     new_replica_cnf = backend.save_replica_cnf(
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:         replica_cnf=replica_cnf,
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:         account_uid=uid,
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:         dry_run=dry_run,
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     )
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:   File "/home/www-data/replica_cnf_api_service/replica_cnf_api_service/backends/envvars_backend.py", line 86, in save_replica_cnf
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     cli = self._get_user_client(user_name=replica_cnf.user)
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:   File "/home/www-data/replica_cnf_api_service/replica_cnf_api_service/backends/envvars_backend.py", line 61, in _get_user_client
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     raise RuntimeError(
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     ...<3 lines>...
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]:     )
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]: RuntimeError: Unable to get kubeconfig for user 'tools.snowybot':
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]: out:b''
Apr 23 07:15:37 tools-nfs-3 uwsgi-toolsdb-replica-cnf-web[1267]: err:b'Traceback (most recent call last):\n  File "/usr/local/bin/load_user_kubeconfig.py", line 77, in <module>\n    main()\n    ~~~~^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1161, in __call__\n    return self.main(*args, **kwargs)\n           ~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1082, in main\n    rv = self.invoke(ctx)\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1443, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 788, in invoke\n    return __callback(*args, **kwargs)\n  File "/usr/local/bin/load_user_kubeconfig.py", line 53, in main\n    kubeconfig = Kubeconfig.load(path=Path(kubeconfig_path))\n  File "/usr/lib/python3/dist-packages/toolforge_weld/kubernetes_config.py", line 98, in load\n    raise KubernetesConfigFileNotFoundException(str(path.resolve()))\ntoolforge_weld.kubernetes_config.KubernetesConfigFileNotFoundException: /srv/tools/project/snowybot/.kube/config\n'

That, in turn, results in the maintain-dbusers error alert firing. The app should handle this case gracefully instead.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
replica_cnf_api_service: check for missing kubeconfig before _create_envvarRaymond Ndibeoperations/puppetproduction+50 -2
handle missing kubeconfig error in replica_cnf_backendRaymond Ndibeoperations/puppetproduction+75 -2
Customize query in gerrit

Event Timeline

taavi created this task.Apr 23 2026, 10:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 23 2026, 10:38 AM
aputhin triaged this task as Medium priority.Apr 29 2026, 2:24 PM
aputhin moved this task from Backlog to Todos on the tools-platform-team board.
gerritbot added a comment.May 18 2026, 11:45 AM

Change #1288526 had a related patch set uploaded (by Raymond Ndibe; author: Raymond Ndibe):

[operations/puppet@production] handle missing kubeconfig error in replica_cnf_backend

https://gerrit.wikimedia.org/r/1288526

gerritbot added a project: Patch-For-Review.May 18 2026, 11:45 AM
Raymond_Ndibe claimed this task.May 18 2026, 12:37 PM
Raymond_Ndibe moved this task from Todos to In review on the tools-platform-team board.
gerritbot added a comment.May 21 2026, 2:01 PM

Change #1288526 merged by FNegri:

[operations/puppet@production] handle missing kubeconfig error in replica_cnf_backend

https://gerrit.wikimedia.org/r/1288526

Raymond_Ndibe closed this task as Resolved.May 21 2026, 2:14 PM
Raymond_Ndibe moved this task from In review to Done on the tools-platform-team board.
Maintenance_bot removed a project: Patch-For-Review.May 21 2026, 2:31 PM
taavi reopened this task as Open.Jun 8 2026, 3:46 PM

The error has changed to something slightly different, but the issue is still there and causing the maintain-dbusers alerts to flap:

Jun 08 14:36:08 cloudcontrol1007 maintain-dbusers[2313455]: ERROR [root.inner:173] Request to create replica.my.cnf file for account_type tool and account_id tools.icobot failed. Reason: Got errors: Skipping account tools.icobot: Home directory (/srv/tools/project/icobot) does not exist yet ,Unable to get kubeconfig for user 'tools.icobot':
Jun 08 14:36:08 cloudcontrol1007 maintain-dbusers[2313455]: out:b''
Jun 08 14:36:08 cloudcontrol1007 maintain-dbusers[2313455]: err:b'Traceback (most recent call last):\n  File "/usr/local/bin/load_user_kubeconfig.py", line 77, in <module>\n    main()\n    ~~~~^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1161, in __call__\n    return self.main(*args, **kwargs)\n           ~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1082, in main\n    rv = self.invoke(ctx)\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1443, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 788, in invoke\n    return __callback(*args, **kwargs)\n  File "/usr/local/bin/load_user_kubeconfig.py", line 53, in main\n    kubeconfig = Kubeconfig.load(path=Path(kubeconfig_path))\n  File "/usr/lib/python3/dist-packages/toolforge_weld/kubernetes_config.py", line 98, in load\n    raise KubernetesConfigFileNotFoundException(str(path.resolve()))\ntoolforge_weld.kubernetes_config.KubernetesConfigFileNotFoundException: /srv/tools/project/icobot/.kube/config\n'
Jun 08 14:36:08 cloudcontrol1007 maintain-dbusers[2313455]: ERROR [root.populate_accountsdb:811] problem populating new account: Request to create replica.my.cnf file for account_type tool and account_id tools.icobot failed. Reason: Got errors: Skipping account tools.icobot: Home directory (/srv/tools/project/icobot) does not exist yet ,Unable to get kubeconfig for user 'tools.icobot':
Jun 08 14:36:08 cloudcontrol1007 maintain-dbusers[2313455]: out:b''
Jun 08 14:36:08 cloudcontrol1007 maintain-dbusers[2313455]: err:b'Traceback (most recent call last):\n  File "/usr/local/bin/load_user_kubeconfig.py", line 77, in <module>\n    main()\n    ~~~~^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1161, in __call__\n    return self.main(*args, **kwargs)\n           ~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1082, in main\n    rv = self.invoke(ctx)\n  File "/usr/lib/python3/dist-packages/click/core.py", line 1443, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3/dist-packages/click/core.py", line 788, in invoke\n    return __callback(*args, **kwargs)\n  File "/usr/local/bin/load_user_kubeconfig.py", line 53, in main\n    kubeconfig = Kubeconfig.load(path=Path(kubeconfig_path))\n  File "/usr/lib/python3/dist-packages/toolforge_weld/kubernetes_config.py", line 98, in load\n    raise KubernetesConfigFileNotFoundException(str(path.resolve()))\ntoolforge_weld.kubernetes_config.KubernetesConfigFileNotFoundException: /srv/tools/project/icobot/.kube/config\n'
Raymond_Ndibe added a comment.Jun 9 2026, 3:35 PM

looking

Raymond_Ndibe changed the task status from Open to In Progress.Jun 9 2026, 3:35 PM
Raymond_Ndibe moved this task from Done to In progress on the tools-platform-team board.
gerritbot added a comment.Jun 10 2026, 4:30 AM

Change #1299671 had a related patch set uploaded (by Raymond Ndibe; author: Raymond Ndibe):

[operations/puppet@production] replica_cnf_api_service: check for missing kubeconfig before _create_envvar

https://gerrit.wikimedia.org/r/1299671

gerritbot added a project: Patch-For-Review.Jun 10 2026, 4:30 AM
Raymond_Ndibe moved this task from In progress to In review on the tools-platform-team board.Jun 10 2026, 4:46 AM
aputhin moved this task from In review to Todos on the tools-platform-team board.Jun 10 2026, 7:34 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits