Page MenuHomePhabricator
Create Task
Maniphest T416588

Deleting ENC projects is broken
Closed, ResolvedPublic
Actions

Description

Deleting a Puppet ENC project fails because it tries to resolve the project name for the project ID:

2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc [None req-c331e2dd-7e5a-4ac8-a10c-004eda907d3a novaadmin admin - - default default] Exception on /v1/admin/project/checkuser-beta-wiki [DELETE]: KeyError: 'checkuser-beta-wiki'
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc Traceback (most recent call last):
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 1511, in wsgi_app
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc     response = self.full_dispatch_request()
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 919, in full_dispatch_request
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc     rv = self.handle_user_exception(e)
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 917, in full_dispatch_request
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc     rv = self.dispatch_request()
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 902, in dispatch_request
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc   File "/usr/local/lib/python3.13/dist-packages/puppet-enc.py", line 786, in delete_project
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc     project_name = project_name_for_id(project)
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc   File "/usr/local/lib/python3.13/dist-packages/puppet-enc.py", line 161, in project_name_for_id
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc     return convertDict[project_id]
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc            ~~~~~~~~~~~^^^^^^^^^^^^
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc KeyError: 'checkuser-beta-wiki'
2026-02-05 12:25:34.708 189710 ERROR uwsgi_file__usr_local_lib_python3_13_dist-packages_puppet-enc

Probably we need to update ENC to store the project ID in a new database table so that mapping is available after a project has been deleted from openstack.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
openstack: encapi: Make new project column NOT NULLMajavahoperations/puppetproduction+1 -1
openstack: puppet-enc: Stop writing and drop old project columnMajavahoperations/puppetproduction+6 -9
openstack: puppet-enc: Exclusively read projects tableMajavahoperations/puppetproduction+29 -10
openstack: wmcs-enc-cli: Add subcommand to manually delete projectsMajavahoperations/puppetproduction+18 -0
openstack: encapi: Always use explicit SQL JOIN syntaxMajavahoperations/puppetproduction+19 -16
opesntack: encapi: Fix column nameMajavahoperations/puppetproduction+1 -1
openstack: encapi: Store project names in databaseMajavahoperations/puppetproduction+89 -27
Customize query in gerrit

Related Objects

Event Timeline

taavi created this task.Feb 5 2026, 12:33 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptFeb 5 2026, 12:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
taavi claimed this task.Feb 17 2026, 2:44 PM
gerritbot added a comment.Feb 17 2026, 3:08 PM

Change #1239965 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: encapi: Store project names in database

https://gerrit.wikimedia.org/r/1239965

gerritbot added a project: Patch-For-Review.Feb 17 2026, 3:08 PM
taavi triaged this task as Medium priority.Feb 18 2026, 11:55 AM
gerritbot added a comment.Feb 18 2026, 1:54 PM

Change #1239965 merged by Majavah:

[operations/puppet@production] openstack: encapi: Store project names in database

https://gerrit.wikimedia.org/r/1239965

Maintenance_bot removed a project: Patch-For-Review.Feb 18 2026, 2:31 PM
gerritbot added a comment.Feb 23 2026, 10:28 AM

Change #1242291 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] opesntack: encapi: Fix column name

https://gerrit.wikimedia.org/r/1242291

gerritbot added a project: Patch-For-Review.Feb 23 2026, 10:28 AM
gerritbot added a comment.Feb 23 2026, 10:31 AM

Change #1242291 merged by Majavah:

[operations/puppet@production] opesntack: encapi: Fix column name

https://gerrit.wikimedia.org/r/1242291

Maintenance_bot removed a project: Patch-For-Review.Feb 23 2026, 11:32 AM
Stashbot added a comment.Apr 23 2026, 10:25 AM

Mentioned in SAL (#wikimedia-cloud) [2026-04-23T10:25:34Z] <taavi> backfill project IDs to enc database with https://phabricator.wikimedia.org/P91349 T416588

taavi added a comment.Apr 23 2026, 10:26 AM

After running the backfill script, we're left with the following stale items in the database:

MariaDB [labspuppet]> select * from prefix where prefix_project_id is null;
+-------+-------------------+---------------------------+----------------------------------------------------------+
| id    | prefix_project_id | project                   | prefix                                                   |
+-------+-------------------+---------------------------+----------------------------------------------------------+
|  2086 |              NULL | monitoring                |                                                          |
|  2159 |              NULL | getstarted                |                                                          |
|  8734 |              NULL | clouddb-services          |                                                          |
|  9796 |              NULL | sso                       |                                                          |
| 10970 |              NULL | mwv-apt                   |                                                          |
| 11113 |              NULL | wikilabels                |                                                          |
| 14418 |              NULL | maps-experiments          |                                                          |
| 17062 |              NULL | terraform                 |                                                          |
| 17557 |              NULL | checkuser-beta-wiki       |                                                          |
| 17800 |              NULL | dump-references-processor |                                                          |
| 14498 |              NULL | openocr                   | test                                                     |
| 17073 |              NULL | wikilabels                | wikilabels-03.wikilabels.eqiad1.wikimedia.cloud          |
| 17183 |              NULL | wikilabels                | wikilabels-database-02.wikilabels.eqiad1.wikimedia.cloud |
| 11683 |              NULL | wikilabels                | wikilabels-staging-02.wikilabels.eqiad1.wikimedia.cloud  |
+-------+-------------------+---------------------------+----------------------------------------------------------+
14 rows in set (0.001 sec)
gerritbot added a comment.May 5 2026, 8:08 AM

Change #1282878 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: encapi: Always use explicit SQL JOIN syntax

https://gerrit.wikimedia.org/r/1282878

gerritbot added a project: Patch-For-Review.May 5 2026, 8:08 AM

Change #1282879 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: wmcs-enc-cli: Add subcommand to manually delete projects

https://gerrit.wikimedia.org/r/1282879

gerritbot added a comment.May 5 2026, 8:59 AM

Change #1282878 merged by Majavah:

[operations/puppet@production] openstack: encapi: Always use explicit SQL JOIN syntax

https://gerrit.wikimedia.org/r/1282878

gerritbot added a comment.May 5 2026, 10:15 AM

Change #1282879 merged by Majavah:

[operations/puppet@production] openstack: wmcs-enc-cli: Add subcommand to manually delete projects

https://gerrit.wikimedia.org/r/1282879

Stashbot added a comment.May 5 2026, 10:20 AM

Mentioned in SAL (#wikimedia-cloud) [2026-05-05T10:20:31Z] <taavi> taavi@cloudcontrol1007 ~ $ sudo wmcs-enc-cli --openstack-project admin delete_project wikilabels # T416588

Maintenance_bot removed a project: Patch-For-Review.May 5 2026, 10:31 AM
gerritbot added a comment.May 5 2026, 10:36 AM

Change #1282943 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: puppet-enc: Exclusively read projects table

https://gerrit.wikimedia.org/r/1282943

gerritbot added a project: Patch-For-Review.May 5 2026, 10:36 AM

Change #1282944 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: puppet-enc: Stop writing and drop old project column

https://gerrit.wikimedia.org/r/1282944

gerritbot added a comment.May 5 2026, 1:12 PM

Change #1282943 merged by Majavah:

[operations/puppet@production] openstack: puppet-enc: Exclusively read projects table

https://gerrit.wikimedia.org/r/1282943

gerritbot added a comment.May 12 2026, 11:19 AM

Change #1282944 merged by Majavah:

[operations/puppet@production] openstack: puppet-enc: Stop writing and drop old project column

https://gerrit.wikimedia.org/r/1282944

gerritbot added a comment.May 12 2026, 11:25 AM

Change #1286315 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: encapi: Make new project column NOT NULL

https://gerrit.wikimedia.org/r/1286315

gerritbot added a comment.May 12 2026, 11:29 AM

Change #1286315 merged by Majavah:

[operations/puppet@production] openstack: encapi: Make new project column NOT NULL

https://gerrit.wikimedia.org/r/1286315

taavi closed this task as Resolved.May 12 2026, 11:30 AM

This is fixed for future projects. T404425 needs to happen to fix the existing cases in the exported Git data.

Maintenance_bot removed a project: Patch-For-Review.May 12 2026, 11:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits