[openstack] Creating a new project returns Gateway Timeout (HTTP 504)
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	fnegri
	Dec 20 2023, 6:13 PM

Project Tags

Referenced Files

None

Subscribers

• Aklapper

Andrew

fnegri

Description

Steps to replicate the issue (include links if applicable):

Create a new Cloud VPS project using the create_project cookbook:

fnegri@cloudcumin1001:~$ sudo cookbook wmcs.vps.create_project --project dhinustestproject --description "A test project managed by Francesco. Can be deleted."

What happens?:

The project is created, but the cookbook fails with

Gateway Timeout (HTTP 504)

The Spicerack logs show that the failing command is the following one:

2023-12-20 18:03:42,671 fnegri 2894973 [INFO] Executing commands [cumin.transports.Command('sudo -i env OS_PROJECT_ID= wmcs-openstack project create --enable \'--description=A test project managed by Francesco. Can be deleted.\' dhinustestproject --os-cloud novaadmin', ok_codes=[0])] on '1' hosts: cloudcontrol1005.eqiad.wmnet

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	keystone haproxy: increase server timeout for admin service to 10m	Andrew Bogott	operations/puppet	production	+16 -8

Customize query in gerrit

Related Objects

Mentioned Here: T345084: OpenStack API response time gets slower over time
T353421: Request creation of Adiutor VPS project

Event Timeline

fnegri created this task.Dec 20 2023, 6:13 PM

Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptDec 20 2023, 6:13 PM

I first spotted this error while creating this project: T353421: Request creation of Adiutor VPS project

This same error appears when creating projects without the cookbook:

root@cloudcontrol1005:~# openstack project create T353829test3
Gateway Timeout (HTTP 504)

And yet, all of the keystone hooks seem to be completing:

root@cloudcontrol1007:~# grep "Completed wmf hooks" /var/log/keystone/keystone.log
2023-12-20 17:23:07.314 2657249 WARNING nova.wmfkeystonehooks.wmfkeystonehooks [None req-514be21d-90e2-4410-839a-fc1d594c7b55 novaadmin admin - - default default] Completed wmf hooks for project creation: adiutor
2023-12-20 18:32:18.041 2657246 WARNING nova.wmfkeystonehooks.wmfkeystonehooks [None req-ff2aa2ea-b4d0-451a-a950-22cd7bd83247 novaadmin admin - - default default] Completed wmf hooks for project creation: T353829test1
2023-12-20 19:25:31.038 2657248 WARNING nova.wmfkeystonehooks.wmfkeystonehooks [None req-4eff5522-7923-474f-9f87-4a27dc948799 novaadmin admin - - default default] Completed wmf hooks for project creation: T353829test3

It's timing out after 120 seconds.

Change 984640 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] keystone haproxy: increase server timeout for admin service to 10m

https://gerrit.wikimedia.org/r/984640

gerritbot added a project: Patch-For-Review.Dec 20 2023, 8:52 PM

Change 984640 merged by Majavah:

[operations/puppet@production] keystone haproxy: increase server timeout for admin service to 10m

https://gerrit.wikimedia.org/r/984640

Maintenance_bot removed a project: Patch-For-Review.Dec 22 2023, 1:30 PM

I've just successfully created a project, Cumin reports the CLI took 5 mins 32 seconds to create the project:

fnegri@cloudcumin1001:~$ sudo cookbook wmcs.vps.create_project --project openvas --description "Security-team vulnerability scanning project using the Greenbone OpenVAS" --task-id T354192
START - Cookbook wmcs.vps.create_project for project openvas in eqiad1
----- OUTPUT of 'sudo -i env OS_P...-cloud novaadmin' -----
PASS |                                                                                                                               |   0% (0/1) [00:00<?, ?hosts/s]
FAIL |                                                                                                                               |   0% (0/1) [00:00<?, ?hosts/s]
+-------------+--------------------------------------------------------------------------+
| Field       | Value                                                                    |
+-------------+--------------------------------------------------------------------------+
| description | Security-team vulnerability scanning project using the Greenbone OpenVAS |
| domain_id   | default                                                                  |
| enabled     | True                                                                     |
| id          | openvas                                                                  |
| is_domain   | False                                                                    |
| name        | openvas                                                                  |
| options     | {}                                                                       |
| parent_id   | default                                                                  |
| tags        | []                                                                       |
+-------------+--------------------------------------------------------------------------+
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [05:32<00:00, 332.44s/hosts]
FAIL |                                                                                                                               |   0% (0/1) [05:32<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'sudo -i env OS_P...-cloud novaadmin'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project openvas in eqiad1

fnegri closed this task as Resolved.Jan 4 2024, 3:11 PM

fnegri claimed this task.

fnegri added a project: cloud-services-team (FY2023/2024-Q1-Q2).Jan 4 2024, 5:39 PM

fnegri moved this task from Backlog to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.

[openstack] Creating a new project returns Gateway Timeout (HTTP 504)Closed, ResolvedPublicBUG REPORTActions

Description

Details

Related Objects

Event Timeline

[openstack] Creating a new project returns Gateway Timeout (HTTP 504)
Closed, ResolvedPublicBUG REPORT
Actions