After trying to create projects for: T213283, T212573 and T212918, I've had some success and some not-so-much. All runs yesterday and today have hung. However, where the wikidata-history-query-service and indico projects appear to have completed most or all of the wmfkeystonehooks material, meza and bstorm-test2 failed to end up with a domain from designate.
On digging in, this is hanging during the POST to http://cloudcontrol1003.wikimedia.org:35357/v3/projects, so the error must be on one of the backend processes.
I can confirm so far that the nova-api logs note that we have an error in our code around the security group modification: novaadmin bstorm-test2 - - -] HTTP exception thrown: Invalid port range 1:65535. For ICMP, the type:code must be valid That seems an easy fix, but I cannot imagine that is the root cause since that's from 11 months ago.
I closed T212573, since it appears to be complete, but I'm keeping the others open until this is resolved. Things aren't quite right.