@Tarrow lead a session looking into this task, here are some (incomplete) notes.

We discussed three possible ways to tackle this task

1. Copy the tf/env/prod folder to tf/env/staging and edit tf/env/prod as needed
   • this would mean taking on some technical debt (staging)
   • but would:
     • allow us to complete this task more quickly and unblocking other tasks
     • reduce the risk of falling down rabbit holes
     • increase our chance of achieving the sprint goal
   • IMO the technical debt is acceptable given that it would reduce risk and speed up completion; further refactoring into modules could be done at the end of the sprint if time or at a later date
2. Refactor files in tf/env/prod to one big singular module to be used in staging and production
   • Wouldn't have to untangle the
   • Probably more difficult than #1
   • Risk of falling into rabbit holes
3. Refactor everything into discrete modules to be used in staging and production
   • Similar to above but probably requires even more effort and risk of rabbit holes

We had a look at what doing #1 might be like

• how to unpick things named prod when really they are staging (e.g. google_storage_buckets)?
• staging and production will need to be unpicked in dns.tf
• uptime.tf won't work as is for production, we probably just want to remove that from the production env (tf/env/prod) for now
• we will need to rename variables named specifically for staging; e.g. recaptcha_v2_staging_site_key -> recaptcha_v2_site_key

These checkboxes could be added to the task description if confirmed to be part of the work needed to complete this task

Further comments

• We want the staging and production terraform configuration to be reused as much as possible (so that staging and prod don't diverge too much) while also allowing us to EASILY change the configuration where needed (so that we can test things on staging before it is deployed to prod)
• If we use modules (#2 or #3) we would need some form of versioning on the modules so that a change to a module for staging doesn't break production. We believe this is possible with terraform but not sure on the details. Another reason to avoid using modules at this time.

All the prod TF resources have now been created following option 1 from Ollie's comment to avoid rabbit holes. The exception to this is the cert-manager secrets because the cert-manager k8s namespace doesn't exist until we run the helmfile stuff.

As part of this we also left DNS code all still managed by staging

The remaining work for this task involves migrating some DNS settings from the "staging" definitions to the "production" ones.

This should be done by defining them in both places first; then importing the resources into the "production" definition before removing them from the staging state. See: https://www.terraform.io/cli/commands/state/rm and https://www.terraform.io/cli/commands/import

Finally we need to add the new DNS resources which are required. These will mimic the staging ones (e.g. NS records, A records etc.)

I believe this is all done now
Other tasks exist for refactorings