in the medbox4 log I see

Jan 03 12:22:57 medbox-iiab puppet-agent-cronjob[120579]: E: The repository 'http://security.debian.org bullseye/updates Release' does not have a Release file.

not sure if this is relevant

I also see in /etc/hosts

127.0.0.1 medbox-iiab.iiab.eqiad1.wikimedia.cloud for medbox4, so I guess the hostsname is medbox-iiab

this was built from the bullseye image. perhaps it is missing some puppet required files.

I notice in medbox3 that 127.0.0.1 is just localhost and the prompt is timmoody@medbox3-iiab so has the true hostname, unlike medbox4 where the prompt is timmoody@medbox-iiab

looks like this is related to https://phabricator.wikimedia.org/T291168

is the hosts entry a separate bug?

do I need to manually change /etc/apt/sources.list

deb http://security.debian.org/ bullseye/updates main
deb-src http://security.debian.org/ bullseye/updates main

to

deb http://security.debian.org/ bullseye-security main
deb-src http://security.debian.org/ bullseye-security main

after this change apt update seems to work.

please advise.

the hosts name problem is mine.

if the manual change to /etc/apt/sources.list is correct this can be closed

Have you done any changes to the instances hostname? Puppet is still failing to run

root@medbox-iiab:~# run-puppet-agent
Exiting; no certificate found and waitforcert is disabled

which is happening due to the hostname mismatch (medbox4-iiab != medbox-iiab). At least during the initial provisioning the instance thought it was medbox4-iiab so something must have changed it after that:

root@medbox-iiab:~# ls /var/lib/puppet/ssl/certs/
buildvm-f575acf4-c566-4258-9a5f-20b9ec80ccad.admin.eqiad1.wikimedia.cloud.pem  ca.pem  medbox4-iiab.iiab.eqiad1.wikimedia.cloud.pem

One of the setup scripts I ran changed hosts and put medbox-iiab as an alias of 127.0.01. (I didn't realize that.)
I have now run hostnamectl set-hostname medbox4-iiab, shut down the instance and restarted.
I now see timmoody@medbox4-iiab

then I ran
timmoody@medbox4-iiab:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for medbox4-iiab.iiab.eqiad1.wikimedia.cloud
Info: Applying configuration version '(dbe4c1c029) cmjohnson - Adding new cloudbackup1003/4 to netboot.cfg'
Notice: The LDAP client stack for this host is: sssd/sudo
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'
Notice: Applied catalog in 4.90 seconds

does this now look OK?

my next steps are to move the cinder volume from medbox3-iiab to medbox4-iiab. I assume its contents will be preserved
and move the fixed ip address

In T298466#7594487, @Tim-moody wrote:

does this now look OK?

yeah, looks good, thanks!