expected due to the update in T383263

expected due to the update in T383263

Great thanks @elukey and Cathal for the netbox research and re reimage. wikikube-worker2022 looks good now, I update the bgp setting, homered and pooled the node. So this issue is resolved now and the host is on bookworm + containerd.

Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host looks good to me now.

Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host looks good to me now.

Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host looks good to me now.

Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host looks good to me now.

Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host looks good to me now.

The following commands have to be executed when the host is back (just noting it down so I don't forget it):

Similar to issues in eqiad, like T381878

In T381878#10441783, @Jclark-ctr wrote:

@Jelto i performed flea power drain and looks to image properly the critical status has cleared will update dell but looks good for now

related to T383263

Thanks @elukey for doublechecking the PXE settings in the BIOS. I tried the reimage with --use-http-for-dhcp but this resulted in the same behavior: the DHCP request timed out and the system booted from disk.

Thanks for digging into the cdincludes parameter option! Bundling the API calls into one indeed seems like a good idea for better performance, especially if the result is cached for the most frequently accessed sites.

Thanks for checking the chanelog!

I removed some logfiles and apt cache. Both hosts have 7.5G free space, this should be enough for the holiday break

The host responses normally and a reimage worked. Thanks @Jhancock.wm for the quick help!

The .git folder was removed from all design miscweb sites and container images. For the other miscweb sites a similar .dockerignore was added as a precaution.

I added .dockerignore files to all miscweb projects.

In T382230#10406800, @dancy wrote:

@Jelto I opened https://gitlab.wikimedia.org/repos/sre/miscweb/design-landing-page/-/merge_requests/5 with a simpler approach.

Fixes for the remaining two design miscweb sites:

A fix was deployed which removed the .git folder from design.wikimedia.org.

I opened https://gitlab.wikimedia.org/repos/sre/miscweb/design-landing-page/-/merge_requests/4 which uses a dedicated builder variant to remove the .git folder and to fix the apache config.

I was not able to find the linked API key in GitLab. Unfortunately the Token information API is not available in 17.4 (our version) and needs at least 17.5.

I tried to use the token but I get Access denied:

remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.wikimedia.org/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.wikimedia.org/repos/sre/miscweb/design-landing-page.git/'

Thanks for raising this issue. I think it makes sense to block that folder in apache. However we should avoid putting this file in the container image at all by fixing the blubber config. Because even if we are blocking the path in apache, you could just pull the image and extract the token locally.

Related to the upgrade in T381969. This will resolve on Friday when the upgrade is finished.

Thanks for finding and fixing the issue with common-query @bking , @Dzahn . I'm glad this is unrelated to the gui migration and not a blocker for the migration of the remaining services.

^ I used the wrong task ID. This depool was for codfw.

In T381504#10394045, @Jhancock.wm wrote:

@Jelto heads up, these are showing up in a netbox report.

Device is Active in Netbox but is missing from PuppetDB (should be ('decommissioning', 'inventory', 'offline', 'planned', 'staged', 'failed'))

wikikube-worker1057
wikikube-worker1069
wikikube-worker1073
wikikube-worker1081

The following commands have to be executed when the host is back (just noting it down so I don't forget it):

The following commands have to be executed when the host is back (just noting it down so I don't forget it):

The following commands have to be executed when the host is back (just noting it down so I don't forget it):

The following commands have to be executed when the host is back (just noting it down so I don't forget it):

wikikube-worker1057 is stuck because of Comm Error: backplane 0, see T381676.

Thanks for the quick help @taavi and @thcipriani !

Yesterday, there were reports of commons-query.wikimedia.org being unavailable, so we rolled back the query-scholarly switch. However, the issue with commons-query persists and continues to return upstream request timeout.

The migration of query-scholarly.wikidata.org to Wikikube has been successfully completed. Basic test queries, headers, and the custom configuration are functioning as expected. A big thanks to @ItamarWMDE and @Lucas_Werkmeister_WMDE for their support!

It should be fine to make the task public.

Reimage of wikikube-worker1006 failed because the node is not in icinga/alertmanager. I'll try to find out why tomorrow

Great thanks, I sent you an invite for tomorrow 14:00 CET.

this is resolved