In T427929#12361917, @Dbrant wrote:

In T427929#12331480, @jasmine_ wrote:

Could you confirm if Mobile Apps would like to serve the entire assetlinks.json served on auth.wikimedia.org [0] to fulfill the DAL/ no redirect requirement specific to android apps? or sufficient to do what we’re doing in the assetlinks.json served on www.wikipedia.org [1], which is to grant the target (android_app) permission to handle all URLs that the source (www.wikimedia.org) can handle via relational string "delegate_permission/common.handle_all_urls" [2]

Thanks @jasmine_, sorry for the delay -- Yes, ideally it should be the same list of items listed on authwiki. The patch looks good!

To recap the intended plan here:

It appears that the cookbook fails here while checking for the records propagation since it uses the old_ttl as clearance for an expected records_propagated_at time.

In T436846#12389151, @SLyngshede-WMF wrote:

@MLechvien-WMF I can certainly take a look :-)

In T439052#12377227, @taavi wrote:

Can we just make this public? The switchover happening will presumably be announced widely, the fact that we had a power outage in codfw and had to roll back switchover preparations is certainly public knowledge at this point (and this task does not contain any possibly non-public details of the codfw DC power setup), as is the analysis of performance issues that led to the original postponing. (Arguably the power outage task or a replacement to collect its follow-ups should be made public as well, but I will leave those arguments for some other place to not clog up the history here.)

In T438896#12371812, @Ladsgroup wrote:

Thank you for the investigation Chris! <3 I updated the title to reflect the fact that thumb migration is likely not the cause. Also the Campaigns is CN feature and I got confused. Sorry for the wrong ping Val.

In T439186#12367967, @MLechvien-WMF wrote:

This dashboard can also be used to see the services in active/active that are not pooled in codfw at the moment.

From there, k8s-ingress-dse-aa should also be checked in addition to the two listed by Jasmine.

In T439186#12364379, @MLechvien-WMF wrote:

We discussed with @DPogorzelski-WMF in Slack:

• all pods are running
• And for services that exist in codfw it seems to work

httpbb /srv/deployment/httpbb-tests/liftwing/production/test_revertrisk.yaml --host inference.svc.codfw.wmnet --https_port 30443
Sending to inference.svc.codfw.wmnet...
PASS: 3 requests sent to inference.svc.codfw.wmnet. All assertions passed.

httpbb /srv/deployment/httpbb-tests/liftwing/production/test_article-descriptions.yaml --host inference.svc.codfw.wmnet --https_port 30443
Sending to inference.svc.codfw.wmnet...
PASS: 1 request sent to inference.svc.codfw.wmnet. All assertions passed.

So from my pov they are reachable and work

In T439187#12362733, @elukey wrote:

Current state:

elukey@puppetserver1001:~$ sudo -i confctl --object-type discovery select 'dnsdisc=pki' get
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=pki"}
{"eqiad": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=pki"}

elukey@puppetserver1001:~$ sudo -i confctl --object-type discovery select 'dnsdisc=puppetboard' get
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=puppetboard"}
{"eqiad": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=puppetboard"}

elukey@puppetserver1001:~$ sudo -i confctl --object-type discovery select 'dnsdisc=puppetdb-api' get
{"eqiad": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=puppetdb-api"}
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=puppetdb-api"}

@jasmine_ The services are marked as active/active in service.yaml, so I'd expect all of them to be pooled in both DCs meanwhile I see only eqiad. IIUC we have rolled back the switchover, so can we repool them in codfw too?

Closing as this has been resolved and fixes have been merged, thanks!