Page MenuHomePhabricator

Can't list instances on Special:NovaInstance
Closed, ResolvedPublic

Description

If I go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance, choose any project, no instances are ever shown, despite logging out and in again. I assume this is the same issue as before that was fixed by restarting … Keystone? Nutcracker?

Event Timeline

scfc raised the priority of this task from to Unbreak Now!.
scfc updated the task description. (Show Details)
scfc added a project: wikitech.wikimedia.org.
scfc subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

After promoting "Tim Landscheidt (Test)" to a projectadmin in Toolsbeta, that user can see the instance list for Toolsbeta. The user "Tim Landscheidt" still sees no instances, which makes me think that it is user-dependent.

Huh. Did you do this in separate browsers or incognito mode or something? If so, tried clearing your cookies?

I did log in as "Tim Landscheidt (Test)" in an incognito window (Firefox; "private window"?).

For "Tim Landscheidt", I just now deleted all cookies for wikitech.wikimedia.org, logged in again, but – as "Tim Landscheidt" – the list in https://wikitech.wikimedia.org/wiki/Special:NovaInstance is still empty.

"Tim Landscheidt" (after deleting all cookies for wikitech.wikimedia.org) still sees no list of instances, but now "Tim Landscheidt (Test)" (in a private window) doesn't either.

@scfc, is this still happening for you? (If so, that's good, because I don't have any other test cases)

Yes, it is. After deleting all cookies for wikitech.wikimedia.org and logging in again, "Tim Landscheidt" still cannot see any instances and "Tim Landscheidt (Test)" in a private window neither.

Are you able to view things as you'd expect within Horizon? Or is your project membership broken there as well?

Also, what does your project filter look like? Are you still able to select projects that you belong to?

ok -- I'm still interested in the answers to the above questions but I've also put a small live-hack in place. Does it affect behavior?

I have tried https://horizon.wikimedia.org/ just now for the first time, and the list of instance there is perfectly fine.

In wikitech, yes, I was able to select the projects I belong to in the project filter as usual. Only the section for the selected projects was (and is) empty:

instance-list-before-logout-cropped.png (396×1 px, 24 KB)

This didn't change after deleting the cookies and logging in again, so I'm afraid your live-hack didn't resolve it yet :-).

Change 235469 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235469

Change 235469 merged by jenkins-bot:
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235469

Change 235479 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235479

Change 235479 merged by jenkins-bot:
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235479

Change 235482 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235482

Change 235482 merged by jenkins-bot:
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235482

@scfc, there is yet another patch in place. If the problem is still present can you ping me on IRC so that I can do a bit of interactive debugging?

The above patch plus a complete purge of keystone tokens, wikitech tokens, and a restart of memcached seems to have fixed things.

I presume that we still have some broken cache-invalidation logic someplace that a few lucky users managed to step in :(

Unfortunately, that didn't last long :-) (or I tested only the "toolsbeta" part of it a few hours ago).

When I delete wikitech cookies, log in and go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance, I can select the project filter to "toolsbeta" (or "shinken" or "project-proxy") and the list shows all instances for this project, but if I choose "tools" (or "bastion") the list is empty. (The list is also empty for "wikivoyage", but that project does not have any instances in reality as well.)

And now toolsbeta is gone as well :-).

ok... I don't know what's going on now, but ping be on irc sometime and I'll turn on some logfiles.

Tools and bastion empty for myself, logged out and back in

Change 235768 had a related patch set uploaded (by Andrew Bogott):
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235768

Change 235768 merged by jenkins-bot:
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235768

Change 235769 had a related patch set uploaded (by Alex Monk):
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235769

Change 235769 merged by jenkins-bot:
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235769

So... once again I believe this to be resolved (mostly because I reverted the patches that most likely caused it.)

Hurray! … in the sense that it works now for both tools and toolsbeta for "Tim Landscheidt" :-).

I'm hesitant to close this task as resolved because I don't understand what the cause of the problem is and if that has been fixed for good.

I'm pretty sure the cause of the problem was my bad cache-refresh code, which is now reverted.