If I go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance, choose any project, no instances are ever shown, despite logging out and in again. I assume this is the same issue as before that was fixed by restarting … Keystone? Nutcracker?
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | scfc | T109933 Labs_lvm::Volume[separate-tmp] is noisy on execution hosts | |||
| Resolved | Andrew | T110629 Can't list instances on Special:NovaInstance |
Event Timeline
Comment Actions
After promoting "Tim Landscheidt (Test)" to a projectadmin in Toolsbeta, that user can see the instance list for Toolsbeta. The user "Tim Landscheidt" still sees no instances, which makes me think that it is user-dependent.
Comment Actions
Huh. Did you do this in separate browsers or incognito mode or something? If so, tried clearing your cookies?
Comment Actions
I did log in as "Tim Landscheidt (Test)" in an incognito window (Firefox; "private window"?).
For "Tim Landscheidt", I just now deleted all cookies for wikitech.wikimedia.org, logged in again, but – as "Tim Landscheidt" – the list in https://wikitech.wikimedia.org/wiki/Special:NovaInstance is still empty.
Comment Actions
"Tim Landscheidt" (after deleting all cookies for wikitech.wikimedia.org) still sees no list of instances, but now "Tim Landscheidt (Test)" (in a private window) doesn't either.
Comment Actions
@scfc, is this still happening for you? (If so, that's good, because I don't have any other test cases)
Comment Actions
Yes, it is. After deleting all cookies for wikitech.wikimedia.org and logging in again, "Tim Landscheidt" still cannot see any instances and "Tim Landscheidt (Test)" in a private window neither.
Comment Actions
Are you able to view things as you'd expect within Horizon? Or is your project membership broken there as well?
Comment Actions
Also, what does your project filter look like? Are you still able to select projects that you belong to?
Comment Actions
ok -- I'm still interested in the answers to the above questions but I've also put a small live-hack in place. Does it affect behavior?
Comment Actions
I have tried https://horizon.wikimedia.org/ just now for the first time, and the list of instance there is perfectly fine.
In wikitech, yes, I was able to select the projects I belong to in the project filter as usual. Only the section for the selected projects was (and is) empty:
This didn't change after deleting the cookies and logging in again, so I'm afraid your live-hack didn't resolve it yet :-).
Comment Actions
Change 235469 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.
Comment Actions
Change 235479 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.
Comment Actions
Change 235482 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.
Comment Actions
@scfc, there is yet another patch in place. If the problem is still present can you ping me on IRC so that I can do a bit of interactive debugging?
Comment Actions
The above patch plus a complete purge of keystone tokens, wikitech tokens, and a restart of memcached seems to have fixed things.
I presume that we still have some broken cache-invalidation logic someplace that a few lucky users managed to step in :(
Comment Actions
Unfortunately, that didn't last long :-) (or I tested only the "toolsbeta" part of it a few hours ago).
When I delete wikitech cookies, log in and go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance, I can select the project filter to "toolsbeta" (or "shinken" or "project-proxy") and the list shows all instances for this project, but if I choose "tools" (or "bastion") the list is empty. (The list is also empty for "wikivoyage", but that project does not have any instances in reality as well.)
Comment Actions
ok... I don't know what's going on now, but ping be on irc sometime and I'll turn on some logfiles.
Comment Actions
Change 235768 had a related patch set uploaded (by Andrew Bogott):
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."
Comment Actions
Change 235768 merged by jenkins-bot:
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."
Comment Actions
Change 235769 had a related patch set uploaded (by Alex Monk):
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."
Comment Actions
Change 235769 merged by jenkins-bot:
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."
Comment Actions
So... once again I believe this to be resolved (mostly because I reverted the patches that most likely caused it.)
Comment Actions
Hurray! … in the sense that it works now for both tools and toolsbeta for "Tim Landscheidt" :-).
I'm hesitant to close this task as resolved because I don't understand what the cause of the problem is and if that has been fixed for good.
Comment Actions
I'm pretty sure the cause of the problem was my bad cache-refresh code, which is now reverted.