Page MenuHomePhabricator

Can't list instances on Special:NovaInstance
Closed, ResolvedPublic

Description

If I go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance, choose any project, no instances are ever shown, despite logging out and in again. I assume this is the same issue as before that was fixed by restarting … Keystone? Nutcracker?

Event Timeline

scfc created this task.Aug 28 2015, 1:29 AM
scfc raised the priority of this task from to Unbreak Now!.
scfc updated the task description. (Show Details)
scfc added a project: wikitech.wikimedia.org.
scfc added a subscriber: scfc.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 28 2015, 1:29 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scfc added a comment.Aug 28 2015, 1:40 AM

After promoting "Tim Landscheidt (Test)" to a projectadmin in Toolsbeta, that user can see the instance list for Toolsbeta. The user "Tim Landscheidt" still sees no instances, which makes me think that it is user-dependent.

Huh. Did you do this in separate browsers or incognito mode or something? If so, tried clearing your cookies?

scfc added a comment.Aug 28 2015, 1:50 AM

I did log in as "Tim Landscheidt (Test)" in an incognito window (Firefox; "private window"?).

For "Tim Landscheidt", I just now deleted all cookies for wikitech.wikimedia.org, logged in again, but – as "Tim Landscheidt" – the list in https://wikitech.wikimedia.org/wiki/Special:NovaInstance is still empty.

scfc added a comment.Aug 29 2015, 12:06 AM

"Tim Landscheidt" (after deleting all cookies for wikitech.wikimedia.org) still sees no list of instances, but now "Tim Landscheidt (Test)" (in a private window) doesn't either.

Andrew added a comment.Sep 1 2015, 7:14 PM

@scfc, is this still happening for you? (If so, that's good, because I don't have any other test cases)

scfc added a comment.Sep 1 2015, 7:32 PM

Yes, it is. After deleting all cookies for wikitech.wikimedia.org and logging in again, "Tim Landscheidt" still cannot see any instances and "Tim Landscheidt (Test)" in a private window neither.

Andrew added a comment.Sep 1 2015, 9:05 PM

Are you able to view things as you'd expect within Horizon? Or is your project membership broken there as well?

Andrew added a comment.Sep 1 2015, 9:06 PM

Also, what does your project filter look like? Are you still able to select projects that you belong to?

Andrew added a comment.Sep 1 2015, 9:34 PM

ok -- I'm still interested in the answers to the above questions but I've also put a small live-hack in place. Does it affect behavior?

Andrew moved this task from To do to Doing on the Labs-sprint-112 board.Sep 1 2015, 9:36 PM
scfc added a comment.Sep 1 2015, 9:49 PM

I have tried https://horizon.wikimedia.org/ just now for the first time, and the list of instance there is perfectly fine.

In wikitech, yes, I was able to select the projects I belong to in the project filter as usual. Only the section for the selected projects was (and is) empty:

This didn't change after deleting the cookies and logging in again, so I'm afraid your live-hack didn't resolve it yet :-).

Change 235469 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235469

Change 235469 merged by jenkins-bot:
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235469

Change 235479 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235479

Change 235479 merged by jenkins-bot:
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235479

Change 235482 had a related patch set uploaded (by Alex Monk):
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235482

Change 235482 merged by jenkins-bot:
Fix logic to refresh the service catalog.

https://gerrit.wikimedia.org/r/235482

Andrew added a comment.Sep 2 2015, 4:16 PM

@scfc, there is yet another patch in place. If the problem is still present can you ping me on IRC so that I can do a bit of interactive debugging?

Andrew closed this task as Resolved.Sep 2 2015, 6:14 PM

The above patch plus a complete purge of keystone tokens, wikitech tokens, and a restart of memcached seems to have fixed things.

I presume that we still have some broken cache-invalidation logic someplace that a few lucky users managed to step in :(

scfc reopened this task as Open.Sep 2 2015, 11:14 PM

Unfortunately, that didn't last long :-) (or I tested only the "toolsbeta" part of it a few hours ago).

When I delete wikitech cookies, log in and go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance, I can select the project filter to "toolsbeta" (or "shinken" or "project-proxy") and the list shows all instances for this project, but if I choose "tools" (or "bastion") the list is empty. (The list is also empty for "wikivoyage", but that project does not have any instances in reality as well.)

scfc added a comment.Sep 2 2015, 11:35 PM

And now toolsbeta is gone as well :-).

Andrew added a comment.Sep 3 2015, 2:38 PM

ok... I don't know what's going on now, but ping be on irc sometime and I'll turn on some logfiles.

Samtar added a subscriber: Samtar.Sep 3 2015, 2:45 PM

Tools and bastion empty for myself, logged out and back in

Change 235768 had a related patch set uploaded (by Andrew Bogott):
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235768

Change 235768 merged by jenkins-bot:
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235768

Change 235769 had a related patch set uploaded (by Alex Monk):
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235769

Change 235769 merged by jenkins-bot:
Revert "Fix logic to refresh the service catalog." Revert "Don't cache the service catalog for nearly as long."

https://gerrit.wikimedia.org/r/235769

Andrew added a comment.Sep 3 2015, 5:40 PM

So... once again I believe this to be resolved (mostly because I reverted the patches that most likely caused it.)

scfc added a comment.Sep 3 2015, 7:27 PM

Hurray! … in the sense that it works now for both tools and toolsbeta for "Tim Landscheidt" :-).

I'm hesitant to close this task as resolved because I don't understand what the cause of the problem is and if that has been fixed for good.

Andrew closed this task as Resolved.Sep 8 2015, 2:47 PM

I'm pretty sure the cause of the problem was my bad cache-refresh code, which is now reverted.

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptDec 27 2018, 5:09 AM