Andrew (Andrew Bogott)
User

Projects (6)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (129 w, 5 d)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott

Recent Activity

Yesterday

Andrew added a comment to T163796: Audit disk usage on labvirts.

That spreadsheet is also available as https://docs.google.com/spreadsheets/d/1TRimo0kT_YzlXl_RD3Z7zOZHdj5Piev31ALIKku7Y8g

Fri, Apr 28, 8:55 PM · Labs-Infrastructure, Labs
Andrew edited the description of T164085: Add domain info to openstack-browser.
Fri, Apr 28, 4:54 PM · Patch-For-Review, Tool-Labs-tools-Other, User-bd808
Andrew committed R2073:57814d7a5573: Added wmflabsdotorg_a_records_for_project and a_records_for_project (authored by Andrew).
Added wmflabsdotorg_a_records_for_project and a_records_for_project
Fri, Apr 28, 4:31 PM
Andrew added a project to T164085: Add domain info to openstack-browser: Tool-Labs-tools-Other.
Fri, Apr 28, 3:32 PM · Patch-For-Review, Tool-Labs-tools-Other, User-bd808
Andrew added a comment to T164085: Add domain info to openstack-browser.

Actually, now that I've said that thing about pulling records from wmflabsdotorg, let me just add a library call for that. Stay tuned...

Fri, Apr 28, 3:31 PM · Patch-For-Review, Tool-Labs-tools-Other, User-bd808
Andrew added a comment to T164085: Add domain info to openstack-browser.

I'm assuming that A records are the only thing the GUI will care about... it's trivial to add queries for other record types if that proves useful.

Fri, Apr 28, 3:27 PM · Patch-For-Review, Tool-Labs-tools-Other, User-bd808
Andrew created T164085: Add domain info to openstack-browser.
Fri, Apr 28, 3:26 PM · Patch-For-Review, Tool-Labs-tools-Other, User-bd808

Thu, Apr 27

Andrew updated subscribers of T163721: Update wikitech-static and develop procedures to keep it maintained.
Thu, Apr 27, 9:06 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew added a comment to T163721: Update wikitech-static and develop procedures to keep it maintained.

I have three different sets of proposed answers:

Thu, Apr 27, 9:05 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew edited the description of T163721: Update wikitech-static and develop procedures to keep it maintained.
Thu, Apr 27, 9:01 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew added a comment to T160611: Make "linter" table available on Labs.

This is now available on all db hosts except for 1001, which is misbehaving.

Thu, Apr 27, 7:15 PM · Patch-For-Review, DBA, Labs, MediaWiki-extensions-Linter
Andrew updated subscribers of T164029: maintain-views needs to do something smarter when trying to update dbs that don't exist.
Thu, Apr 27, 7:03 PM · Labs-Infrastructure, Labs
Andrew added projects to T164029: maintain-views needs to do something smarter when trying to update dbs that don't exist: Labs, Labs-Infrastructure.
Thu, Apr 27, 7:02 PM · Labs-Infrastructure, Labs
Andrew created T164029: maintain-views needs to do something smarter when trying to update dbs that don't exist.
Thu, Apr 27, 7:01 PM · Labs-Infrastructure, Labs
Andrew added a comment to T160611: Make "linter" table available on Labs.

I merged the puppet change. Next we need to update things with

Thu, Apr 27, 5:51 PM · Patch-For-Review, DBA, Labs, MediaWiki-extensions-Linter
Andrew placed T163402: Ensure we can survive a loss of labservices1001 up for grabs.

The particular maintenance that prompted this task is now complete. We still need to improve here, though.

Thu, Apr 27, 5:05 PM · Patch-For-Review, Operations, Labs
Andrew created T164014: Switch primary labservices node back to labservices1001.
Thu, Apr 27, 4:53 PM · Labs, Labs-Infrastructure
Andrew added a comment to T158613: PHP Warning: Memcached::touch(): touch is only supported with binary protocol.

(obviously I vote for option 1, 'switch from PHP5 to HHVM for serving wikitech' since that one is forward and the others are backwards)

Thu, Apr 27, 4:17 PM · Wikimedia-log-errors, wikitech.wikimedia.org, Labs
Andrew added a comment to T158613: PHP Warning: Memcached::touch(): touch is only supported with binary protocol.

So the reason this doesn't make MediaWiki hhvm-only is because normal non-WMF deploys of mediawiki don't include twemproxy?

Thu, Apr 27, 4:17 PM · Wikimedia-log-errors, wikitech.wikimedia.org, Labs
Andrew added a comment to T163823: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs.

When the metaservice is fully down then puppet errors out. If instead the metadata service is responding and returning an empty string for the project id... I can't think of how/why that would happen since the empty string would have to be embedded in an otherwise correct structure.

Thu, Apr 27, 3:41 PM · Operations, Labs
Andrew added a comment to T163796: Audit disk usage on labvirts.

Worst case scenario: What if every one of those instances suddenly partitioned and filled every bit of allocated space?

Thu, Apr 27, 3:16 PM · Labs-Infrastructure, Labs
Andrew added a comment to T163796: Audit disk usage on labvirts.

labvirt1001.eqiad.wmnet:

/dev/sdb1       2.2T  381G  1.9T  18% /var/lib/nova/instances

labvirt1002.eqiad.wmnet:

/dev/sdb1       2.2T  1.2T  1.1T  53% /var/lib/nova/instances

labvirt1003.eqiad.wmnet:

/dev/sdb1       2.2T  1.5T  748G  67% /var/lib/nova/instances

labvirt1004.eqiad.wmnet:

/dev/sdb1       2.2T  1.4T  810G  64% /var/lib/nova/instances

labvirt1005.eqiad.wmnet:

/dev/sdb1       2.2T  1.5T  770G  66% /var/lib/nova/instances

labvirt1006.eqiad.wmnet:

/dev/sdb1       2.2T  1.5T  764G  66% /var/lib/nova/instances

labvirt1007.eqiad.wmnet:

/dev/sdb1       2.2T  1.5T  745G  67% /var/lib/nova/instances

labvirt1008.eqiad.wmnet:

/dev/sdb1       2.2T  1.8T  485G  79% /var/lib/nova/instances

labvirt1009.eqiad.wmnet:

/dev/sdb1       2.2T  1.6T  627G  72% /var/lib/nova/instances

labvirt1010.eqiad.wmnet:

/dev/mapper/tank-data  4.1T  1.4T  2.7T  34% /var/lib/nova/instances

labvirt1011.eqiad.wmnet:

/dev/mapper/tank-data  4.1T  1.7T  2.5T  41% /var/lib/nova/instances

labvirt1012.eqiad.wmnet:

/dev/mapper/tank-data  4.1T  2.1T  2.0T  52% /var/lib/nova/instances

labvirt1013.eqiad.wmnet:

/dev/mapper/tank-data  4.1T  1.9T  2.2T  47% /var/lib/nova/instances

labvirt1014.eqiad.wmnet:

/dev/mapper/tank-data  4.1T   90G  4.0T   3% /var/lib/nova/instances
Thu, Apr 27, 3:11 PM · Labs-Infrastructure, Labs
Andrew reassigned T163390: Update documentation for Tools Proxy failover from Andrew to chasemp.
Thu, Apr 27, 3:06 PM · Operations, Labs
Andrew added a comment to T163390: Update documentation for Tools Proxy failover.

Looks good to me. I added clarification about short-term failovers since that's the most frequent use case: https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource%3ATools%2FAdmin&type=revision&diff=1757725&oldid=1757724

Thu, Apr 27, 3:05 PM · Operations, Labs
Andrew added a comment to T151704: Freenode sometimes throttles bot connections from tools.

Nope, I never heard anything back.

Thu, Apr 27, 2:07 PM · Labs, Tool-Labs
Andrew added a comment to T158613: PHP Warning: Memcached::touch(): touch is only supported with binary protocol.

The stacktrace doesn't implicate anything that's not running on all the other wikis... does anyone have a theory for why this is happening on wikitech and not elsewhere? Is it a php vs hhvm thing?

Thu, Apr 27, 2:06 PM · Wikimedia-log-errors, wikitech.wikimedia.org, Labs
Andrew added a comment to T161473: Stop requiring two-factor authentication for horizon.wikimedia.org.

I dislike the 2FA requirement I do things from many different devices and I dont have access all the time to internet on my phone,

Thu, Apr 27, 2:03 PM · Horizon, Labs
Andrew closed T159021: Wikitech error when adding users to projects as "Resolved".
Thu, Apr 27, 1:59 PM · Labs-Infrastructure, Labs
Andrew closed T159021: Wikitech error when adding users to projects, a subtask of T150091: Support project creation without OpenStackManager, as "Resolved".
Thu, Apr 27, 1:59 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Patch-For-Review, Labs-Infrastructure, Labs
Andrew closed T159021: Wikitech error when adding users to projects, a subtask of T163259: Shorter token life for novaobserver/novaadmin, as "Resolved".
Thu, Apr 27, 1:59 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T163721: Update wikitech-static and develop procedures to keep it maintained.

I updated the password so that's all good now.

Thu, Apr 27, 1:38 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew updated subscribers of T163796: Audit disk usage on labvirts.
Thu, Apr 27, 4:45 AM · Labs-Infrastructure, Labs
Andrew added a comment to T163796: Audit disk usage on labvirts.

For a quick start, I'm attaching a spreadsheet

that shows instance allocated usage (from the instance flavor) next to actual instance disk space usage (via 'du' on the labvirts.) It shows us actually consuming about 50% of allocated disk space.

Thu, Apr 27, 4:42 AM · Labs-Infrastructure, Labs

Wed, Apr 26

Andrew added a comment to T163402: Ensure we can survive a loss of labservices1001.

The answer to the question:

Wed, Apr 26, 9:41 PM · Patch-For-Review, Operations, Labs
Andrew assigned T163942: Ops pwstore currently read-only to MoritzMuehlenhoff.
Wed, Apr 26, 9:26 PM · Operations
Andrew updated subscribers of T163942: Ops pwstore currently read-only.
Wed, Apr 26, 8:52 PM · Operations
Andrew triaged T163942: Ops pwstore currently read-only as "High" priority.
Wed, Apr 26, 8:51 PM · Operations
Andrew created T163942: Ops pwstore currently read-only.
Wed, Apr 26, 8:51 PM · Operations
Andrew added a comment to T163721: Update wikitech-static and develop procedures to keep it maintained.

I've updated https://wikitech.wikimedia.org/wiki/Wikitech-static with some vague maintenance instructions.

Wed, Apr 26, 8:29 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew added a comment to T163721: Update wikitech-static and develop procedures to keep it maintained.

The root password to this host no longer works. Either someone fancied it up to use keys, or we need to do a rescue as per https://support.rackspace.com/how-to/reset-your-server-password/

Wed, Apr 26, 5:06 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew added a comment to T163796: Audit disk usage on labvirts.

In our config, we have disk_allocation_ratio=1.5

Wed, Apr 26, 5:04 PM · Labs-Infrastructure, Labs
Andrew added a comment to T163721: Update wikitech-static and develop procedures to keep it maintained.

Maintenance tasks are just:

Wed, Apr 26, 4:34 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew added a subtask for T163721: Update wikitech-static and develop procedures to keep it maintained: Unknown Object (Task).
Wed, Apr 26, 3:55 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew renamed T163721: Update wikitech-static and develop procedures to keep it maintained from "Keep wikitech-static up to date, at least somewhat" to "Update wikitech-static and develop procedures to keep it maintained".
Wed, Apr 26, 3:54 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew placed T163823: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs up for grabs.
Wed, Apr 26, 3:52 PM · Operations, Labs
Andrew added a comment to T163823: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs.

I still can't reproduce this, even creating the exact set of breakages that were present when the issue appeared in prod.

Wed, Apr 26, 3:40 PM · Operations, Labs
Andrew added a comment to T148506: Rack and setup new eqiad row D switch stack (EX4300/QFX5100).

We're pretty sure that the only Labs thing affected by this is instance creation. I've disabled instance creation for now, with https://gerrit.wikimedia.org/r/#/c/350414/ for Horizon and a live hack in OSM on silver.

Wed, Apr 26, 1:50 PM · Patch-For-Review, Operations, ops-eqiad, netops

Tue, Apr 25

Andrew added a comment to T163823: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs.

Just turning off various dns services (including mdns) does not reproduce this issue.

Tue, Apr 25, 9:59 PM · Operations, Labs
Andrew added a comment to T161899: Investigate ceasing self-service new Trusty instance creation in Labs.

Another thing I'd suggest is that we get Stretch available to users before we start pushing them off Trusty. Jessie isn't in support for much longer than Trusty so we should jump ahead rather than migrating our users over and over.

Tue, Apr 25, 8:24 PM · Operations, Labs
Andrew created T163796: Audit disk usage on labvirts.
Tue, Apr 25, 3:43 PM · Labs-Infrastructure, Labs
Andrew added a comment to T163672: Request creation of Wikidata Concepts labs project.

(No real objection to granting this resource request, if you're sure it's actually going to be useful to you)

Tue, Apr 25, 3:34 PM · Labs
Andrew added a comment to T163672: Request creation of Wikidata Concepts labs project.

Unless there's a public interface for e.g. Hadoop you won't be able to route there from a Labs VM. Having a public IP doesn't help with that.

Tue, Apr 25, 3:33 PM · Labs
Andrew added a comment to T163765: When an instance is deleted, remove proxy records that point to it.

Oh, hm, in theory the DNS record still knows the IP, maybe I can dig it out from there.

Tue, Apr 25, 3:16 PM · Labs-Infrastructure, Labs
Andrew added a comment to T163765: When an instance is deleted, remove proxy records that point to it.

This might turn out to be hard -- sink gets notified of instance deletion but not until after the instance's IP has been freed, so we don't have a good way of following up within sink to find associated proxies. We'd have to include identifying metadata (e.g. instance ID) on the proxy api side in order to correlate an instance with a proxy after the fact.

Tue, Apr 25, 3:14 PM · Labs-Infrastructure, Labs
Andrew added a comment to T161899: Investigate ceasing self-service new Trusty instance creation in Labs.

It should be possible to gather lifespan stats for existing instances and see what people are creating these days. Also, there are some half-measures that we definitely can take immediately:

Tue, Apr 25, 1:54 PM · Operations, Labs
Andrew added a comment to T163402: Ensure we can survive a loss of labservices1001.

For the most part I'd expect DNS to fail over gracefully -- in the cases where it doesn't, that's misbehavior (or misconfig) on the part of the clients. That is https://phabricator.wikimedia.org/T119660.

Tue, Apr 25, 1:42 PM · Patch-For-Review, Operations, Labs
Andrew edited the description of T163402: Ensure we can survive a loss of labservices1001.
Tue, Apr 25, 1:38 PM · Patch-For-Review, Operations, Labs
Andrew added a comment to T161899: Investigate ceasing self-service new Trusty instance creation in Labs.

As soon as we disable Trusty we'll also be violating 'cattle, not pets' for most of our users. It will mean that anytime they need to recreate an instance they will also have to learn how to configure a new OS and adapt their work to run there.

Tue, Apr 25, 1:31 PM · Operations, Labs
Andrew created T163765: When an instance is deleted, remove proxy records that point to it.
Tue, Apr 25, 4:34 AM · Labs-Infrastructure, Labs
Andrew closed T163737: List of leaked dns precords for Andrew to clean up as "Resolved".

All gone!

Tue, Apr 25, 4:11 AM · Labs

Mon, Apr 24

Andrew created T163737: List of leaked dns precords for Andrew to clean up.
Mon, Apr 24, 8:28 PM · Labs
Andrew edited P5322 potential labs proxy updates.
Mon, Apr 24, 7:48 PM
Andrew edited P5322 potential labs proxy updates.
Mon, Apr 24, 7:45 PM
Andrew created P5322 potential labs proxy updates.
Mon, Apr 24, 7:31 PM
Andrew added a subtask for T163721: Update wikitech-static and develop procedures to keep it maintained: Unknown Object (Task).
Mon, Apr 24, 5:58 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew created T163721: Update wikitech-static and develop procedures to keep it maintained.
Mon, Apr 24, 5:58 PM · Patch-For-Review, Operations, wikitech.wikimedia.org, Labs
Andrew added a comment to T162945: The future of service groups and service users on Labs.

Email sent to labs-announce, subject "Does anyone care about service groups?"

Mon, Apr 24, 3:56 PM · MediaWiki-extensions-OpenStackManager, Tool-Labs, Labs
Andrew added a comment to T45580: Automatically updated list of all configured domains.

The result of bd808's work is https://tools.wmflabs.org/openstack-browser/proxy/

Mon, Apr 24, 3:06 PM · User-bd808, Tool-Labs-tools-Other, Labs, Labs-Infrastructure
Andrew renamed T163694: Get labtestwikitech/labtestweb2001 on the deployment train from "Get labtestwikitech/californium on the deployment train" to "Get labtestwikitech/labtestweb2001 on the deployment train".
Mon, Apr 24, 2:34 PM · User-bd808, Release-Engineering-Team, Labs
Andrew added projects to T163694: Get labtestwikitech/labtestweb2001 on the deployment train: Labs, Release-Engineering-Team.
Mon, Apr 24, 2:22 PM · User-bd808, Release-Engineering-Team, Labs
Andrew created T163694: Get labtestwikitech/labtestweb2001 on the deployment train.
Mon, Apr 24, 2:22 PM · User-bd808, Release-Engineering-Team, Labs

Sat, Apr 22

Andrew added a comment to T161898: IO issues for Tools instances flapping with iowait and puppet failure.

OK, 1417 and 1419 are now up and pooled as well.

Sat, Apr 22, 2:09 PM · Labs

Fri, Apr 21

Andrew added a comment to T161898: IO issues for Tools instances flapping with iowait and puppet failure.

-1417 and -1419 are now in the process of puppetizing. This is a note to myself to remember to queue those two over the weekend.

Fri, Apr 21, 11:10 PM · Labs
Andrew added a comment to T161898: IO issues for Tools instances flapping with iowait and puppet failure.

I think we should create 10 tools-webgrid-lighttpd-14* instances to make up for the 20 lost precise ones and see how jobs and load shift. Making 2 tools-webgrid-generic instances seems wise as well.

Fri, Apr 21, 9:30 PM · Labs
Andrew closed T131638: horizon accepts the same 2FA token as wikitech as "Resolved".

I think this is fixed -- Horizon now uses the mediawiki 2fa plugin for verification, same as wikitech.

Fri, Apr 21, 7:48 PM · Labs-Infrastructure, Labs, Security
Andrew added a comment to T161473: Stop requiring two-factor authentication for horizon.wikimedia.org.

I don't know what this means. I've never needed two-factor authentication on wikitech.wikimedia.org. I can log in there just fine.

Fri, Apr 21, 7:16 PM · Horizon, Labs
Andrew added a subtask for T76375: New Labs project requests (tracking): T161554: Provide large disk space to WikiBrain for memory-mapped file.
Fri, Apr 21, 3:27 PM · Tracking, Labs
Andrew added a parent task for T161554: Provide large disk space to WikiBrain for memory-mapped file: T76375: New Labs project requests (tracking).
Fri, Apr 21, 3:27 PM · artificial-intelligence, Labs, Labs-Infrastructure
Andrew added a comment to T161554: Provide large disk space to WikiBrain for memory-mapped file.

OK, sounds good. I'll try to do some capacity assessment and bring this up at our next meeting.

Fri, Apr 21, 3:26 PM · artificial-intelligence, Labs, Labs-Infrastructure
Andrew added a comment to T161554: Provide large disk space to WikiBrain for memory-mapped file.

So, this question relates to both storage needs and also the appropriateness of Labs use: Is this giant storage use something persistent and valuable, or more like a scratch-pad? That is, if we create a 250Gb instance today and then in 2019 you need a 400 Gb instance to handle growth, can you just throw out the old instance and make a new one? Or is the actual storage on the old instance valuable and hard to reproduce such that you'd have to copy or save the file somehow?

Fri, Apr 21, 3:13 PM · artificial-intelligence, Labs, Labs-Infrastructure

Thu, Apr 20

Andrew committed R2073:f61f675fb43d: Add functions to query designate about domains and records for a project. (authored by Andrew).
Add functions to query designate about domains and records for a project.
Thu, Apr 20, 10:31 PM
Andrew committed R2073:4940b62eeb81: Add backend functions for listing proxies for a project (authored by Andrew).
Add backend functions for listing proxies for a project
Thu, Apr 20, 10:31 PM
Andrew closed T163259: Shorter token life for novaobserver/novaadmin as "Resolved".

root@MISC m5[keystone]> SELECT COUNT(*) FROM token;
+----------+

COUNT(*)

+----------+

49197

+----------+
1 row in set (0.02 sec)

Thu, Apr 20, 3:58 PM · Patch-For-Review, Labs-Infrastructure, Labs

Wed, Apr 19

Andrew added a comment to T45580: Automatically updated list of all configured domains.

I figured a big dump that users can search is better than a search widget since it's not that much data -- but, y'know, either way.

Wed, Apr 19, 8:36 PM · User-bd808, Tool-Labs-tools-Other, Labs, Labs-Infrastructure
Andrew added a comment to T45580: Automatically updated list of all configured domains.

This page is just a proof of concept (not live-updating) but is this the kind of thing we're talking about?

Wed, Apr 19, 8:22 PM · User-bd808, Tool-Labs-tools-Other, Labs, Labs-Infrastructure
Andrew added a comment to T159021: Wikitech error when adding users to projects.

@Legoktm are you able to actually produce this issue, or are you still digging back into that previous occurrence?

Wed, Apr 19, 6:28 PM · Labs-Infrastructure, Labs
Andrew added a comment to T159021: Wikitech error when adding users to projects.

If anything it's most likely that OSM is adding a user to a group when the user is already in the group -- much of T150091 involved duplicating OSM behavior in a keystone callback. That said, in my tests it handled the duplication of effort without complaint.

Wed, Apr 19, 6:23 PM · Labs-Infrastructure, Labs
Andrew added a comment to T163259: Shorter token life for novaobserver/novaadmin.

root@MISC m5[keystone]> SELECT COUNT(*) FROM token;
+----------+

COUNT(*)

+----------+

49163

+----------+

Wed, Apr 19, 4:23 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T163259: Shorter token life for novaobserver/novaadmin.

There were about 550,000 tokens found by the queries in those two added crons: novaobserver and novaadmin tokens too young to expire but more than .1 day old. I deleted them all just now to give the cron half a chance. We'll see if 'limit 10,000' once an hour is enough to keep things tidied up.

Wed, Apr 19, 2:49 AM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a parent task for T159021: Wikitech error when adding users to projects: T163259: Shorter token life for novaobserver/novaadmin.
Wed, Apr 19, 2:40 AM · Labs-Infrastructure, Labs
Andrew added a subtask for T163259: Shorter token life for novaobserver/novaadmin: T159021: Wikitech error when adding users to projects.
Wed, Apr 19, 2:40 AM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T159021: Wikitech error when adding users to projects.

I haven't tracked down the relationship, but I suspect this issue is a symptom of token overload, addressed in T163259. That bug should be fixed (albeit poorly) -- does this one look better to you as well?

Wed, Apr 19, 2:40 AM · Labs-Infrastructure, Labs

Tue, Apr 18

Andrew added projects to T163259: Shorter token life for novaobserver/novaadmin: Labs, Labs-Infrastructure.
Tue, Apr 18, 9:26 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T163259: Shorter token life for novaobserver/novaadmin.

On labtest I tried purging all tokens more than 1 day old, and performance gains were considerable. Project deletion took 2.5 minutes with 7-day-old tokens but only 20 seconds with 1-day-old tokens.

Tue, Apr 18, 8:59 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew created T163259: Shorter token life for novaobserver/novaadmin.
Tue, Apr 18, 8:38 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew closed T162615: keystonehooks: Figure out about member role removal as "Resolved".
Tue, Apr 18, 5:40 PM · Patch-For-Review, Horizon, Labs
Andrew closed T126758: Clean up after ldap->mysql keystone migration as "Resolved".
Tue, Apr 18, 5:39 PM · MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)), Patch-For-Review, MediaWiki-extensions-OpenStackManager, Labs, Labs-Infrastructure
Andrew edited the description of T126758: Clean up after ldap->mysql keystone migration.
Tue, Apr 18, 5:38 PM · MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)), Patch-For-Review, MediaWiki-extensions-OpenStackManager, Labs, Labs-Infrastructure
Andrew added a comment to T126758: Clean up after ldap->mysql keystone migration.

And I removed a bunch of other spare role definitions (e.g. 'observer' and 'admin'.)

Tue, Apr 18, 5:38 PM · MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)), Patch-For-Review, MediaWiki-extensions-OpenStackManager, Labs, Labs-Infrastructure
Andrew added a comment to T126758: Clean up after ldap->mysql keystone migration.

...and now I'm going through and removing all members from project entries that are not 'novaadmin.' Novaadmin stays just to keep the schema happy. I'm not sure if we could remove the 'groupofnames' type from project entries; I don't want to risk it.

Tue, Apr 18, 5:28 PM · MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)), Patch-For-Review, MediaWiki-extensions-OpenStackManager, Labs, Labs-Infrastructure