Andrew (Andrew Bogott)
User

Projects (6)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (124 w, 5 d)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott

Recent Activity

Yesterday

Andrew renamed T161006: Convince nova-scheduler to pay attention to CPU metrics from "Rebalance tools exec nodes with an eye towards CPU usage" to "Convince nova-scheduler to pay attention to CPU metrics".
Fri, Mar 24, 9:10 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T159835: Labvirt1001 has insanely slow IO.

The current state of this is: I rebooted labvirt1001 and it got better. I've migrated a handful of tools exec nodes back to labvirt1001 and I'm going to keep an eye on it for a few weeks. If everything is still good come mid-April, we can just shrug and repool it.

Fri, Mar 24, 9:08 PM · ops-eqiad, Operations, Labs-Infrastructure, Labs
Andrew closed T159141: check on the nova-api upstart logs as "Resolved".

This seems to actually work properly now.

Fri, Mar 24, 9:05 PM · Patch-For-Review, Labs
Andrew triaged T161327: bootstrap_vz: Move firstboot.sh out of the base image? as "Normal" priority.
Fri, Mar 24, 9:04 PM · Labs, Labs-Infrastructure
Andrew triaged T161006: Convince nova-scheduler to pay attention to CPU metrics as "Normal" priority.
Fri, Mar 24, 9:04 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew triaged T158103: Horizon Mitaka 'remember me' checkbox immune to keyboard focus as "Low" priority.
Fri, Mar 24, 9:03 PM · Developer-notice, Horizon, Labs
Andrew added a comment to T161265: Rename labtestmetal2001.

I must've done the cleanup out of order... I removed things again and they seem to be actually gone now.

Fri, Mar 24, 8:44 PM · hardware-requests, Operations
Andrew added a comment to T161327: bootstrap_vz: Move firstboot.sh out of the base image?.

(And we should have some kind of validation for the script, probably by putting a hash in the nova metadata.)

Fri, Mar 24, 6:47 PM · Labs, Labs-Infrastructure
Andrew created T161327: bootstrap_vz: Move firstboot.sh out of the base image?.
Fri, Mar 24, 6:46 PM · Labs, Labs-Infrastructure

Thu, Mar 23

Andrew created T161265: Rename labtestmetal2001.
Thu, Mar 23, 10:14 PM · hardware-requests, Operations

Wed, Mar 22

Andrew added a comment to T159835: Labvirt1001 has insanely slow IO.

And post-reboot it's fast again dammit

Wed, Mar 22, 9:26 PM · ops-eqiad, Operations, Labs-Infrastructure, Labs
Andrew added a comment to T159835: Labvirt1001 has insanely slow IO.

The one symptom I'm fixating on is puppet runs. A puppet run on labvirt1001 takes 811.99.

Wed, Mar 22, 9:15 PM · ops-eqiad, Operations, Labs-Infrastructure, Labs
Andrew added a comment to T159835: Labvirt1001 has insanely slow IO.

@hashar, it's nothing to do with load. there are no VMs running on labvirt1001 and it still has the problem.

Wed, Mar 22, 9:01 PM · ops-eqiad, Operations, Labs-Infrastructure, Labs
Andrew added a comment to T146308: Kill limn1 .

Any update on this?

Wed, Mar 22, 1:41 PM · Analytics-Kanban
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Wed, Mar 22, 1:40 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew created T161071: 'novaobserver' user missing from labtest.
Wed, Mar 22, 2:01 AM · Labs

Tue, Mar 21

Andrew added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

It is actually possible to explicitly tell the scheduler to not put multiple nodepool instances on the same labvirt. That would work if the total number of nodepool instances is always < the number of labvirts, which I'm not sure is true.

Tue, Mar 21, 10:18 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew created T161006: Convince nova-scheduler to pay attention to CPU metrics.
Tue, Mar 21, 4:16 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew closed T160995: Wikitech 'Requested domain is invalid' as "Resolved".
Tue, Mar 21, 3:02 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Labs, MediaWiki-extensions-OpenStackManager, Patch-For-Review
Andrew added a project to T160995: Wikitech 'Requested domain is invalid': Labs.
Tue, Mar 21, 3:00 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Labs, MediaWiki-extensions-OpenStackManager, Patch-For-Review
Andrew created T160995: Wikitech 'Requested domain is invalid'.
Tue, Mar 21, 2:05 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Labs, MediaWiki-extensions-OpenStackManager, Patch-For-Review
Andrew triaged T154860: User dschwen unable to log into horizon.wikimedia.org (An error occurred authenticating. Please try again later.) as "Normal" priority.
Tue, Mar 21, 1:41 PM · Labs, Horizon
Andrew reopened T154860: User dschwen unable to log into horizon.wikimedia.org (An error occurred authenticating. Please try again later.) as "Open".

Fixing dschwen's login was a bit of a hack... I'd like to keep this open until the actual cause of the issue is addressed.

Tue, Mar 21, 1:41 PM · Labs, Horizon
Andrew closed T160929: Provision novaobserver credentials on all Labs hosts as "Resolved".
Tue, Mar 21, 12:05 AM · User-bd808, Labs

Mon, Mar 20

Andrew added a comment to T143639: Write a simple script that handles failovering proxies.

This only barely warrants a script, since I just now did it with a single command:

Mon, Mar 20, 9:10 PM · Wikimedia-Incident, Tool-Labs, Labs-Infrastructure, Labs
Andrew added a comment to T160929: Provision novaobserver credentials on all Labs hosts.

I think that the already-existing openstack::observerenv class is just what we want here. I just forgot that it was separate.

Mon, Mar 20, 8:18 PM · User-bd808, Labs
Andrew closed T160862: New Public IP for WikiApiary on WMFLabs as "Invalid".

It sounds like you don't need a quota change, so I'm closing this for now. Feel free to open a quota request if you turn out to really need the IP -- I suspect you'll find the proxy system better though :)

Mon, Mar 20, 2:55 PM · WikiApiary, Labs
Andrew closed T160862: New Public IP for WikiApiary on WMFLabs, a subtask of T140904: Existing Labs project quota increase requests (Tracking), as "Invalid".
Mon, Mar 20, 2:55 PM · Tracking, Labs
Andrew closed T160862: New Public IP for WikiApiary on WMFLabs, a subtask of T149874: move WikiApiary to Labs, as "Invalid".
Mon, Mar 20, 2:55 PM · WikiApiary, Developer-Relations, Labs-project-other
Andrew renamed T160798: Revert temporary quota increase for fastcci project (when ready) from "Temporary quota increase for fastcci project" to "Revert temporary quota increase for fastcci project (when ready)".
Mon, Mar 20, 2:54 PM · Labs
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Mon, Mar 20, 2:42 PM · Patch-For-Review, Labs-Infrastructure, Labs

Fri, Mar 17

Andrew added a comment to T160798: Revert temporary quota increase for fastcci project (when ready).

@dschwen, I think I increased your quota enough for what you need... let me know if it's not enough. Otherwise, let me know when you're cleaned up and I'll drop the quota back down.

Fri, Mar 17, 10:30 PM · Labs
Andrew claimed T160798: Revert temporary quota increase for fastcci project (when ready).
Fri, Mar 17, 10:30 PM · Labs
Andrew created T160798: Revert temporary quota increase for fastcci project (when ready).
Fri, Mar 17, 10:24 PM · Labs
Andrew claimed T154860: User dschwen unable to log into horizon.wikimedia.org (An error occurred authenticating. Please try again later.).
Fri, Mar 17, 10:22 PM · Labs, Horizon
Andrew added a comment to T154860: User dschwen unable to log into horizon.wikimedia.org (An error occurred authenticating. Please try again later.).

This is because of a case mismatch between ldap and mediawiki. The mediawiki user_name table had the username 'Dschwen' but ldap had the cn as 'dschwen'.

Fri, Mar 17, 10:21 PM · Labs, Horizon
Andrew closed T139954: Strategies to avoid OOM on labvirt hosts as "Resolved".

With changes to our provision ration this hasn't been an issue anymore.

Fri, Mar 17, 5:08 PM · Labs-Infrastructure, Labs
Andrew closed T94500: bigbrother doesn't stop as "Resolved".

Yes, I think this is resolved.

Fri, Mar 17, 3:21 PM · Patch-For-Review, Labs, Tool-Labs

Thu, Mar 16

Andrew added a comment to T154860: User dschwen unable to log into horizon.wikimedia.org (An error occurred authenticating. Please try again later.).

Most often this is a result of clock drift on the device providing the 2fa code. Rebooting your phone might help.

Thu, Mar 16, 8:15 PM · Labs, Horizon
Andrew added a comment to T157710: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted.

@Petrb, I'm online now (which is, I believe, 10AM EDT) and will be for six hours at least. I should be available tomorrow during at least the same window.

Thu, Mar 16, 2:12 PM · Huggle, Labs

Wed, Mar 15

Andrew closed T148781: Clean up ldap host entries and references as "Resolved".

I removed all of the ldap host entries.

Wed, Mar 15, 5:36 PM · Patch-For-Review, Labs, LDAP
Andrew closed T148781: Clean up ldap host entries and references, a subtask of T138150: Purge stale data from LDAP, as "Resolved".
Wed, Mar 15, 5:36 PM · LDAP, Labs
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Wed, Mar 15, 5:10 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T157710: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted.

@Petrb, I completed the in-place upgrade on Huggle and it looks ok to me...

Wed, Mar 15, 3:31 PM · Huggle, Labs

Tue, Mar 14

Andrew closed T157760: Review OpenStack monitoring options w/out Mirantis packages as "Resolved".

I moved the related tests to labnet and nrpe -- they seem to be working fine.

Tue, Mar 14, 8:27 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew closed T160344: Request increased quota for recommendation-api labs project as "Resolved".

I've increased your quotas to allow one additional 'bigram' instance. Let me know if I missed anything.

Tue, Mar 14, 3:38 PM · Recommendation-API, Labs
Andrew closed T160344: Request increased quota for recommendation-api labs project, a subtask of T140904: Existing Labs project quota increase requests (Tracking), as "Resolved".
Tue, Mar 14, 3:38 PM · Tracking, Labs
Andrew closed T152518: ldap userkeys broken on labtest as "Invalid".

I just tinkered with my .ssh/config and now this works fine.

Tue, Mar 14, 2:22 PM · Labs
Andrew closed T158099: Upgrade Openstack Horizon to Mitaka as "Resolved".

This is done on Californium and seems fine.

Tue, Mar 14, 2:09 PM · Developer-notice, Patch-For-Review, Horizon, Labs
Andrew added a comment to T157838: Move wm-bot instance to Trusty.

Are there still pending tasks here, or is this resolved?

Tue, Mar 14, 4:13 AM · Labs, WM-Bot

Mon, Mar 13

Andrew added a comment to T159846: Wikmaps Warper - Migrate / Upgrade maps-warper from Precise to Trusty.

@Chippyy Any progress on this? There are two weeks remaining until we start deleting Precise instances.

Mon, Mar 13, 4:07 PM · wikimaps-warper
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Mon, Mar 13, 4:03 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew closed T159737: Labs instance utrs-primary is running Ubuntu Precise and must be rebuilt. as "Resolved".

This instance was deleted and replaced by utrs-database and utrs-production.

Mon, Mar 13, 4:02 PM · Labs-Infrastructure, Labs
Andrew closed T159737: Labs instance utrs-primary is running Ubuntu Precise and must be rebuilt., a subtask of T143349: Deprecate precise instances in Labs by 2017-03-31, as "Resolved".
Mon, Mar 13, 4:02 PM · Patch-For-Review, Labs-Infrastructure, Labs

Fri, Mar 10

Andrew closed T131548: Sort out our libvirt qcow2 hack with the upstream as "Resolved".

I wasn't able to get this change merged upstream. Removing it on a labvirt seems to slightly increase disk usage, but I think it's worth it to have a more-canonical install.

Fri, Mar 10, 8:55 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew closed T131548: Sort out our libvirt qcow2 hack with the upstream, a subtask of T131322: deployment-upload won't start, upload.beta.wmflabs.org down, as "Resolved".
Fri, Mar 10, 8:55 PM · Patch-For-Review, Labs, Beta-Cluster-Infrastructure
Andrew added a subtask for T143349: Deprecate precise instances in Labs by 2017-03-31: T157838: Move wm-bot instance to Trusty.
Fri, Mar 10, 3:30 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a parent task for T157838: Move wm-bot instance to Trusty: T143349: Deprecate precise instances in Labs by 2017-03-31.
Fri, Mar 10, 3:30 PM · Labs, WM-Bot
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Fri, Mar 10, 3:30 PM · Patch-For-Review, Labs-Infrastructure, Labs

Thu, Mar 9

Andrew added a comment to T159990: Remove linux kernel 3.16 from the jessie image on labs.

I just built four different jessie instances, ran 'apt get update && apt-get upgrade' on them and rebooted. All four came up, no problems.

Thu, Mar 9, 5:43 PM · Operations, Labs
Physikerwelt awarded T149109: Account creation failing with "The authentication plugin denied the account creation." a Like token.
Thu, Mar 9, 6:17 AM · Patch-For-Review, Labs, wikitech.wikimedia.org

Tue, Mar 7

Andrew added a comment to T159141: check on the nova-api upstart logs.

There doesn't seem to be a good way to win this one. I already added 'delaycompress' to the logrotate script to prevent cronspam (https://gerrit.wikimedia.org/r/#/c/313558/) but now upstart just writes to the .1 file forever.

Tue, Mar 7, 11:24 PM · Patch-For-Review, Labs
Andrew closed T156655: horizon puppet panel vs. hiera parsing as "Invalid".

I'm not sure this is a real bug so much as me misunderstanding yaml.

Tue, Mar 7, 10:01 PM · Labs
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Tue, Mar 7, 9:48 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Tue, Mar 7, 9:47 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a subtask for T143349: Deprecate precise instances in Labs by 2017-03-31: T159846: Wikmaps Warper - Migrate / Upgrade maps-warper from Precise to Trusty.
Tue, Mar 7, 5:47 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a parent task for T159846: Wikmaps Warper - Migrate / Upgrade maps-warper from Precise to Trusty: T143349: Deprecate precise instances in Labs by 2017-03-31.
Tue, Mar 7, 5:47 PM · wikimaps-warper
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Tue, Mar 7, 5:47 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew renamed T159843: Revert increased quota for maps labs project from "Request increased quota for maps labs project" to "Revert increased quota for maps labs project".
Tue, Mar 7, 4:23 PM · Labs
Andrew added a comment to T159843: Revert increased quota for maps labs project.

I granted this. Will rename the bug to reflect future quota reduction after the precise instance is cleaned up.

Tue, Mar 7, 4:23 PM · Labs
Andrew added a comment to T159843: Revert increased quota for maps labs project.

This is for one Large size instance: 16G and 8 CPUs. And, yes, we'll lower the quota after the corresponding precise instance is gone.

Tue, Mar 7, 4:20 PM · Labs
Andrew renamed T159843: Revert increased quota for maps labs project from "Request increased quota for <Replace Me> labs project" to "Request increased quota for maps labs project".
Tue, Mar 7, 4:18 PM · Labs
Andrew created T159843: Revert increased quota for maps labs project.
Tue, Mar 7, 4:18 PM · Labs
Andrew closed T57691: Rename project bots to wm-bot as "Resolved".

We have a new project, wm-bot, where wm-but is being moved.

Tue, Mar 7, 4:04 PM · Labs, Wikimedia-Labs-General
Andrew added projects to T159835: Labvirt1001 has insanely slow IO: Operations, ops-eqiad.
Tue, Mar 7, 3:24 PM · ops-eqiad, Operations, Labs-Infrastructure, Labs
Andrew created T159835: Labvirt1001 has insanely slow IO.
Tue, Mar 7, 3:23 PM · ops-eqiad, Operations, Labs-Infrastructure, Labs

Mon, Mar 6

Andrew added a comment to T157710: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted.

@Petrb I can't tell what you're saying due to lack of punctuation... maybe "Don't! Instances in huggle project host some essential services!" or maybe "Don't instances in huggle project host some essential services?"

Mon, Mar 6, 10:10 PM · Huggle, Labs
Andrew added a comment to T157710: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted.

If no one lays claim this week I'll probably shut this instance off next Monday, just to see if anyone notices and/or cares.

Mon, Mar 6, 7:55 PM · Huggle, Labs
Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Mon, Mar 6, 7:21 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew created T159737: Labs instance utrs-primary is running Ubuntu Precise and must be rebuilt..
Mon, Mar 6, 7:19 PM · Labs-Infrastructure, Labs
Andrew closed T159309: Request creation of GLAMpipe labs project as "Resolved".

I'm afraid that the camelcase name isn't supported, but I've created the 'glampipe' project with Zache-tool as the project admin. @Zache, you can add additional users or admins as appropriate.

Mon, Mar 6, 4:46 PM · Labs
Andrew closed T159309: Request creation of GLAMpipe labs project, a subtask of T76375: New Labs project requests (tracking), as "Resolved".
Mon, Mar 6, 4:46 PM · Tracking, Labs
Andrew created T159721: labvirt1001 and 1002 cannot launch new VMs.
Mon, Mar 6, 3:58 PM · Labs-Infrastructure, Labs
Andrew closed T159068: Request creation of Wikimedia Incubator labs project as "Resolved".

Sorry for the delay in creation, I was sick most of last week. This project has been created, with @Hydriz as the projectadmin.

Mon, Mar 6, 2:54 PM · Labs
Andrew closed T159068: Request creation of Wikimedia Incubator labs project, a subtask of T76375: New Labs project requests (tracking), as "Resolved".
Mon, Mar 6, 2:54 PM · Tracking, Labs

Fri, Mar 3

Andrew created T159536: Puppet constantly trying to stop the already stopped puppetmaster process on Trusty.
Fri, Mar 3, 3:32 PM · Operations
Andrew added a comment to T159524: backup space is used unwisely.

There's definitely no need to backup labtestweb. Silver is important to back up since it contains our techincal documentation... we have an offsite backup of it at https://wikitech-static.wikimedia.org/wiki/Main_Page but as far as I know that does not preserve all the edit history.

Fri, Mar 3, 3:11 PM · Operations

Mon, Feb 27

Andrew added a comment to T158299: HTTP 500 on Special:NovaSudoers.

It's not a memory issue, that page is just too damn big if 'tools' is selected in the filter. If I increase max_execution_time then it loads just fine

Mon, Feb 27, 9:33 PM · wikitech.wikimedia.org, Labs, MediaWiki-extensions-OpenStackManager
Andrew added a comment to T70100: Sudo Policies can't be displayed for Tools.
Mon, Feb 27, 7:26 PM · MediaWiki-extensions-OpenStackManager
Andrew added a comment to T157710: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted.

Thanks y'all

Mon, Feb 27, 6:13 PM · Huggle, Labs
Andrew added a comment to T143349: Deprecate precise instances in Labs by 2017-03-31.

Email nag sent to labs-announce on 2017-02-27

Mon, Feb 27, 5:52 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T159068: Request creation of Wikimedia Incubator labs project.

(Project request is approved but we need more info re: the floating IP request)

Mon, Feb 27, 4:53 PM · Labs
Andrew added a comment to T159068: Request creation of Wikimedia Incubator labs project.

Can you explain more about needing a public IP for parsoid? Can't the parsoid service run behind a port-specific proxy? It's http right?

Mon, Feb 27, 4:30 PM · Labs
Andrew created T159141: check on the nova-api upstart logs.
Mon, Feb 27, 3:11 PM · Patch-For-Review, Labs
Andrew closed T158645: Request creation of wikidiff2-wmde-dev labs project, a subtask of T76375: New Labs project requests (tracking), as "Resolved".
Mon, Feb 27, 4:45 AM · Tracking, Labs
Andrew closed T158645: Request creation of wikidiff2-wmde-dev labs project as "Resolved".

@jkroll, I just now created this project. You are the only member at the moment but you can add other members or projectadmins as needed.

Mon, Feb 27, 4:45 AM · Labs

Sat, Feb 25

Andrew created T159021: Wikitech error when adding users to projects.
Sat, Feb 25, 12:56 AM · Labs-Infrastructure, Labs
Andrew edited the description of T150091: Support project creation without OpenStackManager.
Sat, Feb 25, 12:53 AM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Patch-For-Review, Labs-Infrastructure, Labs

Fri, Feb 24

Andrew edited the description of T143349: Deprecate precise instances in Labs by 2017-03-31.
Fri, Feb 24, 4:25 PM · Patch-For-Review, Labs-Infrastructure, Labs
Andrew added a comment to T158970: Labs instance snuggle-en.snuggle.eqiad.wmflabs needs to be upgraded, replaced, or deleted.

Aaron, I know we talked about this already, but the clock is ticking and I'd appreciate an update. Thanks!

Fri, Feb 24, 4:24 PM · Labs