Wed, Jan 16
These are now shut down, can be deleted pretty much whenever.
Going forward is looking more promising now! Closing this.
Tue, Jan 15
We can approve this for 200Gb of space. I can't promise that we'll be able to give you a Tb anytime soon but if you're able to demonstrate that the project is useful and popular we can revisit.
Mon, Jan 14
Most of these seem valid, if broken, so I guess they're left to the project owners to deal with.
To test: alias tools.wmflabs.org to 188.8.131.52 (aka tools-proxy-03). Then hit a tool with your browser, and check out /var/log/nginx on tools-proxy-03.tools.eqiad.wmflabs
I have two new nodes, tools-proxy-03 and tools-proxy-04 that are syncing with redis and /should/ be able to serve up webservices. They can't talk to k8s services though.
Sun, Jan 13
Sat, Jan 12
Fri, Jan 11
Thu, Jan 10
After this patch, we should be able to rebuild cloudnet1003 and cloudnet1004 as mitaka/stretch. @Andrew let me know if this is OK
Tue, Jan 8
I emailed most of the above users (everyone with more than ~50k of files) with notice that they have 7 days to do a rescue. A few people have followed up.
Mon, Jan 7
Sat, Jan 5
Thu, Jan 3
Wed, Jan 2
I did run that. I don't know why the _p tables aren't updating properly. Assigning to @Banyek to investigate.
Done -- sorry for the delay.
Well... this started working again and I didn't change anything :(
Chase points out that of course all this traffic goes via the bastion, so that's where I need to look.
Agreed, this hasn't been an issue in quite a while.
I don't see any evidence that this is still happening. Neglect pays off!
Mon, Dec 24
Huh. /etc/hosts pretty much always has a reference to the local hostname. For example:
Sat, Dec 22
Sorry for the delay -- we were waiting on new hardware to add additional projects. I've now created this project; @Est31 is projectadmin and can add additional users as necessary.
Sorry for the delay! This was stalled for lack of hardware but we got a new server online a couple of days ago so I've created the project now. @Videowiki123, you're a project admin and can add other users as you see fit. Note that the 'eqiad' region is being deprecated so you'll want to be sure to select the eqiad1-r region in the project selector before you try to create anything.
Fri, Dec 21
Dec 20 2018
cloudvirt1025 is working properly. The others are stuck in limbo while Arturo and I figure out what to do about stretch vs. jessie.
Dec 19 2018
It's been doing this for a while. There's a hacked version of bootstrap-vz running there ( https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/VM_images#Custom_bootstrap-vz_package ) so I'm inclined to just leave it to suffer in its current state.
In theory those web proxies are updated automatically as part of the migration. I think that what you saw was probably a transitional state -- it seems to be working fine for me now.
Dec 18 2018
Thanks @cscott ! To confirm -- is it OK if I delete this VM and project? Do we want to confirm with Tim before I take action?
I've enabled the new region for this project in case someone wants to create a new Stretch vm in eqiad1-r.
It might be worth looking at https://github.com/cernops/nova-quota-sync/blob/master/nova-quota-sync but really I'm holding out hope that this will make more sense in Newton (although the pike fix is the only smart one!)
Quota calculation is pretty broken in Mitaka; the situation in Newton is mildly better. I just now tried an old fix that used to work (zeroing out the values in the database) and that is making Horizon do different but equally wrong things. Part of this is that nova is trying to keep track of per-user/per-project quotas rather than just per-project quotas even though that isn't really a thing.
@GoranSMilovanovic Thank you for the explanation. We're not likely to take any immediate action with that VM; we'll discuss this and try to get back to you with an adjusted timeline. If you're able to rebuild things on Stretch before we get back to you, all the better :)
Note that on T204695, the wikidataconcepts admin has asked for more time. It seems reasonable to give it to them but we should figure out what the adjusted deadline is and communicate it to them.
Thanks @Tgr! I've deleted this project; we can create new ones as needed.
Dec 17 2018
@Krenair points out that these aren't displayed for the old region either, so this isn't really a bug, just a design choice.
Dec 16 2018
Dec 14 2018
@dschwen, if you reboot maps-wma your nfs mounts should be more like what you'd expect now.
@faidon, who is 'please also construct a draft email' directed to?
I've added an 'Edit Class Parameters' button to this UI, and also (I think) fixed it so it doesn't clobber hiera settings when you apply a class. Let me know how it works!
Project deleted. Thanks!
Hello! I was just about to move your old VM ('wigi.wikidumpparse.eqiad.wmflabs') to the new cloud region, but I see here that you've already built a replacement for it. Can 'wiki' just be deleted rather than moved?
Dec 13 2018
I've deployed @Krenair 's fix for this and it looks good.
It's not perfect but Horizon now does all the things we need. We also have a project tag now, so closing this top-level task.
This is done. The new panel is in gerrit, named 'wmf-member-dashboard'
I deleted those Trusty VMs.