All masters are upgraded, and Debian base images now include puppet 4.
Tue, Dec 12
@ssastry, labtestweb and wikitech (basically anything with 'lab' in its db name) are baby little servers that are unlikely to handle automated traffic very well. Can you just exclude them from future tests?
Tue, Dec 5
Chase killed off all of jem's processes, and CPU usage is plummeting.
Fri, Dec 1
My patch to define 'puppet' works for what it is...
Full instructions for how to set up a working puppet compiler are on https://phabricator.wikimedia.org/T97081
Thu, Nov 30
Wed, Nov 29
Tue, Nov 28
Mon, Nov 27
Sun, Nov 26
@Marostegui this is on my plate but I have no idea how to proceed... is this timeout due to some long-running job that I need to locate and kill?
@Tgr: I created this project with you as a project admin. You should be able to add new members or admins as needed via wikitech -- let me know if you need anything else.
Fri, Nov 24
Thu, Nov 23
gerrit-staging contains one instance, 'gerrit-stretch' which is about 4 hours old. So someone thinks the project is still active :)
Wed, Nov 22
I think this is fixed.
@Hjfocs I've created the new project; you should be able to access it and add additional users or project admins as needed.
@Harej we don't support CamelCase project names, but I've created the 'partnermetrics' with you as an admin. You can add other users as needed.
Tue, Nov 21
Mon, Nov 20
I tried to set MW up on these systems but the puppet classes on Stretch were a total mess so I just downtimed them and set this aside. Probably someone fixed puppet and they sprang to life.
I made our standard Trusty image 'private' by default, and announced here: https://lists.wikimedia.org/pipermail/cloud-announce/2017-November/000010.html
@MarcoAurelio this appears to be a project of the design team, so I'm adding them and removing the cloud team. The site is hosted by WMCS but we don't have anything to do with decisions within the project.
Nov 8 2017
Oct 30 2017
Update: I don't think our custom hiera lookup is working properly with the 4.8 master. That's the next thing to understand/fix.
I have a 4.8 puppetmaster running on andrew-test-master-upgrade.puppet.eqiad.wmflabs. Things I've tested with it are:
Oct 27 2017
Oct 26 2017
Oct 25 2017
Oct 24 2017
I've added the 'bigdisk' flavor to this project, so now when you create a new VM you should see that as an option. Let me know if you run into any trouble.
Oct 23 2017
I've only looked at deployment-cache-text04. Puppet had been broken there for ages due to several puppet changes.
Oct 22 2017
@Sowjanyavemuri, the Outreachy coordinators that I've spoken to are firm that your schedule does not qualify you for an internship for this cycle. I encourage you to re-apply for the May program which should fit your schedule better.
Oct 20 2017
I installed 20 VMs, and ran stress-ng on each of them, like this:
Hello @Sowjanyavemuri! I'm doubt that we'll be able to bend the scheduling rules although I agree that the numbers are close. For number-crunching purposes, can you please answer each of the questions specifically? I know that these are answered in bits and pieces above but I'm a bit lost in the back-scroll.
Oct 19 2017
@Cyberpower678 Can you provide any kind of long-term estimate about how big you expect this database to grow?
Oct 18 2017
I think the VM creation failure was a (mostly? completely?) unrelated issue. I've rescheduled some actually running VMs there, and will see how they do.
cc: everyone who has been active in the toolsbeta project
The future parser has few complaints, so we're ready to move on to actual upgrade testing.
Oct 17 2017
Current theory is that this happens when the labs-private repo is in the process of being rebased.
Oct 16 2017
Oct 12 2017
Here's my latest attempt to describe what works. Once the concerned patches are merged I'll try to get this down on wikitech someplace.
Horizon can't quite delete everything yet, so I generally delete everything that Horizon can see first and then use the 'delete' link in wikitech. Horizon is /close/ to being able to do everything but it needs a bit of work.
It looks to me like this is filter in maintain-views.yaml via logging_whitelist. However, I don't see that 'pagetranslation' has ever been in that list (or at least not since 2016-10-12 which is when the history becomes murky.)
Oct 11 2017
When I refreshed puppet on the affected host, it included this diff:
Oct 10 2017
ok -- I was expecting this table to be present in enwiki. If it's wikidata-specific then we're probably done. @Ladsgroup can you confirm?
I've run maintain-views, but the wb_terms table isn't getting replicated at all. I don't see any evidence of filtering in the sanitarium files but I may be looking in the wrong place... @Marostegui, any ideas?
ok! I've raised the quota to 4 IPs. Lets' leave this task open and you can nudge me when you're ready to clean up.
We don't support CamelCase in project names, so I've created a project called 'mwstake'. @MarkAHershberger is a project admin and can add other users or admins as needed.
Do you already have ram/CPU quota to create the additional instances? Is it really just the IPs that are holding you back?
Approved, will do shortly
This seems to have been caused by https://gerrit.wikimedia.org/r/#/c/382415/, which has now been reverted.
Oct 9 2017
Oct 6 2017
I'm trying to reproduce the tools puppet compiler described here. A few things have clearly changed since this was last built... The hiera setup I seem to need looks like this:
Oct 5 2017
I ran the export and import by hand just now, and I think we're getting the complete wiki.
I've directed shinken-wm to talk in #wikimedia-cloud-feed.
A --current dump is 8.6M, a --full dump is 7.2G. So doing --full may not be practical.
Is there anything I can do to nudge this along, short of 'clone Jaime'?
These boxes are up and installed and seem ok. Actual service implementation is T168470
Adding a regex validation to the instance name in Horizon turns out to be non-trivial in the current version.
rabbit is now much quieter, so this is /maybe/ better. Closing for now, optimistically.
Wikitech is dumped using