I did some digging and found https://gerrit.wikimedia.org/r/#/c/323090/2 probably solved this. It's 1.29.0-wmf.4 so it'll get deployed to group1 wikis today and group2 tomorrow. @Tchanders?

You'll need to be more specific than 'unable', what happens instead of it saving the category?

In T142759#2823498, @jcrespo wrote:

The issue, however, has not been corrected- the tool is not updating this database correctly (maybe it lacks permissions?).

I think this is what was fixed in https://gerrit.wikimedia.org/r/#/c/322414/

Seems ve.ui.MWNoticesPopupTool is getting constructed before ve.init.mw.ArticleTarget.prototype.loadSuccess has run

It doesn't call them out explicitly as a full tab but that doesn't mean it doesn't support them. You can insert them into the internal links input using the interwiki link syntax.

In T146723#2795669, @Ottomata wrote:

Yar, udp2log was notoriously flaky, and I'm not sure if I ever worked with it in Jessie/systemd (maybe I did...?). I know this isn't what you want to hear, but I have to ask. Is there any planned effort to get mw-log off of udp2log? Is it still needed now that logstash/kibana are around? The udp2log-mw instance is the only remaining use of udp2log at the foundation.

duplicate of T150638 ?

So you want to be able to edit the edit summary, *not* preview it, when looking at the diff?

In T148561#2788886, @chasemp wrote:

Are these https://gerrit.wikimedia.org/r/#/c/320804/2/modules/role/templates/labsdb/maintain-views.yaml views that need to be exposed in every DB or some specific DB?

Is there meant to be a table in enwiki and olowiki and on

In T148561#2787065, @jcrespo wrote:

@AlexMonk-WMF, this is not a replication or Database issue, as I just demonstrated.

In T148561#2786440, @jcrespo wrote:

@Krenair may know, but it is not something he has to do (but I think he helped with the new script).

In T150092#2784615, @Andrew wrote:

It compiles properly for labtestcontrol but not for labcontrol, which I'm not yet clear on if that's a real problem or a puppet compiler problem.

In T150058#2775860, @akosiaris wrote:

@AlexMonk-WMF As @Joe said, we 've copied over the CA in production (2 months ago in fact). palladium has already been shutdown. If I had to venture a wild guess, it would be that during in the migration to the new puppetmaster in Beta, the CA was not copied over so this happened.

In T117095#2775967, @Kelson wrote:

We can not really do monthly snapshot (what would a good thing) and we have pretty serious difficulties to create ZIM files of the biggest Wikipedias (EN, DE, FR). The reason are the limitation in hardware resources. The problem is that we have a quota which blocks us to request one or two more large VMs from the XL/large setup (and there is still a doubt that even the large VM will be able to create a ZIM of WPEN within a month due to CPU and/or storage limitation).

In T143136#2772063, @bd808 wrote:

In T143136#2770926, @yuvipanda wrote:

The script works, but is disabled right now. I need to figure out where to put the output JSON. Options are:

1. Somewhere under wikitech.wikimedia.org
2. Somewhere under horizon.wikimedia.org
3. An entirely new domain just for this

Thoughts?

Under horizon is better than under wikitech in the longer term I think unless it can just be made into a tool?

is this a duplicate of T49515: Add access to nova's admin api?

In T150092#2774144, @Andrew wrote:

In T150092#2774119, @AlexMonk-WMF wrote:

In T150092#2774103, @AlexMonk-WMF wrote:

We should probably disallow logins from instance IPs on usernames other than whitelisted, special-purpose accounts like novaobserver. I'm pretty sure we do NOT want individual user's LDAP passwords being exposed to labs instances (anyone's accounts, but also in a lot of cases user's LDAP passwords provide access to production sensitive data via the wmf/nda/ops LDAP groups)

Actually, I'll go further: We should lock (via changing the password to something random) any user other than a whitelisted account that managed to get a successful username+password match from a labs instance IP. Then automatically notify operations/security of a potential breach along with the username and a list of their groups.

@Krenair, do you have in mind a facility for blocking account access by name and IP?

In T150092#2774103, @AlexMonk-WMF wrote:

We should probably disallow logins from instance IPs on usernames other than whitelisted, special-purpose accounts like novaobserver. I'm pretty sure we do NOT want individual user's LDAP passwords being exposed to labs instances (anyone's accounts, but also in a lot of cases user's LDAP passwords provide access to production sensitive data via the wmf/nda/ops LDAP groups)

I'm pretty sure the Nova half of #1 is unnecessary, instances can already hit the nova API (it runs on labnet), they just can't use it due to lack of Keystone access:

krenair@bastion-01:~$ curl http://labnet1001.eqiad.wmnet:8774/v2
Authentication required

krenair@bastion-01:~$ curl -v http://labcontrol1001.wikimedia.org:5000/v3
* Hostname was NOT found in DNS cache
*   Trying 208.80.154.92...

@akosiaris, this cert changed because we changed puppetmasters to a different host. I'm vaguely aware of paladium being retired in production, is this issue going to occur there too?

No, the proper version of the script was replaced in T138450, Ori's DNM version of the old script was useful for this but is no longer necessary.

I shut down the old instance, will probably delete it in a week or two

In T133096#2758125, @Jdforrester-WMF wrote:

We're pretty sure that this is a Beta Cluster infrastructure bug.

Works for me if I disable NWE

Okay. What sort of private data (beyond credentials to external systems) is stored directly on stat1002?

In T149228#2748977, @Ottomata wrote:

• stat1002 - compute node, lots of storage, with private data and Analytics Cluster (Hadoop) access.

We don't expect wikis to actually create pages for tracking categories, we categorise pages in them regardless IIRC. But there is still a problem here - it looks like https://gerrit.wikimedia.org/r/#/c/315465/ accidentally introduced code that adds the tracking category without the appropriate messages to define what the category should be called.

Without an etag we can't ask for html->wikitext snippet conversions via the RB/VirtualRB.

I'm not quite sure, but I think it might mean dirty diffs if you veaction=editsource, make changes, then switch to VE and save?

@GWicke, @mobrovac: So the change was merged but https://beta.wikiversity.org/api/rest_v1/page/html/Main_Page still returns 404. I know RB isn't restarted by puppet like every other service, was that done after the puppet change applied to the servers?

You forgot to specify the correct distribution for jsub. Leaving it unspecified defaults to precise, which is deprecated and will have everything on older versions. The login hosts run trusty:

krenair@tools-bastion-03:~$ jsub -l release=trusty mono --version
Your job 2392342 ("mono-sgen") has been submitted
krenair@tools-bastion-03:~$ cat mono-sgen.out
Mono JIT compiler version 3.2.8 (Debian 3.2.8+dfsg-4ubuntu1.1)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

In T148861#2737349, @Psychoslave wrote:

How long may the deployment train delay be?

Um, the entire domain appears to be missing from the config file. How did this get through?

In T135747#2734583, @RandomDSdevel wrote:

In T135747#2624766, @AlexMonk-WMF wrote:

In T135747#2624486, @RandomDSdevel wrote:

@AlexMonk-WMF: Thanks for the pointers! Are you sure, though, that telemetry will only be sent as part of the wiki installation process? The message on the mailing list seems to indicate otherwise, and that makes me suspect that the telemetry data could be extended to cover things like a list of installed extensions and gadgets and additional statistics like those shown on Special:Version.

It looks to me like it is only ever sent once per wiki (perhaps not from the installer itself, but still). You should ask them though.

@AlexMonk-WMF: Just on IRC or the relevant mailing list some time soon, right? (Hopefully I don't forget amidst the mess that is life; LOL.) Or would I have to open an issue up here in Phabricator since my question might get construed as a sort of feature request?

I suppose we go from one list that needs updating when private wikis are created to one that needs updating when non-private (the majority) wikis are created, in a repository we need to update anyway. Slightly irritating but I can live with that.

In T148560#2730717, @jcrespo wrote:

Right now we maintain a blacklist on realm.pp. We should transform that into a white list- it is technically easy, but it requires some checking with the mediawiki private lists and some human checking.

That might be it? It doesn't have any direct or indirect dependencies on user.options, but it is a sibling of user.options under ext.visualEditor.mediawiki's dependencies

Note deployment requires operations meeting approval (it has sudo privileges) whereas mw-log-readers is just the 3 day wait

(a script run would also handle wikimania2017wiki_p and tcywiki_p which are currently missing)

also note deployment access involves sudo privileges, and so requires more than just the three day waiting period as ops have to discuss it at their weekly meeting

In T148477#2726307, @dr0ptp4kt wrote:

For future reference, what would be the best way to word the request to reflect that?

First this ticket needs to identify groups (or, in the absence of any existing one, specific rights on specific servers - https://wikitech.wikimedia.org/wiki/Production_shell_access#New_users explains this as "A detailed reason for your request. In particular, describe which specific servers you need access to and why."), then management approval can be sought, and then (and I would propose, only then) the three day period can begin. Right now it appears to be approved by @dr0ptp4kt as a blank cheque for any type of production shell - that is vague and can range from login to bastions (which doesn't really make sense on it's own) up to full-sudo-as-root-across-the-cluster (which I suspect would not be given).

yeah, agreed

Can work around this issue using the 'classes' hiera attribute

Well, it doesn't get unset. Are you expecting it to be overwritten? It's not returned to the client by action=visualeditoredit's jsconfigvars return data for overwriting, that comes from action=parse's jsconfigvars prop

If it's fixed on master why not just close this?

In T115194#2714378, @hashar wrote:

Alex can you do the magic SELECT again and see whether DNS entries are still being leaked?