So i guess someone is sending spam subject lines to wikimedia-gh, with a forged from address of wikimedia-co@lists.wikimedia.org, in order for the mailing list software to resend the spam in the form of a pending moderation message. That's a really cute trick.

Yes it was my ip.

I'm not actually sure if it comes from the MaxMind db, and if so what db, but if it does - https://www.maxmind.com/en/geoip2-city-database-accuracy gives some accuracy numbers. Although that's averages per country, I imagine major urban centers are much more accurate than rural places.

In T152297#3360412, @Jseddon wrote:

Do we know how accurate the data is for different countries? What level of
false positives would we be looking at?

Seddon

In T165540#3360407, @Tgr wrote:

@Bawolff do you think that has been resolved now that a notification was sent to wikitech-l or should there be more / elsewhere?

The geoip cookie already records the city (And a lat/long), so the data is already there

In T165540#3359804, @Tgr wrote:

This discussion has become hard to follow, so I'll try to summarize:

• there was lots of debate over whether adding CODE_OF_CONDUCT.md to (almost) every repo is a good or bad idea. Quick tally: 16 were for it (I'm also counting people who awarded tokens), 10 against. (There were maybe half a dozen comments which could be read as implicitly implying one or the other; I tried to err on the side of not overinterpreting and did not count those.) So there was consensus to add the files, although not an overwhelming one. Several people pointed out though that adding the file would be weird or harmful for certain deployment-related repos (such as Debian packaging repos). So we should keep the files which have been added, and probably add CoC files to the other repos as well, as long as we can avoid build/deployment type repos.
• several people objected against pushing directly to the repo (and thus not leaving an audit trail in gerrit). Several other people objected against using gerrit changesets which would cause a torrent of notification mails, and DoS Jenkins with pointless build jobs. Nobody objected to Chad's proposal in T165540#3316733 to use auto-merge so I am assuming that's the consensus approach.
• some argued against using markdown, or a long filename, but neither opinion was popular.

re: the file header I'm assuming it'd be set for both the original and carried over to thumbnails, and purged from all once the file is patroled

In T167400#3356482, @Poyekhali wrote:

Unless Commons have a lot of new file patrollers who really mark new files as patrolled when they are finished reviewing (which I see is not the case), I wouldn't support Bawolff's counter-proposal.

In T167400#3354974, @BBlack wrote:

In T167400#3350622, @Bawolff wrote:

Wikipedia Zero traffic is tied to IP addresses, not users. So it definitely could be performant. Have MediaWiki set an unpatrolled header and purge on patrol. Then (somehow) configure Varnish to understand WP0 IP ranges and block if the unpatrolled header is set.

So the idea would be:

• MediaWiki sets something along the lines of MediaWiki-patrol-status: unpatrolled in File::getContentHeaders()
• Varnish looks for that header when getting files from swift. If the file is unpatrolled, and (maybe) its above a certain size, and the IP address is Zero-rated: Give a 403. Also make sure that cache varrying is set for unpatrolled files based on Zero-ratedness of IP
• On patrol, MediaWiki makes swift backend remove the header, and sends purge to varnish.
  
  Downsides: If anyone using a zero-rated connection is a file patroller, they won't be able to see the file.

Why restrict this mechanism to Zero, making Zero different from other access? We could instead deny access to unpatrolled files for users that aren't logged-in. This would fix the problem at a broader scope re: copy- and other vio content spreading widely before patrolling, since the vast majority of readers are anonymous. Because Patrollers would have accounts, it eliminates that downside as well. This counter-proposal faces a few distinct problem areas of its own, though:

1. I don't know off the top of my head, but there might be complications around CentralAuth and cross-wiki auth/session tokens. If a user has only logged in via enwiki, then sends a request to upload.wikimedia.org for a multimedia file which might be embedded, does that request always send some kind of validate-able CentralAuth session/token information?
2. Currently Varnish's notion of logged-in-ness is very naive - it just looks for a seemingly valid Cookie, and could easily be faked by sending bogus Cookie values. I think this is still sufficient for the majority case though (anon users that don't understand how to set custom cookies or headers to evade our checks). We've talked before (for many other reasons than this particular case) about having varnish sign the outbound session cookie values to validate them in a distributed fashion (it would reject invented ones and strip its signature from valid ones before forwarding on to MW), but that's probably pretty far down our technical roadmap at present.
   
   Back to the meat of the proposal, though: we can do something similar to what you're describing, but not directly via actual cache hash/vary mechanisms. It's only possible to Vary a cache on attributes of a request, not attributes of a response. Thus, it's possible to vary on "Is this a Zero user?" and similarly "Is this a logged-in user?", but it's not possible to sanely vary on "What value is Swift currently sending in the MediaWiki-patrol-status: header of this response?". What we can do is explicitly and dynamically block the response with a 403 under those conditions, something like this pseudocode:
   
   ` sub just_before_final_content_delivery_to_user { // If response data is unpatrolled, deny access to Zero users if (resp.http.MediaWiki-patrol-status != "patrolled" && req.http.X-Carrier) { return synth(403, "Access denied") } } `
   
   However, this carries the caveat that there's no explicit realtime update to the patrol status of a cached file. The first time any user fetches the file through a given cache (regardless of Zero status), the file will be loaded into cache and kept, and the code above simply masks the response to the user when appropriate. Therefore, we'd also need to trigger a cache invalidation on the file when the patrolled status changes, just like we would on content replacement. Otherwise the initial unpatrolled state will persist too long in the cache. We could perhaps hack around that a bit by giving unpatrolled files a shorter TTL cap (e.g. 10 minutes instead of the usual 24h), but invalidate-on-patrol would be much cleaner and saner.

Maybe dupe of T121797?

A bigger question is where to store the hashes. Usually perceptual hashes are compared using hamming distance. This is sort of inefficient to do in a traditional mysql database. I remember years ago that manybubbles talked about how it would make sense to use elasticsearch as the storage backend for image similarity, so itd probably be useful to look into that direction

Wikipedia Zero traffic is tied to IP addresses, not users. So it definitely could be performant. Have MediaWiki set an unpatrolled header and purge on patrol. Then (somehow) configure Varnish to understand WP0 IP ranges and block if the unpatrolled header is set.

Oh, i see, the A record points to a page which is hosted by the registrar for .wiki (who happens to use aws- i didnt originally think of navigating to 54.148.61.160 to find out more info...), so its essential some domain parking service that redirects to our site.

Hmm. W.wiki seems to be similar in that we own the domain but the A record points to AWS. It has the additional interesting thing in that its included in the subject alt name of our main certificate

In T165860#3346704, @MZMcBride wrote:

I'm not happy about how T165540: Add CODE_OF_CONDUCT.md to Wikimedia repositories was handled, though it's weirdly a case of not using +2 and instead bypassing Gerrit altogether, which for some reason is apparently possible.

In T167812#3345184, @Anomie wrote:

• css-sanitizer:
  0001-SECURITY-Escape-angle-brackets-numerically-in-string.patch3 KB

  Download
• TemplateStyles:
  0001-SECURITY-Reject-stylesheets-containing-style.patch5 KB

  Download

I thought this would have been dealt with by T133147

The cl_from entries in those pages are missing (i.e. no page table entry) e.g. https://ba.wikipedia.org/w/index.php?curid=122102&uselang=en gives a badtitle error.

Is this query from labs replica or actual db? In the past labs replicas have had replication issues related to DELETEs on categorylinks table that cause it to retain old rows that arent really there

In T139110#3340566, @hashar wrote:

fawikisourcehas some left over:

[fawikisource]> SELECT cl_collation,count(*) FROM categorylinks GROUP BY cl_collation;
+--------------+----------+
| cl_collation | count(*) |
+--------------+----------+
| uca-fa       |       12 |
| xx-uca-fa    |    53010 |
+--------------+----------+
2 rows in set (0.03 sec)

We would probably want to switch fawikivoyage which currently uses uppercase has a collation.

In T165540#3336220, @demon wrote:

In T165540#3335372, @MZMcBride wrote:

IInstead people are hastily and haphazardly spamming this file to repositories, bypassing Gerrit and other normal approval processes. This is unacceptable.

Point of clarification: this is acceptable. It's what I told people to do when questioned about the best way to do it. I said to push to Gerrit (not directly pushing) and setting them to auto-submit. The changes are all still in gerrit, they just didn't wait for someone to come along and +2. That would've been an even bigger waste of time.

In T162771#3336038, @DFoy wrote:

I checked and there is a problem with the testing timer. I'm attaching a screenshot. If you need credentials to access, please contact me outside of Phabricator.

I know everyone is tired of discussing the code of conduct, but for non-Wikimedia repos hosted in gerrit, I think it would have been more appropriate to have a mailing list discussion before force merging a code of conduct doc into such repos.

ping @DFoy : Everything look good? this has been live for a couple weeks now.

Do we really want to imply to the user that they can just replace the core/ directory? What if we add a new php entry point? Will this make (tarball) users less likely to upgrade skins/ and extensions/?

That's odd. Looks like the skin specific styles for vector are not loading.

In T133408#3316104, @Nirmos wrote:

T40848

Is this referring to a Phabricator task?

Sorry this took so long. I think this is good to go.

Note that an alternative fix for this issue may come in the form of the TemplateStyles extension.

Just as an update, this is almost done and will be finished by the end of the week

In T139110#3286037, @Krinkle wrote:

In T139110#3277667, @gerritbot wrote:

Change 354598 merged by jenkins-bot:
[mediawiki/core@master] Hack around icu breakage for fa sorting

https://gerrit.wikimedia.org/r/354598

@Bawolff If at all convenient, please do add tests for this.

In T139110#3280581, @matmarex wrote:

I suppose we need to run the updateCollation.php script on beta now to verify?

It looks like as of the hackathon all of this is done (Yay!). I'm leaving this open because I'm going to do a quick double check of the fixes later this week, but this bug is essentially done :D

I can't for the life of me figure out how this could have happened

#0 /var/www/core/core/includes/libs/rdbms/TransactionProfiler.php(218): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('writes', 'query-m: REPLAC...')

Maybe its just localization cache screwing with the no-writes in read access requests, transaction profiler thingy

+1

$pageTitle = $rc->getTitle(); in the getDiffHistLinks method can return null, a case not currently handled.

Yay it works, but it seems silly that we now have 2 separate globals for showing sql errors (The $wgShowSQLErrors still seems to be needed for the api)

In T162771#3211931, @DFoy wrote:

Go ahead and try it - let me know when it's live so I can do another test

Sorry for the delay, we will do this soon: https://gerrit.wikimedia.org/r/#/c/354113/

Add a test comment?

Changes made:

• https://en.wikibooks.org/w/index.php?title=MediaWiki%3ACommon.js&type=revision&diff=3219240&oldid=3149903
• https://en.wikibooks.org/w/index.php?title=MediaWiki%3AMobile.js&type=revision&diff=3219244&oldid=3151219
• https://en.wikisource.org/w/index.php?title=MediaWiki:Gadget-Site.js&diff=next&oldid=6790423
• https://as.wikisource.org/w/index.php?title=%E0%A6%AE%E0%A6%BF%E0%A6%A1%E0%A6%BF%E0%A6%AF%E0%A6%BC%E0%A6%BE%E0%A7%B1%E0%A6%BF%E0%A6%95%E0%A6%BF%3ACommon.js&type=revision&diff=10901&oldid=10102
• https://zh.wikisource.org/w/index.php?title=MediaWiki%3ACommon.js&type=revision&diff=867755&oldid=845510
• https://en.wikinews.org/w/index.php?title=MediaWiki%3AMobile.js&type=revision&diff=4314489&oldid=4255006
• https://en.wikinews.org/w/index.php?title=MediaWiki:Common.js&curid=62265&diff=4314487&oldid=4251458

In theory we could have a whitlist and then emit DENY or ALLOW-FROM depending on the origin, but it would have to be implemented in all kinds of things that render/cache wiki pages (MediaWiki, Parsoid, Varnish...) which is a bit of a pain. Maybe it could be limited to authenticated page views (framing an unauthenticated view seems pretty harmless).

unassigning from self. Community-tech has taken over working on this extension.

One handy (ab)use of php serialization is deep cloning

This is probably the header that would improve our security the most. I've been working on this, but progress has been very slow, largely due to lack of time on my part. See https://www.mediawiki.org/wiki/Requests_for_comment/Content-Security-Policy and T135963 for more details. There are different levels of using this header, with different levels of changes required depending on how "strict".

Its related to whether a page is "click-jackable". For ordinary articles, usually that means if there is a "patrol" link on the page.

In T133408#3236205, @Anomie wrote:

In T133408#3233518, @Bawolff wrote:

I'm still working through all this, but some initial things:

• As a precaution, I think there should be a post processing step to kill all U+007F characters, in case someone manages to insert a strip marker. I don't think its possible to insert a strip marker in a valid way into a stylesheet, but just in case, it doesn't hurt to replace U+007F with a unicode replacement character or css \7f escape or something.

You could embed it inside a CSS string. Elsewhere it would probably parse as a <delim-token>, but the sanitizer should filter those out (which is good because there's no way to legally escape them). Or a comment, but the sanitizer will always filter those out.

I think I'll have css-sanitizer use hex escapes for all non-graphic characters except U+0020 in strings and names, and I'll have TemplateStyles replace U+007F with U+FFFD if any sneak through the sanitizer.

I'm still working through all this, but some initial things:

@Legoktm: there is also a report that debian has the wrong version as well (I havent verified this myself)

I sent a warning to mediawiki-l and wikitech-l (https://lists.wikimedia.org/pipermail/mediawiki-l/2017-April/046524.html) arguably an issue of this type deserves a warning to mediawiki-announcements but i dont have send access to that mailing list.

Theres a post about this on oss-security now

We should really issue another release for this right away. The syntax highlight bug was by far the most severe of all the bugs last release. To avoid confusion we should probably bump the mediawiki version number (eventhough that version number technically only applies to mediawiki core and not syntaxhighlight)

+1 to just reverting the pdfhandler config change

The character encoding thing looks like its related to colours. Presumably the escape control character is being stripped by the logs

I mean its being sent as Content-Type: application/x-www-form-urlencoded

Whats the error in question?

This week is already becoming kind of insane due to events on frwiki. How about we do this on monday

This is more a browser UI issue (This is the same for anything else protected by basic http auth e.g https://logstash.wikimedia.org )