Palette PNGs are now fixed as well, I purged those 2 examples you gave @Trlkly and verified the fix. Thanks again for your report.
Fri, Sep 21
That's correct! I'm in the process of unifying the schema column names to make joins between these tables less confusing: T204921: Rename EventLogging column surveyInstanceToken to pageviewToken in QuickSurveysResponses for consistency
Thu, Sep 20
This is now live. From the enwiki main page:
Request added to https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
We're now collecting ResourceTiming data about the top article image. You can find that data in the resourcetiming table on hive:
Data is correctly being recorded in event.resourcetiming on hive.
Wed, Sep 19
Trying out undersampling positive responses to have roughly the same amount as negative ones, in order to circumvent the class imbalance in the dumbest possible way. Default RandomForestClassifier:
Of course :) I'm just trying to explore as much as I can with my limited knowledge and understanding of ML, in order to have a better informed discussion on Monday.
Here's what I get, asking the grid search to optimize for the negative responses' f1 score:
Optimizing the grid search for 0 (negative response)'s precision, I get:
Tue, Sep 18
Trying random forest classifiers with grid search for the parameters on the whole dataset, extracted with this query:
Because of the other things going on that affect the SSIM, it wouldn't really catch the same regression happening, though...
Add test case
As far as I can tell, the math extension is serving images through the REST API (Mathoid) and as such doesn't use the "original" urls:
Simply add palette PNGs to the rule
Actually not without consequence, this is turning RGB thumbnails into RGBA, and the extra transparency layer is making the files bigger. Instead I'll try to restrict to RGBA + indexed.
Mon, Sep 17
It's probably relevant that they're using indexed colors, indeed/
Seems to have worked, job errors on id_internalwikimedia have stopped.
Turns out id_internalwikimedia is probably showing up more because it's having a lot of activity, but all private wikis are affected. The prerender job has always failed for them, apparently.
At a glance, it might be that it's expecting a URL and instead it's only getting a path?
Tested it on WMCS and it works:
Fri, Sep 14
At first glance, I've managed to backport everything. Now I'll need to actually test the program to see if it runs properly...
Nevermind, the source package actually builds both. Back on track...
Now back to trying to backport golang-google-grpc I get:
Undoing the patch that adds the gcc6 option -Wno-error=misleading-indentation worked and allowed me to backport protobuf 3.0.0!
Thu, Sep 13
It worked, yay.
The list keeps growing:
Currently stuck on golang-github-go-kit-kit-dev. Got all its dependencies backported or installed, and then:
For the sid version golang-github-prometheus-common to be backported, the following need backporting as well:
Sure enough, going back to the Debian version of that library, it explains the error:
Next option is to fix the actual error, I guess, on that line: https://github.com/prometheus/haproxy_exporter/blob/v0.9.0/haproxy_exporter.go#L485
Backported kingpin 2.2.6 and I still get the same error. I have a feeling this might be an issue of Go being too old on jessie-backports... It's 1.7 there and the Travis CI configuration for prometheus-haproxy-exporter uses 1.9
After some minor adjustments to the package and what I had installed on my dev machine, I was able to get pretty far, but I'm running into a Go error, probably due to the version of Go itself or of the dependencies:
This probably happens for thumbnails generated with thumbor and then accessed via thumb.php. What's the point of that custom x-object-meta-sha1base36 header for thumbnails, though? The code doesn't say. It could be something that's only really useful for originals and thumbnails just happen to get the same treatment.
Yes, next train is totally fine, thanks!
I'm done, passing the torch
Wed, Sep 12
FYI editing with VE on wikitech is currently broken because of T163438: VisualEditor broken on wikitech when codfw is primary: "Error loading data from server: apierror-visualeditor-docserver-http: HTTP 500."
Tue, Sep 11
ISP requests work on stat machines.
Thu, Sep 6
Created the schema to collect the ResourceTiming data: https://meta.wikimedia.org/wiki/Schema:ResourceTiming
This seems to do the trick:
Identifying the top image in the article (from the content meaningfulness perspective) is a bit tricky, because some infoboxes point directly to images, without them being thumbnails in the wikitext semantic way of the terms (which gets those img elements a special class), and those infoboxes also sometimes contain tiny icons. The tiny icons, while not very significant visually, might be before the first large image people pay attention to in the DOM.
@Ejegg when is the next CentralNotice deployment scheduled? If not scheduled or far in the future, is there any chance this particular change could be backported to production? It's extremely low risk: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CentralNotice/+/436792/1/resources/subscribing/ext.centralNotice.display.js
Wed, Sep 5
Tue, Sep 4
Ops perviously rejected the idea of using the built-in Grafana notification system, as they don't want to maintain another alert/notification/paging system. Which is why we ended up building the Icinga bridge.
Mon, Sep 3
Data collected by MediaViewer about viewport dimensions is currently stored in that EventLogging schema: https://meta.wikimedia.org/wiki/Schema:MultimediaViewerDimensions
This revision has already been merged, see message above in the history
Now that we have transferSize and that it's been collecting for a couple of months, we can go back and see if there's a correlation between that and survey responses:
This appears to work fine, a lot of data is being collected.
My code doesn't seem to be running on enwiki? @AndyRussG Are new versions of CentralNotice not deployed automatically via the train like the rest of MediaWiki extensions? Special:Version on enwiki states "2.6.0 (6492626) 20:21, 6 May 2018".
Hmmm I'm targeted by a WLM banner right now and I'm not seeing the performance mark...
The data is making it to EventLogging as expected and I'm seeing large values as I thought was possible when the survey is initially rendered below the fold. This will be a great signal to eliminate filter out that case or factor it into the machine learning.
We're still getting from 429s, this time I cross-references them and it's the per-IP (ipv4 and not ipv6 this time):