Page MenuHomePhabricator

hashar (Antoine "hashar" Musso)
UserAdministrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:31 PM (322 w, 22 h)
Roles
Administrator
Availability
Available
IRC Nick
hashar
LDAP User
Hashar
MediaWiki User
Unknown

https://www.mediawiki.org/wiki/User:Hashar

Based in Nantes, France CET/CEST (UTC+1, UTC+2)

antoine-approve

Recent Activity

Yesterday

hashar reopened T269339: incorrect display of dates in ISO 8601 format: non-sense displayed everywhere in Phabricator, missing years ! as "Open".

Thank you @Aklapper that unblocked me! I am reopening since I forgot to reopen in my previous comment, feel free to drop the Phabricator tag and replace it with an translation related tag.

Fri, Dec 4, 1:51 PM · Upstream, Phabricator (Upstream)
hashar added a comment to T269396: Parser cache serving old results.

Thanks! As far as I know, we will continue on Monday.

Fri, Dec 4, 1:03 PM · Patch-For-Review, Platform Engineering, MediaWiki-Parser
hashar added a comment to T269339: incorrect display of dates in ISO 8601 format: non-sense displayed everywhere in Phabricator, missing years !.

I have noticed this task yesterday and wanted to elaborate but was busy on other things. I believe it should be reopened cause Phabricator has an issue in how it translates date. I gave it a try by setting the language to French and explicitly setting all the date / time preferences:

Fri, Dec 4, 11:32 AM · Upstream, Phabricator (Upstream)
hashar added a comment to T263186: 1.36.0-wmf.20 deployment blockers.

Rolled back due to an incompatibility of ParserOutput class signature between 1.36.0-wmf.18 and 1.36.0-wmf.20. That is an issue that has often hit us in the past and is tricky enough that we end up hitting it from time to time. The task is being worked on by all the appropriate people so it is on track, though I anticipate itl take a bit of time to address it or find a workaround and get the train unblocked.

Fri, Dec 4, 10:06 AM · Patch-For-Review, Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release, Train Deployments
hashar added a comment to T269396: Parser cache serving old results.

Just for the context, the serialize / unserialize with different class signatures has been crippling for a while:

Fri, Dec 4, 10:03 AM · Patch-For-Review, Platform Engineering, MediaWiki-Parser
hashar updated the task description for T269396: Parser cache serving old results.
Fri, Dec 4, 9:33 AM · Patch-For-Review, Platform Engineering, MediaWiki-Parser

Thu, Dec 3

hashar closed T263186: 1.36.0-wmf.20 deployment blockers as Resolved.

There was one occurrence of a logging issue on a request to a Parsoid server ( T269386 ) but that does not seem worth rolling back. Nothing else is showing in the log at least.

Thu, Dec 3, 8:21 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release, Train Deployments
hashar added a comment to T242751: Update monolog/monolog to 2.1.1.

Another one from Parsoid T269386

Thu, Dec 3, 8:13 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-Logstash, MediaWiki-Debug-Logger, Patch-For-Review, MediaWiki-Vendor
hashar moved T269386: Uncaught Error: Cannot use object of type MediaWiki\Logger\Monolog\LegacyHandler as array in /srv/mediawiki/php-1.36.0-wmf.20/includes/exception/MWExceptionRenderer.php:109 from Untriaged to Dec2020/1.36.0-wmf.20+ on the Wikimedia-production-error board.
Thu, Dec 3, 8:13 PM · Parsoid, Wikimedia-production-error
hashar created T269386: Uncaught Error: Cannot use object of type MediaWiki\Logger\Monolog\LegacyHandler as array in /srv/mediawiki/php-1.36.0-wmf.20/includes/exception/MWExceptionRenderer.php:109.
Thu, Dec 3, 8:12 PM · Parsoid, Wikimedia-production-error
hashar added a comment to T263186: 1.36.0-wmf.20 deployment blockers.

Blockers got solved or ruled out as preexisting conditions. I am going to promote all wikis.

Thu, Dec 3, 7:59 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release, Train Deployments
hashar closed T269314: Argument 5 passed to MediaWiki\Extension\AbuseFilter\Filter\Specs::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.36.0-wmf.20/extensions/AbuseFilter/includes/FilterLookup.php on line 346 as Resolved.

Thank you @DannyS712, @thiemowmde and @Daimona !

Thu, Dec 3, 7:58 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), AbuseFilter, Wikimedia-production-error
hashar closed T269314: Argument 5 passed to MediaWiki\Extension\AbuseFilter\Filter\Specs::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.36.0-wmf.20/extensions/AbuseFilter/includes/FilterLookup.php on line 346, a subtask of T263186: 1.36.0-wmf.20 deployment blockers, as Resolved.
Thu, Dec 3, 7:58 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release, Train Deployments
hashar added a comment to T269314: Argument 5 passed to MediaWiki\Extension\AbuseFilter\Filter\Specs::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.36.0-wmf.20/extensions/AbuseFilter/includes/FilterLookup.php on line 346.

Validated on mwdebug1001 using https://commons.wikimedia.org/wiki/Special:AbuseFilter/history/28/item/127 as the example.

Thu, Dec 3, 7:55 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), AbuseFilter, Wikimedia-production-error
hashar added a comment to T269242: Change deploy-promote to check previous groups have been promoted to.

What I did this week was:

Thu, Dec 3, 4:46 PM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2))
hashar added a project to T269153: Applying security patches should give some output: Release-Engineering-Team (Development services).
Thu, Dec 3, 4:42 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Scap
hashar added a comment to T269153: Applying security patches should give some output.

The lack of output surprised me and I was left wondering whether the patches actually applied.

Thu, Dec 3, 4:42 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Scap
hashar added a comment to T269314: Argument 5 passed to MediaWiki\Extension\AbuseFilter\Filter\Specs::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.36.0-wmf.20/extensions/AbuseFilter/includes/FilterLookup.php on line 346.

https://gerrit.wikimedia.org/r/644966 proposes a fix though it is not ready yet. Is there a change we can easily revert to unblock the train or should we hold on the fix to be completed?

Thu, Dec 3, 2:44 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), AbuseFilter, Wikimedia-production-error
hashar added a comment to T269315: Fatal error: Cannot pass parameter 1 by reference (Wikibase\DataModel\Entity).

Thank you Lucas and Amir for the investigation.

Thu, Dec 3, 2:38 PM · Wikidata-Campsite, Wikidata, Wikibase-DataModel, Wikimedia-production-error
hashar added a comment to T269352: Upgrade Jenkins to 2.263.1.

Turns out our Jenkins servers are using Java 8, notably because of the Gearman plugin. We should eventually switch them to Java 11 but that is a different task really: T269354

Thu, Dec 3, 2:12 PM · Jenkins, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
hashar added a project to T269354: Switch Jenkins servers to Java 11: Jenkins.
Thu, Dec 3, 2:02 PM · Patch-For-Review, Jenkins, Release-Engineering-Team (Development services), Continuous-Integration-Infrastructure
hashar created T269354: Switch Jenkins servers to Java 11.
Thu, Dec 3, 2:02 PM · Patch-For-Review, Jenkins, Release-Engineering-Team (Development services), Continuous-Integration-Infrastructure
hashar created T269352: Upgrade Jenkins to 2.263.1.
Thu, Dec 3, 1:24 PM · Jenkins, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
hashar updated subscribers of T269286: Monolog update removes exception object from logstash.

Nice, thank you @Reedy and @Krinkle!

Thu, Dec 3, 9:49 AM · MW-1.36-notes (1.36.0-wmf.21; 2020-12-08), Patch-For-Review, MediaWiki-Debug-Logger, Wikimedia-production-error

Wed, Dec 2

hashar added a comment to T269286: Monolog update removes exception object from logstash.

I guess the easiest is probably to revert the monolog update in master and 1.36.0-wmf.20 ?

Wed, Dec 2, 7:30 PM · MW-1.36-notes (1.36.0-wmf.21; 2020-12-08), Patch-For-Review, MediaWiki-Debug-Logger, Wikimedia-production-error
hashar added a comment to T263186: 1.36.0-wmf.20 deployment blockers.

@Pchelolo I probably should have filed a single task for those two. I will be out for rest of the day, but you have my blessing to have those fix deployed today!

Wed, Dec 2, 7:29 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release, Train Deployments
hashar created T269278: Out of memory error reaching 700MBytes since 1.36.0-wmf.20.
Wed, Dec 2, 6:38 PM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release, Train Deployments
hashar added a comment to T263186: 1.36.0-wmf.20 deployment blockers.

So yesterday I have promoted the "testwikis" and today I have promoted the group1 wikis. The group0 wikis have NOT been promoted.

Wed, Dec 2, 2:54 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release, Train Deployments
hashar added a parent task for T269235: [X8ehRwpAICgAALHNCUsAAACI] /wiki/clamor ErrorException from line 157 of /srv/mediawiki/php-1.36.0-wmf.20/extensions/CategoryTree/includes/CategoryTreeHooks.php: ParserOutput::mCategoryTreeTag dynamic property write access deprecated [Called from CategoryTreeHooks::parserHook]: T263851: Disallow dynamic property access on ParserOutput.
Wed, Dec 2, 2:36 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), MediaWiki-extensions-CategoryTree, Patch-For-Review, Platform Team Workboards (Green), MediaWiki-Parser, Wikimedia-production-error
hashar added a parent task for T269236: [X8elNgpAAL8AASHtlA8AAADE] /wiki/Category:German_pronunciation_of_prepositions ErrorException from line 4021 of /srv/mediawiki/php-1.36.0-wmf.20/includes/parser/Parser.php: ParserOutput::mNoGallery public write access deprecated [Called from Parser::handleDoubleUnderscore]: T263851: Disallow dynamic property access on ParserOutput.
Wed, Dec 2, 2:36 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), Platform Team Workboards (Green), Patch-For-Review, MediaWiki-Parser, Wikimedia-production-error
hashar added subtasks for T263851: Disallow dynamic property access on ParserOutput: T269235: [X8ehRwpAICgAALHNCUsAAACI] /wiki/clamor ErrorException from line 157 of /srv/mediawiki/php-1.36.0-wmf.20/extensions/CategoryTree/includes/CategoryTreeHooks.php: ParserOutput::mCategoryTreeTag dynamic property write access deprecated [Called from CategoryTreeHooks::parserHook], T269236: [X8elNgpAAL8AASHtlA8AAADE] /wiki/Category:German_pronunciation_of_prepositions ErrorException from line 4021 of /srv/mediawiki/php-1.36.0-wmf.20/includes/parser/Parser.php: ParserOutput::mNoGallery public write access deprecated [Called from Parser::handleDoubleUnderscore].
Wed, Dec 2, 2:36 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), Patch-For-Review, Platform Team Workboards (Green), MediaWiki-Parser, Parsoid
hashar updated the task description for T269235: [X8ehRwpAICgAALHNCUsAAACI] /wiki/clamor ErrorException from line 157 of /srv/mediawiki/php-1.36.0-wmf.20/extensions/CategoryTree/includes/CategoryTreeHooks.php: ParserOutput::mCategoryTreeTag dynamic property write access deprecated [Called from CategoryTreeHooks::parserHook].
Wed, Dec 2, 2:35 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), MediaWiki-extensions-CategoryTree, Patch-For-Review, Platform Team Workboards (Green), MediaWiki-Parser, Wikimedia-production-error
hashar created T269236: [X8elNgpAAL8AASHtlA8AAADE] /wiki/Category:German_pronunciation_of_prepositions ErrorException from line 4021 of /srv/mediawiki/php-1.36.0-wmf.20/includes/parser/Parser.php: ParserOutput::mNoGallery public write access deprecated [Called from Parser::handleDoubleUnderscore].
Wed, Dec 2, 2:35 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), Platform Team Workboards (Green), Patch-For-Review, MediaWiki-Parser, Wikimedia-production-error
hashar added projects to T269235: [X8ehRwpAICgAALHNCUsAAACI] /wiki/clamor ErrorException from line 157 of /srv/mediawiki/php-1.36.0-wmf.20/extensions/CategoryTree/includes/CategoryTreeHooks.php: ParserOutput::mCategoryTreeTag dynamic property write access deprecated [Called from CategoryTreeHooks::parserHook]: MediaWiki-Parser, Platform Engineering.
Wed, Dec 2, 2:33 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), MediaWiki-extensions-CategoryTree, Patch-For-Review, Platform Team Workboards (Green), MediaWiki-Parser, Wikimedia-production-error
hashar added a comment to T263851: Disallow dynamic property access on ParserOutput.

Another one spotted during the train: T269235

Wed, Dec 2, 2:29 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), Patch-For-Review, Platform Team Workboards (Green), MediaWiki-Parser, Parsoid
hashar added a comment to T269235: [X8ehRwpAICgAALHNCUsAAACI] /wiki/clamor ErrorException from line 157 of /srv/mediawiki/php-1.36.0-wmf.20/extensions/CategoryTree/includes/CategoryTreeHooks.php: ParserOutput::mCategoryTreeTag dynamic property write access deprecated [Called from CategoryTreeHooks::parserHook].

Comes from b1a30eb0c40253737b631521b44232f91d158608 / T263851

Wed, Dec 2, 2:28 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), MediaWiki-extensions-CategoryTree, Patch-For-Review, Platform Team Workboards (Green), MediaWiki-Parser, Wikimedia-production-error
hashar created T269235: [X8ehRwpAICgAALHNCUsAAACI] /wiki/clamor ErrorException from line 157 of /srv/mediawiki/php-1.36.0-wmf.20/extensions/CategoryTree/includes/CategoryTreeHooks.php: ParserOutput::mCategoryTreeTag dynamic property write access deprecated [Called from CategoryTreeHooks::parserHook].
Wed, Dec 2, 2:27 PM · MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), MediaWiki-extensions-CategoryTree, Patch-For-Review, Platform Team Workboards (Green), MediaWiki-Parser, Wikimedia-production-error
hashar added a comment to T242751: Update monolog/monolog to 2.1.1.

With https://gerrit.wikimedia.org/r/c/mediawiki/core/+/579668 , the messages formatted for logstash now have the monolog level as an integer when we already have the level name:

Wed, Dec 2, 2:25 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-Logstash, MediaWiki-Debug-Logger, Patch-For-Review, MediaWiki-Vendor
hashar created T269203: releng/phpmetrics should use composer.json / composer.lock.
Wed, Dec 2, 8:37 AM · Continuous-Integration-Config

Tue, Dec 1

hashar added a comment to T262241: Upgrade Gerrit to 3.3.

3.3.0 has been released https://www.gerritcodereview.com/3.3.html

Tue, Dec 1, 8:31 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit
hashar updated the task description for T262241: Upgrade Gerrit to 3.3.
Tue, Dec 1, 8:31 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit
hashar created T269153: Applying security patches should give some output.
Tue, Dec 1, 6:07 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Scap
hashar triaged T267787: Reviewer-bot does not add reviewers when leaving initial WIP mode as Low priority.
Tue, Dec 1, 10:48 AM · Gerrit
hashar added a comment to T267787: Reviewer-bot does not add reviewers when leaving initial WIP mode.

it seems that new users on Gerrit by default have the option Set new changes to "work in progress" by default on.

Tue, Dec 1, 10:48 AM · Gerrit
hashar added a comment to T269058: Unable to search 'namespace' in CodeSearch (as it has more than 5000 matches).

On the first search it queries https://codesearch.wmcloud.org/core/api/v1/search?stats=fosho&repos=*&rng=:20&q=namespace&files=&i=nope and the response has:

Tue, Dec 1, 10:16 AM · VPS-project-Codesearch
hashar closed T268931: Tidy up references to REL1_34 when it is EOL as Resolved.

Thanks for the cleanup!

Tue, Dec 1, 8:37 AM · Continuous-Integration-Config, Wikimedia-General-or-Unknown

Mon, Nov 30

hashar added a comment to T248041: puppetdb on deployment-puppetdb03 keeps getting OOMKilled.

Based on dmesg the oom hasn't happened since beginning of October so that is an improvement. Some request to postgre required more memory than available apparently which triggered the oom killer.

Mon, Nov 30, 5:40 PM · Patch-For-Review, Developer Productivity, Puppet, Beta-Cluster-Infrastructure
hashar added a comment to T268964: phabricator-stage-1001.devtools.eqiad1.wikimedia.cloud fails puppet.

It was broken due to some certificate issue. I tried switching to puppetmaster.cloudinfra.wmflabs.org cause that sometimes fix the cert. Puppet eventually set back the previous puppet master. Maybe there is an issue related to the instance FQDN used in the certificate? Anyway I haven't touched anything else after my short investigation.

Mon, Nov 30, 3:07 PM · Phabricator
hashar updated the task description for T249949: Stop using integration/composer and then archive the repo.
Mon, Nov 30, 2:30 PM · Continuous-Integration-Infrastructure
hashar added a comment to T261207: debmonitor: enhance filtering options for packages.

@Volans @akosiaris can we let the releng latest images be included in a Debmonitor. I really need that functionality to see the packages and it seems to be straightforward: https://gerrit.wikimedia.org/r/c/operations/puppet/+/624096/1/modules/profile/files/docker/reporter/releng_rules.ini

Mon, Nov 30, 12:55 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Continuous-Integration-Infrastructure, Patch-For-Review, git-protocol-v2, SRE-tools
hashar updated subscribers of T268964: phabricator-stage-1001.devtools.eqiad1.wikimedia.cloud fails puppet.

There is something broken with the Puppet master configuration for phabricator-stage-1001 and I can't quite figure it out :-\

Mon, Nov 30, 8:12 AM · Phabricator
hashar added a comment to T268964: phabricator-stage-1001.devtools.eqiad1.wikimedia.cloud fails puppet.

I have manually edit /etc/puppet/puppet.conf to point to the generic puppet master: puppetmaster.cloudinfra.wmflabs.org

Mon, Nov 30, 8:06 AM · Phabricator
hashar created T268964: phabricator-stage-1001.devtools.eqiad1.wikimedia.cloud fails puppet.
Mon, Nov 30, 7:55 AM · Phabricator

Fri, Nov 27

hashar added a comment to T251305: Migrate to helm v3.

The CI Docker image failed to build:

Creating /tmp/helm/repository 
Creating /tmp/helm/repository/cache (image.py:179)
Creating /tmp/helm/repository/local 
Creating /tmp/helm/plugins (image.py:179)
Creating /tmp/helm/starters (image.py:179)
Creating /tmp/helm/cache/archive (image.py:179)
Creating /tmp/helm/repository/repositories.yaml 
Adding stable repo with URL: https://charts.helm.sh/stable (image.py:179)
Adding local repo with URL: http://127.0.0.1:8879/charts (image.py:179)
$HELM_HOME has been configured at /tmp/helm.
Not installing Tiller due to 'client-only' flag having been set (image.py:179)
"wmf-stable" has been added to your repositories (image.py:179)
"incubator" has been added to your repositories (image.py:179)
"wmf-stable" has been added to your repositories (image.py:179)
Error: repo "https://kubernetes-charts-incubator.storage.googleapis.com" is no longer available; try "https://charts.helm.sh/incubator" instead
 (image.py:179)
Fri, Nov 27, 3:44 PM · Patch-For-Review, Kubernetes, serviceops, Operations

Thu, Nov 26

hashar added a comment to T268196: Figure out the future of codesearch in a GitLab world.

@hashar: Hah, I had no idea. Thanks!

Thu, Nov 26, 9:54 AM · GitLab, VPS-project-Codesearch
hashar added a comment to T256768: Archive labs/tools/ZppixBot gerrit repo.
Thu, Nov 26, 9:40 AM · User-MacFan4000, Toolforge, Projects-Cleanup, User-Zppix, ZppixBot
hashar placed T256575: Jenkins comments no longer formatted on Gerrit 3 up for grabs.

We previously had T215658 that eventually has lead to the addition of table of CI results on Gerrit changes. There was some code at https://gerrit.wikimedia.org/r/c/operations/software/gerrit/plugins/wikimedia/+/489483/14/gr-wikimedia/gr-wikimedia-prettify-ci-comments.html

Thu, Nov 26, 9:27 AM · Release-Engineering-Team-TODO, Developer Productivity, Continuous-Integration-Config, Gerrit
hashar added a comment to T215658: Create a plugin that makes ci comments pretty.

With the Gerrit 3.2 upgrade, the CI reports are no pretty (T256575). So maybe we can reuse some part from https://gerrit.wikimedia.org/r/c/operations/software/gerrit/plugins/wikimedia/+/489483/14/gr-wikimedia/gr-wikimedia-prettify-ci-comments.html to prettify ;)

Thu, Nov 26, 9:25 AM · Patch-For-Review, Gerrit
hashar removed a project from T268290: Unable to clone git repo from stat1008: Gerrit.
Thu, Nov 26, 9:21 AM · Release-Engineering-Team (Development services), Analytics
hashar closed T256768: Archive labs/tools/ZppixBot gerrit repo as Resolved.

The repository is already read only since August 11 2020. That has been done by @MacFan4000 : https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/ZppixBot/+/820f147fb851a0c0dffef930aef2b03994842bc5%5E%21/#F0

Thu, Nov 26, 9:19 AM · User-MacFan4000, Toolforge, Projects-Cleanup, User-Zppix, ZppixBot
hashar edited projects for T256768: Archive labs/tools/ZppixBot gerrit repo, added: Projects-Cleanup, Toolforge; removed Diffusion-Repository-Administrators, Gerrit.

Projects-Cleanup is the way to go ;)

Thu, Nov 26, 9:15 AM · User-MacFan4000, Toolforge, Projects-Cleanup, User-Zppix, ZppixBot

Wed, Nov 25

thcipriani awarded T268759: wikifarm-prod.pluggableauth.eqiad1.wikimedia.cloud causing more than 33% of Gerrit http traffic a Party Time token.
Wed, Nov 25, 6:42 PM · Release-Engineering-Team (Development services), Platform Engineering, Gerrit
hashar closed T268759: wikifarm-prod.pluggableauth.eqiad1.wikimedia.cloud causing more than 33% of Gerrit http traffic as Resolved.

Amazing, thank you for the detailed information and I can confirm the HTTP requests are entirely gone. I don't think it was causing much load to Gerrit anyway :]

Wed, Nov 25, 5:01 PM · Release-Engineering-Team (Development services), Platform Engineering, Gerrit
hashar added a comment to T263955: mediawiki/extensions/WSOAuth Github and Gerrit repo have diverged.

The current build is still failing because Selenium is reporting errors. These errors are related to the test user not being able to log in to the wiki. This is to be expected, since WSOAuth is not configured. I am not sure how these errors could be fixed. Can those specific tests be disabled?

Wed, Nov 25, 4:52 PM · Gerrit, Wikimedia-GitHub
hashar added a project to T268759: wikifarm-prod.pluggableauth.eqiad1.wikimedia.cloud causing more than 33% of Gerrit http traffic: Release-Engineering-Team (Development services).
Wed, Nov 25, 3:14 PM · Release-Engineering-Team (Development services), Platform Engineering, Gerrit
hashar added a comment to T226570: Investigate Gerrit https hits spikes.

For wikifarm.pluggableauth.eqiad.wmflabs I have filed T268759

Wed, Nov 25, 3:13 PM · Release-Engineering-Team, Gerrit
hashar added a comment to T226240: Create mirror of Gerrit repositories for consumption by various tools.

For wikifarm.pluggableauth.eqiad.wmflabs I have filed T268759

Wed, Nov 25, 3:13 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit
hashar created T268759: wikifarm-prod.pluggableauth.eqiad1.wikimedia.cloud causing more than 33% of Gerrit http traffic.
Wed, Nov 25, 3:10 PM · Release-Engineering-Team (Development services), Platform Engineering, Gerrit
hashar closed T265615: Terminating MySQL takes several minutes in (Wikibase?) CI jobs as Resolved.

That has been more or less fixed by raising the newish IO quota for the integration instances. It is still not ideal and enhancements will be made later on via the infrastructure task T266777.

Wed, Nov 25, 2:35 PM · Patch-For-Review, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Continuous-Integration-Infrastructure, Wikidata, Quibble
hashar removed a project from T201778: Add pre-commit hook that does basic checks like php -l: Gerrit.
Wed, Nov 25, 9:49 AM · MediaWiki-General
hashar closed T268119: Gerrit: remote unpack failed: Missing tree / blob as Resolved.

The count of all errors we had:

Wed, Nov 25, 9:48 AM · Release-Engineering-Team (Development services), Gerrit
hashar added a comment to T268196: Figure out the future of codesearch in a GitLab world.

Note: Links should prefer https://docs.gitlab.com/ce/* to https://docs.gotlab.com/ee/* (CE, not EE, in case content / functionality differs at some point).

Wed, Nov 25, 8:06 AM · GitLab, VPS-project-Codesearch

Tue, Nov 24

hashar closed T263184: 1.36.0-wmf.18 deployment blockers as Resolved.

Deployed to all wikis. Congratulations everyone.

Tue, Nov 24, 4:49 PM · Patch-For-Review, Release-Engineering-Team-TODO, Release, Train Deployments
hashar added a comment to T268337: New Scap release candidate .deb not visible on deployment-cumin.

/etc/apt/sources.list.d/project-aptly.list comes from the Puppet class aptly::client. The class is applied on every instances of the project via Horizon project puppet config (https://horizon.wikimedia.org/project/puppet/):

classes:
- apt::unattendedupgrades
- role::aptly::client
- role::prometheus::node_exporter
Tue, Nov 24, 12:47 PM · Beta-Cluster-Infrastructure

Mon, Nov 23

hashar added a comment to T263184: 1.36.0-wmf.18 deployment blockers.

It seems to work properly and T267668 has not been reopened. If all goes well over night, I will deploy it on the rest of the wiki during the European time slot at 13:00 UTC (14:00 CET). Checking with SRE before pushing it obviously!

Mon, Nov 23, 11:35 PM · Patch-For-Review, Release-Engineering-Team-TODO, Release, Train Deployments
hashar added a comment to T267668: Some recent Commons uploads not available on other wikis (2020-11).

1.36.0-wmf.18 has been rolled to commons.wikimedia.org Nov 23 at 20:09 UTC.

Mon, Nov 23, 11:34 PM · MW-1.36-notes (1.36.0-wmf.16; 2020-11-03), Patch-For-Review, Wikimedia-production-error, Operations, Commons, MediaWiki-File-management

Fri, Nov 20

hashar added a comment to T268196: Figure out the future of codesearch in a GitLab world.

Gitlab CE comes with a search system backed up by the database ( https://docs.gitlab.com/ee/user/search/#basic-search ). Searches for commit, code or comments are limited to the current project.

Fri, Nov 20, 5:19 PM · GitLab, VPS-project-Codesearch
hashar updated the task description for T165631: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs.
Fri, Nov 20, 9:58 AM · Gerrit, Operations
hashar added a comment to T242855: Undeploy graphoid .

What is the status of the decommissioning of Graphoid ? The reason I ask is that CI still has to maintain images based on Debian Jessie (EOL since April 2020) with Node JS 6 (long EOL) and we would really like to be able to dispose of those legacy bits.

Fri, Nov 20, 9:48 AM · MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), Platform Engineering (Icebox), serviceops, Operations, Graphoid
hashar added a comment to T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects.

I have poked the last three remaining tasks:

Fri, Nov 20, 9:48 AM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Patch-For-Review, Release-Engineering-Team (CI & Testing services), JavaScript, Continuous-Integration-Config
hashar added a comment to T216826: Move Kartotherian and Tilerator to Kubernetes.

Is the migration of Maps (Kartotherian) / Maps (Tilerator) to Kubernetes / Node 10 on some people radar? The reason I ask is that CI still has to maintain images based on Debian Jessie (EOL since April 2020) with Node JS 6 (long EOL) and we would really like to be able to dispose of those legacy bits.

Fri, Nov 20, 9:45 AM · Patch-For-Review, Epic, Operations, Maps (Kartotherian)
hashar added a comment to T225678: Migrate 3d2png to k8s.

Is the migration of #3d2png to Kubernetes / Node 10 on some people radar? The reason I ask is that CI still has to maintain images based on Debian Jessie (EOL since April 2020) with Node JS 6 (long EOL) and we would really like to be able to dispose of those legacy bits.

Fri, Nov 20, 9:45 AM · Structured-Data-Backlog, Performance-Team (Radar), Product-Infrastructure-Team-Backlog, Patch-For-Review, Multimedia, 3D, Release Pipeline
hashar updated the task description for T267006: Puppet failures on many hosts.
Fri, Nov 20, 9:38 AM · Beta-Cluster-Infrastructure
hashar closed T267006: Puppet failures on many hosts as Resolved.

Thank you to everyone that acted on this task.

Fri, Nov 20, 9:38 AM · Beta-Cluster-Infrastructure
hashar closed T256064: Puppet failure on deployment-cache-text06 as Resolved.

That must have un fixed at some point. It is no more happening.

Fri, Nov 20, 9:35 AM · Release-Engineering-Team, Operations, Beta-Cluster-Infrastructure
hashar added a comment to T261513: Puppet Failures on deployment-cache-upload06.

Most probably that was due to T267561, the instance were running Varnish 5 while Puppet deployed VCL files intended for Varnish 6. That probably has been broken for more than a few months and we were not receiving any notification on puppet failure due to another bug somewhere.

Fri, Nov 20, 9:33 AM · Puppet, Beta-Cluster-Infrastructure
hashar merged T261513: Puppet Failures on deployment-cache-upload06 into T267561: Beta needs to be upgraded to Varnish 6.
Fri, Nov 20, 9:33 AM · User-Ryasmeen, Operations, Beta-Cluster-Infrastructure, Traffic
hashar merged task T261513: Puppet Failures on deployment-cache-upload06 into T267561: Beta needs to be upgraded to Varnish 6.
Fri, Nov 20, 9:32 AM · Puppet, Beta-Cluster-Infrastructure
hashar added a comment to T248355: Archive mediawiki/services/recommendation-api/deploy once it's no longer used.

Thank you @Jdforrester-WMF to have taken care of the cleanup!

Fri, Nov 20, 9:28 AM · Patch-For-Review, Recommendation-API, Projects-Cleanup

Thu, Nov 19

hashar changed the status of T268119: Gerrit: remote unpack failed: Missing tree / blob from Stalled to Open.

T268009 is now public. That was a nasty security issue in Gerrit which is addressed by our upgrade to Gerrit version 3.2.5. There was still a couple hits after the upgrade, so the issue is not solved.

Thu, Nov 19, 10:05 PM · Release-Engineering-Team (Development services), Gerrit
hashar added a comment to T268009: Gerrit security advisory 2020-11-17.

Gerrit has been upgraded. Next thing is to find out whether we can drop the work around we have applied to All-Projects / All-Users. Notably the READ permission for refs/* was quite helpful.

Thu, Nov 19, 7:03 PM · Gerrit, Security, Security-Team
hashar closed T268260: URLs in format https://gerrit.wikimedia.org/r/123456 end up on port 80, leading to error SSL_ERROR_RX_RECORD_TOO_LONG as Resolved.

Thank you for the very quick fix and @QChris to have found the upstream issue that had the fix. That saved us from having to roll back the Gerrit upgrade.

Thu, Nov 19, 7:01 PM · Release-Engineering-Team (Development services), Upstream, Regression, Gerrit
hashar added a subtask for T268009: Gerrit security advisory 2020-11-17: T268260: URLs in format https://gerrit.wikimedia.org/r/123456 end up on port 80, leading to error SSL_ERROR_RX_RECORD_TOO_LONG.
Thu, Nov 19, 6:21 PM · Gerrit, Security, Security-Team
hashar added a parent task for T268260: URLs in format https://gerrit.wikimedia.org/r/123456 end up on port 80, leading to error SSL_ERROR_RX_RECORD_TOO_LONG: T268009: Gerrit security advisory 2020-11-17.
Thu, Nov 19, 6:21 PM · Release-Engineering-Team (Development services), Upstream, Regression, Gerrit
hashar added a comment to T268009: Gerrit security advisory 2020-11-17.

An aftermath reported by some persons on IRC is that an URL such as https://gerrit.wikimedia.org/r/123 ends up redirected to an url including :80 which breaks it. T268260 and https://bugs.chromium.org/p/gerrit/issues/detail?id=13705

Thu, Nov 19, 6:21 PM · Gerrit, Security, Security-Team
hashar added a project to T268260: URLs in format https://gerrit.wikimedia.org/r/123456 end up on port 80, leading to error SSL_ERROR_RX_RECORD_TOO_LONG: Upstream.
Thu, Nov 19, 6:21 PM · Release-Engineering-Team (Development services), Upstream, Regression, Gerrit
hashar lowered the priority of T268009: Gerrit security advisory 2020-11-17 from Unbreak Now! to High.
Thu, Nov 19, 5:50 PM · Gerrit, Security, Security-Team
hashar added a comment to T268009: Gerrit security advisory 2020-11-17.

We have deployed Gerrit 3.2.5 and applied the All-Projects and All-Users workaround. This task can now be made public given upstream disclosed the security issues.

Thu, Nov 19, 5:46 PM · Gerrit, Security, Security-Team
hashar added a comment to T268009: Gerrit security advisory 2020-11-17.

I am going to deploy the Gerrit release 3.2.5.

Thu, Nov 19, 4:37 PM · Gerrit, Security, Security-Team
hashar created P13336 gerrit build labels.
Thu, Nov 19, 3:10 PM