hashar (Antoine "hashar" Musso (WMF))
WMF Software developer - Release Engineering

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:31 PM (129 w, 21 h)
Availability
Available
IRC Nick
hashar
LDAP User
Hashar
MediaWiki User
Unknown

https://www.mediawiki.org/wiki/User:Hashar

Based in Nantes, France CET/CEST (UTC+1, UTC+2)

Main IRC channel is #wikimedia-releng

antoine-approve

Recent Activity

Yesterday

hashar added a comment to T137112: migrate mwext-mw-selenium to Nodepool instances.

Just found out that PhantomJS 2.1.1 is available in jessie-backports since March 8th!!! https://packages.debian.org/jessie-backports/phantomjs

Fri, Mar 24, 1:53 PM · User-zeljkofilipin, Patch-For-Review, Continuous-Integration-Scaling, Browser-Tests-Infrastructure
hashar added a comment to T137112: migrate mwext-mw-selenium to Nodepool instances.

I went ahead and just +2ed the patch.

Fri, Mar 24, 1:50 PM · User-zeljkofilipin, Patch-For-Review, Continuous-Integration-Scaling, Browser-Tests-Infrastructure
hashar added a comment to T160989: Revisit Jenkins jobs being triggered for Wikibase.

Moving some jobs to postmerge would at least prevent them from running on every patchsets. The postmerge jobs reports on the Gerrit change that triggered them, so most probably developers will pay attention to the eventual failure message.

Fri, Mar 24, 1:46 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Wikidata
hashar triaged T161305: Merge apps/android/wikipedia Jenkins jobs lint and test as "Normal" priority.
Fri, Mar 24, 1:35 PM · Wikipedia-Android-App-Backlog, Android-app-Bugs, Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar created T161305: Merge apps/android/wikipedia Jenkins jobs lint and test.
Fri, Mar 24, 1:34 PM · Wikipedia-Android-App-Backlog, Android-app-Bugs, Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar added a comment to T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.

Also found out via swift list --lh that most containers are actually empty. Most probably due to container-server.conf having db_preallocation = on.

Fri, Mar 24, 1:08 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
hashar edited the description of T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.
Fri, Mar 24, 1:03 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
hashar added a comment to T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.

From what I understand swift replications ends continuously stats() all the containers and objects sqlite files.

Fri, Mar 24, 1:03 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure

Thu, Mar 23

hashar added a comment to T161208: Create a Phabricator badge for persons involved in CI.

Neat! How does one create a badge? Id like to do one for mobile wizards.

Thu, Mar 23, 8:41 PM · Phabricator
hashar closed T161208: Create a Phabricator badge for persons involved in CI as "Resolved".

Roger @mmodell yeah that makes sense to not auto congratulates ourselves :-}

Thu, Mar 23, 5:25 PM · Phabricator
hashar updated the badge description for Continuous Integrator.
Thu, Mar 23, 5:23 PM
hashar created T161227: Prometheus graph incorrectly sums CPU user and CPU guest.
Thu, Mar 23, 5:02 PM · Graphite, Prometheus-metrics-monitoring
hashar added a project to T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU: Continuous-Integration-Infrastructure (Little Steps Sprint).
Thu, Mar 23, 4:27 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
hashar closed T160923: For operations/puppet : merge tox / rake jobs in a single job? as "Resolved".

Tested, works. I have update the castor cache by rebuilding a build with ZUUL_PIPELINE=postmerge.

Thu, Mar 23, 4:21 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar claimed T160923: For operations/puppet : merge tox / rake jobs in a single job?.
Thu, Mar 23, 3:32 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar placed T155483: Speed up oojs/ui Jenkins jobs up for grabs.
Thu, Mar 23, 3:32 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), OOjs-UI, Continuous-Integration-Config
hashar moved T160923: For operations/puppet : merge tox / rake jobs in a single job? from Backlog to On going on the Continuous-Integration-Infrastructure (Little Steps Sprint) board.
Thu, Mar 23, 3:32 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar created T161222: Remove graphite metrics under zuul.pipeline.check-voter..
Thu, Mar 23, 3:10 PM · Graphite
hashar set Is Sprint to 1 on Continuous-Integration-Infrastructure (Little Steps Sprint).
Thu, Mar 23, 3:01 PM
hashar closed T160476: Disable fundraising CI jobs that are non-voting and always fail as "Resolved".

All done.

Thu, Mar 23, 3:01 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Wikimedia-Fundraising-CiviCRM, FR-Smashpig, MediaWiki-extensions-DonationInterface, Fundraising-Backlog
hashar edited the description of T160476: Disable fundraising CI jobs that are non-voting and always fail.
Thu, Mar 23, 2:57 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Wikimedia-Fundraising-CiviCRM, FR-Smashpig, MediaWiki-extensions-DonationInterface, Fundraising-Backlog
hashar edited the description of T160476: Disable fundraising CI jobs that are non-voting and always fail.
Thu, Mar 23, 2:49 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Wikimedia-Fundraising-CiviCRM, FR-Smashpig, MediaWiki-extensions-DonationInterface, Fundraising-Backlog
hashar closed T161205: Remove check-voter pipeline from Zuul as "Resolved".
Thu, Mar 23, 2:45 PM · Patch-For-Review, Continuous-Integration-Config
hashar updated subscribers of T154894: Phase out jobs "pplint-HEAD" and "erblint-HEAD".

Last patches have been merged by @Ottomata :-}

Thu, Mar 23, 2:23 PM · Patch-For-Review, Continuous-Integration-Config
hashar added a comment to T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.

Each instance uses 300% user CPU and 100% system CPU. So potentially 8 core out of labvirt1004 24 core. All that apparently just for replicating via rsync 23000 sqlite files that are barely changing.

Thu, Mar 23, 2:12 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
hashar added a comment to T161206: Gerrit patchset 99101 cannot be accessed: "500 Internal server error".

gerrit show-caches has:

Thu, Mar 23, 12:44 PM · Upstream, Analytics-Tech-community-metrics, Gerrit
hashar added a comment to T161207: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions.".

I can reach them all now. Seems all those patches are drafts.

Thu, Mar 23, 12:40 PM · Analytics-Tech-community-metrics, Gerrit
hashar edited the description of T161207: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions.".
Thu, Mar 23, 12:39 PM · Analytics-Tech-community-metrics, Gerrit
hashar edited the description of T161206: Gerrit patchset 99101 cannot be accessed: "500 Internal server error".
Thu, Mar 23, 12:38 PM · Upstream, Analytics-Tech-community-metrics, Gerrit
hashar added a comment to T106924: Consider using the Badges application for a few special roles to highlight those users' comments.

We have a few badges now https://phabricator.wikimedia.org/badges/

Thu, Mar 23, 11:39 AM · Phabricator
hashar edited the description of T161208: Create a Phabricator badge for persons involved in CI.
Thu, Mar 23, 11:36 AM · Phabricator
hashar created T161208: Create a Phabricator badge for persons involved in CI.
Thu, Mar 23, 11:35 AM · Phabricator
hashar added a comment to T160667: Create "High Priority" test pipeline.

Three repositories now benefit from the high priority pipeline:

  • operations/mediawiki-config
  • operations/puppet
  • operations/dns
Thu, Mar 23, 11:22 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar removed a project from T160667: Create "High Priority" test pipeline: Patch-For-Review.
Thu, Mar 23, 11:20 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar created T161205: Remove check-voter pipeline from Zuul.
Thu, Mar 23, 11:11 AM · Patch-For-Review, Continuous-Integration-Config
hashar edited the description of T160667: Create "High Priority" test pipeline.
Thu, Mar 23, 11:09 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar added a comment to T160667: Create "High Priority" test pipeline.

On the Grafana Zuul board I have added a graph showing the time to process changes in the test-prio pipeline.

Thu, Mar 23, 11:08 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

The scheduler now spread the Nodepool instances across multiple Compute nodes and CI was responsive again yesterday. Looks like the hack to artificially consume 32GB of RAM on labvirt1004 did the trick.

Thu, Mar 23, 10:18 AM · Patch-For-Review, Labs, Labs-Infrastructure
hashar created T161202: Requesting 'content administrator' access for hashar.
Thu, Mar 23, 10:15 AM · wikitech.wikimedia.org, Labs
hashar lowered the priority of T161198: Enhance deprecated logging to be context aware from "High" to "Normal".
Thu, Mar 23, 9:52 AM · MediaWiki-Logging, MediaWiki-Debug-Logger, Wikimedia-log-errors
hashar created T161198: Enhance deprecated logging to be context aware.
Thu, Mar 23, 9:51 AM · MediaWiki-Logging, MediaWiki-Debug-Logger, Wikimedia-log-errors

Wed, Mar 22

hashar added a comment to T161107: scanner00.security-tools.eqiad.wmflabs has 4 CPU at 100% usage.

Thank you both!

Wed, Mar 22, 11:00 PM · Labs, Security-Core
hashar updated subscribers of T67270: Default license for operations/puppet.

For other repositories on which we wanted to set/change the license, we usually have done a list of non-wmf contributors in a task detail and then reached out to them. Then eventually just moved forward with the licensing.

Wed, Mar 22, 9:57 PM · Patch-For-Review, Operations, Software-Licensing, Documentation, WMF-Legal, Wikimedia-General-or-Unknown
hashar added a comment to T161159: Cannot access the database: Can't connect to MySQL server on '10.192.48.41' (111) (10.192.48.41).

[[ https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=codfw%20prometheus%2Fops&var-server=es2016&from=now-12h&to=now 12 hours view of prometheus ]] shows high disk read and bunch of probes are no more reporting. Started around 17:00UTC

Wed, Mar 22, 9:35 PM · DBA
hashar added a comment to T143349: Deprecate precise instances in Labs by 2017-03-31.

@chasemp wrote:

thank you @hashar

Wed, Mar 22, 8:57 PM · Patch-For-Review, Labs-Infrastructure, Labs
hashar added a comment to T137112: migrate mwext-mw-selenium to Nodepool instances.

test: invoke rspec directly
https://gerrit.wikimedia.org/r/330856

Wed, Mar 22, 5:29 PM · User-zeljkofilipin, Patch-For-Review, Continuous-Integration-Scaling, Browser-Tests-Infrastructure
hashar added a comment to T67270: Default license for operations/puppet.

Patch is https://gerrit.wikimedia.org/r/#/c/183862/

Wed, Mar 22, 4:31 PM · Patch-For-Review, Operations, Software-Licensing, Documentation, WMF-Legal, Wikimedia-General-or-Unknown
hashar moved T161095: Uninitialized string offset warnings with HHVM 3.18 in LanguageAz.php and LanguageKk.php from Backlog to New features on the HHVM board.
Wed, Mar 22, 3:47 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Wikimedia-log-errors, Patch-For-Review, MW-1.29-release-notes, MediaWiki-Internationalization, HHVM, Operations
hashar added a comment to T161095: Uninitialized string offset warnings with HHVM 3.18 in LanguageAz.php and LanguageKk.php.

I have deployed the hotfix on both wmf versions. There is most probably some useless call to ucfirst('') somewhere in the code.

Wed, Mar 22, 3:47 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Wikimedia-log-errors, Patch-For-Review, MW-1.29-release-notes, MediaWiki-Internationalization, HHVM, Operations
hashar placed T161118: Investigate instances with high "steal" CPU up for grabs.
Wed, Mar 22, 3:16 PM · Labs-Infrastructure, Labs
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

Seems Linux kernel in a guest is smart enough to find out instruction execution is being delayed by other instances on the same host (cpu steal). Filled subtask to investigate that T161118. 8 of the top 10 instances are running on labvirt1004.

Wed, Mar 22, 3:15 PM · Patch-For-Review, Labs, Labs-Infrastructure
hashar edited the description of T161118: Investigate instances with high "steal" CPU.
Wed, Mar 22, 3:14 PM · Labs-Infrastructure, Labs
hashar created T161118: Investigate instances with high "steal" CPU.
Wed, Mar 22, 3:12 PM · Labs-Infrastructure, Labs
hashar added a comment to T161107: scanner00.security-tools.eqiad.wmflabs has 4 CPU at 100% usage.

Maybe it is meant to be a short time scan and it ends up taking too long / being blocked in a loop.

Wed, Mar 22, 2:35 PM · Labs, Security-Core
hashar added a comment to T160549: MW-1.29.0-wmf.17 deployment blockers.

HHVM 3.12 -> 3.18 has been done on canaries so there might be some additional log spam such as T161095: Uninitialized string offset warnings with HHVM 3.18 in LanguageAz.php and LanguageKk.php

Wed, Mar 22, 1:50 PM · Release, Release-Engineering-Team (Deployment-Blockers)
hashar added a project to T161095: Uninitialized string offset warnings with HHVM 3.18 in LanguageAz.php and LanguageKk.php: Wikimedia-log-errors.

I am handling the backports to wmf.16 / wmf.17

Wed, Mar 22, 1:50 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Wikimedia-log-errors, Patch-For-Review, MW-1.29-release-notes, MediaWiki-Internationalization, HHVM, Operations
hashar added a parent task for T161095: Uninitialized string offset warnings with HHVM 3.18 in LanguageAz.php and LanguageKk.php: T160549: MW-1.29.0-wmf.17 deployment blockers.
Wed, Mar 22, 1:49 PM · MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), Wikimedia-log-errors, Patch-For-Review, MW-1.29-release-notes, MediaWiki-Internationalization, HHVM, Operations
hashar added a subtask for T160549: MW-1.29.0-wmf.17 deployment blockers: T161095: Uninitialized string offset warnings with HHVM 3.18 in LanguageAz.php and LanguageKk.php.
Wed, Mar 22, 1:49 PM · Release, Release-Engineering-Team (Deployment-Blockers)
hashar created T161107: scanner00.security-tools.eqiad.wmflabs has 4 CPU at 100% usage.
Wed, Mar 22, 1:43 PM · Labs, Security-Core
hashar created T161109: scanner00.security-tools.eqiad.wmflabs has 4 CPU at 100% usage.
Wed, Mar 22, 1:40 PM · Labs, Security-Core
hashar added a comment to T161037: Warning: data error in /srv/mediawiki/php-1.29.0-wmf.17/includes/Revision.php on line 1351.

Typo fixed ExternalStore::decompressRevisionText -> self::decompressRevisionText https://gerrit.wikimedia.org/r/#/c/344109/

Wed, Mar 22, 1:06 PM · MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), MW-1.29-release-notes, Performance-Team, MediaWiki-Cache, MediaWiki-Page-editing, Patch-For-Review, Wikimedia-log-errors
hashar added a comment to T158084: Mediawiki namespace pages, including CentralNotice banners, are slow to save.

Typo fixed ExternalStore::decompressRevisionText -> self::decompressRevisionText https://gerrit.wikimedia.org/r/#/c/344109/

Wed, Mar 22, 1:06 PM · Fundraising Sprint Far Beer, MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), MW-1.29-release-notes, Fundraising Sprint English Cuisine, Patch-For-Review, MediaWiki-Cache, Fundraising Sprint Deferential Equations, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

I have spawned at 10:41 UTC an instance integration-c1.integration (24fe397e-7bd3-4c12-bde3-3e211c5f2671) with 32GB of RAM. It has been scheduled on labvirt1004. Might cause the load to shift to another labvirt.

Wed, Mar 22, 10:48 AM · Patch-For-Review, Labs, Labs-Infrastructure
hashar added a comment to T159835: Labvirt1001 has insanely slow IO.

Might well be related to T161006 which suggest the Scheduler prioritize mostly based on RAM usage. So we end up with Nodepool instances spawning mostly on the same host which most probably overload the CPUs.

Wed, Mar 22, 10:00 AM · ops-eqiad, Operations, Labs-Infrastructure, Labs
hashar created T161086: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon.
Wed, Mar 22, 9:53 AM · Continuous-Integration-Infrastructure
hashar added a commit to T161006: Convince nova-scheduler to pay attention to CPU metrics: rOPUPbcf4179409cd: nova.conf: Collect cpu metrics.
Wed, Mar 22, 9:47 AM · Patch-For-Review, Labs, Labs-Infrastructure
hashar added a comment to T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.

The best I can tell is:

  • lowering number of workers might help
  • most of the time is spent in the process:
    • swift-container-replicator
    • swift-object-replicator
Wed, Mar 22, 9:32 AM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
hashar edited the description of T161084: On beta enable swift statsd metric.
Wed, Mar 22, 9:15 AM · media-storage, Beta-Cluster-Infrastructure
hashar created T161084: On beta enable swift statsd metric.
Wed, Mar 22, 9:15 AM · media-storage, Beta-Cluster-Infrastructure
hashar created T161083: Rebalance deployment-ms-be01 and deployment-ms-be02 so they run on different labvirt.
Wed, Mar 22, 9:04 AM · Labs, media-storage, Beta-Cluster-Infrastructure
hashar added a comment to T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.

On deployment-ms-be01 I reload the object server with 30 workers that might have helped. There is apparently some replication going on between the two back end. Will let them settle.

Wed, Mar 22, 8:58 AM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure

Tue, Mar 21

hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

And a paper that happens to mention the case we have https://01.org/sites/default/files/utilization_based_scheduing_in_openstack_compute_nova_1.docx that shows up the default weighters is all about spreading RAM usage and does not take in account I/O or CPU usage :]

Tue, Mar 21, 11:31 PM · Patch-For-Review, Labs, Labs-Infrastructure
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

To summarize the wild guesses I made to andrew over IRC:

Tue, Mar 21, 11:29 PM · Patch-For-Review, Labs, Labs-Infrastructure
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

I guess that prevents the scheduler to select a compute node that already has an instance in that antiaffinity group isn't it ?

Tue, Mar 21, 10:28 PM · Patch-For-Review, Labs, Labs-Infrastructure
hashar added a comment to T159855: Upgrade deployment-prep to elasticsearch 5.x.

It seems all patches have been merged on March 9th and deployment-prep apparently runs ElasticSearch 5. So I guess this task can be closed?

Tue, Mar 21, 10:20 PM · Discovery-Search (Current work), Discovery
hashar added a comment to T160476: Disable fundraising CI jobs that are non-voting and always fail.

The REL1_28 flavor I guess we can just move it to the experimental pipeline.

Tue, Mar 21, 10:01 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Wikimedia-Fundraising-CiviCRM, FR-Smashpig, MediaWiki-extensions-DonationInterface, Fundraising-Backlog
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

I lack data from the OpenStack side but a theory would be that a lot of Nodepool instances ends up being scheduled on the same host. Maybe because that is the one having the less vCPU allocated or the favorite candidate. With instances having little load that makes sense, but the CI instances typically consume a lot of CPU when being used.

Tue, Mar 21, 9:50 PM · Patch-For-Review, Labs, Labs-Infrastructure
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

labvirt1004 had its load bump since March 7th

Tue, Mar 21, 9:45 PM · Patch-For-Review, Labs, Labs-Infrastructure
hashar edited the description of T160667: Create "High Priority" test pipeline.
Tue, Mar 21, 9:19 PM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar triaged T160476: Disable fundraising CI jobs that are non-voting and always fail as "Normal" priority.

Thank you @awight to have taken the extra time to fill this one. I am adding that to our sprint to deal with similar jobs.

Tue, Mar 21, 8:56 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Wikimedia-Fundraising-CiviCRM, FR-Smashpig, MediaWiki-extensions-DonationInterface, Fundraising-Backlog
hashar added a comment to T161006: Convince nova-scheduler to pay attention to CPU metrics.

Went creating a lame graph that for each labvirt node graph the CPU usage * 2:

Tue, Mar 21, 4:29 PM · Patch-For-Review, Labs, Labs-Infrastructure
hashar added a comment to T107067: Measure capacity and utilization of labvirt**** servers.

I have added a graph of the CPU sum of {system,user,nice,iowait,irq,softirq} per labvirt hosts using a 1 day moving median.

Tue, Mar 21, 2:14 PM · Labs
hashar added a comment to T141673: Track labs instances hanging .

Potentially this one is solved for good. I closed the umbrella task I had (T152599) and haven't noticed such hang for a while now.

Tue, Mar 21, 2:12 PM · Patch-For-Review, Labs-Infrastructure, Labs
hashar created T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.
Tue, Mar 21, 12:41 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
hashar created T160989: Revisit Jenkins jobs being triggered for Wikibase.
Tue, Mar 21, 12:27 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Wikidata
hashar updated subscribers of T160737: Merge search/ javadoc jobs in the main maven job.

@thcipriani Over the last 2 months that would have saved 82 instances. Not so much but that sprint is named "Little Steps".

Tue, Mar 21, 10:02 AM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Discovery-Search, Discovery, Elasticsearch
hashar closed T160737: Merge search/ javadoc jobs in the main maven job as "Resolved".

Aced by the java cabal. The search repos now trigger the verify goal and it is up to developers to define in their pom whatever they want CI to run.

Tue, Mar 21, 9:55 AM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Discovery-Search, Discovery, Elasticsearch
hashar assigned T160737: Merge search/ javadoc jobs in the main maven job to dcausse.

That came up last friday was @dcausse / @Gehel . The jobs got changed from goal package to verify which includes the javadoc source generation. An excellent opportunity to remove some jobs \O/

Tue, Mar 21, 9:22 AM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Discovery-Search, Discovery, Elasticsearch
hashar moved T160737: Merge search/ javadoc jobs in the main maven job from Backlog to On going on the Continuous-Integration-Infrastructure (Little Steps Sprint) board.
Tue, Mar 21, 9:21 AM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Discovery-Search, Discovery, Elasticsearch
hashar triaged T160668: Create "High Priority" gate-and-submit pipeline as "Normal" priority.
Tue, Mar 21, 9:12 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar triaged T160737: Merge search/ javadoc jobs in the main maven job as "Normal" priority.
Tue, Mar 21, 9:12 AM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), Discovery-Search, Discovery, Elasticsearch
hashar triaged T160923: For operations/puppet : merge tox / rake jobs in a single job? as "Normal" priority.
Tue, Mar 21, 9:12 AM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar triaged T160667: Create "High Priority" test pipeline as "Normal" priority.
Tue, Mar 21, 9:12 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar moved T160667: Create "High Priority" test pipeline from Backlog to On going on the Continuous-Integration-Infrastructure (Little Steps Sprint) board.
Tue, Mar 21, 9:10 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar claimed T160667: Create "High Priority" test pipeline.

Deployed and I will monitor it over the day. The tests probably offer a good enough coverage for now.

Tue, Mar 21, 9:10 AM · Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar added a comment to T147779: MediaWiki code coverage no longer runs parser tests.

I guess I got confused because the parser tests have been made way faster. Thank you for the task cleanup.

Tue, Mar 21, 8:07 AM · MediaWiki-Unit-tests, Continuous-Integration-Config, Release-Engineering-Team
hashar added a comment to T144667: Update puppet-lint to 2.*.

Got merged as well: https://gerrit.wikimedia.org/r/#/c/342637/ - bump version

Tue, Mar 21, 8:05 AM · Patch-For-Review, Puppet, Operations

Mon, Mar 20

Smalyshev awarded T100987: "git review -d XXX" doesn't work for http gerrit a Manufacturing Defect? token.
Mon, Mar 20, 6:34 PM · Upstream, Gerrit
hashar added a comment to T94149: Get rid of zend tests for wmf branches.

Talked about it again during the release engineering meeting. We believe that although maintenance script are still using Zend php5.5, running tests on every single patch is unlikely to catch any issue and thus have little purpose beside slowing down merges.

Mon, Mar 20, 4:35 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint)
hashar updated subscribers of T160923: For operations/puppet : merge tox / rake jobs in a single job?.
Mon, Mar 20, 4:33 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint)