Page MenuHomePhabricator

Gehel (Guillaume Lederrey)
Operations Engineer - Discovery

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Nov 9 2015, 9:18 PM (179 w, 3 d)
Availability
Available
IRC Nick
gehel
LDAP User
Gehel
MediaWiki User
GLederrey (WMF) [ Global Accounts ]

Recent Activity

Yesterday

Gehel added a comment to T141324: Look into shoving gerrit logs into logstash.

I had a conversation with @hashar about this topic. So here are a few idea:

Thu, Apr 18, 2:39 PM · Release-Engineering-Team (Backlog), Technical-Debt, Wikimedia-Logstash, Gerrit
Gehel moved T220830: data reimport on wdqs1009 and wdqs1010 from Backlog to Done on the Discovery-Wikidata-Query-Service-Sprint board.

Data transfer completed with the new cookbook, everything seems fine.

Thu, Apr 18, 9:18 AM · Operations, Discovery-Wikidata-Query-Service-Sprint

Tue, Apr 16

Gehel closed T219849: Tilerator crashed on maps200[1-3].codfw.wmnet, a subtask of T198622: migrate maps servers to stretch with the current style, as Resolved.
Tue, Apr 16, 3:08 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Operations, Maps
Gehel closed T219849: Tilerator crashed on maps200[1-3].codfw.wmnet as Resolved.

Stretch migration is completed. This should be fixed, we'll reopen if this happens again.

Tue, Apr 16, 3:08 PM · Maps (Tilerator), Operations
Gehel assigned T221055: Collect metrics on maps cassandra to Mathew.onipe.
Tue, Apr 16, 3:04 PM · Patch-For-Review, Operations, Maps, Cassandra
Gehel moved T221013: prometheus-wmf-elasticsearch-exporter interferes with prometheus-wmf-elasticsearch-exporter-9* unit on elastic nodes from in progress to Done on the Discovery-Search (Current work) board.
Tue, Apr 16, 1:00 PM · Discovery-Search (Current work), Elasticsearch
Gehel added a comment to T221013: prometheus-wmf-elasticsearch-exporter interferes with prometheus-wmf-elasticsearch-exporter-9* unit on elastic nodes.

redundant units have been cleaned via cumin:

Tue, Apr 16, 1:00 PM · Discovery-Search (Current work), Elasticsearch
Gehel created P8404 (An Untitled Masterwork).
Tue, Apr 16, 12:44 PM

Mon, Apr 15

Gehel added a comment to T220982: maps hosts have bad permissions under /srv/deployment.

Deployment seems to be a noop:

Mon, Apr 15, 2:57 PM · Operations
Gehel added a comment to T220982: maps hosts have bad permissions under /srv/deployment.

permissions reset via:

Mon, Apr 15, 2:23 PM · Operations
Gehel removed a project from T202898: Decommission maps-test cluster: Maps.

Removing maps from this ticket, since there isn't any work left on our side.

Mon, Apr 15, 7:44 AM · Reading-Infrastructure-Team-Backlog, ops-codfw, decommission, Operations

Fri, Apr 12

Gehel created T220830: data reimport on wdqs1009 and wdqs1010.
Fri, Apr 12, 2:58 PM · Operations, Discovery-Wikidata-Query-Service-Sprint

Thu, Apr 11

Gehel moved T219799: Create cookbook to reset readonly indices on elasticsearch clusters from Needs review to Done on the Discovery-Search (Current work) board.
Thu, Apr 11, 12:13 PM · Patch-For-Review, Operations, Wikimedia-Incident, Discovery-Search (Current work)
Gehel closed T217557: Socket timeout on wdqs.svc.eqiad.wmnet as Resolved.

I don't think there is anything actionable at this point. Let's close.

Thu, Apr 11, 7:26 AM · Wikidata, Operations, Wikidata-Query-Service, Discovery-Wikidata-Query-Service-Sprint

Wed, Apr 10

Gehel added a comment to T220625: Initialize CirrusSearch on cloudelastic.

Open firewall on cloudelsatic machines to allow connections from mwmaint*, mw job runners and cloudelastic

Wed, Apr 10, 4:24 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery

Tue, Apr 9

Gehel claimed T219799: Create cookbook to reset readonly indices on elasticsearch clusters.
Tue, Apr 9, 5:17 PM · Patch-For-Review, Operations, Wikimedia-Incident, Discovery-Search (Current work)
Gehel moved T220038: Degraded RAID on elastic2048 from in progress to Done on the Discovery-Search (Current work) board.
Tue, Apr 9, 2:43 PM · Discovery-Search (Current work), Operations, ops-codfw
Gehel added a project to T220038: Degraded RAID on elastic2048: Discovery-Search (Current work).

Reimage was problematic, with first a puppet failure and then the server not booting over PXE. Manually booting in PXE (F12) finally fixed the issue.

Tue, Apr 9, 2:43 PM · Discovery-Search (Current work), Operations, ops-codfw
Gehel created P8376 (An Untitled Masterwork).
Tue, Apr 9, 1:51 PM

Mon, Apr 8

Gehel moved T219799: Create cookbook to reset readonly indices on elasticsearch clusters from in progress to Needs review on the Discovery-Search (Current work) board.
Mon, Apr 8, 5:39 PM · Patch-For-Review, Operations, Wikimedia-Incident, Discovery-Search (Current work)

Fri, Apr 5

Mill <mill@mail.com> committed rCUMIN4d1480f7c3f0: 0kbaaaaaaaaaaa (authored by Gehel).
0kbaaaaaaaaaaa
Fri, Apr 5, 10:29 PM
Mill <mill@mail.com> committed rCUMIN6964d19b3846: )ccaaaaaaaaaaa (authored by Gehel).
)ccaaaaaaaaaaa
Fri, Apr 5, 10:29 PM
Gehel created T220205: Define constraints for cloudelastic use cases.
Fri, Apr 5, 2:08 PM · Discovery-Search (Current work)
Gehel committed rDPOM88f102a989f4: [maven-release-plugin] prepare for next development iteration (authored by Gehel).
[maven-release-plugin] prepare for next development iteration
Fri, Apr 5, 9:41 AM
Gehel committed rDPOMe280fd8f3a01: [maven-release-plugin] prepare release discovery-parent-pom-1.28 (authored by Gehel).
[maven-release-plugin] prepare release discovery-parent-pom-1.28
Fri, Apr 5, 9:41 AM
Gehel committed rDPOM1a3d9f7828fe: Update surefire / failsafe to latest milestone. (authored by Gehel).
Update surefire / failsafe to latest milestone.
Fri, Apr 5, 9:25 AM

Thu, Apr 4

Gehel added a comment to T220038: Degraded RAID on elastic2048.

From syslog:

Thu, Apr 4, 2:57 PM · Discovery-Search (Current work), Operations, ops-codfw
Gehel added a comment to T220038: Degraded RAID on elastic2048.
ehel@elastic2048:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0](F) sdb1[1]
      29279232 blocks super 1.2 [2/1] [_U]
Thu, Apr 4, 2:50 PM · Discovery-Search (Current work), Operations, ops-codfw
Gehel committed rDPOM74e45ead7878: [maven-release-plugin] prepare for next development iteration (authored by Gehel).
[maven-release-plugin] prepare for next development iteration
Thu, Apr 4, 2:04 PM
Gehel committed rDPOM6c83509ea9a6: [maven-release-plugin] prepare release discovery-parent-pom-1.27 (authored by Gehel).
[maven-release-plugin] prepare release discovery-parent-pom-1.27
Thu, Apr 4, 2:04 PM
Gehel committed rDPOMa9282b3e056c: duplicate-finder: ignore Java 9 module-info files (authored by Gehel).
duplicate-finder: ignore Java 9 module-info files
Thu, Apr 4, 2:04 PM

Wed, Apr 3

Gehel updated subscribers of T220038: Degraded RAID on elastic2048.

Node is depooled and excluded from the cluster. @Papaul if you have a spare, feel free to do what needs doing. Ping me when done and I'll reimage.

Wed, Apr 3, 8:43 PM · Discovery-Search (Current work), Operations, ops-codfw
Gehel created P8339 (An Untitled Masterwork).
Wed, Apr 3, 3:04 PM

Tue, Apr 2

Gehel moved T218833: Migrate mjolnir to stdout/syslog/cee logging output from in progress to Done on the Discovery-Search (Current work) board.
Tue, Apr 2, 5:26 PM · Patch-For-Review, Discovery-Search (Current work), Operations, Wikimedia-Logstash
Gehel moved T217967: Publish both shaded and unshaded artifacts from analytics refinery from needs triage to watching / waiting on the Discovery-Search board.
Tue, Apr 2, 5:22 PM · Discovery-Search, Patch-For-Review, Analytics
Gehel edited projects for T217967: Publish both shaded and unshaded artifacts from analytics refinery, added: Discovery-Search; removed Discovery-Search (Current work).
Tue, Apr 2, 5:22 PM · Discovery-Search, Patch-For-Review, Analytics
Gehel moved T215135: Run sonar analysis as a pre-merge step for search platform maven projects from needs triage to Ops / SRE on the Discovery-Search board.
Tue, Apr 2, 5:19 PM · Discovery-Search, Code-Health-Metrics, Continuous-Integration-Config
Gehel edited projects for T215135: Run sonar analysis as a pre-merge step for search platform maven projects, added: Discovery-Search; removed Patch-For-Review, Discovery-Search (Current work).
Tue, Apr 2, 5:19 PM · Discovery-Search, Code-Health-Metrics, Continuous-Integration-Config

Mon, Apr 1

Gehel created T219799: Create cookbook to reset readonly indices on elasticsearch clusters.
Mon, Apr 1, 3:48 PM · Patch-For-Review, Operations, Wikimedia-Incident, Discovery-Search (Current work)
Gehel moved T218878: Upgrade to elasticsearch 6.5.4 for cirrus / codfw from in progress to Done on the Discovery-Search (Current work) board.
Mon, Apr 1, 3:45 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel moved T219640: Make spicerack more robust when unfreezing writes to elasticsearch / cirrus from in progress to Done on the Discovery-Search (Current work) board.
Mon, Apr 1, 3:45 PM · Patch-For-Review, Wikimedia-Incident, Operations, Discovery-Search (Current work)
Gehel moved T219638: Create cookbook to reset frozen write state on elasticsearch / cirrus from in progress to Done on the Discovery-Search (Current work) board.
Mon, Apr 1, 3:44 PM · Patch-For-Review, Discovery-Search (Current work), Operations, Wikimedia-Incident

Fri, Mar 29

Gehel claimed T219640: Make spicerack more robust when unfreezing writes to elasticsearch / cirrus.
Fri, Mar 29, 3:25 PM · Patch-For-Review, Wikimedia-Incident, Operations, Discovery-Search (Current work)
Gehel triaged T219638: Create cookbook to reset frozen write state on elasticsearch / cirrus as High priority.
Fri, Mar 29, 3:25 PM · Patch-For-Review, Discovery-Search (Current work), Operations, Wikimedia-Incident
Gehel claimed T219638: Create cookbook to reset frozen write state on elasticsearch / cirrus.
Fri, Mar 29, 3:25 PM · Patch-For-Review, Discovery-Search (Current work), Operations, Wikimedia-Incident
Gehel created T219640: Make spicerack more robust when unfreezing writes to elasticsearch / cirrus.
Fri, Mar 29, 3:25 PM · Patch-For-Review, Wikimedia-Incident, Operations, Discovery-Search (Current work)
Gehel added a project to T219601: Create checks that alerts on cirrussearch update lags: Wikimedia-Incident.
Fri, Mar 29, 3:24 PM · Discovery-Search (Current work), Patch-For-Review, Wikimedia-Incident, Operations, CirrusSearch, Elasticsearch
Gehel created T219638: Create cookbook to reset frozen write state on elasticsearch / cirrus.
Fri, Mar 29, 3:21 PM · Patch-For-Review, Discovery-Search (Current work), Operations, Wikimedia-Incident

Thu, Mar 28

Gehel created T219507: Create cookbook to reindex into elasticsearch / cirrus.
Thu, Mar 28, 2:10 PM · Operations, Discovery-Search

Wed, Mar 27

Gehel committed rDPOM8a65118be1cc: Add documentation on checkstyle configuration. (authored by Gehel).
Add documentation on checkstyle configuration.
Wed, Mar 27, 8:10 PM
Gehel committed rDPOMf3f7bd813170: Add documentation on checkstyle configuration. (authored by Gehel).
Add documentation on checkstyle configuration.
Wed, Mar 27, 8:10 PM

Mon, Mar 25

Gehel added a comment to T218994: Epic: Deprecation warning on elasticsearch 6 .

I suspect the following would mute the error messages:

curl -XPUT https://search.svc.eqiad.wmnet:9243/_cluster/settings -d '{"transient":{"logger.org.elasticsearch.deprecation.common.ParseField": "ERROR"}}'
Mon, Mar 25, 10:39 AM · Epic, CirrusSearch, Discovery-Search, Operations
hashar awarded T218994: Epic: Deprecation warning on elasticsearch 6 a Love token.
Mon, Mar 25, 8:16 AM · Epic, CirrusSearch, Discovery-Search, Operations

Fri, Mar 22

Gehel added a comment to T218994: Epic: Deprecation warning on elasticsearch 6 .

The elasticsearch security manager is preventing log4j2 to auto-reload it's configuration (more precisely, it can't restart the GELF appender, as socket access is denied). So we will require a full cluster restart to reload the logging configuration. This will be done next week, bundled with the JVM upgrade.

Fri, Mar 22, 3:40 PM · Epic, CirrusSearch, Discovery-Search, Operations
Gehel moved T218879: Upgrade to elasticsearch 6.5.4 for cirrus / eqiad from in progress to Done on the Discovery-Search (Current work) board.
Fri, Mar 22, 3:09 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel claimed T216235: cleanup reprepro configuration for elasticsearch-curator.
Fri, Mar 22, 3:09 PM · Discovery-Search (Current work), Patch-For-Review, User-fgiunchedi, Elasticsearch, Operations
Gehel claimed T218991: update elasticsearch curator to 5.6.0.
Fri, Mar 22, 3:09 PM · Patch-For-Review, Discovery-Search (Current work), Logstash, Operations
Gehel moved T218991: update elasticsearch curator to 5.6.0 from in progress to Done on the Discovery-Search (Current work) board.
Fri, Mar 22, 3:01 PM · Patch-For-Review, Discovery-Search (Current work), Logstash, Operations
Gehel moved T216235: cleanup reprepro configuration for elasticsearch-curator from in progress to Done on the Discovery-Search (Current work) board.
Fri, Mar 22, 2:57 PM · Discovery-Search (Current work), Patch-For-Review, User-fgiunchedi, Elasticsearch, Operations
Gehel edited projects for T216235: cleanup reprepro configuration for elasticsearch-curator, added: Discovery-Search (Current work); removed Discovery-Search.
Fri, Mar 22, 2:57 PM · Discovery-Search (Current work), Patch-For-Review, User-fgiunchedi, Elasticsearch, Operations
Gehel renamed T218995: re-enable deprecation warning logger on elasticsearch once issues are solved from re-enable deprecation warning logger once issues are solved to re-enable deprecation warning logger on elasticsearch once issues are solved.
Fri, Mar 22, 2:30 PM · CirrusSearch, Discovery-Search, Operations
Gehel created T218995: re-enable deprecation warning logger on elasticsearch once issues are solved.
Fri, Mar 22, 2:30 PM · CirrusSearch, Discovery-Search, Operations
Gehel added a comment to T218994: Epic: Deprecation warning on elasticsearch 6 .

disabling this logger for now, let's not forget to re-enable it once we've fixed the underlying issues!

Fri, Mar 22, 2:29 PM · Epic, CirrusSearch, Discovery-Search, Operations
Gehel created T218994: Epic: Deprecation warning on elasticsearch 6 .
Fri, Mar 22, 2:21 PM · Epic, CirrusSearch, Discovery-Search, Operations
Gehel added a comment to T218991: update elasticsearch curator to 5.6.0.

Note that we should take this as an opportunity to fix T216235 as well.

Fri, Mar 22, 12:37 PM · Patch-For-Review, Discovery-Search (Current work), Logstash, Operations
Gehel created T218991: update elasticsearch curator to 5.6.0.
Fri, Mar 22, 12:36 PM · Patch-For-Review, Discovery-Search (Current work), Logstash, Operations

Thu, Mar 21

Gehel added a comment to T218879: Upgrade to elasticsearch 6.5.4 for cirrus / eqiad.

Archived settings were reset. For reference, the settings before the reset:

Thu, Mar 21, 6:50 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel triaged T218879: Upgrade to elasticsearch 6.5.4 for cirrus / eqiad as High priority.
Thu, Mar 21, 1:17 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel triaged T218878: Upgrade to elasticsearch 6.5.4 for cirrus / codfw as High priority.
Thu, Mar 21, 1:17 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel created T218879: Upgrade to elasticsearch 6.5.4 for cirrus / eqiad.
Thu, Mar 21, 1:17 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel created T218878: Upgrade to elasticsearch 6.5.4 for cirrus / codfw.
Thu, Mar 21, 1:16 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel moved T218116: Upgrade relforge to elasticsearch 6.5.4 from in progress to Done on the Discovery-Search (Current work) board.
Thu, Mar 21, 1:15 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel updated subscribers of T218608: OAuth doesn't work when $wgBlockDisablesLogin is true.
Thu, Mar 21, 9:50 AM · cloud-services-team (Kanban), Security-Team, MediaWiki-Authentication-and-authorization, MediaWiki-extensions-OAuth, Security
Gehel committed rDPOMf881c08127c7: Add dependency-check-maven plugin to check for vulnerabilities. (authored by Gehel).
Add dependency-check-maven plugin to check for vulnerabilities.
Thu, Mar 21, 12:14 AM
Gehel committed rDPOM34b27e3baaf3: [maven-release-plugin] prepare for next development iteration (authored by Gehel).
[maven-release-plugin] prepare for next development iteration
Thu, Mar 21, 12:14 AM
Gehel committed rDPOM9b90f16dc024: [maven-release-plugin] prepare release discovery-parent-pom-1.26 (authored by Gehel).
[maven-release-plugin] prepare release discovery-parent-pom-1.26
Thu, Mar 21, 12:14 AM
Gehel committed rDPOMe437081a2990: Add dependency-check-maven plugin to check for vulnerabilities. (authored by Gehel).
Add dependency-check-maven plugin to check for vulnerabilities.
Thu, Mar 21, 12:14 AM
Gehel committed rDPOM3f22d19b40ec: Update dependencies to latest. (authored by Gehel).
Update dependencies to latest.
Thu, Mar 21, 12:14 AM

Wed, Mar 20

Gehel renamed T218550: post-merge jenkins job not run after merge for maven based search projects from post-merge jenkins job not run after merge for search/glent project to post-merge jenkins job not run after merge for maven based search projects.
Wed, Mar 20, 3:17 PM · Continuous-Integration-Config

Mar 19 2019

Gehel moved T218247: Upgrade deployment-elastic* to 6.5.4 from in progress to Done on the Discovery-Search (Current work) board.
Mar 19 2019, 5:26 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel moved T217967: Publish both shaded and unshaded artifacts from analytics refinery from in progress to Waiting/Blocked on the Discovery-Search (Current work) board.
Mar 19 2019, 5:23 PM · Discovery-Search, Patch-For-Review, Analytics
Gehel created T218683: extract throttling filter form wdqs so that it can be reused in other projects.
Mar 19 2019, 3:01 PM · Restricted Project, Wikidata, Wikidata-Query-Service
Gehel added a comment to T218315: cleanup the custom elasticsearch_${version}@ systemd unit in favor of an override configuration.

actually, we're deploying a new unit as a template, so I'm not sure if we can just override the standard one. This will need discussion with someone who understand systemd better than I do.

Mar 19 2019, 9:33 AM · Patch-For-Review, Elasticsearch, Operations, Discovery-Search

Mar 18 2019

Gehel created T218550: post-merge jenkins job not run after merge for maven based search projects.
Mar 18 2019, 10:21 AM · Continuous-Integration-Config

Mar 14 2019

Gehel triaged T218315: cleanup the custom elasticsearch_${version}@ systemd unit in favor of an override configuration as High priority.
Mar 14 2019, 4:02 PM · Patch-For-Review, Elasticsearch, Operations, Discovery-Search
Gehel moved T218315: cleanup the custom elasticsearch_${version}@ systemd unit in favor of an override configuration from needs triage to Ops / SRE on the Discovery-Search board.
Mar 14 2019, 4:02 PM · Patch-For-Review, Elasticsearch, Operations, Discovery-Search
Gehel created T218315: cleanup the custom elasticsearch_${version}@ systemd unit in favor of an override configuration.
Mar 14 2019, 4:01 PM · Patch-For-Review, Elasticsearch, Operations, Discovery-Search

Mar 13 2019

Gehel created T218247: Upgrade deployment-elastic* to 6.5.4.
Mar 13 2019, 7:37 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel moved T218113: Upload elastic 6.5.4 packages to reprepro from in progress to Done on the Discovery-Search (Current work) board.
Mar 13 2019, 7:37 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel moved T218099: Maven Wrapper does not support XDG_CACHE_HOME from in progress to Done on the Discovery-Search (Current work) board.
Mar 13 2019, 8:55 AM · Patch-For-Review, Discovery-Search (Current work), Continuous-Integration-Config

Mar 12 2019

Gehel moved T217945: Updater dashboards broken from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.
Mar 12 2019, 5:13 PM · Discovery-Wikidata-Query-Service-Sprint, Wikidata, Wikidata-Query-Service
Gehel assigned T217945: Updater dashboards broken to Mathew.onipe.

The above patch will allow prometheus to collect the metrics after the domain was changed. We still need to update the dashboards.

Mar 12 2019, 5:13 PM · Discovery-Wikidata-Query-Service-Sprint, Wikidata, Wikidata-Query-Service
fgiunchedi awarded T216052: upgrade logstash and the logstash elasticsearch cluster to 5.6.14 a Like token.
Mar 12 2019, 5:00 PM · Discovery-Search (Current work), Patch-For-Review, Wikimedia-Logstash, Operations
Gehel created T218116: Upgrade relforge to elasticsearch 6.5.4.
Mar 12 2019, 3:49 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel created T218113: Upload elastic 6.5.4 packages to reprepro.
Mar 12 2019, 3:32 PM · Patch-For-Review, Discovery-Search (Current work)
Gehel moved T193654: [epic] Run multiple elasticsearch clusters on same hardware from in progress to Done on the Discovery-Search (Current work) board.
Mar 12 2019, 3:30 PM · Discovery-Search (Current work), Epic
Gehel edited projects for T193654: [epic] Run multiple elasticsearch clusters on same hardware, added: Discovery-Search (Current work); removed Discovery-Search.
Mar 12 2019, 3:30 PM · Discovery-Search (Current work), Epic
Gehel moved T217196: Adapt elasticsearch puppet module for elasticsearch6 from Needs review to Done on the Discovery-Search (Current work) board.
Mar 12 2019, 3:29 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch
Gehel awarded T204506: cloudvps: maps project trusty deprecation a Love token.
Mar 12 2019, 2:08 PM · cloud-services-team (Kanban), User-TheDJ, Cloud-VPS (Ubuntu Trusty Deprecation), Maps
Gehel created T218099: Maven Wrapper does not support XDG_CACHE_HOME.
Mar 12 2019, 1:59 PM · Patch-For-Review, Discovery-Search (Current work), Continuous-Integration-Config