Page MenuHomePhabricator

Mathew.onipe (onimisionipe)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Aug 22 2018, 7:21 AM (67 w, 2 d)
Availability
Available
LDAP User
Mathew.onipe
MediaWiki User
Unknown

Recent Activity

Today

Mathew.onipe updated the task description for T239728: Re-import OSM data at eqiad and codfw to temporarily fix current OSM replication issues..
Fri, Dec 6, 9:37 AM · Operations, Maps
Mathew.onipe closed T230088: cloudelastic1002: SMART/disk error as Resolved.
Fri, Dec 6, 9:07 AM · ops-eqiad, DC-Ops, Operations, cloud-services-team (Kanban)

Yesterday

Mathew.onipe added a comment to T230088: cloudelastic1002: SMART/disk error.
onimisionipe@cloudelastic1002:~$ sudo smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-9-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Thu, Dec 5, 10:32 PM · ops-eqiad, DC-Ops, Operations, cloud-services-team (Kanban)

Wed, Dec 4

Mathew.onipe added a comment to T239728: Re-import OSM data at eqiad and codfw to temporarily fix current OSM replication issues..
Wed, Dec 4, 12:19 PM · Operations, Maps

Tue, Dec 3

Mathew.onipe created T239728: Re-import OSM data at eqiad and codfw to temporarily fix current OSM replication issues..
Tue, Dec 3, 4:46 PM · Operations, Maps

Thu, Nov 28

Mathew.onipe updated subscribers of T239389: Grant merge permission to discovery-team members on https://gerrit.wikimedia.org/r/admin/projects/wikidata/query/deploy.
Thu, Nov 28, 12:36 PM · Release-Engineering-Team, Gerrit-Privilege-Requests, Discovery-Search (Current work)
Mathew.onipe triaged T239389: Grant merge permission to discovery-team members on https://gerrit.wikimedia.org/r/admin/projects/wikidata/query/deploy as Medium priority.
Thu, Nov 28, 8:59 AM · Release-Engineering-Team, Gerrit-Privilege-Requests, Discovery-Search (Current work)
Mathew.onipe created T239389: Grant merge permission to discovery-team members on https://gerrit.wikimedia.org/r/admin/projects/wikidata/query/deploy.
Thu, Nov 28, 8:59 AM · Release-Engineering-Team, Gerrit-Privilege-Requests, Discovery-Search (Current work)

Mon, Nov 25

Mathew.onipe added a comment to T238822: XSS in Wikidata Query Service UI (i18n messages) - CVE-2019-19327.

I have deployed Lucas's patch. Kindly test this to confirm everything is fine.

Mon, Nov 25, 9:31 AM · User-Addshore, Security-Team, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikidata Query UI, Security

Wed, Nov 20

Mathew.onipe created T238733: Push rights on https://gerrit.wikimedia.org/r/admin/projects/wikidata/query/blazegraph for onimisionipe.
Wed, Nov 20, 11:02 AM · Gerrit-Privilege-Requests, Wikidata, Release-Engineering-Team (Unit & Int & System Tooling), Operations, Wikidata-Query-Service

Mon, Nov 18

Mathew.onipe committed rWDQGa5c60a2aaf62: 0.3.7-SNAPSHOT (authored by Mathew.onipe).
0.3.7-SNAPSHOT
Mon, Nov 18, 5:43 PM
Mathew.onipe moved T238408: Metrics from the wdqs updater are no longer collected from In Progress to Done on the Discovery-Search (Current work) board.
Mon, Nov 18, 5:04 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Fri, Nov 15

Mathew.onipe moved T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs from In Progress to Done on the Discovery-Search (Current work) board.
Fri, Nov 15, 11:55 AM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata

Wed, Nov 13

Mathew.onipe added a comment to T238232: blazegraph journal on wdqs1005 is oversized.

@Igorkim78 file has been uploaded.

Wed, Nov 13, 5:28 PM · Wikidata-Query-Service, Discovery-Search (Current work), Wikidata
Mathew.onipe added a comment to T238232: blazegraph journal on wdqs1005 is oversized.

Wed, Nov 13, 5:26 PM · Wikidata-Query-Service, Discovery-Search (Current work), Wikidata

Nov 6 2019

Physikerwelt awarded T233213: XSS in Wikidata Query Service UI, DATATYPE_MATHML - CVE-2019-19329 a Dislike token.
Nov 6 2019, 9:13 AM · Discovery-Search (Current work), User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Vuln-XSS, Wikidata, Wikidata Query UI, Security
Mathew.onipe added a comment to T233213: XSS in Wikidata Query Service UI, DATATYPE_MATHML - CVE-2019-19329.

Sorry all.. I just did another deployment and patch has been correctly applied. Also test works fine now

Nov 6 2019, 9:12 AM · Discovery-Search (Current work), User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Vuln-XSS, Wikidata, Wikidata Query UI, Security
Mathew.onipe added a comment to T233213: XSS in Wikidata Query Service UI, DATATYPE_MATHML - CVE-2019-19329.

I've deployed @Lucas_Werkmeister_WMDE patch. But I can still run:

Nov 6 2019, 7:55 AM · Discovery-Search (Current work), User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Vuln-XSS, Wikidata, Wikidata Query UI, Security

Nov 5 2019

Mathew.onipe added a comment to T231446: Reindex commonswiki as shards have grown beyond critical threshold.

T230746 is almost here. So we won't be resharding any index for now

Nov 5 2019, 4:58 PM · Discovery-Search, Patch-For-Review, Operations, Elasticsearch

Nov 4 2019

Mathew.onipe added a comment to T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day.

see https://phabricator.wikimedia.org/T237228 for current OSM replication issues

Nov 4 2019, 9:57 AM · Operations, Maps
Mathew.onipe triaged T237228: OSM Replication failed at eqiad and codfw as High priority.
Nov 4 2019, 8:52 AM · Operations, Maps
Mathew.onipe created T237228: OSM Replication failed at eqiad and codfw.
Nov 4 2019, 8:52 AM · Operations, Maps
Mathew.onipe closed T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day as Invalid.
Nov 4 2019, 8:45 AM · Operations, Maps
Mathew.onipe added a comment to T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day.

I'm closing this task as there are icinga alerts for osm sync

Nov 4 2019, 8:44 AM · Operations, Maps
Mathew.onipe updated subscribers of T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day.
Nov 4 2019, 8:43 AM · Operations, Maps
Mathew.onipe updated subscribers of T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day.
Nov 4 2019, 8:42 AM · Operations, Maps
Mathew.onipe raised the priority of T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day from Medium to High.
Nov 4 2019, 8:41 AM · Operations, Maps
Mathew.onipe added a comment to T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day.

We have an alert for this. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=icinga1001&service=Maps+-+OSM+synchronization+lag+-+eqiad

Nov 4 2019, 8:41 AM · Operations, Maps
Mathew.onipe lowered the priority of T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day from High to Medium.
Nov 4 2019, 8:40 AM · Operations, Maps
Mathew.onipe triaged T237209: Alert SRE if Wikipedia Maps replication sync lag exceeds 1 day as High priority.
Nov 4 2019, 8:39 AM · Operations, Maps

Nov 1 2019

Mathew.onipe triaged T237089: Create CQS puppet configs by applying query_service module as Medium priority.
Nov 1 2019, 8:05 AM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe created T237089: Create CQS puppet configs by applying query_service module.
Nov 1 2019, 8:05 AM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe renamed T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs from Create puppet configs for SDC query to Refactor Puppet WDQS module to make it usable for wdqs and cqs.
Nov 1 2019, 8:01 AM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata

Oct 29 2019

Mathew.onipe added a comment to T233213: XSS in Wikidata Query Service UI, DATATYPE_MATHML - CVE-2019-19329.

I’ve updated P9315 after a MathJax developer commented on the MathML incompatibility issue (it seems to be very limited in scope and can be worked around); I think it should be good enough to deploy as a first version that will fix the XSS.
@Mathew.onipe, @Gehel: I think the process to deploy this without making the fix public prematurely would be:

  1. Turn P9315 into a patch against wikidata/query/gui-deploy
  2. On the deployment server (is that deployment.eqiad.wmnet or a different one, btw?), apply that patch in the gui submodule of the wikidata/query/deploy checkout (/src/deployment/wdqs/wdqs?)
  3. Commit that patch to the gui submodule, and then commit the submodule update in the main deploy repository? Not sure if this is necessary
  4. Deploy the current tree (scap deploy?)
  5. Verify that the vulnerability is fixed
  6. Coordinate pre-announcements?
  7. Upload the fix to Gerrit to the wikidata/query/gui repository
  8. Merge it, causing a corresponding change to wikidata/query/gui-deploy to be built
  9. Merge that change
  10. Update gui submodule in wikidata/query/deploy repository, bringing it in line with the deployment server’s version
  11. Announce the fix
  12. Discuss any further changes to math rendering in T214980: Support mathematical formulae in Wikidata Query Service UI on all browsers (public)

Does that sound right to you?

Oct 29 2019, 9:22 AM · Discovery-Search (Current work), User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Vuln-XSS, Wikidata, Wikidata Query UI, Security

Oct 27 2019

Mathew.onipe updated subscribers of T236601: Degraded RAID on elastic1039.
Oct 27 2019, 6:53 PM · Discovery-Search (Current work), ops-eqiad, Operations
Mathew.onipe added a comment to T236601: Degraded RAID on elastic1039.

This is causing mjolnir deploy directory to become unavailable/missing and also causing puppet to fail.

Oct 27 2019, 6:53 PM · Discovery-Search (Current work), ops-eqiad, Operations

Oct 21 2019

Mathew.onipe edited projects for T233403: Unassigned shards in eqiad, added: Discovery-Search (Current work); removed Discovery-Search.
Oct 21 2019, 12:58 PM · Discovery-Search (Current work), Operations, Elasticsearch
Mathew.onipe added a comment to T233403: Unassigned shards in eqiad.

Issue still persist: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=search.svc.eqiad.wmnet&service=ElasticSearch+unassigned+shard+check+-+9243

Oct 21 2019, 12:57 PM · Discovery-Search (Current work), Operations, Elasticsearch

Oct 16 2019

Mathew.onipe added a comment to T235540: SPARQL query causes StackOverflowError and fails to execute.
Oct 16 2019, 12:09 PM · Wikidata, Wikidata-Query-Service

Oct 10 2019

Mathew.onipe triaged T235159: Enable write access for Mathew.onipe(onimisionipe) and gehel on wikidata gui repo as Medium priority.
Oct 10 2019, 9:37 AM · Discovery-Search (Current work), Wikidata-Query-Service, Gerrit-Privilege-Requests, Wikidata
Mathew.onipe created T235159: Enable write access for Mathew.onipe(onimisionipe) and gehel on wikidata gui repo.
Oct 10 2019, 9:37 AM · Discovery-Search (Current work), Wikidata-Query-Service, Gerrit-Privilege-Requests, Wikidata

Oct 9 2019

Mathew.onipe committed rMSKA217ddd64b9d3: allow npm install devDeps (authored by Mathew.onipe).
allow npm install devDeps
Oct 9 2019, 10:35 PM
Mathew.onipe committed rMSKAa6bafe419667: Add copies directive to build stage (authored by Mathew.onipe).
Add copies directive to build stage
Oct 9 2019, 10:35 PM
Mathew.onipe added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.

@dduvall Thanks!. I removed the test stage also forced devdeps to install. We should definitely look at a better way to handle this later. but Its fine as it is.
Currently, Build is passing but not publishing yet. Do we need to enable CI publish stage for the repo?

Oct 9 2019, 10:31 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
Mathew.onipe added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.

@dduvall Thanks. I will implement this.

Oct 9 2019, 11:24 AM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)

Oct 3 2019

Mathew.onipe added a comment to T233403: Unassigned shards in eqiad.

This issue has come up again. Currently, we have only enwiki_content_1546970425 unassigned with too many shards [1] allocated to this node for index [enwiki_content_1546970425], index setting index.routing.allocation.total_shards_per_node=1] error from _cluster/allocation/explain.

Oct 3 2019, 1:05 PM · Discovery-Search (Current work), Operations, Elasticsearch

Oct 1 2019

Mathew.onipe added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.

Post merge builds seems to fail.
https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/539209

Oct 1 2019, 12:31 AM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)

Sep 24 2019

Mathew.onipe added a comment to T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline.

We should talk to elastic to see how we can move this forward.
Currently, we require jackson-databind 2.8.11 and jackson-annotation 2.8.11 for JsonLayout to work when using SyslogAppender. Version 2.8.6 is provided by debian for this packages. We should use the correct version to make sure everything work as expected.

Sep 24 2019, 5:49 PM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash
Mathew.onipe moved T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline from Needs review to Blocked on the Discovery-Search (Current work) board.
Sep 24 2019, 5:45 PM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash
Mathew.onipe moved T232184: MIgrate WDQS to new logging pipeline from Waiting to Done on the Discovery-Search (Current work) board.
Sep 24 2019, 5:40 PM · Discovery-Search (Current work), Wikimedia-Logstash, Operations, observability, Product-Analytics, Discovery-Analysis (Current work), Wikidata-Query-Service, Wikidata

Sep 23 2019

Mathew.onipe triaged T233578: hw troubleshooting: Memory correctable errors -EDAC- for elastic1029.eqiad.wmnet as Medium priority.
Sep 23 2019, 8:06 AM · Operations, ops-eqiad, DC-Ops
Mathew.onipe created T233578: hw troubleshooting: Memory correctable errors -EDAC- for elastic1029.eqiad.wmnet.
Sep 23 2019, 8:05 AM · Operations, ops-eqiad, DC-Ops

Sep 20 2019

Mathew.onipe created T233403: Unassigned shards in eqiad.
Sep 20 2019, 12:51 PM · Discovery-Search (Current work), Operations, Elasticsearch

Sep 18 2019

Mathew.onipe moved T232184: MIgrate WDQS to new logging pipeline from In Progress to Waiting on the Discovery-Search (Current work) board.
Sep 18 2019, 1:23 PM · Discovery-Search (Current work), Wikimedia-Logstash, Operations, observability, Product-Analytics, Discovery-Analysis (Current work), Wikidata-Query-Service, Wikidata

Sep 16 2019

Mathew.onipe created T233039: hw troubleshooting: <type of hardware failre> for <fqhn of server>.
Sep 16 2019, 5:53 PM · DC-Ops
Mathew.onipe closed T201991: Broken memory on elastic1029 as Resolved.
Sep 16 2019, 5:50 PM · Operations, ops-eqiad
Mathew.onipe reopened T201991: Broken memory on elastic1029 as "Open".
Sep 16 2019, 5:46 PM · Operations, ops-eqiad

Sep 12 2019

Mathew.onipe added a comment to T176875: Allow access to wdqs.svc.eqiad.wmnet on port 8888.

@Ladsgroup there's no TLS termination on that port for now. We should have and I will work on it in the nearest future. Please use HTTP for now

Sep 12 2019, 10:24 AM · Patch-For-Review, Traffic, Wikidata-Query-Service, Operations, WMDE-Analytics-Engineering, User-Addshore, Discovery, Wikidata
Mathew.onipe added a comment to T176875: Allow access to wdqs.svc.eqiad.wmnet on port 8888.

@Addshore @Ladsgroup @WMDE-leszek, can you test that you can reach wdqs.svc.eqiad.wmnet on port 8888. LVS and other appropriate changes have been merged and It should work. Thanks

Sep 12 2019, 8:54 AM · Patch-For-Review, Traffic, Wikidata-Query-Service, Operations, WMDE-Analytics-Engineering, User-Addshore, Discovery, Wikidata

Sep 11 2019

Mathew.onipe updated the task description for T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs.
Sep 11 2019, 6:45 AM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata

Sep 10 2019

Mathew.onipe moved T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline from In Progress to Needs review on the Discovery-Search (Current work) board.
Sep 10 2019, 1:50 PM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash
Mathew.onipe added a project to T232184: MIgrate WDQS to new logging pipeline: Discovery-Search (Current work).
Sep 10 2019, 3:12 AM · Discovery-Search (Current work), Wikimedia-Logstash, Operations, observability, Product-Analytics, Discovery-Analysis (Current work), Wikidata-Query-Service, Wikidata

Sep 9 2019

Mathew.onipe updated the task description for T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs.
Sep 9 2019, 2:49 PM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe updated the task description for T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs.
Sep 9 2019, 2:46 PM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe updated the task description for T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs.
Sep 9 2019, 2:42 PM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe added a project to T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs: Operations.
Sep 9 2019, 2:36 PM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe updated the task description for T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs.
Sep 9 2019, 2:21 PM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe triaged T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs as Medium priority.
Sep 9 2019, 7:42 AM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata
Mathew.onipe created T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs.
Sep 9 2019, 7:41 AM · Structured-Data-Backlog, Structured Data Engineering, Operations, Discovery-Search (Current work), SDC General, Wikidata

Sep 6 2019

Mathew.onipe added a comment to T232224: September 2019 DoS attacks [Public].

This is a know issue. The SRE team is finding a quick solution to restore these services. Thanks

Sep 6 2019, 6:22 PM · Wikimedia-Incident, Operations
Mathew.onipe added a comment to T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline.

JsonLayout requires other dependencies for log4j. This include jackson databind. See https://logging.apache.org/log4j/2.x/runtime-dependencies.html.
Two options:

  1. Rebuild log4j with this dependencies
  2. Fall back to shipping logs with PatternLayout.
Sep 6 2019, 11:19 AM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash
Mathew.onipe updated subscribers of T228483: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos.

Let's wait for @MSantos or @Mholloway opinion before deleting those repos please

Sep 6 2019, 7:40 AM · User-MarcoAurelio, Release-Engineering-Team-TODO, Release-Engineering-Team (Development services), Repository-Admins, Maps, Cleanup
Mathew.onipe triaged T232184: MIgrate WDQS to new logging pipeline as Medium priority.
Sep 6 2019, 7:36 AM · Discovery-Search (Current work), Wikimedia-Logstash, Operations, observability, Product-Analytics, Discovery-Analysis (Current work), Wikidata-Query-Service, Wikidata
Mathew.onipe created T232184: MIgrate WDQS to new logging pipeline.
Sep 6 2019, 7:36 AM · Discovery-Search (Current work), Wikimedia-Logstash, Operations, observability, Product-Analytics, Discovery-Analysis (Current work), Wikidata-Query-Service, Wikidata

Sep 4 2019

Mathew.onipe added a comment to T231928: CI service-pipeline-test-and-publish job assumes blubber config has a single production image.

Not sure but seems we are missing some configs in our config.yaml patch

Sep 4 2019, 11:20 AM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO, Maps (Kartotherian), Product-Infrastructure-Team-Backlog

Sep 3 2019

Mathew.onipe added a comment to T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline.

rsyslog Json requires the @cee token which must be provided according to standard via profile::rsyslog::udp_localhost_compat. Let's use profile::rsyslog::udp_json_logback_compat instead as it permits parsing of json from log4j without the token.

Sep 3 2019, 3:50 PM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash

Sep 2 2019

Mathew.onipe moved T231516: Alert when a jvm hits more than 100 old gc ops/hour from Needs review to Done on the Discovery-Search (Current work) board.
Sep 2 2019, 9:33 AM · observability, Operations, Discovery-Search (Current work)

Aug 29 2019

Mathew.onipe added a comment to T231516: Alert when a jvm hits more than 100 old gc ops/hour.

On another note, I think this check make sense for other clusters as well

Aug 29 2019, 11:10 AM · observability, Operations, Discovery-Search (Current work)
Mathew.onipe edited projects for T231516: Alert when a jvm hits more than 100 old gc ops/hour, added: Discovery-Search (Current work), Operations; removed Discovery-Search.
Aug 29 2019, 8:32 AM · observability, Operations, Discovery-Search (Current work)
Mathew.onipe claimed T231516: Alert when a jvm hits more than 100 old gc ops/hour.
Aug 29 2019, 8:31 AM · observability, Operations, Discovery-Search (Current work)
Mathew.onipe reopened T214283: Memory correctable errors -EDAC- elastic1029 as "Open".

elastic1029 is back on icinga showing memory errors. see https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=elastic1029&service=Memory+correctable+errors+-EDAC-

Aug 29 2019, 3:03 AM · Discovery-Search (Current work), ops-eqiad, Discovery, DC-Ops, Operations

Aug 28 2019

Mathew.onipe added a comment to P8995 Khmer samples.

My screenshot from Windows 10/Version 76.0.3809.100 (Official Build) (64-bit)

Aug 28 2019, 2:44 PM · Discovery-Search
Mathew.onipe triaged T231446: Reindex commonswiki as shards have grown beyond critical threshold as Medium priority.
Aug 28 2019, 1:03 PM · Discovery-Search, Patch-For-Review, Operations, Elasticsearch
Mathew.onipe created T231446: Reindex commonswiki as shards have grown beyond critical threshold.
Aug 28 2019, 1:03 PM · Discovery-Search, Patch-For-Review, Operations, Elasticsearch
Mathew.onipe added a comment to T230774: Run jstack / jmap / etc... with PrivateTmp=true.

@Gehel I think you meant: https://wikitech.wikimedia.org/wiki/Search#Using_jstack_or_jmap_or_other_similar_tools_to_view_logs

Aug 28 2019, 3:47 AM · Discovery-Search (Current work), Operations
Mathew.onipe added a comment to T229980: Need help to create and deploy Debian-packaged Python 3 app.

I changed the priority of this to normal. Feel free to change it as you see fit

Aug 28 2019, 3:27 AM · serviceops, Operations, Packaging, CPT Initiatives (Session Management Service (CDP2))
Mathew.onipe triaged T229980: Need help to create and deploy Debian-packaged Python 3 app as Medium priority.
Aug 28 2019, 3:26 AM · serviceops, Operations, Packaging, CPT Initiatives (Session Management Service (CDP2))
Mathew.onipe updated subscribers of T231274: Have a link to the alert in the icinga alert email.
Aug 28 2019, 3:25 AM · Icinga, observability, Operations
Mathew.onipe triaged T231274: Have a link to the alert in the icinga alert email as Medium priority.
Aug 28 2019, 3:25 AM · Icinga, observability, Operations

Aug 27 2019

Mathew.onipe claimed T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline.
Aug 27 2019, 5:18 PM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash
Mathew.onipe triaged T231010: Change partitioning scheme for elasticsearch from RAID to JBOD as Medium priority.
Aug 27 2019, 2:51 AM · Operations, Discovery-Search, Elasticsearch
Mathew.onipe updated subscribers of T230994: labweb100[12]: Search backend error during get of .[array] after 0: unknown: No enabled connection.
Aug 27 2019, 2:45 AM · Operations, Discovery-Search (Current work), CirrusSearch
Mathew.onipe assigned T230994: labweb100[12]: Search backend error during get of .[array] after 0: unknown: No enabled connection to dcausse.
Aug 27 2019, 2:43 AM · Operations, Discovery-Search (Current work), CirrusSearch
Mathew.onipe added a comment to T227529: Request rename of "waldir" to "waldyrious" on LDAP.

@thcipriani any update on this? seems stalled or partially resolved.

Aug 27 2019, 2:42 AM · LDAP-Access-Requests
Mathew.onipe added a comment to T227695: Requesting access to analytics-privatedata-users for mbsantos.

@MSantos
what's the latest on this? Do you want to follow up on Nuria?

Aug 27 2019, 2:35 AM · SRE-Access-Requests, Operations
Mathew.onipe closed T231111: Access to HUE for cchen as Resolved.

I'm guessing everyone is happy so I'm going to close this.

Aug 27 2019, 2:32 AM · Analytics-Kanban, SRE-Access-Requests, Operations, Analytics

Aug 26 2019

Mathew.onipe edited projects for T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline, added: Discovery-Search (Current work); removed Discovery-Search.
Aug 26 2019, 2:26 PM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash
Mathew.onipe added a comment to T231009: Make jobprocessor's test not depend on external files.

@Joe here is the error: https://integration.wikimedia.org/ci/blue/organizations/jenkins/service-pipeline-test/detail/service-pipeline-test-and-publish/3100/pipeline/
Thanks!

Aug 26 2019, 9:20 AM · Release Pipeline, Operations, Maps (Kartotherian)

Aug 22 2019

Mathew.onipe created T231010: Change partitioning scheme for elasticsearch from RAID to JBOD.
Aug 22 2019, 2:05 PM · Operations, Discovery-Search, Elasticsearch
Mathew.onipe created T231009: Make jobprocessor's test not depend on external files.
Aug 22 2019, 1:52 PM · Release Pipeline, Operations, Maps (Kartotherian)