Page MenuHomePhabricator

RKemper (Ryan Kemper)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
May 1 2020, 10:28 PM (20 w, 3 d)
Availability
Available
LDAP User
Unknown
MediaWiki User
RKemper (WMF) [ Global Accounts ]

Recent Activity

Today

RKemper moved T260083: Reshard commonswiki_file elasticsearch index from Ready for Development to In Progress on the Discovery-Search (Current work) board.
Tue, Sep 22, 2:21 AM · Patch-For-Review, Operations, Discovery-Search (Current work)
RKemper added a comment to T263073: Large, steady increase in unprocessed cloudelastic job.cirrusSearchElasticaWrite messages.

Not-at-all-important question about ProductionServices.php's format:

Tue, Sep 22, 1:54 AM · Patch-For-Review, Discovery-Search (Current work)

Fri, Sep 18

RKemper added a comment to T263073: Large, steady increase in unprocessed cloudelastic job.cirrusSearchElasticaWrite messages.

Dumping some stream-of-conscioussness / notes here:

Fri, Sep 18, 1:37 AM · Patch-For-Review, Discovery-Search (Current work)

Thu, Sep 17

RKemper added a comment to T233351: Increase poolcounter for wikibase prefix search.

We might want to add alerting on the value of scaleToSeconds(consolidateBy(transformNull(MediaWiki.CirrusSearch.poolCounter.Search.failureMs.sample_rate, 0), 'max'), 60), for example if it averages above 1000 for a 5 minute period. See https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?viewPanel=65&orgId=1&from=now-6h&to=now for a graph of pool counter rejections over time

Thu, Sep 17, 5:56 PM · Discovery-Search, Elasticsearch

Wed, Sep 16

RKemper updated the task description for T263073: Large, steady increase in unprocessed cloudelastic job.cirrusSearchElasticaWrite messages.
Wed, Sep 16, 10:36 PM · Patch-For-Review, Discovery-Search (Current work)
RKemper created T263073: Large, steady increase in unprocessed cloudelastic job.cirrusSearchElasticaWrite messages.
Wed, Sep 16, 10:35 PM · Patch-For-Review, Discovery-Search (Current work)

Mon, Sep 14

RKemper updated the task description for T262630: ProductionServices.php has cloudelastic-{psi,omega}-eqiad ports mixed up.
Mon, Sep 14, 3:44 PM · Discovery-Search
RKemper added a comment to T262694: Search is currently too busy.

Moving forward we've decided to add the above proposed alert before closing this ticket.

Mon, Sep 14, 3:41 PM · Discovery-Search (Current work), Operations, Elasticsearch, CirrusSearch

Sat, Sep 12

RKemper moved T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder from Needs review to Needs Reporting on the Discovery-Search (Current work) board.

We'll want to monitor across the next week to verify all looks good.

Sat, Sep 12, 10:26 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service, Dumps-Generation
RKemper added a comment to T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder.
dumpsgen  58563  0.0  0.0   4276   700 ?        Ss   20:50   0:00 /bin/sh -c python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py --configfile /etc/dumps/confs/addschanges.conf --dumptype incrdumps --quiet
dumpsgen  58565  0.0  0.0  57976 18128 ?        S    20:50   0:03 python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py --configfile /etc/dumps/confs/addschanges.conf --dumptype incrdumps --quiet
dumpsgen  63246  3.9  0.0 456952 64728 ?        S    21:08   2:51 /usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php dumpTextPass.php --wiki=commonswiki --stub=gzip:/mnt/dumpsdata/otherdumps/incr/commonswiki/20200912/commonswiki-20200912-stubs-meta-hist-incr.xml.gz --quiet --spawn=/usr/bin/php7.2 --output=bzip2:/mnt/dumpsdata/otherdumps/incr/commonswiki/20200912/commonswiki-20200912-pages-meta-hist-incr.xml.bz2
dumpsgen  63248  0.0  0.0   4276   740 ?        S    21:08   0:00 sh -c bzip2 > '/mnt/dumpsdata/otherdumps/incr/commonswiki/20200912/commonswiki-20200912-pages-meta-hist-incr.xml.bz2'
dumpsgen  63249  2.9  0.0  13604  8492 ?        S    21:08   2:08 bzip2
dumpsgen  63250  0.0  0.0   4276   736 ?        S    21:08   0:00 sh -c '/usr/bin/php7.2' '/srv/mediawiki/php-1.36.0-wmf.8/../multiversion/MWScript.php' 'fetchText.php' '--wiki' 'commonswiki'
dumpsgen  63251 18.5  0.0 453812 63484 ?        S    21:08  13:24 /usr/bin/php7.2 /srv/mediawiki/php-1.36.0-wmf.8/../multiversion/MWScript.php fetchText.php --wiki commonswiki
Sat, Sep 12, 10:25 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service, Dumps-Generation

Fri, Sep 11

RKemper added a comment to T262694: Search is currently too busy.

Following up here:

Fri, Sep 11, 10:18 PM · Discovery-Search (Current work), Operations, Elasticsearch, CirrusSearch

Thu, Sep 10

RKemper moved T261373: Pool and depool scripts not present on cloudelastic100[5,6].wikimedia.org from Incoming to Needs Reporting on the Discovery-Search (Current work) board.
Thu, Sep 10, 5:57 PM · Discovery-Search (Current work), Patch-For-Review, Discovery
RKemper added a project to T261373: Pool and depool scripts not present on cloudelastic100[5,6].wikimedia.org: Discovery-Search (Current work).
Thu, Sep 10, 5:56 PM · Discovery-Search (Current work), Patch-For-Review, Discovery
RKemper added a comment to T261373: Pool and depool scripts not present on cloudelastic100[5,6].wikimedia.org.
~ % ssh cloudelastic1005.wikimedia.org
Linux cloudelastic1005 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64
Debian GNU/Linux 9.13 (stretch)
cloudelastic1005 is a elasticsearch cloud elastic cirrus (elasticsearch::cloudelastic)
The last Puppet run was at Thu Sep 10 17:53:38 UTC 2020 (0 minutes ago).
Last puppet commit: (c2f9c42c5c) Ryan Kemper - cloudelastic: we do want to use conf_tool
Debian GNU/Linux 9 auto-installed on Tue May 5 14:54:52 UTC 2020.
Last login: Fri Aug 28 21:43:55 2020 from 2620:0:863:1:198:35:26:6
ryankemper@cloudelastic1005:~$ which pool
/usr/local/bin/pool
ryankemper@cloudelastic1005:~$ which depool
/usr/local/bin/depool
ryankemper@cloudelastic1005:~$
Thu, Sep 10, 5:55 PM · Discovery-Search (Current work), Patch-For-Review, Discovery
RKemper updated the task description for T261373: Pool and depool scripts not present on cloudelastic100[5,6].wikimedia.org.
Thu, Sep 10, 5:55 PM · Discovery-Search (Current work), Patch-For-Review, Discovery
RKemper added a comment to T260271: (Need By: TBD) rack/setup/install maps20[05-10].codfw.wmnet.

@Papaul We should be fine with 256k.

Thu, Sep 10, 5:20 PM · Operations, Maps, ops-codfw, DC-Ops
RKemper added a comment to T260271: (Need By: TBD) rack/setup/install maps20[05-10].codfw.wmnet.

@Papaul Sorry for the delayed response!

Thu, Sep 10, 6:04 AM · Operations, Maps, ops-codfw, DC-Ops

Tue, Sep 8

RKemper added a comment to T260271: (Need By: TBD) rack/setup/install maps20[05-10].codfw.wmnet.

@Papaul We'd like to use RAID10 as our hardware RAID.

Tue, Sep 8, 6:35 PM · Operations, Maps, ops-codfw, DC-Ops
RKemper updated the task description for T260271: (Need By: TBD) rack/setup/install maps20[05-10].codfw.wmnet.
Tue, Sep 8, 6:29 PM · Operations, Maps, ops-codfw, DC-Ops
RKemper triaged T261786: WDQS 0.3.46 deploy failure as Medium priority.
Tue, Sep 8, 5:37 PM · Discovery-Search (Current work)

Fri, Sep 4

RKemper created T262002: Improve visibility into blazegraph queries causing errors.
Fri, Sep 4, 1:11 AM · Wikidata-Query-Service, Wikidata, Sustainability (Incident Followup)

Thu, Sep 3

RKemper triaged T261920: docker-registry.wikimedia.org/golang:1.11 should no more depends on stretch-backports as Medium priority.
Thu, Sep 3, 9:33 PM · Operations
RKemper triaged T261966: Move cassandra puppet code to profile::java as Medium priority.
Thu, Sep 3, 9:32 PM · Cassandra, Operations
RKemper triaged T261962: Migrate install_server::web_server (apt*) to nginx-light as Medium priority.
Thu, Sep 3, 9:32 PM · User-ArielGlenn, Traffic, Operations

Wed, Sep 2

RKemper added a comment to T247556: [Epic] Search platform - Hardware requests for 2020-2021.

For maps, I synced with DC-Ops and earliest we can realistically expect the MAPS servers to be racked is late October. (Naturally it could take longer)

Wed, Sep 2, 5:09 PM · Epic, Discovery-Search

Tue, Sep 1

RKemper updated the task description for T261786: WDQS 0.3.46 deploy failure.
Tue, Sep 1, 8:26 PM · Discovery-Search (Current work)
RKemper removed a project from T101914: Search service monitoring should fail if search results only return exact matches and suggestions don't work: Operations.
Tue, Sep 1, 7:43 PM · Discovery-Search, Discovery, MediaWiki-Search, observability
RKemper updated the task description for T143553: Switching search traffic between datacenters should be faster.
Tue, Sep 1, 7:21 PM · Discovery, Discovery-Search, Elasticsearch
RKemper added a comment to T233351: Increase poolcounter for wikibase prefix search.

Context: This is not referring to the Elasticsearch threadpool; rather this pool is our own concept (in mediawiki-land) that puts an artificial limit on our side

Tue, Sep 1, 7:08 PM · Discovery-Search, Elasticsearch
RKemper updated the task description for T261786: WDQS 0.3.46 deploy failure.
Tue, Sep 1, 7:05 PM · Discovery-Search (Current work)
RKemper created T261786: WDQS 0.3.46 deploy failure.
Tue, Sep 1, 7:05 PM · Discovery-Search (Current work)

Mon, Aug 31

RKemper added a comment to T261528: SSL cert renewal warnings for cloudelastic100[5-6].wikimedia.org.

@Dzahn Yeah I'd left it open because I'd just learned that on our team we move stuff to a "Reporting" section on workboard when they're resolved before officially closing the ticket (but at the time had some questions around our process). Closed is fine, thanks for moving it.

Mon, Aug 31, 8:16 PM · Discovery-Search
RKemper added a comment to T261239: Reboot (restart) Elasticsearch nodes.
ryankemper@cumin2001:~$ sudo -i python3
Python 3.7.3 (default, Dec 20 2019, 18:57:59)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from spicerack import Spicerack
>>> s = Spicerack(verbose=True, dry_run=True)
>>>
>>>
>>> CLUSTERGROUPS = ('search_eqiad', 'search_codfw', 'relforge')
>>>
>>> ecs = s.elasticsearch_clusters(CLUSTERGROUPS[1])
>>> ecs.wait_for_all_write_queues_empty()
WARNING:spicerack.decorators:Reduce tries from 60 to 1 in DRY-RUN mode
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): prometheus.svc.eqiad.wmnet:80
DEBUG:urllib3.connectionpool:http://prometheus.svc.eqiad.wmnet:80 "GET /ops/api/v1/query?query=kafka_burrow_partition_lag%7B++++group%3D%22cpjobqueue-cirrusSearchElasticaWrite%22%2C++++topic%3D~%22%5B%5B%3Aalpha%3A%5D%5D%2A.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite%22%7D HTTP/1.1" 200 303
DEBUG:spicerack.elasticsearch_cluster:Prom query kafka_burrow_partition_lag{    group="cpjobqueue-cirrusSearchElasticaWrite",    topic=~"[[:alpha:]]*.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite"} returned queue_results of [('eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite', '0', 4699), ('eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite', '1', 4640), ('eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite', '2', 5993)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/spicerack/decorators.py", line 103, in wrapper
    return func(*args, **kwargs)  # type: ignore
  File "/usr/lib/python3/dist-packages/spicerack/elasticsearch_cluster.py", line 355, in wait_for_all_write_queues_empty
    "of topic {}.".format(queue_size, partition, topic))
spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError: Write queue not empty (had value of 4699) for partition 0of topic eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.
Mon, Aug 31, 7:21 PM · Patch-For-Review, Discovery-Search (Current work)
RKemper added a comment to T261239: Reboot (restart) Elasticsearch nodes.

Realized the issue above was that I needed to sudo apt-get install --only-upgrade spicerack on cumin2001 to get access to the new function.

import logging
logging.basicConfig(level=logging.DEBUG)
from spicerack import Spicerack
s = Spicerack(verbose=True, dry_run=True)
ecs = s.elasticsearch_clusters('search_codfw')
ecs.wait_for_all_write_queue_empty()
Mon, Aug 31, 2:31 AM · Patch-For-Review, Discovery-Search (Current work)
RKemper added a comment to T261239: Reboot (restart) Elasticsearch nodes.
sudo -i python3
import logging
logging.basicConfig(level=logging.DEBUG)
from spicerack import Spicerack
s = Spicerack(verbose=True, dry_run=True)
sreal = Spicerack(verbose=True, dry_run=False)
Mon, Aug 31, 2:19 AM · Patch-For-Review, Discovery-Search (Current work)

Fri, Aug 28

RKemper updated the task description for T261528: SSL cert renewal warnings for cloudelastic100[5-6].wikimedia.org.
Fri, Aug 28, 9:56 PM · Discovery-Search
RKemper updated the task description for T261528: SSL cert renewal warnings for cloudelastic100[5-6].wikimedia.org.
Fri, Aug 28, 8:58 PM · Discovery-Search
RKemper updated the task description for T261528: SSL cert renewal warnings for cloudelastic100[5-6].wikimedia.org.
Fri, Aug 28, 8:56 PM · Discovery-Search
RKemper created T261528: SSL cert renewal warnings for cloudelastic100[5-6].wikimedia.org.
Fri, Aug 28, 8:53 PM · Discovery-Search

Thu, Aug 27

RKemper triaged T261373: Pool and depool scripts not present on cloudelastic100[5,6].wikimedia.org as Medium priority.
Thu, Aug 27, 4:24 AM · Discovery-Search (Current work), Patch-For-Review, Discovery
RKemper created T261373: Pool and depool scripts not present on cloudelastic100[5,6].wikimedia.org.
Thu, Aug 27, 4:23 AM · Discovery-Search (Current work), Patch-For-Review, Discovery
RKemper closed T254353: Service implementation for cloudelastic100[56], a subtask of T249062: (Need by: TDB) rack/setup/install cloudelastic100[56], as Resolved.
Thu, Aug 27, 4:07 AM · Patch-For-Review, Operations, ops-eqiad
RKemper closed T254353: Service implementation for cloudelastic100[56] as Resolved.
Thu, Aug 27, 4:07 AM · Discovery-Search (Current work)
RKemper added a comment to T254353: Service implementation for cloudelastic100[56].

Nodes brought into service like so:

Thu, Aug 27, 4:06 AM · Discovery-Search (Current work)

Wed, Aug 26

RKemper set the point value for T261239: Reboot (restart) Elasticsearch nodes to 5.
Wed, Aug 26, 5:19 PM · Patch-For-Review, Discovery-Search (Current work)
RKemper triaged T261239: Reboot (restart) Elasticsearch nodes as High priority.
Wed, Aug 26, 4:34 PM · Patch-For-Review, Discovery-Search (Current work)

Tue, Aug 25

RKemper added a comment to T217742: Rework the data flow between logstash and cirrus elasticsearch cluster for ApiFeatureUsage.

We discussed this and we think sticking kafka between logstash and elasticsearch will help improve the synchronous flow and thus will give us a good value:effort tradeoff.

Tue, Aug 25, 7:25 PM · Discovery-Search
RKemper created T261239: Reboot (restart) Elasticsearch nodes.
Tue, Aug 25, 6:07 PM · Patch-For-Review, Discovery-Search (Current work)

Aug 10 2020

RKemper created T260083: Reshard commonswiki_file elasticsearch index.
Aug 10 2020, 6:46 PM · Patch-For-Review, Operations, Discovery-Search (Current work)

Jul 30 2020

RKemper added a member for Discovery: RKemper.
Jul 30 2020, 7:12 PM
RKemper added a comment to T244073: Cross-wiki search result should display namespace.

We (Erik, really) invested time in looking into this, got rabbit-holed, and so we've put this back into the backlog.

Jul 30 2020, 7:08 PM · Discovery-Search, Patch-For-Review, CirrusSearch

Jul 27 2020

RKemper added a comment to T257387: Perform In-Place Re-index to apply newest shard settings.

I'd forgotten to update the state but this ticket's been done for > a week.

Jul 27 2020, 5:07 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper closed T257387: Perform In-Place Re-index to apply newest shard settings, a subtask of T256928: Tune Elasticsearch Shard counts, as Resolved.
Jul 27 2020, 5:06 PM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery
RKemper closed T257387: Perform In-Place Re-index to apply newest shard settings as Resolved.
Jul 27 2020, 5:06 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper added a comment to T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines.

We discussed this during this week's SRE meeting and resolved to enable full root access for wdqs admins, rather than granularly expanding access one file at a time. This also lines up better with how we currently manage Elasticsearch, where admins have root access.

Jul 27 2020, 4:29 PM · Sustainability (Incident Followup), Patch-For-Review, Wikidata-Query-Service, Wikidata, Operations, SRE-Access-Requests

Jul 24 2020

RKemper added a comment to T258245: Move mjolnir kafka daemon from ES to search-loader VMs.

@elukey In response to your earlier ping in this thread - yup, let's work to get this moving next week

Jul 24 2020, 5:58 PM · Discovery-Search (Current work), Patch-For-Review, Discovery, Analytics-Clusters
RKemper added a project to T258754: Define SLOs and error budget for WDQS: Sustainability (Incident Followup).
Jul 24 2020, 12:51 AM · Wikidata, Wikidata-Query-Service, Sustainability (Incident Followup), Discovery
RKemper added a project to T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines: Sustainability (Incident Followup).
Jul 24 2020, 12:51 AM · Sustainability (Incident Followup), Patch-For-Review, Wikidata-Query-Service, Wikidata, Operations, SRE-Access-Requests

Jul 23 2020

RKemper updated the task description for T258754: Define SLOs and error budget for WDQS.
Jul 23 2020, 9:56 PM · Wikidata, Wikidata-Query-Service, Sustainability (Incident Followup), Discovery
RKemper created T258754: Define SLOs and error budget for WDQS.
Jul 23 2020, 9:29 PM · Wikidata, Wikidata-Query-Service, Sustainability (Incident Followup), Discovery
RKemper renamed T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines from wdqs admins should have access to nginx logs on wdqs machines to wdqs admins should have access to nginx logs, jstack on wdqs machines.
Jul 23 2020, 9:14 PM · Sustainability (Incident Followup), Patch-For-Review, Wikidata-Query-Service, Wikidata, Operations, SRE-Access-Requests

Jul 20 2020

RKemper committed rLPRIca46e875193b: maps: remove no-longer-used slave files (authored by RKemper).
maps: remove no-longer-used slave files
Jul 20 2020, 5:54 PM

Jul 10 2020

RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 10 2020, 9:54 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 10 2020, 9:31 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 10 2020, 9:29 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 10 2020, 9:26 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 10 2020, 9:23 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper added a comment to T257387: Perform In-Place Re-index to apply newest shard settings.

Updated the main ticket body with the results after performing the two optimizations I spoke about here:

Jul 10 2020, 9:07 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 10 2020, 9:06 PM · Discovery-Search (Current work), Elasticsearch, Discovery

Jul 9 2020

RKemper added a comment to T257387: Perform In-Place Re-index to apply newest shard settings.

Erik had two optimizations that both make sense. I've listed them in descending order of forecasted performance gain:

Jul 9 2020, 7:19 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper added a comment to T257387: Perform In-Place Re-index to apply newest shard settings.

@Gehel: Commands are ready for review, see the above task description (final codeblock specifically). See changes in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/608965/8/wmf-config/InitialiseSettings.php for/if verifying that the list of wikis I used in the ruby script is correct.

Jul 9 2020, 7:18 AM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 9 2020, 7:14 AM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 9 2020, 7:13 AM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 9 2020, 6:45 AM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 9 2020, 6:45 AM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery

Jul 7 2020

RKemper created T257387: Perform In-Place Re-index to apply newest shard settings.
Jul 7 2020, 11:06 PM · Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 7 2020, 11:03 PM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 7 2020, 11:02 PM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery

Jul 3 2020

RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 3 2020, 4:21 AM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 3 2020, 3:40 AM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 3 2020, 2:34 AM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery

Jul 2 2020

RKemper added a comment to T241791: (Need by: 2020-04-02) rack/setup/install relforge100[34].

(Meant to comment earlier, but I'm looking into the cause of the RAID failure)

Jul 2 2020, 11:08 PM · Patch-For-Review, ops-eqiad, Discovery-Search (Current work), Operations

Jul 1 2020

RKemper edited projects for T256928: Tune Elasticsearch Shard counts, added: Discovery-Search (Current work); removed Discovery-Search.
Jul 1 2020, 10:57 PM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery
RKemper updated the task description for T256928: Tune Elasticsearch Shard counts.
Jul 1 2020, 10:52 PM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery
RKemper created T256928: Tune Elasticsearch Shard counts.
Jul 1 2020, 10:51 PM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Discovery

Jun 2 2020

RKemper added a comment to T253114: Search autocompletion broken for recent articles (after April 30, 2020?) for some users / browsers.

Thanks all for reporting this behavior. The Search team is working on a fix.

Jun 2 2020, 10:31 PM · Discovery-Search (Current work), CirrusSearch, Regression, Discovery

May 19 2020

RKemper closed T252068: WQDS Data Reload, a subtask of T230588: Wikidata Query Service is swapping items and properties, as Resolved.
May 19 2020, 4:49 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
RKemper closed T252068: WQDS Data Reload, a subtask of T240831: Sometimes Q77104211 shows up in sparql query as having a wrong value for P2397, however item is deleted since December 3rd 2019, as Resolved.
May 19 2020, 4:49 PM · Wikidata, Wikidata-Query-Service
RKemper closed T252068: WQDS Data Reload, a subtask of T245135: Too many values for wikibase:geoLatitude and wikibase:geoLongitude in WDQS, as Resolved.
May 19 2020, 4:49 PM · Wikidata, Wikidata-Query-Service
RKemper closed T252068: WQDS Data Reload, a subtask of T246568: Deepcategory returns only very few results, as Resolved.
May 19 2020, 4:49 PM · MW-1.36-notes (1.36.0-wmf.3; 2020-08-04), Discovery-Search (Current work), CirrusSearch, Commons
RKemper closed T252068: WQDS Data Reload as Resolved.

Still need to circle back to dependent tickets and verify that those problems are solved

May 19 2020, 4:49 PM · Operations, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
RKemper added a comment to T252068: WQDS Data Reload.

Data transfer is done across all instances as of last friday, along with a wdqs-categories reload that we tacked on.

May 19 2020, 4:47 PM · Operations, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata

May 8 2020

RKemper added a comment to T251572: SRE Onboarding - Ryan Kemper, Search Platform team.
-----BEGIN PGP PUBLIC KEY BLOCK-----
May 8 2020, 4:30 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Discovery-Search (Current work)
RKemper added a comment to T251572: SRE Onboarding - Ryan Kemper, Search Platform team.

@Dzahn Oh, thanks for catching/fixing that! rkemper@ is fine.

May 8 2020, 4:06 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Discovery-Search (Current work)

May 7 2020

RKemper added a comment to T251572: SRE Onboarding - Ryan Kemper, Search Platform team.

Initial context:

May 7 2020, 10:25 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Discovery-Search (Current work)
RKemper updated the task description for T251572: SRE Onboarding - Ryan Kemper, Search Platform team.
May 7 2020, 9:59 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Discovery-Search (Current work)
RKemper added a comment to T251572: SRE Onboarding - Ryan Kemper, Search Platform team.

Icinga access is working for me

May 7 2020, 5:51 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Discovery-Search (Current work)
RKemper updated the task description for T251572: SRE Onboarding - Ryan Kemper, Search Platform team.
May 7 2020, 5:50 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Discovery-Search (Current work)

May 6 2020

RKemper added a comment to T252068: WQDS Data Reload.

should also circle back and fix https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L2207-L2242, wdqs200[78] are declared twice

May 6 2020, 8:45 PM · Operations, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
RKemper added a comment to T252068: WQDS Data Reload.

wdqs1010 is one of our test servers

May 6 2020, 8:43 PM · Operations, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata