dcausse (David Causse)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Jun 9 2015, 9:03 AM (136 w, 1 d)
Availability
Available
IRC Nick
dcausse
LDAP User
DCausse
MediaWiki User
DCausse (WMF)

Recent Activity

Yesterday

dcausse triaged T185108: Refactor query parsing logic as Normal priority.
Wed, Jan 17, 3:38 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse claimed T185108: Refactor query parsing logic.
Wed, Jan 17, 3:38 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T185108: Refactor query parsing logic from Needs triage to Current work on the Discovery-Search board.
Wed, Jan 17, 3:37 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse created T185108: Refactor query parsing logic.
Wed, Jan 17, 3:37 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T175179: Create selenium-CirrusSearch-jessie daily Jenkins job from In progress to Done on the Discovery-Search (Current work) board.
Wed, Jan 17, 3:34 PM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, User-zeljkofilipin, Release-Engineering-Team (Kanban), Discovery-Search (Current work), Discovery
dcausse added a comment to T135021: index names must be lowercased.

@Gopavasanth index names mean the name of the indices created by the extension CirrusSearch in elasticsearch.
CirrusSearch is the extension that provides search functionalities using elasticsearch as a backend.

Wed, Jan 17, 3:01 PM · Easy, Discovery-Search, CirrusSearch, Discovery

Tue, Jan 16

dcausse removed a project from T87892: Convert CirrusSearch to use extension registration: Patch-For-Review.
Tue, Jan 16, 5:11 PM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Discovery-Search, Discovery, CirrusSearch
dcausse added a comment to T87892: Convert CirrusSearch to use extension registration.

@Oetterer I don't think this is related, this task is just to track progress on making the extension CirrusSearch compatible with the new extension registration process. It is just listing the pieces of code that make this refactoring problematic not actual problems regarding Config factories.
Perhaps we'll end up having the same issues but I don't think we have code that need to be run just after the extension is loaded.

Tue, Jan 16, 1:51 PM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Discovery-Search, Discovery, CirrusSearch

Fri, Jan 5

dcausse awarded T166248: Upgrade Analytics Cluster to Java 8 a Like token.
Fri, Jan 5, 2:26 PM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics-Cluster
dcausse awarded T166248: Upgrade Analytics Cluster to Java 8 a Like token.
Fri, Jan 5, 10:44 AM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics-Cluster

Wed, Jan 3

dcausse added a comment to T184008: Language fallback for search can fail when rescore profile doesn't exist on target wiki.

I remember that it's due to the type of API param we use. When setting an array as ApiBase::PARAM_TYPE a default must be provided IIRC.
The use of arrays was a way to expose the list of possible profiles to use but the drawback was that the API would fail if you provide an unknown param. I think this is wrong, I agree, cirrus should be able to know if a profile was explicitly set by the user.

Wed, Jan 3, 9:30 AM · Discovery-Search (Current work), MW-1.31-release-notes (WMF-deploy-2018-01-09 (1.31.0-wmf.16)), Patch-For-Review, Discovery, CirrusSearch

Thu, Dec 21

dcausse added a comment to T172260: Default namespaces is what has been defined in user preferences.

This is not only the namespaces selected and saved by the users but also the list of default namespace searched by default.
Currently when the extension is enabled you can encouter a strange behavior that looks like a bug:

Thu, Dec 21, 5:31 PM · TCB-Team, Advanced-Search

Wed, Dec 20

dcausse added a comment to T183053: New Wikidata items appear in search with a delay.

Same for me I'd be for trying to increase the refresh rate on wikidata_content.

Wed, Dec 20, 10:02 AM · Discovery-Search (Current work), Discovery, Wikidata

Tue, Dec 19

dcausse updated the task description for T183279: Implement profile management in cirrus.
Tue, Dec 19, 6:15 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Discovery
dcausse added a parent task for T183279: Implement profile management in cirrus: T87892: Convert CirrusSearch to use extension registration.
Tue, Dec 19, 6:13 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Discovery
dcausse added a subtask for T87892: Convert CirrusSearch to use extension registration: T183279: Implement profile management in cirrus.
Tue, Dec 19, 6:13 PM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Discovery-Search, Discovery, CirrusSearch
dcausse triaged T183279: Implement profile management in cirrus as Normal priority.
Tue, Dec 19, 6:12 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Discovery
dcausse moved T183279: Implement profile management in cirrus from Backlog to In progress on the Discovery-Search (Current work) board.
Tue, Dec 19, 6:12 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Discovery
dcausse moved T183279: Implement profile management in cirrus from Needs triage to Current work on the Discovery-Search board.
Tue, Dec 19, 6:12 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Discovery
dcausse created T183279: Implement profile management in cirrus.
Tue, Dec 19, 6:11 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Discovery
dcausse added a comment to T181627: Port elasticsearch metrics to Prometheus.

I ported elasticsearch-memory and elasticsearch-indexing.

Tue, Dec 19, 1:14 PM · Patch-For-Review, Discovery-Search (Current work), cloud-services-team (Kanban), User-fgiunchedi, Goal, Operations

Dec 18 2017

dcausse added a comment to T183101: Items missing from Wikidata index due to LinksUpdate lock failures.

Q45825730 is me, I used this one just to test.

Dec 18 2017, 8:26 PM · MediaWiki-JobQueue, Performance-Team, Wikimedia-log-errors, Discovery-Search (Current work), Wikidata
dcausse added a comment to T183053: New Wikidata items appear in search with a delay.

If a large majority of such usecases involve searching the entity id (QXXX) of the newly created item we can perform an additional db match to compensate the lag of the search index.
It's what we do for normal wikis, a db match is run in addition to the query sent to the search index.
If users search for the label or aliases of the newly created then this solution is pointless.

Dec 18 2017, 6:02 PM · Discovery-Search (Current work), Discovery, Wikidata
dcausse awarded F11851691: plot.png a Love token.
Dec 18 2017, 3:55 PM

Dec 14 2017

dcausse reassigned T115756: Search suggests non-existent title due to namespace/redirect mixup from dcausse to EBernhardson.
Dec 14 2017, 11:06 AM · Patch-For-Review, Discovery-Search (Current work), MediaWiki-Search, WorkType-Maintenance, CirrusSearch, Discovery

Dec 13 2017

dcausse added a comment to T175179: Create selenium-CirrusSearch-jessie daily Jenkins job.

The error EADDRINUSE /tmp/cirrussearch-integration-tagtracker means that the tests are running in the background or that we failed to cleanup the socket when the tests finished or was killed.
It's perfectly fine to delete /tmp/cirrussearch-integration-tagtracker if you think the test is no longer running.

Dec 13 2017, 1:36 PM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, User-zeljkofilipin, Release-Engineering-Team (Kanban), Discovery-Search (Current work), Discovery
dcausse added a comment to T182717: Move fine tuning of search configs to mediawiki-config.

A decent place for profiles has always been a pain and I could not find something sane. I'd like to address (improve) this problem adding a ProfileManager in cirrus.

Dec 13 2017, 11:02 AM · Discovery-Search (Current work), CirrusSearch, Discovery, Wikidata
dcausse claimed T182136: English labels in wikidata prefix search in non-English have low ranking.
Dec 13 2017, 10:37 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Discovery-Search (Current work), Patch-For-Review, CirrusSearch, Wikidata, Discovery
dcausse claimed T115756: Search suggests non-existent title due to namespace/redirect mixup.
Dec 13 2017, 10:20 AM · Patch-For-Review, Discovery-Search (Current work), MediaWiki-Search, WorkType-Maintenance, CirrusSearch, Discovery

Dec 12 2017

dcausse moved T182293: Tune wikidata fulltext search similarity parameters from In progress to Needs review on the Discovery-Search (Current work) board.
Dec 12 2017, 4:10 PM · MW-1.31-release-notes (WMF-deploy-2017-12-12 (1.31.0-wmf.12)), Patch-For-Review, User-Smalyshev, Discovery, Discovery-Search (Current work), Wikidata

Dec 7 2017

dcausse moved T182293: Tune wikidata fulltext search similarity parameters from Backlog to In progress on the Discovery-Search (Current work) board.
Dec 7 2017, 10:56 AM · MW-1.31-release-notes (WMF-deploy-2017-12-12 (1.31.0-wmf.12)), Patch-For-Review, User-Smalyshev, Discovery, Discovery-Search (Current work), Wikidata
dcausse moved T175179: Create selenium-CirrusSearch-jessie daily Jenkins job from Backlog to In progress on the Discovery-Search (Current work) board.
Dec 7 2017, 10:56 AM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, User-zeljkofilipin, Release-Engineering-Team (Kanban), Discovery-Search (Current work), Discovery
dcausse moved T174662: [epic] Upgrade to elastic 5.5.x from In progress to Done on the Discovery-Search (Current work) board.
Dec 7 2017, 10:56 AM · Discovery-Search (Current work), Epic, CirrusSearch, Discovery, Elasticsearch
dcausse triaged T182293: Tune wikidata fulltext search similarity parameters as Normal priority.
Dec 7 2017, 9:03 AM · MW-1.31-release-notes (WMF-deploy-2017-12-12 (1.31.0-wmf.12)), Patch-For-Review, User-Smalyshev, Discovery, Discovery-Search (Current work), Wikidata
dcausse added a comment to T175179: Create selenium-CirrusSearch-jessie daily Jenkins job.

@zeljkofilipin we might be ready to port our selenium-CirrusSearch jenkins job to nodejs, I uploaded https://gerrit.wikimedia.org/r/#/c/395872/ to try to comply with the structure expected by jenkins:

  • tests in tests/selenium/specs
  • wdio config in tests/selenium/wdio.conf.jenkins.js
Dec 7 2017, 8:27 AM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, User-zeljkofilipin, Release-Engineering-Team (Kanban), Discovery-Search (Current work), Discovery

Dec 6 2017

dcausse created P6435 php foreach vs array_filter.
Dec 6 2017, 2:03 PM

Dec 1 2017

dcausse added a comment to T179945: Re-index English-language wikis to pick up kana mapping.

Reindex is done, @TJones could you check few indices to make sure it worked as expected?

Dec 1 2017, 8:52 AM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T179945: Re-index English-language wikis to pick up kana mapping from In progress to Done on the Discovery-Search (Current work) board.
Dec 1 2017, 8:51 AM · Discovery-Search (Current work), Discovery, CirrusSearch

Nov 30 2017

dcausse created T181716: forceSearchIndex.php hangs at the end of the process when running on large wikis.
Nov 30 2017, 1:37 PM · MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch

Nov 29 2017

TJones awarded T181479: Requesting access to terbium/wasat for Trey Jones a Doubloon token.
Nov 29 2017, 6:33 PM · Patch-For-Review, Ops-Access-Requests, Operations

Nov 28 2017

dcausse created P6388 prefer-recent mess.
Nov 28 2017, 5:15 PM
dcausse created T181479: Requesting access to terbium/wasat for Trey Jones.
Nov 28 2017, 8:53 AM · Patch-For-Review, Ops-Access-Requests, Operations

Nov 27 2017

dcausse closed T180523: Error while creating vagrant VM: psysh cannot find cURL as Resolved.

fixed in https://gerrit.wikimedia.org/r/#/c/392998/

Nov 27 2017, 11:26 AM · Patch-For-Review, MediaWiki-Vagrant

Nov 21 2017

dcausse created P6358 Error: function timed out after 30000 milliseconds.
Nov 21 2017, 6:56 PM
dcausse added a comment to T180051: Reduce the number of fields declared in elasticsearch by logstash.

I'm investigating two approaches here:

  1. provide a way inside logstash filters to blacklist some known fields (move them into a debug_blob field that is not indexed)
  2. investigate disabling dynamic mapping where the first step would be to log all elastic queries to discover what are the fields we currently use. It'll allow to create the first static mapping.
Nov 21 2017, 9:22 AM · Discovery-Search, Services (watching), Operations, Wikimedia-Logstash

Nov 20 2017

dcausse closed T180805: Re-enable the token count router, a subtask of T180795: Elastic 5.5 rolling restart causes some search queries to fail, as Resolved.
Nov 20 2017, 2:44 PM · Discovery-Search (Current work), Wikimedia-log-errors, Patch-For-Review, Discovery, CirrusSearch
dcausse closed T180805: Re-enable the token count router as Resolved.
Nov 20 2017, 2:44 PM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T180805: Re-enable the token count router from Needs review to Done on the Discovery-Search (Current work) board.
Nov 20 2017, 2:44 PM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse updated the task description for T177520: Experiment with different grouping of queries that get fed into the DBN.
Nov 20 2017, 12:54 PM · Discovery-Search (Current work), Discovery
dcausse placed T177520: Experiment with different grouping of queries that get fed into the DBN up for grabs.

Moving back to backlog as this task actually covers 2 experiments and thought it was new:

Nov 20 2017, 12:53 PM · Discovery-Search (Current work), Discovery
dcausse added a comment to T179945: Re-index English-language wikis to pick up kana mapping.

List of affected wikis:

labtestwiki
mediawikiwiki
test2wiki
testwiki
testwikidatawiki
zerowiki
advisorywiki
auditcomwiki
betawikiversity
bewikimedia
boardgovcomwiki
boardwiki
cawikimedia
chairwiki
chapcomwiki
checkuserwiki
collabwiki
commonswiki
donatewiki
electcomwiki
enwikibooks
enwikinews
enwikiquote
enwikisource
enwikiversity
enwikivoyage
enwiktionary
execwiki
fdcwiki
foundationwiki
grantswiki
iegcomwiki
incubatorwiki
internalwiki
labswiki
legalteamwiki
loginwiki
metawiki
movementroleswiki
nostalgiawiki
nycwikimedia
nzwikimedia
officewiki
ombudsmenwiki
otrs_wikiwiki
outreachwiki
pa_uswikimedia
projectcomwiki
qualitywiki
searchcomwiki
simplewiktionary
sourceswiki
spcomwiki
specieswiki
stewardwiki
strategywiki
techconductwiki
transitionteamwiki
usabilitywiki
votewiki
wikidatawiki
wikimania2005wiki
wikimania2006wiki
wikimania2007wiki
wikimania2008wiki
wikimania2009wiki
wikimania2010wiki
wikimania2011wiki
wikimania2012wiki
wikimania2013wiki
wikimania2014wiki
wikimania2015wiki
wikimania2016wiki
wikimania2017wiki
wikimania2018wiki
wikimaniateamwiki
arbcom_enwiki
enwiki
simplewiki
tenwiki
wg_enwiki
Nov 20 2017, 10:14 AM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse claimed T179945: Re-index English-language wikis to pick up kana mapping.
Nov 20 2017, 10:08 AM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T177520: Experiment with different grouping of queries that get fed into the DBN from Backlog to In progress on the Discovery-Search (Current work) board.
Nov 20 2017, 9:57 AM · Discovery-Search (Current work), Discovery
dcausse renamed T177520: Experiment with different grouping of queries that get fed into the DBN from Experiement with different grouping of queries that get fed into the DBN to Experiment with different grouping of queries that get fed into the DBN.
Nov 20 2017, 9:57 AM · Discovery-Search (Current work), Discovery
dcausse placed T178412: Upgrade logstash cluster to elastic 5.5.x up for grabs.
Nov 20 2017, 9:56 AM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Elasticsearch, Discovery
dcausse moved T178411: Upgrade cirrus elasticsearch clusters to 5.5.x from Backlog to Done on the Discovery-Search (Current work) board.
Nov 20 2017, 9:55 AM · Discovery-Search (Current work), CirrusSearch, Elasticsearch, Discovery
dcausse claimed T180805: Re-enable the token count router.
Nov 20 2017, 9:45 AM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch

Nov 17 2017

dcausse closed T180807: Undefined index: _score in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/in cludes/Query/CompSuggestQueryBuilder.php on line 224 as Declined.

It was just a brief spike of 300 errors today around 15:00 UTC, looking at the code I see no obvious reasons why it could happen, except a broken response from elastic.
I'll assume that this error was due to the rolling restart and won't try to hide it by calling isset on the result sets.

Nov 17 2017, 5:20 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse triaged T180807: Undefined index: _score in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/in cludes/Query/CompSuggestQueryBuilder.php on line 224 as High priority.

moving to high because this code should not reach group2

Nov 17 2017, 4:33 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse edited projects for T180807: Undefined index: _score in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/in cludes/Query/CompSuggestQueryBuilder.php on line 224, added: Discovery-Search (Current work); removed Discovery-Search.
Nov 17 2017, 4:31 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse created T180807: Undefined index: _score in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/in cludes/Query/CompSuggestQueryBuilder.php on line 224.
Nov 17 2017, 4:28 PM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse closed T180795: Elastic 5.5 rolling restart causes some search queries to fail as Resolved.
Nov 17 2017, 4:26 PM · Discovery-Search (Current work), Wikimedia-log-errors, Patch-For-Review, Discovery, CirrusSearch
dcausse removed a project from T180805: Re-enable the token count router: Wikimedia-log-errors.
Nov 17 2017, 4:26 PM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T180805: Re-enable the token count router from Needs triage to Current work on the Discovery-Search board.
Nov 17 2017, 4:23 PM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse created T180805: Re-enable the token count router.
Nov 17 2017, 4:22 PM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T180795: Elastic 5.5 rolling restart causes some search queries to fail from Backlog to Done on the Discovery-Search (Current work) board.
Nov 17 2017, 4:22 PM · Discovery-Search (Current work), Wikimedia-log-errors, Patch-For-Review, Discovery, CirrusSearch
dcausse moved T180795: Elastic 5.5 rolling restart causes some search queries to fail from Needs triage to Current work on the Discovery-Search board.
Nov 17 2017, 4:21 PM · Discovery-Search (Current work), Wikimedia-log-errors, Patch-For-Review, Discovery, CirrusSearch
dcausse created T180795: Elastic 5.5 rolling restart causes some search queries to fail.
Nov 17 2017, 3:41 PM · Discovery-Search (Current work), Wikimedia-log-errors, Patch-For-Review, Discovery, CirrusSearch

Nov 14 2017

dcausse merged T180440: Search suggestions ignore settings for logged in users on de.wikipedia, always performing a prefix search (no typo correction) into T178474: Completion suggester disabled on some wikis.
Nov 14 2017, 10:22 AM · MW-1.31-release-notes (WMF-deploy-2017-12-12 (1.31.0-wmf.12)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse merged task T180440: Search suggestions ignore settings for logged in users on de.wikipedia, always performing a prefix search (no typo correction) into T178474: Completion suggester disabled on some wikis.
Nov 14 2017, 10:22 AM · Discovery-Search, CirrusSearch, Discovery
dcausse added a comment to T180382: Emptying description on Wikidata doesn't remove it from ElasticSearch database used by wbsearchentities.

@Smalyshev it's certainly the case yes, tuning the noop script should fix the issue.

Nov 14 2017, 8:10 AM · MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)), Patch-For-Review, Discovery-Search (Current work), MediaWiki-extensions-WikibaseRepository, Elasticsearch, Regression, Discovery, Wikidata

Nov 13 2017

dcausse claimed T180298: Catchable fatal error: Argument 1 passed to CirrusSearch\DataSender::reportUpdateMetrics() must be an instance of Elastica\Bulk\ResponseSet, null given.
Nov 13 2017, 1:17 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Discovery-Search (Current work), Patch-For-Review, Discovery, CirrusSearch
dcausse moved T180298: Catchable fatal error: Argument 1 passed to CirrusSearch\DataSender::reportUpdateMetrics() must be an instance of Elastica\Bulk\ResponseSet, null given from This Quarter to Current work on the Discovery-Search board.
Nov 13 2017, 1:17 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Discovery-Search (Current work), Patch-For-Review, Discovery, CirrusSearch
dcausse moved T180298: Catchable fatal error: Argument 1 passed to CirrusSearch\DataSender::reportUpdateMetrics() must be an instance of Elastica\Bulk\ResponseSet, null given from Needs triage to This Quarter on the Discovery-Search board.
Nov 13 2017, 1:17 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Discovery-Search (Current work), Patch-For-Review, Discovery, CirrusSearch
dcausse added a comment to T180298: Catchable fatal error: Argument 1 passed to CirrusSearch\DataSender::reportUpdateMetrics() must be an instance of Elastica\Bulk\ResponseSet, null given.

It's a bug but the only explanation I have to explain why this error is frequent is that your index may have run out of date. Reading the code I understand that this error could only happen when trying to update some metadata in elasticsearch concerning a document that is not indexed.

Nov 13 2017, 1:15 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Discovery-Search (Current work), Patch-For-Review, Discovery, CirrusSearch

Nov 9 2017

dcausse updated subscribers of T180169: Make list of languages where using stemmed analyzer for Wikibase is beneficial.
Nov 9 2017, 9:41 PM · MediaWiki-extensions-WikibaseRepository, Wikidata, Discovery-Search (Current work), Discovery
dcausse added a comment to T180051: Reduce the number of fields declared in elasticsearch by logstash.

My fear is that the "too many fields" problem is going to be more painful than the mapping conflicts one.
Speaking only about short term solutions:

  • For mapping conflicts we can rename
  • For too many fields I don't have yet a short term solution
Nov 9 2017, 3:16 PM · Discovery-Search, Services (watching), Operations, Wikimedia-Logstash
dcausse added a comment to T180051: Reduce the number of fields declared in elasticsearch by logstash.

Typically logstash/elastic is not able to sustain this kind of events: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2017.11.08/mediawiki?id=AV-ZBn5-gaOKEclNGWio&_g=h@8b5b71a
EventBus.events.params should be marked as debug only in some ways

Nov 9 2017, 11:28 AM · Discovery-Search, Services (watching), Operations, Wikimedia-Logstash
dcausse added a comment to T180051: Reduce the number of fields declared in elasticsearch by logstash.

I think we should introduce a pattern where log emitters can freely send large and complex objects that would be only available for debugging purposes on a per event basis: no need to search/aggregate them.
The current strategy we have is to index everything and to allow aggregation we index 2 elastic fields per json field.
Looking at some EventBus logs I see : https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2017.11.09/mediawiki?id=AV-gV9NNSUnOz-leF_9O&_g=h@44136fa

Nov 9 2017, 10:53 AM · Discovery-Search, Services (watching), Operations, Wikimedia-Logstash

Nov 7 2017

dcausse moved T178474: Completion suggester disabled on some wikis from In progress to Needs review on the Discovery-Search (Current work) board.
Nov 7 2017, 8:00 PM · MW-1.31-release-notes (WMF-deploy-2017-12-12 (1.31.0-wmf.12)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch

Nov 3 2017

dcausse added a comment to T177957: Analysis of: A/B test to test relaxing the retrieval query filter.

@chelsyx thanks!

Nov 3 2017, 8:47 AM · Discovery-Analysis (Current work), Discovery

Nov 2 2017

dcausse moved T178906: Refactor the Searcher class and the SearchContext to allow easy code reuse from Needs review to Done on the Discovery-Search (Current work) board.
Nov 2 2017, 10:41 AM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch

Nov 1 2017

dcausse closed T179270: TTMServerMessageUpdateJob fails in labs as Resolved.

The mirror config is now empty in labs and should not cause issues anymore.

Nov 1 2017, 1:46 PM · Patch-For-Review, User-Nikerabbit, Wikimedia-Site-requests

Oct 31 2017

dcausse moved T178906: Refactor the Searcher class and the SearchContext to allow easy code reuse from In progress to Needs review on the Discovery-Search (Current work) board.
Oct 31 2017, 5:16 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch

Oct 30 2017

dcausse moved T179270: TTMServerMessageUpdateJob fails in labs from Backlog to To deploy on the Wikimedia-Site-requests board.
Oct 30 2017, 5:00 PM · Patch-For-Review, User-Nikerabbit, Wikimedia-Site-requests
dcausse claimed T179270: TTMServerMessageUpdateJob fails in labs.

@Nikerabbit nothing that I'm aware of. If this error is new I have no idea what could have happened.
Reading the code in CommonSettings I think there's bug there, it properly checks for wmfAllServices to not add a cluster but it still adds the mirror config.
(So, I don't think this error is new).

Oct 30 2017, 4:40 PM · Patch-For-Review, User-Nikerabbit, Wikimedia-Site-requests

Oct 27 2017

Liuxinyu970226 awarded T177871: Re-index un-fallbacked languages a Baby Tequila token.
Oct 27 2017, 12:01 PM · User-notice, Discovery-Search (Current work), Discovery, I18n

Oct 26 2017

dcausse added a comment to T115756: Search suggests non-existent title due to namespace/redirect mixup.

We partially fixed this in the completion suggester in T129575. Sadly the completion suggester was recently disabled on some wikis including mw.org and meta (T178474).

Oct 26 2017, 6:02 PM · Patch-For-Review, Discovery-Search (Current work), MediaWiki-Search, WorkType-Maintenance, CirrusSearch, Discovery

Oct 25 2017

dcausse moved T177956: Turn off: A/B test to test relaxing the retrieval query filter from In progress to Done on the Discovery-Search (Current work) board.
Oct 25 2017, 4:11 PM · MW-1.31-release-notes (WMF-deploy-2017-10-17 (1.31.0-wmf.4)), Patch-For-Review, Discovery-Search (Current work), Discovery
dcausse added a comment to T174960: Varnish does not vary elasticsearch query by request body.

@dbarratt sadly I don't know all the details of this cluster, but you could get it working by not specifying an index:

Oct 25 2017, 2:41 PM · Operations, Traffic, Wikimedia-Logstash
dcausse added a comment to T174960: Varnish does not vary elasticsearch query by request body.

Yes the syntax is slightly different:

  • you need to set Content-Type: application/x-ndjson
  • every request must be formed of 2 lines:
    • first line some metadata such as the index you want to query
    • second line the search request body
Oct 25 2017, 2:08 PM · Operations, Traffic, Wikimedia-Logstash
dcausse claimed T177956: Turn off: A/B test to test relaxing the retrieval query filter.
Oct 25 2017, 12:57 PM · MW-1.31-release-notes (WMF-deploy-2017-10-17 (1.31.0-wmf.4)), Patch-For-Review, Discovery-Search (Current work), Discovery

Oct 24 2017

dcausse moved T178274: Review mjolnir kafka implementation on large featuresets from Needs review to Done on the Discovery-Search (Current work) board.
Oct 24 2017, 5:10 PM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse claimed T178906: Refactor the Searcher class and the SearchContext to allow easy code reuse.
Oct 24 2017, 2:49 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T178906: Refactor the Searcher class and the SearchContext to allow easy code reuse from Needs triage to Current work on the Discovery-Search board.
Oct 24 2017, 2:48 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse added projects to T178906: Refactor the Searcher class and the SearchContext to allow easy code reuse: CirrusSearch, Discovery-Search (Current work).
Oct 24 2017, 2:48 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse created T178906: Refactor the Searcher class and the SearchContext to allow easy code reuse.
Oct 24 2017, 2:48 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T177871: Re-index un-fallbacked languages from In progress to Done on the Discovery-Search (Current work) board.
Oct 24 2017, 10:52 AM · User-notice, Discovery-Search (Current work), Discovery, I18n

Oct 23 2017

RandomDSdevel awarded T177871: Re-index un-fallbacked languages a Baby Tequila token.
Oct 23 2017, 8:01 PM · User-notice, Discovery-Search (Current work), Discovery, I18n
dcausse moved T178274: Review mjolnir kafka implementation on large featuresets from In progress to Needs review on the Discovery-Search (Current work) board.
Oct 23 2017, 3:54 PM · Patch-For-Review, Discovery-Search (Current work), Discovery, CirrusSearch