Maniphest T200508

Run WikibaseLexeme tests with Wikibase CI
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Tarrow
	Jul 27 2018, 10:54 AM

Description

When a change happens in Wikibase that breaks something in Lexeme we don't know until it has already been merged.

This is a problem because we could plausibly have a release cut at this stage. AFAICT no additional CI jobs are run for Lexeme when the release branch is cut so I think these breaks could get as far as wikidata.org

An example of a (minor) breaking change is: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/448005/
Which caused breaks in some of the Lexeme tests such as: https://integration.wikimedia.org/ci/job/mwext-testextension-php70-composer-jessie/1083/console
We fixed this in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseLexeme/+/448016/

In order to prevent this happening we may want to run the Lexeme test suite on changes to Wikibase. One question is should this test be voting? If so does it put us in catch-22 and stop any breaking changes (even desired ones) in Wikibase.

We may also want to run a nightly test to give us a warning.

Details

Subject	Repo	Branch	Lines +/-
Add extension-quibble template to Wikibase	integration/config	master	+1 -0
Revert "Temporarily exclude WikibaseLexeme from jobs including Wikibase"	integration/config	master	+2 -1
phpcs ignore phan stubs	mediawiki/extensions/Wikibase	master	+1 -0
Run WikibaseLexeme Jobs on Wikibase	integration/config	master	+2 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Addshore	T200508 Run WikibaseLexeme tests with Wikibase CI
Resolved	Addshore	T201336 Use Quibble for WikibaseLexeme CI
Resolved	Addshore	T197771 Couple errors with Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest
Resolved	Addshore	T201356 Fix QUnit failures blocking using quibble for WikibaseLexeme in CI
Resolved	Addshore	T200891 Warnings in qunit tests on quibble: "Cannot read property 'errors' of undefined'"
Resolved	Addshore	T200890 Warnings in qunit tests on quibble: "Cannot read property 'id' of undefined'"
Resolved	Addshore	T201337 Quibble & WikibaseLexeme: selenium, TypeError: Cannot read property 'value' of undefined

Event Timeline

Tarrow created this task.Jul 27 2018, 10:54 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 27 2018, 10:54 AM

Tarrow updated the task description. (Show Details)Jul 27 2018, 10:54 AM

Jakob_WMDE subscribed.Jul 27 2018, 10:56 AM

Change 448468 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[integration/config@master] Run WikibaseLexeme Jobs on Wikibase

https://gerrit.wikimedia.org/r/448468

gerritbot added a project: Patch-For-Review.Jul 27 2018, 11:20 AM

I've uploaded a "strawman" patch to be clear what I'm suggesting and give a place to track the discussion.

WMDE-leszek subscribed.Jul 27 2018, 11:29 AM

Change 448468 merged by jenkins-bot:
[integration/config@master] Run WikibaseLexeme Jobs on Wikibase

https://gerrit.wikimedia.org/r/448468

Not very helpful comment: AFAIK (but can be wrong, so pinging @Addshore so an expert) there is no CI job which is run when deployment branches are cut. Such thing could also be an option for this particular problem, and for developing extension to be deployed on WMF wikis in general. But I imagine this would come with the own list of challenges (e.g. I assume no single wiki has ALL the extensions, there is N combinations instead, etc).

oh, that was fast :)

Mentioned in SAL (#wikimedia-releng) [2018-07-27T11:38:52Z] <addshore> reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/ T200508

Validating CI for Wikibase using https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/447080/ (which I just rebased).
If PS2 there passes then we can close this

Addshore claimed this task.Jul 27 2018, 11:42 AM

Addshore added a project: Continuous-Integration-Config.

Restricted Application added a project: User-Addshore. · View Herald TranscriptJul 27 2018, 11:42 AM

Addshore moved this task from Unsorted 💣 to Blocked 🚧 on the User-Addshore board.Jul 27 2018, 11:42 AM

Addshore moved this task from Backlog to In progress on the Continuous-Integration-Config board.

https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/1899/console

11:49:20 FILE: .../phpunit/includes/Search/Elastic/DispatchingQueryBuilderTest.php
11:49:20 ----------------------------------------------------------------------
11:49:20 FOUND 0 ERRORS AND 1 WARNING AFFECTING 1 LINE
11:49:20 ----------------------------------------------------------------------
11:49:20  12 | WARNING | Duplicate interface name
11:49:20     |         | "CirrusSearch\Query\FullTextQueryBuilder" found; first
11:49:20     |         | defined in
11:49:20     |         | /workspace/src/extensions/Wikibase/tests/phan/stubs/cirrussearch.php
11:49:20     |         | on line 193 (Generic.Classes.DuplicateClassName.Found)
11:49:20 ----------------------------------------------------------------------
11:49:20 
11:49:20 Time: 2 mins, 29.15 secs; Memory: 17.52Mb

sounds like phpcs should ignore the phan stubs?

Addshore mentioned this in T195906: Phan fails for Wikibase on wmf.x branches.Jul 27 2018, 12:53 PM

Change 448511 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] phpcs ignore phan stubs

https://gerrit.wikimedia.org/r/448511

Change 448511 abandoned by Addshore:
phpcs ignore phan stubs

https://gerrit.wikimedia.org/r/448511

https://gerrit.wikimedia.org/r/448510

The patch linked above fixes the 1 issue

Addshore moved this task from Blocked 🚧 to Closing ✔️ on the User-Addshore board.Jul 27 2018, 1:46 PM

Re opening as this needs to be re solved as we had to stop WikibaseLexeme from running with Wikibase due to all of the recent CI issues.

My recent test with Lexeme running with Wikibase with the ContentTranslation CI produced the following: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/9972/console

16:09:26 There were 2 errors:
16:09:26 
16:09:26 1) Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest::testAddedStatementsWithLexemesAsTargetDisplayLemma
16:09:26 Hamcrest\AssertionError: Expected: is valid html piece having child (with tag name "ins" and having child (with tag name "a" and having text contents "foo"))
16:09:26      but: valid html piece having no children (with tag name "ins" and having child (with tag name "a" and having text contents "foo"))
16:09:26 Actual html:
16:09:26 <tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">property / <a title="Property:P1" href="/index.php/Property:P1">P1</a></td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span><a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span></span></ins></div></td></tr><tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">property / <a title="Property:P1" href="/index.php/Property:P1">P1</a>: <a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span> / rank</td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span>Normal rank</span></ins></div></td></tr>
16:09:26 
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:115
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:52
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest.php:29
16:09:26 /workspace/src/tests/phpunit/HamcrestPHPUnitIntegration.php:31
16:09:26 /workspace/src/extensions/WikibaseLexeme/tests/phpunit/mediawiki/Diff/LexemeDiffVisualizerIntegrationTest.php:114
16:09:26 /workspace/src/tests/phpunit/MediaWikiTestCase.php:475
16:09:26 /workspace/src/maintenance/doMaintenance.php:94
16:09:26 
16:09:26 2) Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest::testAddedStatementsOnFormsTargettingFormsAreDisplayedAsLinks
16:09:26 Hamcrest\AssertionError: Expected: is valid html piece having child (with tag name "ins" and having child (with tag name "a" and having text contents "baz"))
16:09:26      but: valid html piece having no children (with tag name "ins" and having child (with tag name "a" and having text contents "baz"))
16:09:26 Actual html:
16:09:26 <tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">Form / property / <a title="Property:P1" href="/index.php/Property:P1">P1</a></td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span><a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1-F1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span></span></ins></div></td></tr><tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">Form / property / <a title="Property:P1" href="/index.php/Property:P1">P1</a>: <a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1-F1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span> / rank</td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span>Normal rank</span></ins></div></td></tr>
16:09:26 
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:115
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:52
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest.php:29
16:09:26 /workspace/src/tests/phpunit/HamcrestPHPUnitIntegration.php:31
16:09:26 /workspace/src/extensions/WikibaseLexeme/tests/phpunit/mediawiki/Diff/LexemeDiffVisualizerIntegrationTest.php:448
16:09:26 /workspace/src/tests/phpunit/MediaWikiTestCase.php:475
16:09:26 /workspace/src/maintenance/doMaintenance.php:94
16:09:26 
16:09:26 FAILURES!
16:09:26 Tests: 3789, Assertions: 13750, Errors: 2, Skipped: 25.

This matches up to T197771

Addshore removed Addshore as the assignee of this task.Aug 6 2018, 4:28 PM

Addshore triaged this task as Medium priority.

Addshore added a subtask: T197771: Couple errors with Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest.

Addshore added a subtask: T201336: Use Quibble for WikibaseLexeme CI.Aug 6 2018, 5:55 PM

Addshore removed a subtask: T197771: Couple errors with Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest.

Addshore renamed this task from Changes in Wikibase that break WikibaseLexeme are not visible before merge to Run WikibaseLexeme tests with Wikibase CI.Aug 6 2018, 5:57 PM

Addshore mentioned this in T201496: Wikibase CI: Quibble job should possibly include Math extension.Aug 8 2018, 10:29 AM

Addshore claimed this task.Aug 13 2018, 1:34 PM

Change 452361 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Revert "Temporarily exclude WikibaseLexeme from jobs including Wikibase"

https://gerrit.wikimedia.org/r/452361

Addshore closed subtask T201336: Use Quibble for WikibaseLexeme CI as Resolved.Aug 13 2018, 1:35 PM

Change 452361 merged by jenkins-bot:
[integration/config@master] Revert "Temporarily exclude WikibaseLexeme from jobs including Wikibase"

https://gerrit.wikimedia.org/r/452361

Looks like everything is working and ContentTransltion ( which had issues before T200693 ) is green.

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/10565/consoleFull
Triggered by a recheck of https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ContentTranslation/+/248855/

And a validation run for Wikibase also looks green:
https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/2923/console :)

Addshore mentioned this in T201901: Selenium test Lexeme:Undoing removes grammatical features fails.Aug 14 2018, 11:54 AM

Addshore mentioned this in T201988: FormIdHtmlFormatter: Call to undefined method Item::getRepresentations().Aug 15 2018, 8:34 AM

Addshore mentioned this in T202066: Run WikibaseMediaInfo with Wikibase CI.Aug 16 2018, 3:48 PM

Even though the patch is merged and apparently the Lexeme tests get run with the ContentTranslation CI due to their including through Wikibase apparently they don't actually get run during the Wikibase CI itself.

Looking at the CLI debug log for https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/3199/ which has phpunit debugging turned on so it would output all tests run I do not see Lexeme tests, for example SpecialnewLexemeTest...

I discovered this while figuring out how T202114 got through our CI.....

addshore@deployment-deploy01:~$ cat test-cli-log.txt| grep "Start test" | grep SpecialNew
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Hooks\HtmlPageLinkRendererBeginHookHandlerTest::testDoHtmlPageLinkRendererBegin_overrideSpecialNewEntityLink with data set #0
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Hooks\HtmlPageLinkRendererBeginHookHandlerTest::testDoHtmlPageLinkRendererBegin_overrideSpecialNewEntityLink with data set #1
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testAllNecessaryFormFieldsArePresent_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testSiteAndPageInputFieldsWithPredefinedValuesPresent_WhenRenderedWithGetParametersPassed
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testLabelAndDescriptionValuesAreSetAccordingToSubpagePath_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenItemWithTheSameLabelAndDescriptionInThisLanguageAlreadyExists
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorAboutNonExistentPageIsDisplayed_WhenSiteExistsButPageDoesNot
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenLabelIsInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenDescriptionIsInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenAliasIsInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenAliasesAreInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only label is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "another language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only description is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "single alias"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "multiple aliases"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "nontrimmed label"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "nontrimmed description"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "all input is present"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown site identifier"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "all fields are empty"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "empty label and description, aliases contain only spaces and pipe symbols"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testMediaWikiTestCaseParentSetupCalled
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testValidCovers
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testAllNecessaryFormFieldsArePresent_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testLabelAndDescriptionAndDataTypeValuesAreSetAccordingToSubpagePath_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testFailsAndDisplaysAnError_WhenTryToCreateSecondPropertyWithTheSameLabel
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only label is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "another language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only description is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "single alias"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "multiple aliases"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "another datatype is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "all input is present"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown datatype"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "all fingerprint fields are empty"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "empty label and description, aliases contain only spaces and pipe symbols"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "empty datatype"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testMediaWikiTestCaseParentSetupCalled
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testValidCovers

Even though WikibaseLexme is listed to be loaded with Wikibase in parameter_functions.py it doesn't seem to actually be loading that, even after reloading zuul again just now.

so Wikibase doesn't run the Lexeme tests
it is only loaded in the "old" wikibase jobs, mwext-Wikibase-{kind}-tests-{dbflavor}-{phpflavor}-{image}
but that runs with either --group Wikibase,WikibaseAPI,Purtle or --group Wikibase,WikibaseClient
so lexeme jobs will never run there
and the wmf- prefixed quibble doesn't actually load Wikibase lexeme
ContentTranslation that keeps seeming to break (as in T202114) is running quibble-vendor-mysql-hhvm-docker (without the wmf- prefix) that actually loads Wikibase and Lexeme together and runs the tests...

Change 453385 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Add extension-quibble template to Wikibase

https://gerrit.wikimedia.org/r/453385

Change 453385 merged by jenkins-bot:
[integration/config@master] Add extension-quibble template to Wikibase

https://gerrit.wikimedia.org/r/453385

Mentioned in SAL (#wikimedia-releng) [2018-08-17T12:32:21Z] <addshore> reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)

Resulting job list now looks like this:

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/11201/console is the first example run and Lexeme and other Wikibase extensions can be seen loaded...

Addshore moved this task from Back Burner 🏛️ to Closing ✔️ on the User-Addshore board.Aug 17 2018, 12:38 PM

	F25059248: image.png
	Aug 17 2018, 12:35 PM

Run WikibaseLexeme tests with Wikibase CIClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Run WikibaseLexeme tests with Wikibase CI
Closed, ResolvedPublic
Actions

Related Objects
Search...