Page MenuHomePhabricator
Create Task
Maniphest T207920

Test spicerack elasticsearch module
Closed, ResolvedPublic
Actions

Description

This will serve as an integration test to test this module.

Test spicerack elasticsearch module on cumin host(s).

  • Using python repl test elasticsearch_clusters module. Include result or result summary here.
  • Test rolling restart cookbook using relforge as entry point.

Details

SubjectRepoBranchLines +/-
elasticsearch: fix typo (xarg instead of xargs)operations/software/spicerackmaster+4 -4
elasticsearch: systemctl iterates explicitly on elasticsearch instancesoperations/software/spicerackmaster+10 -9
elasticsearch: add a file with the list on elasticsearch instancesoperations/puppetproduction+13 -0
elasticsearch: get_next_clusters_nodes raises ElasticsearchClusterErroroperations/software/spicerackmaster+2 -2
elasticsearch: support cluster names which have '-' in themoperations/software/spicerackmaster+11 -10
elasticsearch: access production clusters over HTTPSoperations/software/spicerackmaster+6 -6
elasticsearch: retry on TransportError while waiting for node to be upoperations/software/spicerackmaster+34 -10
elasticsearch: add doc type to delete queryoperations/software/spicerackmaster+2 -1
add temporary lvs enabled optionoperations/cookbooksmaster+13 -3
elasticsearch_cluster: fix issues from test resultoperations/software/spicerackmaster+18 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Mathew.onipe triaged this task as Medium priority.Oct 25 2018, 3:38 AM
• Mathew.onipe created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 25 2018, 3:38 AM
• Mathew.onipe renamed this task from Test spicerack elasticsearch module on relforge or similar environment to Test spicerack elasticsearch module on relforge.Jan 8 2019, 3:08 AM
• Mathew.onipe renamed this task from Test spicerack elasticsearch module on relforge to Test spicerack elasticsearch module.Jan 10 2019, 1:52 PM
• Mathew.onipe updated the task description. (Show Details)
• Mathew.onipe added a comment.Jan 24 2019, 3:37 AM

Gehel did some testing and here is the link to the results: https://etherpad.wikimedia.org/p/spicerack-elastic-test-plan
The summary are as follows:

  • Need to fix .get_next_clusters_nodes(datetime.utcnow(), 0) calling this method with nodes_per_run as 0, throws error. The error raised here should be caught and reraised to alert the user/client.
  • Fix .frozen_writes(reason) method. Currently it throws an error about an unexpected keywork arg - doc_id.
  • Fix .start_elasticsearch(). Here, elasticsearch instances were not started. wildcard may not be working the way we think it is working. more investigation to follow.

More test to follow after this before making patches to fix these.

• Mathew.onipe added a project: Elasticsearch.Jan 24 2019, 3:38 AM
Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptJan 24 2019, 3:38 AM
• Mathew.onipe edited projects, added Discovery-Search (Current work); removed Discovery-Search.Jan 24 2019, 3:39 AM
• Mathew.onipe moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:34 PM
gerritbot added a comment.Jan 28 2019, 1:45 PM

Change 486858 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/software/spicerack@master] elasticsearch_cluster: fix issues from test result

https://gerrit.wikimedia.org/r/486858

gerritbot added a project: Patch-For-Review.Jan 28 2019, 1:45 PM
• Mathew.onipe moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Jan 30 2019, 8:29 AM
gerritbot added a comment.Feb 11 2019, 7:48 AM

Change 489628 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/cookbooks@master] add temp lvs enabled option

https://gerrit.wikimedia.org/r/489628

gerritbot added a comment.Feb 11 2019, 10:40 AM

Change 489628 merged by Gehel:
[operations/cookbooks@master] add temporary lvs enabled option

https://gerrit.wikimedia.org/r/489628

gerritbot added a comment.Feb 11 2019, 10:42 AM

Change 486858 merged by Gehel:
[operations/software/spicerack@master] elasticsearch_cluster: fix issues from test result

https://gerrit.wikimedia.org/r/486858

gerritbot added a comment.Feb 13 2019, 6:30 PM

Change 490383 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/software/spicerack@master] elasticsearch: add type to delete query

https://gerrit.wikimedia.org/r/490383

gerritbot added a comment.Feb 14 2019, 2:01 PM

Change 490383 merged by Gehel:
[operations/software/spicerack@master] elasticsearch: add doc type to delete query

https://gerrit.wikimedia.org/r/490383

gerritbot added a comment.Feb 15 2019, 3:27 PM

Change 490866 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/software/spicerack@master] elasticsearch: retry on TransportError while waiting for node to be up

https://gerrit.wikimedia.org/r/490866

gerritbot added a comment.Feb 18 2019, 10:40 AM

Change 490866 merged by jenkins-bot:
[operations/software/spicerack@master] elasticsearch: retry on TransportError while waiting for node to be up

https://gerrit.wikimedia.org/r/490866

• Mathew.onipe moved this task from Needs review to Incoming on the Discovery-Search (Current work) board.Feb 19 2019, 6:31 PM
• Mathew.onipe removed a project: Patch-For-Review.
Gehel moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Feb 20 2019, 9:42 AM

Upgrade to elasticsearch 5.6.x was performed on relforge with spicerack, so testing is complete. There are always things to improve, but this is working well enough !

gerritbot added a comment.Feb 20 2019, 2:08 PM

Change 491750 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/software/spicerack@master] elasticsearch: access production clusters over HTTPS

https://gerrit.wikimedia.org/r/491750

gerritbot added a project: Patch-For-Review.Feb 20 2019, 2:08 PM
gerritbot added a comment.Feb 20 2019, 2:15 PM

Change 491750 merged by Gehel:
[operations/software/spicerack@master] elasticsearch: access production clusters over HTTPS

https://gerrit.wikimedia.org/r/491750

gerritbot added a comment.Feb 20 2019, 4:29 PM

Change 491797 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/software/spicerack@master] elasticsearch: support cluster names which have '-' in them

https://gerrit.wikimedia.org/r/491797

gerritbot added a comment.Feb 20 2019, 4:58 PM

Change 491797 merged by Gehel:
[operations/software/spicerack@master] elasticsearch: support cluster names which have '-' in them

https://gerrit.wikimedia.org/r/491797

gerritbot added a comment.Feb 20 2019, 5:01 PM

Change 491803 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/software/spicerack@master] elasticsearch: get_next_clusters_nodes raises ElasticsearchClusterError

https://gerrit.wikimedia.org/r/491803

gerritbot added a comment.Feb 20 2019, 5:25 PM

Change 491808 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/software/spicerack@master] elasticsearch: systemctl iterates explicitly on elasticsearch instances

https://gerrit.wikimedia.org/r/491808

gerritbot added a comment.Feb 20 2019, 8:32 PM

Change 491850 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: add script to execute systemctl on each elasticsearch instance

https://gerrit.wikimedia.org/r/491850

gerritbot added a comment.Feb 21 2019, 8:54 AM

Change 491803 merged by Gehel:
[operations/software/spicerack@master] elasticsearch: get_next_clusters_nodes raises ElasticsearchClusterError

https://gerrit.wikimedia.org/r/491803

gerritbot added a comment.Feb 21 2019, 9:48 AM

Change 491850 merged by Gehel:
[operations/puppet@production] elasticsearch: add a file with the list on elasticsearch instances

https://gerrit.wikimedia.org/r/491850

gerritbot added a comment.Feb 21 2019, 10:04 AM

Change 491808 merged by Gehel:
[operations/software/spicerack@master] elasticsearch: systemctl iterates explicitly on elasticsearch instances

https://gerrit.wikimedia.org/r/491808

gerritbot added a comment.Feb 21 2019, 1:23 PM

Change 491960 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/software/spicerack@master] elasticsearch: fix typo (xarg instead of xargs)

https://gerrit.wikimedia.org/r/491960

gerritbot added a comment.Feb 22 2019, 8:38 AM

Change 491960 merged by Gehel:
[operations/software/spicerack@master] elasticsearch: fix typo (xarg instead of xargs)

https://gerrit.wikimedia.org/r/491960

debt closed this task as Resolved.Feb 22 2019, 8:39 PM
joanna_borun added a project: Spicerack.Jun 15 2022, 10:46 AM
Restricted Application added a project: SRE-tools. · View Herald TranscriptJun 15 2022, 10:46 AM
• Mathew.onipe mentioned this in rCCKB3cfd3e9d324e: add temporary lvs enabled option.Dec 14 2022, 3:23 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL