Page MenuHomePhabricator
Create Task
Maniphest T129290

[keyresult] Migrate remaining trebuchet deployed services
Closed, ResolvedPublic
Actions

Description

https://wikitech.wikimedia.org/wiki/Services/Scap_Migration

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• MoritzMuehlenhoffT164780 Sunset our use of Salt
 ResolvedthciprianiT129290 [keyresult] Migrate remaining trebuchet deployed services
 InvalidNoneT129155 Deploy sentry with scap3
 ResolvedthciprianiT129154 Deploy iegreview with scap3
 DeclinedNoneT129153 Deploy rcstream with scap3
 ResolvedakosiarisT129152 Deploy servermon with scap3
 ResolvedelukeyT129151 Deploy analytics-refinery with scap3
 ResolvedYurikT129150 Deploy kartotherian with scap3
 ResolvedthciprianiT129149 Deploy discovery-analytics with scap3
 ResolvedthciprianiT129148 Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner)
 ResolvedthciprianiT167098 scap should allow restarting multiple services
 ResolvedthciprianiT167104 Figure out how to disable starting of jobrunner/jobchron in the non-active DC
 Resolved• bearNDT129147 Deploy mobileapps/deploy with scap3
 ResolvedYurikT129146 Deploy tilerator with scap3
 ResolvedoriT129145 remove fluoride from the deployments directory
 ResolvedSmalyshevT129144 Deploy wdqs with scap3
 ResolvedDzahnT138628 Add wdqs-admins to deploy-services group
 Resolved• mobrovacT129143 Deploy zotero/translators with scap3
 DeclinedNoneT129142 Deploy ocg with scap3
 Resolved• mobrovacT129140 Deploy zotero/translation-server with scap3
 ResolvedfgiunchediT129139 Deploy statsv with scap3
 ResolvedakosiarisT129136 Deploy libreNMS with scap3
 ResolvedNiharikaT129134 Deploy scholarships with scap3
 Resolved• mmodellT114363 Deploy Phabricator with scap3
 ResolvedthciprianiT116207 enforcing deployment from `/srv/deployment` is wrong
 Resolved• mmodellT125853 Move /srv/phab/repos to /srv/repos
 Resolved• mmodellT125851 Refactor phabricator module in puppet to remove git tag pinning behavior
 Resolved• mmodellT113072 Make puppet provider for scap3
 ResolvedfgiunchediT126660 rebuild scap debian package (we forgot to include refreshCdbJsonFiles)
 Resolved• mmodellT127215 scap::target should use scap's debian package instead of trebuchet
 DeclinedNoneT151996 Deploy elasticsearch plugins with scap3 (Trebuchet elasticsearch/plugins)
 ResolvedthciprianiT165748 Deploy logstash/plugins with scap3
 Resolved• mobrovacT116340 Deploy logstash logback encoder with scap3
 ResolveddduvallT129420 Support git-fat in scap3 deployment
 Resolved• mobrovacT137371 Deploy cassandra metrics collector via scap3
 InvalidNoneT170839 Migrate dropwizard/metrics to scap3
 Resolved• demonT170881 Cleanup /srv/deployment
 DeclinedKrinkleT118772 Use scap3 to deploy eventlogging/eventlogging
 ResolvedOttomataT131263 Stop using global python setup.py install for eventlogging deploy
 ResolvedOttomataT131354 Make sure ALL eventlogging dependencies are available in both jessie and trusty via .deb packages
 ResolvedKrinkleT131977 Stop using global eventlogging install on hafnium (and any other eventlogging lib user)
 Resolved• mobrovacT130948 Scap3 promote stage not working

Event Timeline

greg created this task.Mar 8 2016, 11:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 8 2016, 11:09 PM
greg renamed this task from Migrate remaining trebuchet deployed services to [keyresult] Migrate remaining trebuchet deployed services.Mar 8 2016, 11:09 PM
greg triaged this task as Medium priority.
greg added subtasks: T129155: Deploy sentry with scap3, T129154: Deploy iegreview with scap3, T129153: Deploy rcstream with scap3, T129152: Deploy servermon with scap3, T129151: Deploy analytics-refinery with scap3, T129150: Deploy kartotherian with scap3, T129149: Deploy discovery-analytics with scap3, T129148: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner), T129147: Deploy mobileapps/deploy with scap3, T129146: Deploy tilerator with scap3, T129145: remove fluoride from the deployments directory, T129144: Deploy wdqs with scap3, T129143: Deploy zotero/translators with scap3, T129142: Deploy ocg with scap3, T129140: Deploy zotero/translation-server with scap3, T129139: Deploy statsv with scap3, T129138: Deploy kibana with debian packages, T129136: Deploy libreNMS with scap3, T129134: Deploy scholarships with scap3, T114363: Deploy Phabricator with scap3.Mar 8 2016, 11:13 PM
• mobrovac closed subtask T129143: Deploy zotero/translators with scap3 as Resolved.Apr 15 2016, 2:18 PM
• mobrovac closed subtask T129140: Deploy zotero/translation-server with scap3 as Resolved.Apr 15 2016, 2:38 PM
• bearND updated the task description. (Show Details)May 3 2016, 5:21 PM
• mobrovac closed subtask T129147: Deploy mobileapps/deploy with scap3 as Resolved.May 11 2016, 4:07 PM
Yurik closed subtask T129150: Deploy kartotherian with scap3 as Resolved.Jun 16 2016, 9:55 PM
Yurik closed subtask T129146: Deploy tilerator with scap3 as Resolved.
Smalyshev closed subtask T129144: Deploy wdqs with scap3 as Resolved.Jun 28 2016, 9:02 PM
greg removed a subtask: T129138: Deploy kibana with debian packages.Jun 28 2016, 10:49 PM
• Nuria closed subtask T129151: Deploy analytics-refinery with scap3 as Resolved.Aug 8 2016, 3:20 PM
ori closed subtask T129145: remove fluoride from the deployments directory as Resolved.Nov 19 2016, 7:05 PM
thcipriani closed subtask T129155: Deploy sentry with scap3 as Invalid.Nov 28 2016, 6:43 PM
hashar created subtask T151996: Deploy elasticsearch plugins with scap3 (Trebuchet elasticsearch/plugins).Nov 30 2016, 11:42 AM
akosiaris closed subtask T129136: Deploy libreNMS with scap3 as Resolved.May 5 2017, 1:45 PM
faidon added a parent task: T164780: Sunset our use of Salt.May 8 2017, 7:00 PM
thcipriani added subscribers: cscott, Ottomata, bd808 and 3 others.May 8 2017, 7:45 PM

Went through this somewhat recently to see what yet remains on tin, here's a dump of my notes:

$ ls /srv/deployment/*/*/.git/FETCH_HEAD | sed -e 's#/.git/FETCH_HEAD##' | sort > ~/deployed
$ ls /srv/deployment/*/*/.git/DEPLOY_HEAD | sed -e 's#/.git/DEPLOY_HEAD##' | sort > ~/deployed-with-scap
$ comm -23 deployed deployed-with-scap
/srv/deployment/abacist/abacist
/srv/deployment/cassandra/logstash-logback-encoder
/srv/deployment/cassandra/metrics-collector
/srv/deployment/dropwizard/metrics
/srv/deployment/elasticsearch/plugins
/srv/deployment/eventlogging/eventlogging
/srv/deployment/gdash/gdash
/srv/deployment/grafana/grafana
/srv/deployment/iegreview/iegreview
/srv/deployment/integration/mediawiki-tools-codesniffer
/srv/deployment/integration/php-coveralls
/srv/deployment/integration/phpcs
/srv/deployment/integration/phpunit
/srv/deployment/integration/slave-scripts
/srv/deployment/jobrunner/jobrunner
/srv/deployment/kibana/kibana
/srv/deployment/librenms/librenms.a
/srv/deployment/logstash/plugins
/srv/deployment/mathoid/mathoid
/srv/deployment/ocg/ocg
/srv/deployment/parsoid/deploy-old
/srv/deployment/parsoid/Parsoid
/srv/deployment/rcstream/rcstream
/srv/deployment/restbase/deploy.bak
/srv/deployment/restbase/deploy.old
/srv/deployment/scholarships/scholarships
/srv/deployment/servermon/servermon
/srv/deployment/statsv/statsv
/srv/deployment/tessera/tessera

Need manual removed from tin:

/srv/deployment/parsoid/deploy-old
/srv/deployment/restbase/deploy.bak
/srv/deployment/restbase/deploy.old
/srv/deployment/librenms/librenms.a
/srv/deployment/mathoid/mathoid
integration/php-coveralls
integration/phpunit
integration/slave-scripts
integration/php-coveralls
integration/phpcs

Have owners

Services:

/srv/deployment/cassandra/logstash-logback-encoder
/srv/deployment/cassandra/metrics-collector

Blocked?

eventlogging/eventlogging

In the README there shows blocked by T131977: Stop using global eventlogging install on hafnium (and any other eventlogging lib user), but that's closed so maybe blocked no longer @Ottomata may know.

Unknown:

  • dropwizard/metrics - super old, early 2015
  • iegreview/iegreview
  • logstash/plugins - @bd808 may know
  • ocg/ocg - @cscott may know
  • rcstream/rcstream
  • statsv/statsv - @Ottomata may know
bd808 added subscribers: Niharika, EBernhardson.May 8 2017, 8:31 PM
In T129290#3245438, @thcipriani wrote:
  • iegreview/iegreview

This one is me & @Niharika (same as scholarships).

  • logstash/plugins - @bd808 may know

@EBernhardson owns this one these days. I gave up my Logstash root powers and ownership responsibility in rOPUP9a43626e8c9a: Remove bd808 from logstash-roots.

EBernhardson added a comment.May 9 2017, 4:15 PM

I'm not really sure what has to be done to switch logstash-plugins from trebuchet to scap, but basically the only thing it needs to do:

  • Ship updates to the repository to logstash100[123].eqiad.wmnet
  • Utilize git-fat to pull binary blobs from archiva
hashar added a comment.May 9 2017, 4:46 PM

jobrunner/jobrunner - @hashar - T129148

I can't tackle that one sorry :-( Though hypothetically it is something we can pair up on next week. Would be an opportunity for me to level up.

Stashbot subscribed.May 19 2017, 10:08 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-19T10:08:28Z] <_joe_> moved stale repos to /srv/deployment/STALE on tin, T129290

thcipriani added a subtask: T165748: Deploy logstash/plugins with scap3.May 19 2017, 12:48 PM
greg added a project: releng-201718-q1.May 20 2017, 11:55 AM
greg added a project: Trebuchet.
hashar unsubscribed.May 23 2017, 7:43 AM
dcausse closed subtask T151996: Deploy elasticsearch plugins with scap3 (Trebuchet elasticsearch/plugins) as Declined.May 31 2017, 7:15 AM
akosiaris closed subtask T129152: Deploy servermon with scap3 as Resolved.Jun 6 2017, 11:29 AM
thcipriani closed subtask T129148: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) as Resolved.Jun 9 2017, 6:09 PM
hashar reopened subtask T129148: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) as Open.Jun 22 2017, 2:11 PM
thcipriani closed subtask T165748: Deploy logstash/plugins with scap3 as Resolved.Jun 22 2017, 5:29 PM
• demon closed subtask T129153: Deploy rcstream with scap3 as Declined.Jul 3 2017, 6:05 PM
greg added a subtask: T116340: Deploy logstash logback encoder with scap3.Jul 12 2017, 10:46 PM
thcipriani added a subtask: T137371: Deploy cassandra metrics collector via scap3.Jul 17 2017, 3:55 PM
thcipriani added a subtask: T170839: Migrate dropwizard/metrics to scap3.
thcipriani added a subtask: T170881: Cleanup /srv/deployment.Jul 17 2017, 8:58 PM
fgiunchedi closed subtask T129139: Deploy statsv with scap3 as Resolved.Jul 19 2017, 10:02 AM
• mobrovac closed subtask T116340: Deploy logstash logback encoder with scap3 as Resolved.Aug 10 2017, 7:37 PM
• mobrovac closed subtask T137371: Deploy cassandra metrics collector via scap3 as Resolved.Aug 10 2017, 8:53 PM
greg added a subtask: T118772: Use scap3 to deploy eventlogging/eventlogging.Aug 16 2017, 5:00 PM
Krinkle added a subtask: T110903: Make webperf eventlogging consumers use eventlogging on Kafka.Aug 17 2017, 9:53 PM
Krinkle closed subtask T118772: Use scap3 to deploy eventlogging/eventlogging as Declined.
Krinkle changed the status of subtask T110903: Make webperf eventlogging consumers use eventlogging on Kafka from Stalled to Open.
thcipriani closed subtask T170839: Migrate dropwizard/metrics to scap3 as Invalid.Aug 23 2017, 3:28 PM
Krinkle removed a subtask: T110903: Make webperf eventlogging consumers use eventlogging on Kafka.Aug 31 2017, 1:42 AM
• demon closed subtask T170881: Cleanup /srv/deployment as Resolved.Sep 1 2017, 5:34 PM
thcipriani closed subtask T129154: Deploy iegreview with scap3 as Resolved.Sep 13 2017, 3:47 PM
thcipriani closed subtask T129134: Deploy scholarships with scap3 as Resolved.
thcipriani closed subtask T129149: Deploy discovery-analytics with scap3 as Resolved.Sep 14 2017, 5:19 PM
thcipriani closed subtask T129142: Deploy ocg with scap3 as Declined.Sep 15 2017, 12:54 PM
thcipriani closed subtask T129148: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) as Resolved.Sep 18 2017, 9:10 PM
thcipriani closed this task as Resolved.Sep 18 2017, 9:12 PM
thcipriani claimed this task.

All subtasks closed. Closing this one out.

greg added a comment.Sep 18 2017, 9:14 PM

\o/

bd808 awarded a token.Sep 18 2017, 9:21 PM
• mmodell awarded a token.Sep 19 2017, 12:43 AM
• mmodell subscribed.

itshappening

• Phabricator_maintenance added a project: RelEng-Archive-FY201718-Q1.Dec 21 2017, 7:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits