Page MenuHomePhabricator
Create Task
Maniphest T90487

Deploy graphoid service into production
Closed, ResolvedPublic
Actions

Description

Time has come to let the baby fly. The graphoid code has been rewritten on top of the service-template-node & service-runner, instrumented for production and needs to be deployed. Once done, it will make graphs available as images, allowing support for non-js browsers, and possibly others.

Expected load - very very low at first.
If we make images the default (instead of client-side rendering), the load may increase significantly once graphs are widely used.

Needed help: any guide or similar service that i can copy/paste.

Details

SubjectRepoBranchLines +/-
Graphoid: Varnish configurationoperations/puppetproduction+22 -0
Graphoid: LVS configurationoperations/puppetproduction+20 -0
Assign LVS IPs to the graphoid serviceoperations/dnsmaster+5 -0
Graphoid: service deployment on SCAoperations/puppetproduction+117 -1
WIP: Graphoid: Puppet bitsoperations/puppetproduction+144 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Yurik created this task.Feb 23 2015, 9:03 PM
Yurik claimed this task.
Yurik raised the priority of this task from to High.
Yurik updated the task description. (Show Details)
Yurik added projects: MediaWiki-extensions-Graph, Services, service-runner, service-template-node.
Yurik added subscribers: Yurik, • GWicke, ori and 2 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2015, 9:03 PM
Yurik moved this task from Backlog to Graphoid on the MediaWiki-extensions-Graph board.Feb 23 2015, 9:05 PM
Yurik added a parent task: T87406: Deploy Graph Extension to all Wikipedias.Feb 23 2015, 9:07 PM
Yurik updated the task description. (Show Details)Feb 23 2015, 9:12 PM
Yurik set Security to None.
Eloquence added a comment.Feb 24 2015, 3:08 AM

@GWicke - are we at a point yet where someone on your team could help Yuri get this all the way into prod? Obviously lower priority than VE dependencies ... but would still be great to get out the door.

ori added a comment.Feb 24 2015, 3:38 AM
In T90487#1061216, @Eloquence wrote:

@GWicke - are we at a point yet where someone on your team could help Yuri get this all the way into prod? Obviously lower priority than VE dependencies ... but would still be great to get out the door.

Are the VE dependencies actually done? If nothing else, the code to call RESTbase from VE has not been written yet, so if we have a surplus of engineering resources, it'd be great to get that out the door, too.

Eloquence added a comment.Feb 24 2015, 4:39 AM

*nod* Fair point :)

• GWicke added a comment.Feb 24 2015, 6:06 AM

@Eloquence: We helped @Yurik over the weekend, but this week we really want to get RESTBase & the VE integration ready so that we can test it thoroughly next week.

Eloquence added a comment.Feb 24 2015, 6:07 AM

sounds good - let's revisit later. yuri - hold your horses for a bit longer :)

Yurik added a comment.Feb 24 2015, 6:09 AM

@GWicke, thanks for all your help over the weekend, couldn't have implemented the whole service so quickly without it! Could someone provide some starting points for me so I could poke at it myself? Such as: which repo/files contain something similar to what I would need? I would do all the preparatory work to simplify the final review. Thanks!

• GWicke added a comment.Feb 24 2015, 6:21 AM

@Yurik, you can take a look at the restbase Puppet module. It has slightly more things than you need, but should give you an idea & already uses service-runner.

Most of the general service deploy tasks are spelled out in T89901, which is about creating a module to address the recurring parts generically. @ori signalled his intention to look into this as well, so I'm pretty optimistic that we can create something soon.

• GWicke moved this task from Backlog to Next on the Services board.Mar 17 2015, 8:01 PM
• mobrovac added a project: acl*sre-team.Mar 24 2015, 1:15 PM
• mobrovac added a subtask: T93881: Improve Graphoid code.Mar 25 2015, 1:59 PM
Mvolz added subscribers: Mvolz, Mooeypoo.Mar 25 2015, 2:02 PM
Yurik closed subtask T93881: Improve Graphoid code as Resolved.Apr 8 2015, 1:06 AM
Liuxinyu970226 subscribed.Apr 8 2015, 1:10 AM
• mobrovac reopened subtask T93881: Improve Graphoid code as Open.Apr 8 2015, 9:16 AM
• mobrovac closed subtask T93881: Improve Graphoid code as Resolved.Apr 9 2015, 8:15 AM
• mobrovac added a subtask: T95533: Unify SCA Service Puppet Modules / Roles.Apr 9 2015, 9:41 AM
• mobrovac closed subtask T95533: Unify SCA Service Puppet Modules / Roles as Resolved.Apr 14 2015, 12:37 PM
• mobrovac added a subscriber: akosiaris.Apr 15 2015, 8:58 AM

@akosiaris has started work on this, ETA: end of April

Ricordisamoa subscribed.Apr 16 2015, 3:02 PM
Yurik mentioned this in T87407: Implement server-side graph rendering.Apr 16 2015, 3:55 PM
Yurik added a parent task: T73227: Graph extension does not have a fallback for non-JavaScript browsers.
Yurik reassigned this task from Yurik to akosiaris.Apr 16 2015, 4:18 PM
Anomie mentioned this in T73227: Graph extension does not have a fallback for non-JavaScript browsers.Apr 16 2015, 7:21 PM
Yurik edited projects, added Graphoid; removed MediaWiki-extensions-Graph.Apr 16 2015, 9:19 PM
• mobrovac added a subtask: T96557: List domains and aliases for Graphoid in production.Apr 20 2015, 2:17 PM
• mobrovac closed subtask T96557: List domains and aliases for Graphoid in production as Invalid.Apr 20 2015, 4:54 PM
gerritbot subscribed.Apr 20 2015, 7:41 PM

Change 205350 had a related patch set uploaded (by Mobrovac):
WIP: Graphoid: Puppet bits

https://gerrit.wikimedia.org/r/205350

gerritbot added a project: Patch-For-Review.Apr 20 2015, 7:41 PM
akosiaris added a comment.Apr 22 2015, 2:24 PM

There are a couple of things missing from this ticket. Those are:

  • A clear description of what this service does. Preferably a link to a wiki page

https://www.mediawiki.org/wiki/Extension:Graph seems to be a good start with a section for Graphoid under https://www.mediawiki.org/wiki/Extension:Graph#Graphoid_Service but a dedicated page with some more information would be welcome. Specifically I 'd love:

  • A link to a simplified proposed architecture diagram. Nothing too fancy,
    • just how requests flow from
      1. The browser(end-user) to mediawiki (well the Graph Extension)
      2. The extension to the service (if any)
      3. The browser to the service (if any)
    • Jobs that might need to run via jobrunners (if any)
    • Data store dependencies (if any)
    • Anything architecturally significant I might have missed

The idea is to have a very very clear diagram for everyone to understand at first glance so it can serve as documentation. No intermediate layers like nginx/varnish/restbase or whatever. These can/will probably change at some point in the future, especially if too much info is provided (like which varnish cluster is being used) without updating the diagram making the documentation out of date and less helpful.

  • Who's running point

I am assuming the services team, but I may be wrong and we definitely want to have that documented

gerritbot added a comment.Apr 22 2015, 2:27 PM

Change 205856 had a related patch set uploaded (by Alexandros Kosiaris):
Assign LVS IPs to the graphoid service

https://gerrit.wikimedia.org/r/205856

• mobrovac added a comment.Apr 22 2015, 2:41 PM
In T90487#1227653, @akosiaris wrote:

There are a couple of things missing from this ticket. Those are:

  • A clear description of what this service does. Preferably a link to a wiki page

https://www.mediawiki.org/wiki/Extension:Graph seems to be a good start with a section for Graphoid under https://www.mediawiki.org/wiki/Extension:Graph#Graphoid_Service but a dedicated page with some more information would be welcome. Specifically I 'd love:

  • A link to a simplified proposed architecture diagram. Nothing too fancy,
    • just how requests flow from
      1. The browser(end-user) to mediawiki (well the Graph Extension)
      2. The extension to the service (if any)
      3. The browser to the service (if any)
    • Jobs that might need to run via jobrunners (if any)
    • Data store dependencies (if any)
    • Anything architecturally significant I might have missed

The idea is to have a very very clear diagram for everyone to understand at first glance so it can serve as documentation. No intermediate layers like nginx/varnish/restbase or whatever. These can/will probably change at some point in the future, especially if too much info is provided (like which varnish cluster is being used) without updating the diagram making the documentation out of date and less helpful.

Great idea, @akosiaris! @Yurik, could you see to it? As always, Services can help if you need any.

  • Who's running point

I am assuming the services team, but I may be wrong and we definitely want to have that documented

Actually, it should be @Yurik. The Services team is here only to facilitate the deployment (he's already got +2 on the deploy repo).

akosiaris added a comment.Apr 22 2015, 3:32 PM

>> - Who's running point

I am assuming the services team, but I may be wrong and we definitely want to have that documented

Actually, it should be @Yurik. The Services team is here only to facilitate the deployment (he's already got +2 on the deploy repo).

Good thing I asked then :-)

Yurik added a comment.Apr 22 2015, 10:34 PM

I will create a doc page with a diagram in the next few days. Please ping for any other info.

gerritbot added a comment.Apr 23 2015, 10:36 AM

Change 205350 abandoned by Mobrovac:
WIP: Graphoid: Puppet bits

Reason:
As per IRC discussion with _joe_, abandoning this change in order to have a complete blueprint for all of the bits needed for service deploy. I will create new patches for graphoid split in three parts.

https://gerrit.wikimedia.org/r/205350

gerritbot added a comment.Apr 23 2015, 12:43 PM

Change 206105 had a related patch set uploaded (by Mobrovac):
Graphoid: service deployment on SCA

https://gerrit.wikimedia.org/r/206105

gerritbot added a comment.Apr 23 2015, 12:59 PM

Change 206106 had a related patch set uploaded (by Mobrovac):
Graphoid: LVS configuration

https://gerrit.wikimedia.org/r/206106

gerritbot added a comment.Apr 23 2015, 1:05 PM

Change 206108 had a related patch set uploaded (by Mobrovac):
Graphoid: Varnish configuration

https://gerrit.wikimedia.org/r/206108

• csteipp closed subtask T93126: Perform a security review of graphoid as Resolved.Apr 23 2015, 5:52 PM
• mobrovac added a subtask: T97123: Graphoid's routes should be RESTful.Apr 24 2015, 12:21 PM
Ricordisamoa awarded a token.Apr 24 2015, 2:44 PM
gerritbot added a comment.Apr 27 2015, 3:28 PM

Change 206105 merged by Alexandros Kosiaris:
Graphoid: service deployment on SCA

https://gerrit.wikimedia.org/r/206105

• mobrovac mentioned this in rOPUPf56e53bb2dcf: Graphoid: service deployment on SCA.Apr 27 2015, 3:28 PM
• mobrovac closed subtask T97123: Graphoid's routes should be RESTful as Resolved.Apr 28 2015, 12:40 PM
Yurik added a comment.Apr 28 2015, 2:28 PM

Diagram has been added to the https://www.mediawiki.org/w/index.php?title=Extension:Graph#Graphoid_Service

gerritbot added a comment.Apr 28 2015, 2:35 PM

Change 205856 merged by Alexandros Kosiaris:
Assign LVS IPs to the graphoid service

https://gerrit.wikimedia.org/r/205856

akosiaris mentioned this in rODNSa7c919bbd78d: Assign LVS IPs to the graphoid service.Apr 28 2015, 2:37 PM
gerritbot added a comment.Apr 28 2015, 2:38 PM

Change 206106 merged by Alexandros Kosiaris:
Graphoid: LVS configuration

https://gerrit.wikimedia.org/r/206106

• mobrovac mentioned this in rOPUP8e6c60dcb924: Graphoid: LVS configuration.Apr 28 2015, 2:38 PM
• mobrovac added a comment.Apr 28 2015, 3:06 PM
In T90487#1241635, @Yurik wrote:

Diagram has been added to the https://www.mediawiki.org/w/index.php?title=Extension:Graph#Graphoid_Service

+1, LGTM. Nice one, @Yurik

gerritbot added a comment.Apr 29 2015, 2:46 PM

Change 206108 merged by Alexandros Kosiaris:
Graphoid: Varnish configuration

https://gerrit.wikimedia.org/r/206108

• mobrovac mentioned this in rOPUP3101479e2e92: Graphoid: Varnish configuration.Apr 29 2015, 2:46 PM
akosiaris closed this task as Resolved.Apr 29 2015, 3:03 PM

The corresponding changes have been merged, @Yurik has updated the wiki page with a architectural diagram, simple tests at
https://graphoid.wikimedia.org/_info/home work, icinga is happy, resolving this. Thanks everyone!

Liuxinyu970226 unsubscribed.Apr 29 2015, 11:39 PM
• mobrovac mentioned this in T98803: Varnish does not honor Cache-Control for Graphoid.May 14 2015, 10:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL