For the record, just saying pointing out that the question of a new VM versus mwmaint1002 is probably irrelevant here. We can do both with what looks like minimal repercussions.

In T216238#4960997, @elukey wrote:

So after reading https://wikitech.wikimedia.org/wiki/Ganeti#Create_a_VM this is what I'd do:

1. Review/Merge https://gerrit.wikimedia.org/r/491219 to add a Private ip allocation in row A (after checking gnt-group list on ganeti1003 it seems to me that either A or C are fine, but lemme know if there is more to verify)
2. Use makevm on ganeti1003 to create the VM and annotate the MAC address
3. Create a puppet code change to add the node to DHCP and partman configs. Run puppet on install[12]002 before proceeding.
4. Run gnt-instance start kerberos1001.eqiad.wnet on ganeti1003, and then attach to the console via gnt-instance console kerberos1001.eqiad.wnet
5. Wait for the OS install to finish, and then before the end execute gnt-instance modify --hypervisor-parameters=boot_order=disk kerberos1001.eqiad.wnet

In T213566#4934895, @Ottomata wrote:

they will also not allow them to send the SYN/ACK packet required for the second (of the three) phase of the TCP handshake,

? Is that true? I'm pretty sure it isn't...unless we have some special rules I'm forgetting.

In T215319#4928541, @thcipriani wrote:

Clarification needed from serviceops folks: is it only the base-image for the production variant we want to restrict?

FrEx: blubber uses the golang image to build blubberoid and then copies that artifact to the production image based on docker-registry.wikimedia.org/wikimedia-stretch

In T215320#4933573, @mobrovac wrote:

Bonus points if these jobs could be created by someone adding a .pipeline/config.yaml to their project.

At least 100000 bonus points for this! It would be really really useful if these semi-custom jobs could be created automatically for people without human intervention and/or git commits and deploys.

In T177868#3673182, @thcipriani wrote:

In talking with @dduvall the ideal would be, on failure, providing a url that could be passed to docker pull that lets the developer pull down an image that didn't pass muster.

In T211881#4952874, @Jrbranaa wrote:

In any case, and with the risk of repeating myself, the service is under a code stewardship request cause it does not have currently a maintainer. If we want to keep it around, we need first and foremost a maintainer.

@akosiaris From what I gathered in this discussion, the course of action on this should be to find a Code Steward for graphoid.

In T196478#4919507, @Cmjohnson wrote:

@akosiaris Sorry for the really late response to this....the task got buried. No, I don't know why mgmt would not be working now unless it's disconnected or the cable is bad. I will check it next week after all hands.

In T213116#4943317, @Ladsgroup wrote:

Thanks for the blogpost @akosiaris I had a complete misunderstanding of what blue/green deployment means. What I wanted to suggest for now was to have half/one third of the nodes as canary, Basically having a rather big canary. Do you think it would work?

Change has been deployed across the fleet. WMCS IP space 172.16.0.0/12 should now be exempt of rate limiting rules. @Cyberpower678, @Kelson could you please confirm?

In T213475#4941129, @Cyberpower678 wrote:

Question, when will this patch go live?

Seems like we forgot to close this one

In T213566#4934835, @Ottomata wrote:

How is the data going to make it from Hadoop, which resides in the analytics cluster and is firewalled at the router level (aka network ACLs) to whichever machine is chosen for this? Has this been already worked out (cause I see no mention of this)?

@akosiaris: T213976

It hasn't been totally worked out, but a solution is rsync pull. The network firewalls don't allow analytics VLAN to initiate connections, but they will accept incoming ones.

How is the data going to make it from Hadoop, which resides in the analytics cluster and is firewalled at the router level (aka network ACLs) to whichever machine is chosen for this? Has this been already worked out (cause I see no mention of this)?

I 've removed the memory part cause https://grafana.wikimedia.org/d/000000377/host-overview?refresh=5m&orgId=1&var-server=mwdebug1001&var-datasource=eqiad%20prometheus%2Fops&var-cluster=appserver&from=now-7d&to=now shows that mwdebug1002 is never pressed for more memory. I 've also bumped vpu count to 4. I 'll resolve this for now, if we need more resources feel free to reopen.

In T213371#4932956, @pmiazga wrote:

@Tgr I assume you're still waiting for answers from @ema? Is there anything I can help you with?

I 've added the capacity to varnish puppet code to augment the wikimedia_trust and wikimedia_nets constructs, followed by a patch adding the new WMCS IP space to wikimedia_nets in order to exempt that IP space from rate limiting. @BBlack lemme know what you think.

Assuming we go for option (1), how would we go around and install these packages? And how would we instruct the app to load them?
It seems like using NODE_PATH is discouraged these days[1] and would anyway require changes to blubber to set NODE_PATH. We used to have that variable set and have moved away from it in https://gerrit.wikimedia.org/r/#/c/blubber/+/460997/

The restart of varnish-frontend on cp3030 indeed resolved the issue. I 'll lower priority but leave task open. Feel free to resolve however.

cp3030 seems to be in some trouble since approximately 04:30 [1]

In T213116#4929800, @Halfak wrote:

@akosiaris I'd like your input here. We're running into a lot of "After moving to kubernetes, none of this would matter" for some maintenance tasks. For a problem like this one, do you think it is worthwhile to invest time into our current deployment process or if it is reasonable to wait for k8s?

Graphs in codfw mail[1] and eqiad mail[2] point out that this behavior has not reemerged since Jan 25, so I 'll tentatively close this as resolved. Feel free to reopen

Cleaned up some 10k emails from 2 more host with the same pattern as yesterday and blocked them as well.

In T178690#4909073, @jcrespo wrote:

@akosiaris we had some chat about details, I don't mind the USE pattern, but a poor graph using USE doesn't mean it is good, if the chosen metrics are poor, like the above example. Note also they were probably in a worse state before my comments 0:-)

In T178690#4877021, @jcrespo wrote:

In T178690#4876994, @CDanis wrote:

Jaime, going to have to guess here; are you referring to "Prometheus machine stats" (marked for deletion) vs "Host overview"?

Yes.

info-en-c seems to be down to 167 messages now and the hosts participating in the storm remain blocked. I 'll lower priority for now.

The email storm can be witnessed at https://grafana.wikimedia.org/d/000000451/mail?orgId=1&from=1548346803405&to=1548357438520&var-datasource=codfw%20prometheus%2Fops (this is the secondary DC) and https://grafana.wikimedia.org/d/000000451/mail?orgId=1&from=1548346740714&to=1548358041101&var-datasource=eqiad%20prometheus%2Fops (for the primary DC). The 2 distinct phases are there because of me freezing a ton of emails which later got thawed and eventually delivered.

In T182914#4905675, @Halfak wrote:

Right. Only when we're approaching overload does the celery key contain entries. All workers need to be busy before we see anything.

Yes indeed. I 've switched the ownership of the list to the email provided and issued a new password for it. It should arrive with an email to the new admin email. Let me know if anything goes wrong. I am closing this as resolved. Thanks!

Change has been merged and deployed but up to now no data has been exported. The celery key currently looks empty so I guess this is expected?

Adding that a, from our esams DC, traceroute to this IP seems to stop before beelive.ru

We can probably get away with reusing https://github.com/oliver006/redis_exporter that we already use. It does have a check-keys parameter that allow us to count a lists elements. It's a bit slower as an implementation that in the next version as it uses scan but in my tests it did return within ~1s. I get in prometheus the following as an example

In T213475#4883423, @Kelson wrote:

I'm not sure to fully understand the technical explanation. Is the problem confirmed? If "yes", what is the plan to solve it?

In T119043#4888995, @Yurik wrote:

In T119043#4887415, @akosiaris wrote:

In T119043#4885517, @Yurik wrote:

Most of the time, Vega is used via a template, because otherwise you have a massive copy/paste of code without any benefit, while having no way to fix issues or improve appearance of all graphs en mass. Thus, per what @Anomie said - MCR is an orthogonal (in its current form) to the generated content. This actually has more similarities with the image thumb service than MCR (content is generated from "master" - wiki markup, and cached for usage by both the rendering service like Graphoid and directly from the client via the dynamic graph loading).

This does contradict however with requirement 6. BonusB: When user looks at an older revision of an article, they should see the graphs for that revision. given above. Just noting it, effectively reiterating what I think Tim has better phrased it in his comment at T119043#1868557

@akosiaris why is it a contradiction? The bonus B is similar to being able to see an older version of an article with every type of dependent resource, not just graph, i.e. older images, templates, Lua modules, and even data tables from Commons. On the other hand, the current (master) version should auto-refresh when dependencies are updated.

Upgrade done.

The Graph extension could potentially use an MCR slot to store the Vega JSON rather than embedding it in the wikitext inside a <graph> tag. But that wouldn't support the existing uses where templates and modules are being used to generate the Vega JSON.

In T211881#4882359, @MSantos wrote:

In T211881#4878426, @akosiaris wrote:

@akosiaris, the logic in <An unorthodox architecture of the API of the service> is fundamentally flawed. The client only knows about the graph's hash because MediaWiki parser knew the exact graph data, calculated a hash, and stored that data under that hash in a key-value store (page_props), and included that hash in the HTML. Also, this structure is identical to the way maps function -- map data is calculated by the parser, stored in the key-value page_props with the hash as the key, and kartotherian service does exactly the same steps as graphoid - pulls that data out of page_props to render a static image (so that Leaflet libs are not downloaded until the user interacts with the map). In short - the ONLY component that knows what user wants to draw is MediaWiki parser.

[...]
By the way, page_props.pp_value field is a blob and hence bound to 65k. That does not sound like enough space to store the representation. In fact from what I see at T184128 this has already happened?

Considering that what do you think about giving a little push on the following discussion? T119043: Graph/Graphoid/Kartographer - data storage architecture

In T119043#3887878, @brion wrote:

Is this still needed open? Currently Graphs and Maps development is on hold and we may rethink some of how this is done later, but it's not on the immediate agenda.

Adding performance-team and core platform team per SoS recommendation to request for help.