In T174596#4681328, @ayounsi wrote:

This is also the reason we have to have the following route on cr1/2-eqiad static route 172.16.0.0/21 next-hop 10.64.22.4 so when hosts in the 172.16/21 subnet reach hosts outside (without NAT), return traffic knows what path to take.

I think there are several action items here, the end goal being OpenStack to be fully separated from the remaining of the infrastructure.
1/ Standardize the list of target subnets to not use NAT as mentioned in T174596#4530414 (use 1 list instead of 4, the most complete one)

For the record, I don't think we should move openstack components inside openstack just to avoid complex chicken-egg problems.

We have dedicated hardware for this, in the case of labmon1002, a fairly new server which was put into works some months ago.

According to the timeline described here https://wikitech.wikimedia.org/wiki/News/Trusty_deprecation#Cloud_VPS_projects we should be sending pings to the deprecation subtasks starting today.

Reopening and reverting patch. I can confirm is causing at least 2 issues:

Does this still affect us? If so, which concrete subnets are affected?

hey @herron it should be working now. This was my test:

I'm checking with @ayounsi trying to debug what's happening here.

For the record, I refreshed the docs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployments

In T207327#4676888, @ayounsi wrote:

1. I renamed the switch ports descriptions from labvirt1018 to cloudvirt1018
2. The switch ports configurations are similar to cloudvirt1024 (eth0 access for cloud-hosts1-b-eqiad, eth1 trunk with cloud-instances1-b-eqiad/cloud-instances2-b-eqiad)

Agreed.

• openstack mitaka is in jessie-backports (already using it in eqiad1)
• openstack newton is in stretch (mitaka+1)
• openstack ocata is in stretch-backports (newton+1)

I just created the project, so you can start doing test and some work. However, I didn't do anything special regarding the disk space, hopefully @Andrew can say better if we are ready for the disk space allocation or we have to wait a bit more.
Keeping the task open for now.

From https://blogs.gnome.org/dcbw/2018/07/27/the-ascendance-of-nftables/ :

This still doesn't work. Same behavior as before: packets go out of the interface but nothing enters, which may indicate a miscommunication somewhere in the switching stack.

Idea 1: have a checklist in phabricator tasks when developing new features/fixing things that points us to update the relevant docs.
This way we don't consider a task as completed until docs have been updated/reviewed.

Ok, it seems I'm wrong, will have to review my own docs lol

Something is wrong in that page:

Wait, the NAT is only applied to connections in the egress path (from VM to the internet), and connections internal to CloudVPS are not affected by this egress NAT, i.e, VMs in the 172.16.0.0/12 range can contact 185.15.56.18/31 without source NAT being applied.

We have a mechanisms called dmz_cidr which we can use to exclude NATs between certain IP ranges.
See a more detailed explanation here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron#dmz_cidr

Could you please also patch profile::prometheus::openstack_exporter?

I created a new grafaa dashboard and we have now 3 dashboards to monitor eqiad1:

The obvious step seems to be to update all the machine firmware, as suggested by @RobH, which will take the server down for a couple of hours. Updating firmware is also what HP usually ask first when opening a support ticket with them.

Thanks for filling the task, we are evaluating this.

After T205986: cloudnet1004: spontaneous reboot today, I see issues with agents again all over the place, including python tracebacks in the logs :-(

In T205524#4632558, @GTirloni wrote:

I can't find the error messages in the logs anymore but it's very likely I'm not looking right. @aborrero does the situation look better now?

Status of this: code has been deployed (after a lot of back and forth with puppet) and we get the Forbidden: Policy doesn't allow os_compute_api:os-hypervisors to be performed. (HTTP 403) error.
Also, not sure if we need something else to instruct prometheus to read from this exporter.

With the current status of package and puppet patch, I see this error message from the exporter script:

The other day I created this page in wikitech:

In T199003#4623257, @Harej wrote:

I will defer to you on the email since you're probably more knowledgeable in the particulars of the transition.

This machine migration had a disk failure when copying which apparently solved by itself:

In T199003#4595029, @Harej wrote:

See also https://wikitech.wikimedia.org/wiki/News/Trusty_deprecation

Feel free to reopen if more issues or questions appears.

I think this is solved. Feel free to reopen if more issues or questions arise.

Current state of this:

For a working agent<->server communication I can see messages flowing in both directions:

All the restarts do nothing.

I'm tracking neutron agent/server issues in T205524

Issue seems to be that neutron is having a hard time with agents:

The region-migrate script failed:

Next can be: mwv-apt, wikimania-support

Since this is my clinic-duty week, I took the freedom of creating this project.

In T205159#4607290, @herron wrote:

Thanks for adding quota! Question -- What is the difference between pools wan-transport-eqiad and compat-net? Which should be used?

That's all by now. I don't want to upgrade docker/linux/grub related packages by now.

hey @RobH @Cmjohnson what is the status of this? Could we move forward with OS installation?

I still don't know why this failed.

For the record @Andrew created this dashboard: https://grafana.wikimedia.org/dashboard/db/cloud-capacity-planning?orgId=1

For the record, this is what I did:

Project has been created! It should be in the eqiad1-r region in the eqiad1 deployment. Mind the horizon option in the top-left drop-down menu.

@GoranSMilovanovic, find more info here:

In T204003#4583866, @aborrero wrote:

Please refresh https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployments with these changes or let me know and I'll do it :-)

For the record, this is what I did:

Done :-)

In T204698#4599707, @Jdlrobson wrote:

@aborrero if we don't hear from Sumit, i suggest we remove this instance, however I'm not sure what the protocol and grace period is for doing so.