Recording this for posterity, so others can review/discuss/fix as appropriate tomorrow or at a later time!

First there was the creation of two new public IPs in the high-traffic2 space in eqiad. This was done via Netbox and worked flawlessly :)

https://netbox.wikimedia.org/ipam/ip-addresses/7642/
https://netbox.wikimedia.org/ipam/ip-addresses/7643/

After a fair amount of debugging and thinking and compiling, the main patch to configure the LVS service ended up being:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/655533/16

Basic Followups:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/659409/2 (turn on production lvs config)
https://gerrit.wikimedia.org/r/c/operations/puppet/+/659414 (allow $PRODUCTION_NETWORKS access to wikireplicas via ferm, for at least LVS healthchecks to work, but seems sane in general)

At this point things seemed like they should Just Work, but connections through the new LVS-based public IPs weren't working correctly from <anywhere reasonable>. After a bunch of debugging, eventually we figured out that I had completely missed the fact that dbproxy1018 and dbproxy1019, the service hosts that LVS was intended to route the traffic to, were not in the normal production vlans, and thus LVS didn't have a direct route to them, and was sending the traffic back to the eqiad juniper routers. The traffic would then loop between Juniper and LVS until the TTL expired.

These hosts are in cloud-support1-c-eqiad. So we decided the most-expedient thing to do for the evening was just to add the bare minimum LVS interface configurations to make just this service work for its primary and backup LVS hosts (lvs1014 and lvs1016). This isn't a general-case solution yet, but i didn't want to take more risks by affecting more LVSes and/or subnets. The result is basically this patch to add just this one extra vlan, to just those two LVS boxes:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/659439

I would have assigned these IPs in netbox before merging this patch, but netbox's interface-name seemed to force me to use only the existing interfaces from the dropdown menu, so I couldn't really add it there first since the (virtual, tagged) interfaces didn't exist yet in puppetdb. So I just skipped netbox on those two IPs for now, more on that later...

The switch side also had to be manually fixed up to add the vlan to the available set on the row C interface for these two hosts manually:

bblack@asw2-c-eqiad# set interfaces xe-4/0/5 unit 0 family ethernet-switching vlan members cloud-support1-c-eqiad 
bblack@asw2-c-eqiad# set interfaces xe-2/0/13 unit 0 family ethernet-switching vlan members cloud-support1-c-eqiad

I realize this should have come from homer, but when I dug into homer's control of this stuff, it turns out it pulls this from netbox, which we skipped earlier, so on with the show...

WIth the interfaces now functioning, I ran the puppet agents on lvs101[46] to ensure puppetdb got its updates, and then was able to ask netbox to update from puppetdb for these two hosts, which did correctly create correct tagged-vlan address objects for them AFAICS, at:

https://netbox.wikimedia.org/ipam/ip-addresses/7650/
https://netbox.wikimedia.org/ipam/ip-addresses/7654/

And I'm assuming at tihs point that this would now close the loop and eventually homer would report no switch config diffs from my manual change, but I haven't checked on that yet!

We'll have to revisit all of this later/next-week and decide in the short- and medium- terms how we'll handle this sort of case and whether these hacks need to be re-done in a different way, or expanded on (more vlans and lvs hosts) to avoid future problems, etc.