Page MenuHomePhabricator

WDQS graph split hosts: Remove throttling/banning mechanisms and investigate external connectivity
Closed, ResolvedPublic

Description

Per IRC conversation with @dcausse , we have a couple of issues with the graph split hosts:

  • Querying the test hosts is resulting in bans: "You have been banned until 2024-01-09T19:15:34.264Z" . These hosts (all members of the 'test' tier) are not publicly accessible, so throttling/banning mechanisms can be removed.
  • Access to WDQS (from stat machines?) works via only cleartext port. Investigate why we can't access the test hosts' TLS ports.

Event Timeline

Change 989186 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: disable query throttling on test servers

https://gerrit.wikimedia.org/r/989186

Change 989186 merged by Bking:

[operations/puppet@production] wdqs: disable query throttling on test servers

https://gerrit.wikimedia.org/r/989186

bking renamed this task from WDQS graph split hosts: Remove throttling/banning mechanisms and improve hadoop access to WDQS graph split hosts: Remove throttling/banning mechanisms and investigate external connectivity.Jan 9 2024, 5:09 PM
bking changed the task status from Open to In Progress.EditedJan 9 2024, 5:13 PM
bking added a subscriber: BTullis.

Per pairing session with @BTullis , the wdqs test hosts are not running envoy, so only the cleartext ports are available. If TLS is required, we can enable it in puppet by inculding the profile::tlsproxy::envoy profile, as we do in modules/role/manifests/wdqs/public.pp.

dcausse let us know if you think we should enable TLS. If not, feel free to close out this ticket.

Update: We're going to take a stab at this for a couple of reasons:

Change 989236 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: Enable TLS for test tier

https://gerrit.wikimedia.org/r/989236

Change 989244 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs-test: Enable PKI

https://gerrit.wikimedia.org/r/989244

Change 989236 abandoned by Bking:

[operations/puppet@production] wdqs: Enable TLS for test tier

Reason:

superseded by 989244

https://gerrit.wikimedia.org/r/989236

Change 989244 merged by Bking:

[operations/puppet@production] wdqs-test: Enable PKI

https://gerrit.wikimedia.org/r/989244

bking claimed this task.

Merging/applying the above patch added TLS to the test hosts, using the domain "wdqs-test.eqiad.wmnet". You can also access individual hosts by their FQDN (such as https://wdqs1022.eqiad.wmnet). Closing...