clean up commons-query on miscweb setup
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dzahn
	Mar 30 2023, 12:17 AM

Description

https://commons-query.wikimedia.org somehow redirects to the appserver cluster for a rewrite rule but ALSO exists as a virtual host on miscweb machines.

But it has not been added to the TLS certs used by envoy on miscweb* machines. So apparently something speaks to it unencrypted on port 80!

So you have a virtual host but if you ask the machines for it over TLS you get a certificate error that this is not on the cert.

Since we put effort into making everything else use TLS from the caching servers and terminate with envoy and because this causes monitoring alerts or workarounds.. we should get this cleaned up.

Also we should probably drop access to port 80 from external completely to prevent this from happening with new sites and so that we can rely on TLS being used for everything.

Needs coordination with maintainers of the service of course.

Details

Subject	Repo	Branch	Lines +/-
query_service: force TLS for monitoring for search-platform team	operations/puppet	production	+1 -2
ssl: update certificate for webserver-misc-apps.discovery.wmnet	operations/puppet	production	+36 -36
microsites/query_service: enable TLS when monitoring commons-query	operations/puppet	production	+1 -2
microsites: do not expect 301 for commons-query.wikimedia.org	operations/puppet	production	+6 -8
microsites: commons-query.wm.org only works on port 80/http	operations/puppet	production	+1 -0
microsites: do not use TLS when monitoring commons-query.wikimedia.org	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dzahn	T333510 clean up commons-query on miscweb setup
Resolved	Dzahn	T333507 ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb)
Resolved	Dzahn	T333515 ProbeDown - commons_query_wikimedia_org
Resolved	Dzahn	T333517 ProbeDown - commons_query_wikimedia_org
Resolved	Dzahn	T352941 ProbeDown (commons_query)

Event Timeline

Dzahn created this task.Mar 30 2023, 12:17 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2023, 12:17 AM

Screenshot from 2023-03-29 17-19-24.png (468×1 px, 111 KB)

host commons-query.wikimedia.org
commons-query.wikimedia.org is an alias for dyna.wikimedia.org.
dyna.wikimedia.org has address 208.80.154.224
dyna.wikimedia.org has IPv6 address 2620:0:861:ed1a::1

root@miscweb1002:/etc/apache2/sites-enabled# grep ServerName 50-commons-query-wikimedia-org.conf 
    ServerName commons-query.wikimedia.org

Dzahn updated the task description. (Show Details)Mar 30 2023, 12:21 AM

Change 904377 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites: do not use TLS when monitoring commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904377

gerritbot added a project: Patch-For-Review.Mar 30 2023, 12:21 AM

Dzahn mentioned this in T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).Mar 30 2023, 12:24 AM

Change 904377 merged by Dzahn:

[operations/puppet@production] microsites: do not use TLS when monitoring commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904377

Change 904382 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites: commons-query.wm.org only works on port 80/http

https://gerrit.wikimedia.org/r/904382

Change 904382 merged by Dzahn:

[operations/puppet@production] microsites: commons-query.wm.org only works on port 80/http

https://gerrit.wikimedia.org/r/904382

Change 904384 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites: do not expect 301 for commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904384

Dzahn mentioned this in T333515: ProbeDown - commons_query_wikimedia_org.Mar 30 2023, 1:12 AM

Change 904384 merged by Dzahn:

[operations/puppet@production] microsites: do not expect 301 for commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904384

Dzahn added a subtask: T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).Mar 30 2023, 1:23 AM

Dzahn closed subtask T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb) as Resolved.

Dzahn added a subtask: T333515: ProbeDown - commons_query_wikimedia_org.

Dzahn added a subtask: T333517: ProbeDown - commons_query_wikimedia_org.Mar 30 2023, 1:29 AM

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2023, 1:29 AM

Dzahn closed subtask T333517: ProbeDown - commons_query_wikimedia_org as Resolved.Mar 30 2023, 1:30 AM

LSobanski triaged this task as Low priority.Apr 3 2023, 3:52 PM

LSobanski moved this task from Incoming to Backlog on the collaboration-services board.

LSobanski awarded a token.Oct 16 2023, 3:56 PM

Dzahn claimed this task.Nov 6 2023, 7:02 PM

bking subscribed.Dec 6 2023, 9:32 PM

Change 980950 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites/query_service: enable TLS when monitoring commons-query

https://gerrit.wikimedia.org/r/980950

gerritbot added a project: Patch-For-Review.Dec 6 2023, 9:56 PM

Change 980950 merged by Dzahn:

[operations/puppet@production] microsites/query_service: enable TLS when monitoring commons-query

https://gerrit.wikimedia.org/r/980950

Maintenance_bot removed a project: Patch-For-Review.Dec 6 2023, 10:10 PM

Jelto mentioned this in T352941: ProbeDown (commons_query).Dec 7 2023, 8:20 AM

Jelto added a subtask: T352941: ProbeDown (commons_query).

Jelto moved this task from Backlog to Work in Progress on the collaboration-services board.Dec 7 2023, 10:19 AM

Dzahn closed subtask T352941: ProbeDown (commons_query) as Resolved.Dec 7 2023, 9:05 PM

Change 983470 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ssl: update certificate for webserver-misc-apps.discovery.wmnet

https://gerrit.wikimedia.org/r/983470

gerritbot added a project: Patch-For-Review.Dec 15 2023, 6:40 PM

Change 983470 merged by Dzahn:

[operations/puppet@production] ssl: update certificate for webserver-misc-apps.discovery.wmnet

https://gerrit.wikimedia.org/r/983470

Maintenance_bot removed a project: Patch-For-Review.Dec 15 2023, 7:10 PM

revert:revert. use TLS for monitoring: https://gerrit.wikimedia.org/r/c/operations/puppet/+/983303

Screenshot from 2023-12-15 14-15-43.png (690×1 px, 72 KB)

^ screenshot shows how service name commons-query.wikimedia.org gets 200 on https

bking awarded a token.Dec 15 2023, 10:18 PM

Change 983491 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] query_service: force TLS for monitoring for search-platform team

https://gerrit.wikimedia.org/r/983491

gerritbot added a project: Patch-For-Review.Dec 15 2023, 10:19 PM

re: original ticket description:

fixed the TLS cert and added the missing domain to SANs
switched monitoring for our team to force TLS, like the other checks
confirmed it actually works
the firewall part is already the case, only localhost (from envoy) and deployment servers (for httpbb) are allowed port 80, it was only working because monitoring services are allowed everything

resolved

Change 983491 merged by Dzahn: