Page MenuHomePhabricator

New Service Request miscweb
Closed, ResolvedPublic

Description

Description: Miscellaneous (static) websites currently hosted on miscweb ganeti VMs
Timeline: _One_ of the hosted sites, static-bugzilla, until end of quarter
Diagram:

miscweb-network-diagram.png (597×448 px, 15 KB)

Technologies: apache, envoy, HTML
Point person: @Dzahn

port: 4111

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -125
operations/deployment-chartsmaster+2 -2
operations/container/miscwebmaster+0 -0
operations/puppetproduction+1 -1
operations/puppetproduction+2 -0
operations/puppetproduction+1 -1
operations/dnsmaster+2 -0
operations/puppetproduction+1 -1
operations/deployment-chartsmaster+3 -3
operations/container/miscwebmaster+1 -1
operations/puppetproduction+1 -1
operations/deployment-chartsmaster+7 -2
operations/deployment-chartsmaster+7 -2
operations/deployment-chartsmaster+3 -0
operations/deployment-chartsmaster+0 -3
operations/deployment-chartsmaster+3 -0
operations/puppetproduction+1 -1
operations/puppetproduction+39 -1
operations/container/miscwebmaster+1 K -0
operations/container/miscwebmaster+0 -37
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+0 -0
operations/container/miscwebmaster+10 M -0
operations/container/miscwebmaster+120 K -0
operations/container/miscwebmaster+3 -0
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+6 -0
operations/deployment-chartsmaster+54 -0
operations/deployment-chartsmaster+7 -104
operations/deployment-chartsmaster+689 -0
operations/container/miscwebmaster+1 -0
operations/container/miscwebmaster+24 -0
operations/container/miscwebmaster+72 -21
operations/puppetproduction+7 -0
operations/container/miscwebmaster+51 -57
operations/container/miscwebmaster+181 -0
operations/container/miscwebmaster+3 -4
operations/container/miscwebmaster+1 -0
operations/container/miscwebmaster+6 -6
operations/container/miscwebmaster+1 -1
operations/container/miscwebmaster+0 -12
operations/container/miscwebmaster+4 -0
operations/container/miscwebmaster+33 -168
operations/container/miscwebmaster+46 -36
operations/container/miscwebmaster+38 -33
operations/container/miscwebmaster+305 -1
operations/container/miscwebmaster+1 -1
operations/dnsmaster+4 -0
integration/configmaster+8 -0
integration/configmaster+9 -0
operations/container/miscwebmaster+15 -0
operations/puppetproduction+12 -1
operations/puppetproduction+2 -0
labs/privatemaster+4 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
ResolvedDzahn
ResolvedDzahn

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 715237 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: bump production version to 2021-08-27-115701-production

https://gerrit.wikimedia.org/r/715237

Change 715727 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] load mod_alias to be able to use Redirect Directive

https://gerrit.wikimedia.org/r/715727

Change 715727 merged by jenkins-bot:

[operations/container/miscweb@master] load mod_alias to be able to use Redirect Directive

https://gerrit.wikimedia.org/r/715727

Change 715237 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: bump production version to 2021-09-01-143556-production

https://gerrit.wikimedia.org/r/715237

Change 717347 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] add HTML of the first 10000 bugs

https://gerrit.wikimedia.org/r/717347

Change 717347 merged by jenkins-bot:

[operations/container/miscweb@master] add HTML of the first 10000 bugs

https://gerrit.wikimedia.org/r/717347

Change 728668 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: compress bug HTML with gzip and add 10k more bugs

https://gerrit.wikimedia.org/r/728668

Change 728668 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: compress bug HTML with gzip and add 10k more bugs

https://gerrit.wikimedia.org/r/728668

Change 730275 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 20000 - 30000

https://gerrit.wikimedia.org/r/730275

Change 730281 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 30000 - 40000

https://gerrit.wikimedia.org/r/730281

Change 730285 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 40000 - 50000

https://gerrit.wikimedia.org/r/730285

Change 730287 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 50000 - 60000

https://gerrit.wikimedia.org/r/730287

Change 730275 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 20000 - 30000

https://gerrit.wikimedia.org/r/730275

Change 730288 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 60000 - 70000

https://gerrit.wikimedia.org/r/730288

Change 730292 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 60001 - 73681

https://gerrit.wikimedia.org/r/730292

Change 730314 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 1 - 20000

https://gerrit.wikimedia.org/r/730314

Change 730281 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 30000 - 40000

https://gerrit.wikimedia.org/r/730281

Change 730285 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 40000 - 50000

https://gerrit.wikimedia.org/r/730285

Change 730316 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 20001 - 40000

https://gerrit.wikimedia.org/r/730316

Change 730287 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 50000 - 60000

https://gerrit.wikimedia.org/r/730287

Change 730330 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 40001 - 60000

https://gerrit.wikimedia.org/r/730330

Change 730288 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 60000 - 70000

https://gerrit.wikimedia.org/r/730288

Change 730334 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 60001 - 73681

https://gerrit.wikimedia.org/r/730334

Change 730336 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add all_bugs, all_activies pages, gzip index

https://gerrit.wikimedia.org/r/730336

Change 730292 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed bug content, 60001 - 73681

https://gerrit.wikimedia.org/r/730292

Change 730339 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] static-bugzilla: add skins directory

https://gerrit.wikimedia.org/r/730339

Change 730314 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 1 - 20000

https://gerrit.wikimedia.org/r/730314

Change 730316 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 20001 - 40000

https://gerrit.wikimedia.org/r/730316

Change 730330 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 40001 - 60000

https://gerrit.wikimedia.org/r/730330

Change 730334 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add compressed activity content, 60001 - 73681

https://gerrit.wikimedia.org/r/730334

Change 730336 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add all_bugs, all_activies pages, gzip index

https://gerrit.wikimedia.org/r/730336

Change 730339 merged by jenkins-bot:

[operations/container/miscweb@master] static-bugzilla: add skins directory

https://gerrit.wikimedia.org/r/730339

Change 694625 merged by Giuseppe Lavagetto:

[operations/puppet@production] add miscweb to LVS

https://gerrit.wikimedia.org/r/694625

Change 694628 merged by Giuseppe Lavagetto:

[operations/puppet@production] service/miscweb: switch state from service_setup to lvs_setup

https://gerrit.wikimedia.org/r/694628

Change 739675 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: enable TLS, fix public port in defaults

https://gerrit.wikimedia.org/r/739675

Change 739675 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: enable TLS

https://gerrit.wikimedia.org/r/739675

Change 739810 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: remove nodePort and re-enable TLS

https://gerrit.wikimedia.org/r/739810

Change 739810 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: remove nodePort and re-enable TLS

https://gerrit.wikimedia.org/r/739810

Change 739848 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: before enabling TLS, first remove nodePort line separately

https://gerrit.wikimedia.org/r/739848

Change 739848 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: before enabling TLS, first remove nodePort line separately

https://gerrit.wikimedia.org/r/739848

Change 739945 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: try enabling TLS after nodePort is removed and we deployed

https://gerrit.wikimedia.org/r/739945

Change 739945 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: try enabling TLS after nodePort is removed and we deployed

https://gerrit.wikimedia.org/r/739945

Change 742819 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: try again to enable TLS, remove nodePort

https://gerrit.wikimedia.org/r/742819

Change 742819 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: try again to enable TLS, remove nodePort

https://gerrit.wikimedia.org/r/742819

Change 694629 merged by Dzahn:

[operations/puppet@production] service/miscweb: switch state from lvs_setup to monitoring_setup

https://gerrit.wikimedia.org/r/694629

I could debug the gzip encoding issue and at the same time test using my production image in cloud VPS in k8splay project.

dzahn@dzahn:~$ curl --compressed dzahn.k8splay.eqiad1.wikimedia.cloud:8080/bug10001.html

works and gets the 10001st bug that only exists as bug10001.html.gz on disk in the container.

also, for testing, something like:

dzahn@dzahn:~$ docker run -d -p 8080:8080 docker-registry.wikimedia.org/wikimedia/operations-container-miscweb:production -t miscweb

works to run it in cloud VPS and forward the port

after also passing it through the cloud VPS proxy and adding a security rule for it, both in Horizon:

https://staticbz.wmcloud.org/bug10001.html

so close.. but not quite

Change 752235 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] fix content type for HTML, it's not CSS

https://gerrit.wikimedia.org/r/752235

Change 752235 merged by jenkins-bot:

[operations/container/miscweb@master] fix content type for HTML, it's not CSS

https://gerrit.wikimedia.org/r/752235

Change 752750 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: bump staging and prod version to 2022-01-10-220730-production

https://gerrit.wikimedia.org/r/752750

Change 752750 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: bump staging and prod version to 2022-01-10-220730-production

https://gerrit.wikimedia.org/r/752750

Since the latest deploy now content can be fetched from staging, codfw and eqiad, it is all gzipped inside the image, which should be much smaller now and content is now properly delivered as content-type text/html.

A client accepting/requesting gzip encoded gets proper content. This can be tested with curl --compress.

Also we are using the service names from LVS setup.

from deploy1002:

curl --compressed https://staging.svc.eqiad.wmnet:4111/bug10023.html
curl --compressed https://miscweb.svc.codfw.wmnet:4111/bug10023.html -I
curl --compressed https://miscweb.svc.eqiad.wmnet:4111/bug10023.html -I
..

content-encoding: gzip
content-type: text/html

Change 753813 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] trafficserver: switch static-bugzilla from ganeti-miscweb to k8s-miscweb

https://gerrit.wikimedia.org/r/753813

Change 694630 merged by Dzahn:

[operations/puppet@production] service/miscweb: switch state from monitoring_setup to production

https://gerrit.wikimedia.org/r/694630

after switching service state to production and running puppet on authdns1001 and alert1001 the new monitoring appeared in Icinga

could confirm manually too and scheduled downtimes to make sure really nobody gets paged

the new file /var/lib/gdnsd/discovery-miscweb.state has been created.

authdns1001:/etc/nagios/nrpe.d] $ /usr/local/lib/nagios/plugins/check_confd_template '/var/lib/gdnsd/discovery-miscweb.state'
No errors detected

running puppet on all dns and icinga servers.. pending checks coming in

all green on DNS servers:

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=miscweb.state

Change 693968 merged by Dzahn:

[operations/dns@master] Add discovery DNS for miscweb

https://gerrit.wikimedia.org/r/693968

Discovery DNS added! :)

Now this works, too:

[deploy1002:~] $ curl --compressed https://miscweb.discovery.wmnet:4111/bug10023.html -I

Change 753813 merged by Dzahn:

[operations/puppet@production] trafficserver: switch static-bugzilla from ganeti-miscweb to k8s-miscweb

https://gerrit.wikimedia.org/r/753813

Change 753846 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] add miscweb to disc_desired_state.py

https://gerrit.wikimedia.org/r/753846

Change 753846 merged by Dzahn:

[operations/puppet@production] add miscweb to disc_desired_state.py

https://gerrit.wikimedia.org/r/753846

Mentioned in SAL (#wikimedia-operations) [2022-01-19T23:35:59Z] <mutante> deploy1002 - checked freshly generated cert in /etc/helmfile-defaults/private/main_services/miscweb/eqiad.yaml with 'openssl x509 -noout -text -in .. | grep DNS'. now has static-bz on it. (T281538)

23:35 mutante: puppetmaster1001 - revoked puppet cert miscweb.discovery.wmnet; updated kube_services.crts.yaml to include static-bugzilla.wikimedia.org, removed miscweb.discovery.wmnet.crt and .csr.pem, used cergen to check and regenerate cert, committed in private repo, ran puppet on deploy1001

Change 755681 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] Revert \"Revert \"trafficserver: switch static-bugzilla from ganeti-miscweb to k8s-miscweb\"\"

https://gerrit.wikimedia.org/r/755681

Change 755681 merged by Dzahn:

[operations/puppet@production] Revert \"Revert \"trafficserver: switch static-bugzilla from ganeti-miscweb to k8s-miscweb\"\"

https://gerrit.wikimedia.org/r/755681

Change 755748 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/container/miscweb@master] add a foot note to the index.html that this is now a Kubernetes service

https://gerrit.wikimedia.org/r/755748

Change 755748 merged by jenkins-bot:

[operations/container/miscweb@master] add a foot note to the index.html that this is now a Kubernetes service

https://gerrit.wikimedia.org/r/755748

Change 755751 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: bump version to 2022-01-20-183807-production

https://gerrit.wikimedia.org/r/755751

Change 755751 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: bump version to 2022-01-20-183807-production

https://gerrit.wikimedia.org/r/755751

This is resolved! :)

Proof is the footnote in https://static-bugzilla.wikimedia.org/ that is only shown when served from k8s.

Screenshot from 2022-01-20 11-33-21.png (584×1 px, 82 KB)

Change 755761 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] delete bugzilla_static after it moved from puppet to k8s

https://gerrit.wikimedia.org/r/755761

Change 755761 merged by Dzahn:

[operations/puppet@production] delete bugzilla_static after it moved from puppet to k8s

https://gerrit.wikimedia.org/r/755761

deployed @JMeybohm's change https://gerrit.wikimedia.org/r/c/operations/puppet/+/765572

Now this is behind the new istio ingress.

I can see fresh traffic here:

Screenshot from 2022-02-24 13-59-38.png (442×1 px, 37 KB)

and logs here:

Screenshot from 2022-02-24 13-59-48.png (898×1 px, 287 KB)

after searching for by the "authority" field. authority=static-bugzilla.wikimedia.org:30443

I got the links to the dashboards from https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress