Page MenuHomePhabricator
Create Task
Maniphest T285355

Set up an-web1001 and decommission thorium
Closed, ResolvedPublic
Actions

Description

an-web1001 is ready, see also {T264344}.

Details

ProjectBranchLines +/-Subject
operations/deployment-chartsmaster+2 -14Remove thorium from linkrecommendation egress rules
operations/puppetproduction+0 -51statistics: remove leftovers for stats.wikimedia.org
operations/puppetproduction+5 -20statistics: decom old httpd Directory for stats.wikimedia.org
operations/puppetproduction+3 -9profile::analytics::httpd: disable CGI
operations/dnsmaster+1 -1Point analytics-web CNAME at an-web1001
operations/puppetproduction+3 -3Point trafficserver at analytics-web cname instead of thorium hostname
operations/puppetproduction+1 -1statistics::rsync::published - push to analytics-web.discovery.wmnet cname
operations/deployment-chartsmaster+13 -1linkrecommendation: use CNAME for analytics-web
operations/puppetproduction+11 -0analytics-web: temporary rsync module for thorium
operations/puppetproduction+1 -1analytics: commission new webserver
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Jun 22 2021, 8:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 22 2021, 8:41 PM
Ottomata updated the task description. (Show Details)Jun 22 2021, 8:42 PM
Ottomata mentioned this in T255145: Analytics Hardware for Fiscal Year 2020/2021.
Ottomata assigned this task to BTullis.Jul 26 2021, 3:37 PM
Ottomata triaged this task as Medium priority.
Ottomata moved this task from Backlog to Q1 2021/2022 on the Analytics-Clusters board.
Ottomata mentioned this in T234629: Move the Analytics infrastructure to Debian Buster.Jul 29 2021, 7:31 PM
Ottomata reassigned this task from BTullis to RKemper.Aug 2 2021, 3:46 PM
Ottomata added a project: Analytics-Kanban.
Ottomata added a subscriber: BTullis.
BTullis added a comment.Aug 2 2021, 3:48 PM

I wonder if this needs to be changed.
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/statistics/manifests/rsync/published.pp#13

$destination = 'thorium.eqiad.wmnet::published-destination', # TODO: hiera-ize thorium.eqiad.wmnet
Ottomata added a comment.Aug 2 2021, 4:08 PM

Def ^

gerritbot added a comment.Aug 2 2021, 7:42 PM

Change 709530 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] analytics: commission new webserver

https://gerrit.wikimedia.org/r/709530

gerritbot added a project: Patch-For-Review.Aug 2 2021, 7:42 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Aug 3 2021, 4:13 PM
gerritbot added a comment.Aug 3 2021, 7:25 PM

Change 709530 merged by Ryan Kemper:

[operations/puppet@production] analytics: commission new webserver

https://gerrit.wikimedia.org/r/709530

Stashbot added a comment.Aug 3 2021, 7:31 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-03T19:31:21Z] <ryankemper> T285355 ryankemper@an-web1001:~$ sudo run-puppet-agent to establish role(analytics_cluster::webserver) on the host in preparation for upcoming cutover from thorium->an-web1001

RKemper added a comment.Aug 3 2021, 7:34 PM

Seeing this error upon running puppet on an-web1001:

Notice: /Stage[main]/Statistics::Sites::Stats/File[/etc/apache2/htpasswd.stats]/ensure: defined content as '{md5}763c9b521fb88366d3bf0f99da3406a1'
Notice: /Stage[main]/Statistics::Sites::Stats/File[/etc/apache2/htpasswd.stats-geowiki]/ensure: defined content as '{md5}0d7d507562b2505e351d5cb177c01663'
Notice: /Stage[main]/Statistics::Sites::Stats/File[/srv/src]/ensure: created
Notice: /Stage[main]/Statistics::Sites::Stats/File[/srv/src/wikistats-v2]/ensure: created
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 60)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 60)
Wrapped exception:
No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs
Error: /Stage[main]/Statistics::Sites::Stats/File[/srv/stats.wikimedia.org/htdocs/v2]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 60)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 69)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 69)
Wrapped exception:
No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs
Error: /Stage[main]/Statistics::Sites::Stats/File[/srv/stats.wikimedia.org/htdocs/index.html]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 69)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 74)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 74)
Wrapped exception:
No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs
Error: /Stage[main]/Statistics::Sites::Stats/File[/srv/stats.wikimedia.org/htdocs/assets-v2]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 74)

Investigating...

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2021, 8:10 PM
RKemper added a comment.Aug 3 2021, 8:31 PM

After applying https://gerrit.wikimedia.org/r/c/operations/puppet/+/709817:

ryankemper@an-web1001:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for an-web1001.eqiad.wmnet
Info: Applying configuration version '(f654a1b556) Ryan Kemper - analytics web: create wikistats webdir if absent'
Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:
--- /etc/ssh/ssh_known_hosts    2021-08-03 20:16:03.777519388 +0000
+++ /tmp/puppet-file20210803-16918-yan6dm       2021-08-03 20:29:55.151226774 +0000
@@ -714,6 +714,7 @@
 furud.codfw.wmnet,furud,10.192.16.65,2620:0:860:102:10:192:16:65 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHsxtgYVN6ckgW52t9U2wmMnDZhrIA4qhFyT8Ffz3LPq1WIOx1QDEv8X5LkNKVIXAx9tTMMvFuf1dfpwZvg7J9Q=
 ganeti-test2001.codfw.wmnet,ganeti-test2001,10.192.0.73,2620:0:860:101:10:192:0:73 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGUPK0yz/yzxTVDY7mnzd87wg9xWuaglPdPh2kP0WTmh1zJMclXORS7g67k7IZi9mOOZ7xvYlu81A1rhbCbI8M4=
 ganeti-test2002.codfw.wmnet,ganeti-test2002,10.192.0.74,2620:0:860:101:10:192:0:74 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG26BcF03JVXwG70VTNRXziOJ6P9gIhy6KcCV5iUin2qNZtbVTqPrMy4RYpWkkVeop4Ge0vv0C5SGg8jHSMZqg0=
+ganeti-test2003.codfw.wmnet,ganeti-test2003,10.192.0.93,2620:0:860:101:10:192:0:93 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANVFoqYiKKcu8P4gS9diVjAWKGZ3E5Q9VvuhaNBqfBbrN3wXuZhEBXUffGMyNoPaUEaBthNpos27lCoUiafqLs=
 ganeti1005.eqiad.wmnet,ganeti1005,10.64.0.245,2620:0:861:101:1a66:daff:fea3:af25 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ67MWaWh9EPfyXkydN22oUyur6XLcsrLOIDHnoxxzV6lnMoWc6u0Jy0Y+g5EFit99z05kiMgegRG+5B7DxWf2Y=
 ganeti1006.eqiad.wmnet,ganeti1006,10.64.0.246,2620:0:861:101:1a66:daff:fea3:455d ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ67MWaWh9EPfyXkydN22oUyur6XLcsrLOIDHnoxxzV6lnMoWc6u0Jy0Y+g5EFit99z05kiMgegRG+5B7DxWf2Y=
 ganeti1007.eqiad.wmnet,ganeti1007,10.64.0.248,2620:0:861:101:1a66:daff:fea3:ac69 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ67MWaWh9EPfyXkydN22oUyur6XLcsrLOIDHnoxxzV6lnMoWc6u0Jy0Y+g5EFit99z05kiMgegRG+5B7DxWf2Y=

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content: content changed '{md5}3a95c661934874a3ccbe006114c7d061' to '{md5}6b6c62d3e887b56831afd604ab7848f8'
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[cgi]/Exec[ensure_present_mod_cgi]/returns: executed successfully (corrective)
Info: /Stage[main]/Httpd/Httpd::Mod_conf[cgi]/Exec[ensure_present_mod_cgi]: Scheduling refresh of Service[apache2]
Notice: /Stage[main]/Statistics::Sites::Stats/File[/srv/stats.wikimedia.org]/ensure: created
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 68)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 68)
Wrapped exception:
No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs
Error: /Stage[main]/Statistics::Sites::Stats/File[/srv/stats.wikimedia.org/htdocs/v2]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 68)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 77)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 77)
Wrapped exception:
No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs
Error: /Stage[main]/Statistics::Sites::Stats/File[/srv/stats.wikimedia.org/htdocs/index.html]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 77)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 82)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 82)
Wrapped exception:
No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs
Error: /Stage[main]/Statistics::Sites::Stats/File[/srv/stats.wikimedia.org/htdocs/assets-v2]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/stats.wikimedia.org/htdocs (file: /etc/puppet/modules/statistics/manifests/sites/stats.pp, line: 82)
Info: Class[Statistics::Sites::Stats]: Unscheduling all events on Class[Statistics::Sites::Stats]
Notice: /Stage[main]/Httpd/Service[apache2]: Triggered 'refresh' from 1 event
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 18.82 seconds
RKemper added a comment.Aug 5 2021, 10:00 PM

Realized I never linked the gerrit patches to this ticket. See https://gerrit.wikimedia.org/r/c/operations/puppet/+/709822 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/709817 for patches related to this ticket.

Anyway, https://gerrit.wikimedia.org/r/c/operations/puppet/+/709822 fixed the above error(s) and an-web1001's puppet run is now successful

RKemper added a comment.EditedAug 5 2021, 10:00 PM

(Ignore this comment)

gerritbot added a comment.Aug 5 2021, 10:30 PM

Change 710371 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] analytics-web: temporary rsync module for thorium

https://gerrit.wikimedia.org/r/710371

gerritbot added a project: Patch-For-Review.Aug 5 2021, 10:30 PM
gerritbot added a comment.Aug 5 2021, 10:34 PM

Change 710371 merged by Ryan Kemper:

[operations/puppet@production] analytics-web: temporary rsync module for thorium

https://gerrit.wikimedia.org/r/710371

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2021, 11:10 PM
gerritbot added a comment.Aug 19 2021, 8:01 PM

Change 713934 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/deployment-charts@master] link recco: use CNAME for analytics-web

https://gerrit.wikimedia.org/r/713934

gerritbot added a project: Patch-For-Review.Aug 19 2021, 8:01 PM
gerritbot added a comment.Aug 23 2021, 11:33 PM

Change 713934 merged by jenkins-bot:

[operations/deployment-charts@master] linkrecommendation: use CNAME for analytics-web

https://gerrit.wikimedia.org/r/713934

Stashbot added a comment.Aug 23 2021, 11:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-23T23:41:14Z] <ryankemper> T285355 helmfile -e staging -i apply on /srv/deployment-charts/helmfile.d/services/linkrecommendation/ from ryankemper@deploy1002

Maintenance_bot removed a project: Patch-For-Review.Aug 24 2021, 12:10 AM
elukey added a comment.Aug 24 2021, 7:13 AM

@RKemper Great work on helm, but I am now wondering what are the expectations of availability for thorium/an-web1001 from the link recommendation point of view. For example, what happens to the service if an-web1001/thorium is unavailable for a day? I assume nothing terrible but if you have more info or a contact to follow up with I'd be happy to know, just to be sure :D

RKemper added a comment.Aug 24 2021, 7:04 PM

@elukey Agreed, that's definitely something we should hash out with them. I'll hop in their channel and ask about it and report back here as well as in the (fka) analytics irc channel.

BTullis reassigned this task from RKemper to Ottomata.Sep 14 2021, 4:21 PM
BTullis added a subscriber: RKemper.
Ottomata added a comment.Sep 14 2021, 7:09 PM

@elukey IIRC we told them not to rely on it. Its used for occasionally loading data, and I believe they were told to point to the internal address rather than go through the public routers. I suppose a cname would be helpful here, but for now I'm just going to move forward with this.

@RKemper I've claimed this task and am going to try to complete it. Is this status summary correct?

  • an-web1001 provisioned and puppetized.
  • An initial rsync from thorium has been run, but not since mid August.
  • linkrecommendation has been updated in staging to use an-web1001, but not in prod eqiad or codfw?

If so, I'll do the following:

  1. Do another rsync from thorium
  2. deploy linkrecommendation in prod clusters
  3. update any (non LVS) remaining references in puppet to point at an-web1001
  4. change LVS entries to point to an-web1001
  5. do a final rsync from thorium
  6. wait a week or two
  7. decom thorium

Sound ok @BTullis @razzi @RKemper?

RKemper added a comment.Sep 14 2021, 8:07 PM

@Ottomata Looks like I didn't make separate SAL logs when doing the eqiad / codfw (production) side of the helmfile stuff, but here's the automated logs for peace of mind:

https://sal.toolforge.org/log/dFG9p3sB8Fs0LHO5573D
https://sal.toolforge.org/log/c1G9p3sB8Fs0LHO52b2F
https://sal.toolforge.org/log/DPPBp3sB1jz_IcWuhWxP
https://sal.toolforge.org/log/ewbBp3sBa_6PSCT9lTdh

So the only amendment I have right now is to this:

linkrecommendation has been updated in staging to use an-web1001, but not in prod eqiad or codfw?

The actual state is that linkrecco has been updated to use the analytics-web cname in both staging and prod, and that cname is still pointing to thorium. So when ready to cutover the cname will need to be changed, but no further changes should need to be made to the helmfile

Ottomata added a comment.EditedSep 14 2021, 8:14 PM

Oh! There is a cname! analytics-web.discovery.wmnet. new plan:

Will be editing this comment as plan adjusts.

  1. Use analytics-web.discovery.wmnet everywhere instead of thorium.
  2. Do another rsync from thorium
  3. change analytics-web cname to point to an-web1001
  4. wait N minutes for DNS to propagate.
  5. do a final rsync from thorium
  6. Remove thorium egress rules from linkrecommendation k8s helmfiles
  7. wait a week or two
  8. decom thorium
gerritbot added a comment.Sep 14 2021, 8:48 PM

Change 721061 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] statistics::rsync::published - push to analytics-web.discovery.wmnet cname

https://gerrit.wikimedia.org/r/721061

gerritbot added a project: Patch-For-Review.Sep 14 2021, 8:48 PM
gerritbot added a comment.Sep 14 2021, 8:51 PM

Change 721062 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Point trafficserver at analytics-web cname instead of thorium hostname

https://gerrit.wikimedia.org/r/721062

gerritbot added a comment.Sep 15 2021, 1:25 PM

Change 721061 merged by Ottomata:

[operations/puppet@production] statistics::rsync::published - push to analytics-web.discovery.wmnet cname

https://gerrit.wikimedia.org/r/721061

gerritbot added a comment.Sep 15 2021, 1:30 PM

Change 721062 merged by Ottomata:

[operations/puppet@production] Point trafficserver at analytics-web cname instead of thorium hostname

https://gerrit.wikimedia.org/r/721062

Stashbot added a comment.Sep 15 2021, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-15T13:33:08Z] <ottomata> pointing {stats,analytics}.wikimedia.org at analytics-web.discovery.wmnet cname - T285355

gerritbot added a comment.Sep 15 2021, 2:07 PM

Change 721327 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/dns@master] Point analytics-web CNAME at an-web1001

https://gerrit.wikimedia.org/r/721327

Ottomata added a comment.Sep 15 2021, 2:56 PM

Running rsync /srv thorium -> an-web1001:

time sudo rsync -av thorium.eqiad.wmnet::transfer_from_thorium/ /srv/
Ottomata moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Sep 15 2021, 4:04 PM
Ottomata added a comment.EditedSep 15 2021, 8:27 PM

Oops, have to tell rsync to preserve hardlinks. Starting over:

sudo rsync -av -H --delete --exclude '.hardsync.*'  thorium.eqiad.wmnet::transfer_from_thorium/ /srv/
gerritbot added a comment.Sep 16 2021, 7:25 PM

Change 721327 merged by Ottomata:

[operations/dns@master] Point analytics-web CNAME at an-web1001

https://gerrit.wikimedia.org/r/721327

Stashbot added a comment.Sep 16 2021, 7:25 PM

Mentioned in SAL (#wikimedia-analytics) [2021-09-16T19:25:48Z] <ottomata> pointing analytics-web cname at new an-web1001, this moves stats and analytics .wm.org from thorium to an-web1001 - T285355

Ottomata added a comment.Sep 16 2021, 7:32 PM

stats.wikimedia.org and analyitcs.wikimedia.org are now being served by an-web1001

gerritbot added a comment.Sep 17 2021, 7:05 AM

Change 721755 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::analytics::httpd: disable CGI

https://gerrit.wikimedia.org/r/721755

gerritbot added a comment.Sep 17 2021, 8:11 AM

Change 721755 merged by Elukey:

[operations/puppet@production] profile::analytics::httpd: disable CGI

https://gerrit.wikimedia.org/r/721755

elukey added a comment.Sep 17 2021, 8:21 AM

To keep archives happy - I took the liberty to disable cgi/cgid httpd modules and configs on an-web1001, those were related to the execution of old perl scripts for Wikistats 1 that nobody really read anymore. There was an alarm in icinga related to puppet changing things on every run (there was some issue with the cgi module) so while checking I decided to finally do something that I have been planning for years :D Please let me know in case we want to revert back (even if I don't see a good reason now).

Side note: the following httpd config seems very stale and not used, can we get rid of it? (I recall I had this chat with Dan in the past but we didn't proceed for some reasons)

<Directory "/srv/stats.wikimedia.org/htdocs/reportcard/staff">
    AllowOverride None
    AuthName "Password protected area"
    AuthType Basic
    AuthUserFile /etc/apache2/htpasswd.stats
    Require user wmf
</Directory>
<Directory "/srv/stats.wikimedia.org/htdocs/reportcard/extended">
    AllowOverride None
    AuthName "Password protected area"
    AuthType Basic
    AuthUserFile /etc/apache2/htpasswd.stats
    Require user internal
</Directory>
<Directory "/srv/stats.wikimedia.org/htdocs/reportcard/pediapress">
    AllowOverride None
    AuthName "Password protected area"
    AuthType Basic
    AuthUserFile /etc/apache2/htpasswd.stats
    Require user pediapress
</Directory>



# --- Internal Geowiki ---
# Force https and use http auth for geowiki's private data
<Directory "<%= scope.lookupvar('::statistics::sites::stats::geowiki_private_directory') %>">
    RewriteEngine On
    RewriteCond %{HTTP:X-Forwarded-Proto} !https
    RewriteRule ^/(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,E=ProtoRedirect]

    AllowOverride None
    AuthName "Geowiki's 'foundation only' files"
    AuthType Basic
    AuthUserFile "<%= scope.lookupvar('::statistics::sites::stats::geowiki_private_htpasswd_file') %>"
    Require valid-user
</Directory>
BTullis added a comment.Sep 17 2021, 10:00 AM

We're getting cronspam from the published-sync jobs running on the client and launcher nodes, when they're running their rsync tasks and publishing to an-web1001.

My guess is that it's because the stats user uid is different between thorium and an-web1001.

btullis@an-web1001:~$ id stats
uid=499(stats) gid=499(stats) groups=499(stats)
btullis@an-web1001:~$ id systemd-coredump
uid=999(systemd-coredump) gid=999(systemd-coredump) groups=999(systemd-coredump)
btullis@an-web1001:~$ ls -l /srv/published-rsynced/|tail -n 5
drwxrwxr-x 3 systemd-coredump wikidev 4096 Oct 30  2019 stat1004
drwxrwxr-x 4 systemd-coredump wikidev 4096 May 21  2020 stat1005
drwxrwxr-x 3 systemd-coredump wikidev 4096 Oct 30  2019 stat1006
drwxrwxr-x 6 systemd-coredump wikidev 4096 May 20  2020 stat1007
drwxrwxr-x 4 systemd-coredump wikidev 4096 Jun 23  2020 stat1008
btullis@thorium:~$ id stats
uid=999(stats) gid=999(stats) groups=999(stats)
btullis@thorium:~$ ls -l /srv/published-rsynced/|tail -n 5
drwxrwxr-x 3 stats wikidev 4096 Oct 30  2019 stat1004
drwxrwxr-x 4 stats wikidev 4096 May 21  2020 stat1005
drwxrwxr-x 3 stats wikidev 4096 Oct 30  2019 stat1006
drwxrwxr-x 6 stats wikidev 4096 May 20  2020 stat1007
drwxrwxr-x 4 stats wikidev 4096 Jun 23  2020 stat1008

I'll have a look and see if there's a way we can fix this relatively quickly, so as to avoid the frequent cron mail.

BTullis added a comment.Sep 17 2021, 11:05 AM

I chowned the files on an-web1001 with sudo find /srv/published-rsynced -user systemd-coredump -exec chown stats {} \;

I hesitated to make sure that we didn't have to keep uids in sync between the source servers (an-launcher1002, stat100* etc) and the two [possible destination servers (thorium and an-web1001).
However, it seems like the syncs are only going from the sources to a single destination server and the uids don't have to match. I think that the issue must have been caused by the rsync directly from thorium to an-web1001 mentioned in T285355#7357055

Logspam has now stopped and I don't think we need to run another sync from thorium, so I think that this one-off action has fixed it.

elukey added a comment.Sep 17 2021, 1:29 PM
In T285355#7361119, @elukey wrote:

Side note: the following httpd config seems very stale and not used, can we get rid of it? (I recall I had this chat with Dan in the past but we didn't proceed for some reasons)

After a chat with Dan on Slack it seems so, let's backup (if needed, files from 2012..) + drop those dirs and then clean up httpd config.

Ottomata added a comment.Sep 17 2021, 2:21 PM

Thank you both!

I hesitated to make sure that we didn't have to keep uids in sync

Ah, we should probably do this. We recently gained the ability to make Puppet do this for us, but likely have not done so yet for the stats user.

gerritbot added a comment.Sep 20 2021, 6:43 AM

Change 722252 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] statistics: decom old httpd Directory for stats.wikimedia.org

https://gerrit.wikimedia.org/r/722252

gerritbot added a comment.Sep 20 2021, 8:10 AM

Change 722252 merged by Elukey:

[operations/puppet@production] statistics: decom old httpd Directory for stats.wikimedia.org

https://gerrit.wikimedia.org/r/722252

elukey added a subscriber: Milimetric.Sep 20 2021, 8:18 AM

The following dirs/files are no longer be accessible from stats.wikimedia.org (not sure if anybody really used them in years):

elukey@an-web1001:~$ ls -l /srv/stats.wikimedia.org/htdocs/reportcard/{pediapress,extended,staff}
/srv/stats.wikimedia.org/htdocs/reportcard/extended:
total 3352
-rw-r--r--  1 523 wikidev     54 Aug 27  2012 asc.gif
drwxr-xr-x  2 523 wikidev   4096 Aug 27  2012 assets
-rw-r--r--  1 523 wikidev     64 Aug 27  2012 bg.gif
drwxr-xr-x 32 523 wikidev   4096 Aug 27  2012 charts
-rw-r--r--  1 523 wikidev     54 Aug 27  2012 desc.gif
-rw-r--r--  1 523 wikidev  34379 Aug 27  2012 html
-rw-r--r--  1 523 wikidev  18437 Aug 27  2012 index.html
-rwxr-xr-x  1 523 wikidev   2853 Aug 27  2012 jquery.tabs.css
-rwxr-xr-x  1 523 wikidev    541 Aug 27  2012 jquery.tabs-ie.css
-rwxr-xr-x  1 523 wikidev   6449 Aug 27  2012 RC_0000_prints.html
-rwxr-xr-x  1 523 wikidev  43344 Aug 27  2012 RC_2009_05_columns.html
-rwxr-xr-x  1 523 wikidev  41507 Aug 27  2012 RC_2009_05_detailed.html
-rwxr-xr-x  1 523 wikidev  43811 Aug 27  2012 RC_2009_05_summary.html
-rwxr-xr-x  1 523 wikidev  43395 Aug 27  2012 RC_2009_06_columns.html
-rwxr-xr-x  1 523 wikidev  41832 Aug 27  2012 RC_2009_06_detailed.html
-rwxr-xr-x  1 523 wikidev  43863 Aug 27  2012 RC_2009_06_summary.html
-rwxr-xr-x  1 523 wikidev  16760 Aug 27  2012 RC_2009_07_columns.html
-rwxr-xr-x  1 523 wikidev  33886 Aug 27  2012 RC_2009_07_detailed.html
-rwxr-xr-x  1 523 wikidev  15603 Aug 27  2012 RC_2009_07_summary.html
-rwxr-xr-x  1 523 wikidev    640 Aug 27  2012 RC_2009_07_synopsis.txt
-rwxr-xr-x  1 523 wikidev  16765 Aug 27  2012 RC_2009_08_columns.html
-rwxr-xr-x  1 523 wikidev  34089 Aug 27  2012 RC_2009_08_detailed.html
-rwxr-xr-x  1 523 wikidev  15608 Aug 27  2012 RC_2009_08_summary.html
-rwxr-xr-x  1 523 wikidev    640 Aug 27  2012 RC_2009_08_synopsis.txt
-rwxr-xr-x  1 523 wikidev  17082 Aug 27  2012 RC_2009_09_columns.html
-rwxr-xr-x  1 523 wikidev  29634 Aug 27  2012 RC_2009_09_detailed.html
-rwxr-xr-x  1 523 wikidev  15925 Aug 27  2012 RC_2009_09_summary.html
-rwxr-xr-x  1 523 wikidev    640 Aug 27  2012 RC_2009_09_synopsis.txt
-rwxr-xr-x  1 523 wikidev  17096 Aug 27  2012 RC_2009_10_columns.html
-rwxr-xr-x  1 523 wikidev  29667 Aug 27  2012 RC_2009_10_detailed.html
-rwxr-xr-x  1 523 wikidev  15939 Aug 27  2012 RC_2009_10_summary.html
-rw-r--r--  1 523 wikidev  17623 Aug 27  2012 RC_2009_11_columns.html
-rwxr-xr-x  1 523 wikidev  30307 Aug 27  2012 RC_2009_11_detailed.html
-rw-r--r--  1 523 wikidev  16474 Aug 27  2012 RC_2009_11_summary.html
-rw-r--r--  1 523 wikidev   6811 Aug 27  2012 RC_2009_11_synopsis.html
-rw-r--r--  1 523 wikidev   6352 Aug 27  2012 RC_2009_11_synopsis.txt
-rw-r--r--  1 523 wikidev  17488 Aug 27  2012 RC_2009_12_columns.html
-rw-r--r--  1 523 wikidev  30187 Aug 27  2012 RC_2009_12_detailed.html
-rw-r--r--  1 523 wikidev   6199 Aug 27  2012 RC_2009_12_errata.html
-rw-r--r--  1 523 wikidev  16339 Aug 27  2012 RC_2009_12_summary.html
-rw-r--r--  1 523 wikidev   9017 Aug 27  2012 RC_2009_12_synopsis.html
-rw-r--r--  1 523 wikidev   7535 Aug 27  2012 RC_2009_12_synopsis.txt
-rw-r--r--  1 523 wikidev  17462 Aug 27  2012 RC_2010_01_columns.html
-rw-r--r--  1 523 wikidev  30177 Aug 27  2012 RC_2010_01_detailed.html
-rw-r--r--  1 523 wikidev   6198 Aug 27  2012 RC_2010_01_errata.html
-rw-r--r--  1 523 wikidev  16313 Aug 27  2012 RC_2010_01_summary.html
-rw-r--r--  1 523 wikidev  11170 Aug 27  2012 RC_2010_01_synopsis.html
-rw-r--r--  1 523 wikidev  11065 Aug 27  2012 RC_2010_01_synopsis.txt
-rw-r--r--  1 523 wikidev  17468 Aug 27  2012 RC_2010_02_columns.html
-rw-r--r--  1 523 wikidev  30389 Aug 27  2012 RC_2010_02_detailed.html
-rw-r--r--  1 523 wikidev   7216 Aug 27  2012 RC_2010_02_errata.html
-rw-r--r--  1 523 wikidev  16319 Aug 27  2012 RC_2010_02_summary.html
-rw-r--r--  1 523 wikidev  10643 Aug 27  2012 RC_2010_02_synopsis.html
-rw-r--r--  1 523 wikidev  10538 Aug 27  2012 RC_2010_02_synopsis.txt
-rw-r--r--  1 523 wikidev  18084 Aug 27  2012 RC_2010_03_columns.html
-rw-r--r--  1 523 wikidev  31186 Aug 27  2012 RC_2010_03_detailed.html
-rw-r--r--  1 523 wikidev   7213 Aug 27  2012 RC_2010_03_errata.html
-rw-r--r--  1 523 wikidev  16935 Aug 27  2012 RC_2010_03_summary.html
-rw-r--r--  1 523 wikidev  10872 Aug 27  2012 RC_2010_03_synopsis.html
-rw-r--r--  1 523 wikidev  10767 Aug 27  2012 RC_2010_03_synopsis.txt
-rw-r--r--  1 523 wikidev  17655 Aug 27  2012 RC_2010_04_columns.html
-rw-r--r--  1 523 wikidev  30590 Aug 27  2012 RC_2010_04_detailed.html
-rw-r--r--  1 523 wikidev   7213 Aug 27  2012 RC_2010_04_errata.html
-rw-r--r--  1 523 wikidev  16506 Aug 27  2012 RC_2010_04_summary.html
-rw-r--r--  1 523 wikidev  11324 Aug 27  2012 RC_2010_04_synopsis.html
-rw-r--r--  1 523 wikidev  11219 Aug 27  2012 RC_2010_04_synopsis.txt
-rw-r--r--  1 523 wikidev  18062 Aug 27  2012 RC_2010_05_columns.html
-rw-r--r--  1 523 wikidev  31143 Aug 27  2012 RC_2010_05_detailed.html
-rw-r--r--  1 523 wikidev   7211 Aug 27  2012 RC_2010_05_errata.html
-rw-r--r--  1 523 wikidev  16913 Aug 27  2012 RC_2010_05_summary.html
-rw-r--r--  1 523 wikidev  11702 Aug 27  2012 RC_2010_05_synopsis.html
-rw-r--r--  1 523 wikidev  11582 Aug 27  2012 RC_2010_05_synopsis.txt
-rw-r--r--  1 523 wikidev  18074 Aug 27  2012 RC_2010_06_columns.html
-rw-r--r--  1 523 wikidev  31373 Aug 27  2012 RC_2010_06_detailed.html
-rw-r--r--  1 523 wikidev   7212 Aug 27  2012 RC_2010_06_errata.html
-rw-r--r--  1 523 wikidev  16925 Aug 27  2012 RC_2010_06_summary.html
-rw-r--r--  1 523 wikidev  11401 Aug 27  2012 RC_2010_06_synopsis.html
-rw-r--r--  1 523 wikidev  11296 Aug 27  2012 RC_2010_06_synopsis.txt
-rw-r--r--  1 523 wikidev  17899 Aug 27  2012 RC_2010_07_columns.html
-rw-r--r--  1 523 wikidev  31207 Aug 27  2012 RC_2010_07_detailed.html
-rw-r--r--  1 523 wikidev  16750 Aug 27  2012 RC_2010_07_summary.html
-rw-r--r--  1 523 wikidev  11552 Aug 27  2012 RC_2010_07_synopsis.html
-rw-r--r--  1 523 wikidev  11549 Aug 27  2012 RC_2010_07_synopsis.txt
-rw-r--r--  1 523 wikidev  18233 Aug 27  2012 RC_2010_08_columns.html
-rw-r--r--  1 523 wikidev  31825 Aug 27  2012 RC_2010_08_detailed.html
-rw-r--r--  1 523 wikidev  17084 Aug 27  2012 RC_2010_08_summary.html
-rw-r--r--  1 523 wikidev  11979 Aug 27  2012 RC_2010_08_synopsis.html
-rw-r--r--  1 523 wikidev  11976 Aug 27  2012 RC_2010_08_synopsis.txt
-rw-r--r--  1 523 wikidev  18348 Aug 27  2012 RC_2010_09_columns.html
-rw-r--r--  1 523 wikidev  31948 Aug 27  2012 RC_2010_09_detailed.html
-rw-r--r--  1 523 wikidev   6237 Aug 27  2012 RC_2010_09_errata.html
-rw-r--r--  1 523 wikidev  17199 Aug 27  2012 RC_2010_09_summary.html
-rw-r--r--  1 523 wikidev  11650 Aug 27  2012 RC_2010_09_synopsis.html
-rw-r--r--  1 523 wikidev  11545 Aug 27  2012 RC_2010_09_synopsis.txt
-rw-r--r--  1 523 wikidev  19100 Aug 27  2012 RC_2010_10_columns.html
-rw-r--r--  1 523 wikidev  32948 Aug 27  2012 RC_2010_10_detailed.html
-rw-r--r--  1 523 wikidev  17951 Aug 27  2012 RC_2010_10_summary.html
-rw-r--r--  1 523 wikidev  10037 Aug 27  2012 RC_2010_10_synopsis.html
-rw-r--r--  1 523 wikidev  10034 Aug 27  2012 RC_2010_10_synopsis.txt
-rw-r--r--  1 523 wikidev  19003 Aug 27  2012 RC_2010_11_columns.html
-rw-r--r--  1 523 wikidev  32857 Aug 27  2012 RC_2010_11_detailed.html
-rw-r--r--  1 523 wikidev  17854 Aug 27  2012 RC_2010_11_summary.html
-rw-r--r--  1 523 wikidev  10279 Aug 27  2012 RC_2010_11_synopsis.html
-rw-r--r--  1 523 wikidev  10276 Aug 27  2012 RC_2010_11_synopsis.txt
-rw-r--r--  1 523 wikidev  19201 Aug 27  2012 RC_2010_12_columns.html
-rw-r--r--  1 523 wikidev  33557 Aug 27  2012 RC_2010_12_detailed.html
-rw-r--r--  1 523 wikidev  18052 Aug 27  2012 RC_2010_12_summary.html
-rw-r--r--  1 523 wikidev  10597 Aug 27  2012 RC_2010_12_synopsis.html
-rw-r--r--  1 523 wikidev  10594 Aug 27  2012 RC_2010_12_synopsis.txt
-rw-r--r--  1 523 wikidev  19505 Aug 27  2012 RC_2011_01_columns.html
-rw-r--r--  1 523 wikidev  34132 Aug 27  2012 RC_2011_01_detailed.html
-rw-r--r--  1 523 wikidev   6429 Aug 27  2012 RC_2011_01_errata.html
-rw-r--r--  1 523 wikidev  18356 Aug 27  2012 RC_2011_01_summary.html
-rw-r--r--  1 523 wikidev  11499 Aug 27  2012 RC_2011_01_synopsis.html
-rw-r--r--  1 523 wikidev  11394 Aug 27  2012 RC_2011_01_synopsis.txt
-rw-r--r--  1 523 wikidev  19257 Aug 27  2012 RC_2011_02_columns.html
-rw-r--r--  1 523 wikidev  33741 Aug 27  2012 RC_2011_02_detailed.html
-rw-r--r--  1 523 wikidev  18108 Aug 27  2012 RC_2011_02_summary.html
-rw-r--r--  1 523 wikidev  11098 Aug 27  2012 RC_2011_02_synopsis.html
-rw-r--r--  1 523 wikidev  11095 Aug 27  2012 RC_2011_02_synopsis.txt
-rw-r--r--  1 523 wikidev  19297 Aug 27  2012 RC_2011_03_columns.html
-rw-r--r--  1 523 wikidev  33588 Aug 27  2012 RC_2011_03_detailed.html
-rw-r--r--  1 523 wikidev  18148 Aug 27  2012 RC_2011_03_summary.html
-rw-r--r--  1 523 wikidev  12323 Aug 27  2012 RC_2011_03_synopsis.html
-rw-r--r--  1 523 wikidev  12131 Aug 27  2012 RC_2011_03_synopsis.txt
-rw-r--r--  1 523 wikidev  19841 Aug 27  2012 RC_2011_04_columns.html
-rw-r--r--  1 523 wikidev  34141 Aug 27  2012 RC_2011_04_detailed.html
-rw-r--r--  1 523 wikidev  18692 Aug 27  2012 RC_2011_04_summary.html
-rw-r--r--  1 523 wikidev  12245 Aug 27  2012 RC_2011_04_synopsis.html
-rw-r--r--  1 523 wikidev  12242 Aug 27  2012 RC_2011_04_synopsis.txt
-rw-r--r--  1 523 wikidev   9282 Aug 27  2012 RC_2011_05_detailed.html
-rw-r--r--  1 523 wikidev   9282 Aug 27  2012 RC_2011_08_detailed.html
-rw-r--r--  1 523 wikidev  19976 Aug 27  2012 RC_2011_09_columns.html
-rw-r--r--  1 523 wikidev  34276 Aug 27  2012 RC_2011_09_detailed.html
-rw-r--r--  1 523 wikidev   5789 Aug 27  2012 RC_2011_09_errata.html
-rw-r--r--  1 523 wikidev  18827 Aug 27  2012 RC_2011_09_summary.html
-rw-r--r--  1 523 wikidev  11580 Aug 27  2012 RC_2011_09_synopsis.html
-rw-r--r--  1 523 wikidev  11475 Aug 27  2012 RC_2011_09_synopsis.txt
-rw-r--r--  1 523 wikidev  19896 Aug 27  2012 RC_2011_10_columns.html
-rw-r--r--  1 523 wikidev  34222 Aug 27  2012 RC_2011_10_detailed.html
-rw-r--r--  1 523 wikidev  18747 Aug 27  2012 RC_2011_10_summary.html
-rw-r--r--  1 523 wikidev  11415 Aug 27  2012 RC_2011_10_synopsis.html
-rw-r--r--  1 523 wikidev  11412 Aug 27  2012 RC_2011_10_synopsis.txt
-rw-r--r--  1 523 wikidev  19841 Aug 27  2012 RC_2011_11_columns.html
-rw-r--r--  1 523 wikidev  34173 Aug 27  2012 RC_2011_11_detailed.html
-rw-r--r--  1 523 wikidev  18692 Aug 27  2012 RC_2011_11_summary.html
-rw-r--r--  1 523 wikidev  10703 Aug 27  2012 RC_2011_11_synopsis.html
-rw-r--r--  1 523 wikidev  10700 Aug 27  2012 RC_2011_11_synopsis.txt
-rw-r--r--  1 523 wikidev  19842 Aug 27  2012 RC_2011_12_columns.html
-rw-r--r--  1 523 wikidev  34159 Aug 27  2012 RC_2011_12_detailed.html
-rw-r--r--  1 523 wikidev  18693 Aug 27  2012 RC_2011_12_summary.html
-rw-r--r--  1 523 wikidev  12568 Aug 27  2012 RC_2011_12_synopsis.html
-rw-r--r--  1 523 wikidev  12565 Aug 27  2012 RC_2011_12_synopsis.txt
-rw-r--r--  1 523 wikidev  19819 Aug 27  2012 RC_2012_01_columns.html
-rw-r--r--  1 523 wikidev  34116 Aug 27  2012 RC_2012_01_detailed.html
-rw-r--r--  1 523 wikidev  18670 Aug 27  2012 RC_2012_01_summary.html
-rw-r--r--  1 523 wikidev  13483 Aug 27  2012 RC_2012_01_synopsis.html
-rw-r--r--  1 523 wikidev  13480 Aug 27  2012 RC_2012_01_synopsis.txt
-rw-r--r--  1 523 wikidev  19586 Aug 27  2012 RC_2012_02_columns.html
-rw-r--r--  1 523 wikidev  34111 Aug 27  2012 RC_2012_02_detailed.html
-rw-r--r--  1 523 wikidev  18437 Aug 27  2012 RC_2012_02_summary.html
-rw-r--r--  1 523 wikidev  12281 Aug 27  2012 RC_2012_02_synopsis.html
-rw-r--r--  1 523 wikidev  12278 Aug 27  2012 RC_2012_02_synopsis.txt
-rw-r--r--  1 523 wikidev   9099 Aug 27  2012 RC_2012_03_detailed.html
-rwxr-xr-x  1 523 wikidev  31904 Aug 27  2012 RT_yyyy_mm.html
-rwxr-xr-x  1 523 wikidev 161879 Aug 27  2012 SamplePrint.png
-rwxr-xr-x  1 523 wikidev   1006 Aug 27  2012 tab.png
-rwxr-xr-x  1 523 wikidev  18011 Aug 27  2012 WikiReportsGenerateReportCards.pl
-rwxr-xr-x  1 523 wikidev  17444 Aug 27  2012 WikiStatsComScoreReferenceSites.pl

/srv/stats.wikimedia.org/htdocs/reportcard/pediapress:
total 0

/srv/stats.wikimedia.org/htdocs/reportcard/staff:
total 1556
-rwxr--r-- 1 523 wikidev  308309 Apr 16  2013 ComScoreReferenceSites.htm
-rwxr--r-- 1 523 wikidev 1061183 Apr 16  2013 ComScoreTop1000Properties.htm
-rw-r--r-- 1 523 wikidev   13478 Aug 27  2012 EzLib.pm
-rw-r--r-- 1 523 wikidev     384 Aug 27  2012 index.html
-rw-r--r-- 1 523 wikidev     597 Aug 27  2012 PediaPress.html
-rw-r--r-- 1 523 wikidev    1136 Aug 27  2012 PediaPressSalesMonthlyConciseBooksShippedOnly.csv
-rw-r--r-- 1 523 wikidev   12727 Aug 27  2012 PediaPressSalesMonthlyDetailedBreakdownPerCountry.csv
-rw-r--r-- 1 523 wikidev   68608 Aug 27  2012 PediaPressSalesMonthlyDetailedBreakdownPerCountry.xls
-rw-r--r-- 1 523 wikidev    9609 Aug 27  2012 PediaPressSalesMonthlyDetailedOneRowPerMonth.csv
-rw-r--r-- 1 523 wikidev   20720 Aug 27  2012 PediaPress_Sales.png
-rw-r--r-- 1 523 wikidev   20235 Aug 27  2012 ReportCardStaff.pl
drwxr-xr-x 2 523 wikidev    4096 Aug 27  2012 squids
-rw-r--r-- 1 523 wikidev   19363 Aug 27  2012 WikipediaStatistics13.js
-rw-r--r-- 1 523 wikidev   19371 Aug 27  2012 WikipediaStatistics14.js

@Milimetric since these dirs were password protected, should we archive/drop everything from an-web1001? (just as precaution)

After it I'll do the final clean up :)

gerritbot added a comment.Sep 20 2021, 8:37 AM

Change 722270 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] statistics: remove leftovers for stats.wikimedia.org

https://gerrit.wikimedia.org/r/722270

Ottomata mentioned this in T291384: Standardize the stats system user uid.Sep 20 2021, 1:46 PM

For reference, here is Dan's response from Slack:

The old geowiki data has been disabled for years and everyone I know uses geoeditors instead, so that can be dropped and deleted. The other directories were deprecated before that even, so same for those.
The only thing that remains from the original wikistats is a set of reports that we should have migrated by now but we have no way of prioritizing with all the other work we need to do. So I'm fine with decomissioning everything and looking at it with fresh eyes once we get more of the shared data platform established.

elukey added a comment.Sep 21 2021, 6:53 AM

@Ottomata what should we do about the dirs mentioned above? (just trying to confirm if we can drop them or not to avoid a "OH NOESSS" moment :))

Ottomata added a comment.Sep 21 2021, 1:11 PM

Oh I think we should drop them, I wasn't sure where the communication was between you and @Milimetric. Let's verify with him and I'll follow up.

Milimetric added a comment.Sep 27 2021, 2:52 PM

I agree we should drop them. I vaguely remember this kind of special access to some versions of the old reportcard, and even back then use was very limited. These kinds of partnerships should be going through different channels these days anyway.

Ottomata added a comment.Sep 29 2021, 2:21 PM

@elukey, deleted.

14:19:19 [@an-web1001:/srv/stats.wikimedia.org/htdocs/reportcard] $ sudo rm -rf ./pediapress ./staff ./extended
gerritbot added a comment.Sep 29 2021, 2:36 PM

Change 724754 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] Remove thorium from linkrecommendation egress rules

https://gerrit.wikimedia.org/r/724754

elukey added a comment.Sep 29 2021, 2:36 PM

@Ottomata nice! To double check, I see the following to drop in puppet private:

class passwords::geowiki {
        # NOTE!  If you change these, you should also change
        # the password for the stats user in the htpasswd.geowiki file
        # in the private repository at files/apache/htpasswd.geowiki.
        $user = 'stats'
        $pass = REDACTED
elukey@puppetmaster1001:/srv/private$ find -name htpasswd*
./modules/secret/secrets/apache/htpasswd.stats-geowiki
./modules/secret/secrets/apache/htpasswd.stats

Green light to drop them too?

Ottomata added a comment.Sep 29 2021, 2:38 PM

Yes please! @elukey I'm going to proceed with the decom of thorium.

Ottomata added a subtask: T292075: decommission thorium.eqiad.wmnet.Sep 29 2021, 2:42 PM
gerritbot added a comment.Sep 29 2021, 2:45 PM

Change 722270 merged by Elukey:

[operations/puppet@production] statistics: remove leftovers for stats.wikimedia.org

https://gerrit.wikimedia.org/r/722270

gerritbot added a comment.Sep 29 2021, 2:47 PM

Change 724754 merged by Ottomata:

[operations/deployment-charts@master] Remove thorium from linkrecommendation egress rules

https://gerrit.wikimedia.org/r/724754

elukey added a comment.Sep 29 2021, 2:53 PM

All cleaned up!

Ottomata added a comment.Sep 29 2021, 2:54 PM

Thank you!

Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Sep 30 2021, 6:52 PM
Ottomata closed this task as Resolved.Oct 21 2021, 5:23 PM
Cmjohnson closed subtask T292075: decommission thorium.eqiad.wmnet as Resolved.Dec 9 2021, 6:12 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL