It also seems that in order to use pyenv to switch between python versions, pyenv should be used to build these versions from source; see https://github.com/pyenv/pyenv/issues/1244 We may be able to work around it, though this is unclear. We would have to go through the python patches in the pyenv python-build directory for the versions we want and see how necessary they are.

These have been tested: all the shell scripts in the repo, all python scripts with an entry point, with both checkpoint and regular dump setups. I plan to deploy this weekend; it will have to go at the same time as some puppet changes to call at least one of these scripts using python3.

What sort of help do you need from my end, or is it too early?

The lbzip2 code doesn't produce chunked bzip2 streams (like e.g. the multistream xml pages-articles dumps). It's one stream only. I expect that is why php runs ok on it.

I note that there do not seem to be debian packages for pyenv.

Ah, I misread! Before we arbitrarily throttle though, I'd like the dbas to weigh in. It might be that having the flood actually flood may be helpful sometimes.

Of the above options, I prefer the idea of explicitly increasing the LAG_WARN_THRESHOLD in scripts where that's desired. Anything that depends on a particular db server makes me uneasy, and I don't like the idea of turning off load monitoring completely either.

The commit that broke the rsync is this one: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/471035/
(BTW that commit has a typo in compute.pp that is still around in the manifests, "publshed")

It's running a sequence of jobs so you might not get all the hashes added to the final output file until it completes the particular sequence.
In the meantime you can look for and grab sha1sums-fiwiki-20181201-pages-articles.xml.bz2.txt or md5sums-fiwiki-20181201-pages-articles.xml.bz2.txt from the same directory; they are written right after the output file for each step is created.

Just a note that while it's taken 5 years for that number of records to go from 154 million to 517 million, some of those increases are 80 million records in a day, let's plan for that. (See https://www.troyhunt.com/ive-just-added-2844-new-data-breaches-with-80m-records-to-have-i-been-pwned/ for more.)

Forgot to add, we now can check all db hosts and wikis' table structures against one presumptive master db server/wiki; this is how I collected a small list of hosts and wikis with differences. That's this script again: https://github.com/apergos/misc-wmf-crap/blob/master/checktables/check_table_structures.py Yeah I know it's awful. But for right now it gets the job done.

I've got a few scripts written and they're gross. But they work. Here's one that generates values I need for in a format that my show explain script can read:
https://github.com/apergos/misc-wmf-crap/blob/master/runqueries/genrqsettings.py
I've run that for the following wikis: frwiki, jawiki, plwiki, commonswiki, enwiki, wikidatawik, frwiktionary. That output, with a bit of cleanup, will get fed to:
https://github.com/apergos/misc-wmf-crap/blob/master/runqueries/explain_sql_query.py
which will run the query of interest on the specified host, show explain it from a subprocess, then shoot the original query in case it happens to take time to complete.
Still in the works is a script to collect the sql query text right out of the WikiExporter class as it runs, by monkey-patching (eww); we need this so we can keep up with changes to the code.

They are indeed for dumps; one is set up to be the web server, and a reboot of that means that the broader public will notice. The other provides NFS service of dumps to stat100* and to labs vms (includes toolforge I think), and this is a bit finicky. Definitely @Bstorm will want to weigh in on these.

Needless to say I'd like a bunch of eyes on this to make sure it's right. Thanks in advance.

Ok, I have read all the dang back tckets and thanks everyone for their comments. I am skipping wikisource and betawikiversity because it's more complication than I want to deal with. But here's a changeset that *should* cover everything else so we can not worry about this again, for everything else, except of course the wikipedia rename.

Can we think about ways to detect the problem within short period of time? I'd like to see that happen at least.

I could only find one user account creation on wikitech; where is the other one? That's what concerns me, one account there and yet two ldap entries.

I'm ignoring zh-min-nan for the purposes of this ticket.

Are there other zh-yue and yue projects that also need to be addressed? I f we are going to add redirects, we might as well do all that are needed.

Does the community want that, if there is a community of users on the incubator?

This move is complete, no? Is there anything left to do before closing?

What do folks think about this for a first step? When we're happy that these are ok, we can roll out to the rdf dumps. Right now the last dumps of the weekly (lexeme) finish on Sunday so that's just not sustainable going forwards.

I see content in the current lexemes dump:

There is no rewrite rule for zh-yue wiktionary; there is one for yue wikipedia. See line 97: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/mediawiki/files/apache/sites/redirects/redirects.dat

This is now dependent on the bandwith caps for labstore 1006,7. There's a task for that: T191491

This got done years ago. Closing.

In half an hour or so this will be live everywhere and you can check that it's working.

Going to go ahead and close this, since Turnilo is working.

https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#Access This is the problem (see the end of that section. Someone in Analytics will have to create an account for you manually.

I see some attempts at access with kzeta and 401 (meaning 'unauthorized'). Maybe there's more needed, like adding you to a whitelist of some kind. Hopefully Ottomata can shed some light on this.

kzeta would be the right name; it apparently does want your shell account over there. And that image looks like you found a bug. Adding @Ottomata who knows anything about superset's setup and service. I'll snoop around a bit though.

Done.

What user name did you give when trying to log in to Superset?

Adding @Smalyshev who has been working with the wikidata weekly dumps recently; if someone else is a better contact person, please feel free to remove yourself and add them.

If you view the wikitext for the page by choosing to edit the source, you will see that there are two categories listed at the bottom. Those are the categories in the wikitext of the dumped page as well.

Note that the three checklists are for whatever sres are working on these tasks; as we see different users doing the steps we will double-check them and then mark them off. No need for you to do so :-)

And this is delayed yet again; I have a 'run queries with variable substitution' script but in fact now what I need is a script that will run the real query, grab the thread id, show explain for that thread id, then kill that query. This would ensure that the explain we get is really what runs. Working on that script right now.

Who in Cloud Services is able to look at this?

Does this still need to happen or did the cleanup save us from further incidents?

While I don't want to open up this discussion here, it may be the case that some scripts moved out of puppet might share a repo, not necessarily every short 2-3 file script having its own repo. But in general I agree with the sentiment. I prefer to have individual modules with their own testing definitions, according to the needs of the module's scripts and the time and resources of the script maintainers.

I mean, it's fine, but maybe it's better to just provide them as is done on stat100? (5? 7?) via nfs mount from labstore1006 (7?). Looping @Bstorm in for her opinion, as she is one of the point people for those servers. I don't mean to slow this down at all, just if there's a simpler solution than copy them over, maybe we should go for it.

Excuse me for butting in at this late date but these files are already available from labstore1006,7 to labs instances and on stats100? (I forget which one now). Do you need them to be available somewhere else?

In T208750#4739774, @Addshore wrote:

In T208750#4739752, @ArielGlenn wrote:

Hrm, good question. What team would be using/benefiting from this work on the wmf side, any idea?

As it is Graphite the only team I think it relates to at all is Operations really.

In T208750#4739015, @Addshore wrote:

...

I can easily get WMDE manager sign off, I'm not really sure who from the WMF would have to / want to sign off on this.. ?

I hear ya, just looking to get it off our clinic duty dashboard :-)

Am I right in assuming this list's archives would be public?

@Legoktm Any idea what the non-wacky version of the process is going to look like? Should we care about this or close it?

@spatton any updates?

I'm holding this up for a bit because mmarble@wikimedia.org has two ldap entries, one with uid marble, cn Marble and the other with uid mmarble, cn Michal Marble. I can only find one user account creation in the logs at wikitech, which would be User:Marble. @Andrew can you shed any light on this?

I've verified that kzeta is now a member of the wmf ldap group. Please check that you can do things with it :-)

Verified that kchapman is now in the wmf ldap group. Please try using it for something :-)

I have verified that these three folks already are in the wmf ldap group as atgomex, nirzar and prtksxna respectively. Is anything else needed for them?

The patch that was pushes out cuts down the amount of time that a script may have failed to notice a db configuration change, but it doesn't address the root cause, which is rather harder.

@Addshore It looks like you need write access to /var/lib/carbon/whisper on graphite1001 and 2001, which means being able to remove or move things in there as the _graphite user. If so, we will need new group for this; I don't see any existing group that does what you want. Will that also cover merge of metrics?

@ayounsi What is needed for this to move ahead?