If you want, you can now lower CPU quota to 65 cores and RAM quota to 146.048 MB.

In T283293#7114669, @aborrero wrote:

Hi,

Let me see if I understand the request correctly. Are you talking about +36 CPU, +36 GB ram and +160 GB cinder storage?

Specifically:

• CPU from 45 to 81
• RAM from 182GB to 218GB
• Storage from 160GB to 320GB

Please, confirm this is your request.

I confirm. It also includes the new flavor.

Um, no. We can't move forward with transition yet. Afais, instance resizing isn't enabled yet, but that's a prerequisite. After that, we'll probably need a few days, depending on my and Doc Taxon's availability.

Should be fixed now.

I wonder why it worked with the new tclsh binary and tcl libraries on the old tcl container then.

Ok, so now my kubernetes webservice of type tcl doesn't work anymore, it is missing libmariadbclient.so.18.

I thought I needed the packages as runtime on the exec nodes. I don't know if anything has changed on them but I do no longer seem to need this as it just works fine now.

As it turns out, I found an old workaround that does away with the dependency of bc in the build process.

In T210519#4785439, @Bjs wrote:

The first sentence above of course applies only if the user has the corresponding rights, which I have. I also checked other cases: All my edits were auto-reviewed, only the moves from user namespace to article namespace not.

The repo apparently was archived in gerrit directly. This is now undone. There also was no +2 right for me, which is now added. A jenkins integration would be nice to have but in the meantime I can make do with submitting directly.

Despite https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL and IRC suggesting otherwise, the repo is still archived and gerrit still refuses patches.

Could you please un-archive labs/tools/giftbot?

You're right, it seems that we don't need so much RAM at the moment. At least it was necessary in the past or otherwise the instance would be really slow. I think we could first try with an xlarge instance.

I went through my table creation statements and added engine statements. I dropped four tables that I want to recreate and changed the engine of the remaining to InnoDB. Thank you for your prompt reaction!

< yuvipanda> !log tools set complex_values slots=300,release=trusty for tools-exec-gift-trusty-01.tools.eqiad.wmflabs
This seems to have done the trick, thank you!
@chasemp The old instance can be cleaned up now.

Killing the Precise instance (and defaulting to Trusty) shouldn't pose a problem. I switched my code to Trusty (I didn't even know anymore that I had to hardcode that) and I will test it with the next regular run starting on the 1st of March. (Also, I'm kinda swamped with school stuff but I'm glad to have found a minute for this.)

< annika> chasemp: […] I guess you mean: You will provide a trusty exec node with an appropriate queue? And I will just use that instead of the current one?
< chasemp> annika: basically, a second trusty node in the same queue for you to migrate to
< annika> that would be fine

So, we deleted the old instace before creating the new one anyway because we wanted to keep the name. You can now readjust the quota.

Considering my earlier comment (bigram instance), the needed numbers would be 1+8=9 cores and 2+36=38GB. But otherwise you're right. And I actually planned to do the transition without additional temporary resources. But it surely would give us more security.

I'm not aware of any script errors. Maybe there are db queries or other things that cause it to hang.

We need more RAM for data processing. We are not aware of other possibilities. Afais, "bigram" has even more RAM, so we'd like to have enough room for 1 small and 1 bigram instance.

Seems to work again.

I do.

@doctaxon apparently they do, but it seems the claim is unsubstantiated

(Ya.)

I had no illusions of resolving the ticket with my summary alone (which should be clear if you read my comment carefully). I just wanted to document what's happening. But thank you for mentioning the root mail (which I couldn't remember).

DrTrigon added me to the service group, I will keep him updated about anything I will do regarding the tool.

I've got an answer from DrTrigon: It's been too long ago, he doesn't remember how to configure the tool. He wishes the admins to add me. Because that's still impossible (yay), I reminded him of the process of adding me as a maintainer. I also offered him to reply in this ticket (which I hope to suffice).

Because there's no policy in place yet, per @Andrew, I tried and sent an e-mail to DrTrigon, who hopefully will respond.

Resolved for both of us.

There are also URLs that don't redirect to the domain root but show the same content as the domain root default document.

Especially interesting portions are:
dwllib.tcl, proc check_curl*: curl -gLksm200 -A $user_agent -w %{http_code} -o /dev/null $url
List of good status codes: 200 201 202 206 226 229 301 302 304 401 412 503 507 999 (and the next 4 lines)
The file "exceptions" contains tcl [string match] url patterns mostly of sites that don't want to be tested/crawled (they may have a tool labs block). This means that you cannot determine if they're dead or not. (I test with a throttling of 1 second, there may be other methods that prevent blocking.)

Well, you could add a check if the API response is valid json. If not, you should repeat the request after a reasonable timespan.

This error originates from the Wikimedia API.

2016-01-01 00:00 UTC and 2016-01-15 00:00 UTC: 0 0 1,15 * * jlocal ./dwla.sh

I have had the same problem (tools.giftbot):

The last run of my job started at 24/11/2015 21:15 UTC and ended at 22:43 (then I got the error that I lost connection). The code then tries to reconnect in a loop, then I get the error can't connect to mysql. It connects just fine when I (update the data file and) restart the job.

Please make the form respecting the settings (e.g. 30 days) when applying form changes.

https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/Cyberbot_II_5: "Future feature requests, such as detecting unmarked dead links, should be made under a subsequent BRFA." From that I conclude that the workload won't be overlapping with my efforts. What is consuming the resources is probably keeping all data in memory instead of writing it to disk (which should be preferable I guess?). A point which concerns me, is that the bot traverses over the complete article namespace via the API (for template transclusion and article text), which should better be done via the DB and/or dumps.

1. giftbot
2. tcl
3. webservice and 5 continuous jobs
4. yes
5. no
6. no

error:

mysqlreceive/db server: Lost connection to MySQL server during query
   while executing
"mysqlreceive [set db [get_db dewiki]] {select distinct el_to from externallinks join page on el_from = page_id where page_namespace = 0} url {
       if [ca..."
   (file "/data/project/giftbot/dwl1a.tcl" line 18)

The instance is now broken. You can't log in, when it tries to create the home directory, it fails.

Do it.

Yes, I can.