Page MenuHomePhabricator

Planning for Xtools beta
Closed, ResolvedPublic3 Story Points

Description

I believe the best move for the Xtools rewrite would be to stage a beta. This task is to plan and execute that.

We should start by placing a banner on all the pages of the old Xtools:

Xtools is beta-testing a new version. Please test it here

And on the new Xtools:

This is a beta version of xtools. If you find any bugs, please report them to Phabricator

Now for the discussion part: How long should we run the two xtools in parallel before we replace the old xtools? When should be our start date (July 1?) ?

Event Timeline

kaldari set the point value for this task to 3.Jun 6 2017, 11:14 PM

Once we're up and running at xtools.wmflabs.org will we keep tools.wmflabs.org/xtools-dev as the staging area? Or would it be better to have xtools.wmflabs.org/dev? I'm just wondering, because want to set up an oauth consumer (or two) and need to specify the callback URL.

Sorry, only just realised we've got xtools-dev.wmflabs.org for that.

kaldari updated the task description. (Show Details)Jun 20 2017, 11:30 PM
kaldari updated the task description. (Show Details)

The new prod site is up and running, and the notice added there. I don't have access to old XTools, so can't add the notice there (I'd send a PR, but it looks like the code in the GH repo is out of date with what's actually running). @MusikAnimal can you give me access?

kaldari raised the priority of this task from Low to High.Jul 5 2017, 11:10 PM

There's a new prod server in town that needs setting up. I can try but I was hoping @Samwilson would be willing to do it since he set up the existing prod server :) See T169590#3422038 for more

Samwilson claimed this task.

The new prod server is set up, and the former live01 turned off (but not deleted). Start of documentation: https://wikitech.wikimedia.org/wiki/Tool:XTools

  • Do we need to back up the database?
  • Is the continuous deployment thing ok? It checks for updates every ten minutes and if there are any it gets them and then updates composer, runs migrations, clears the cache, and rebuilds assets. This means master should stay stable!! Once we're in proper production (i.e. after next week) I think it'd be good to return to a pull-request based workflow of development.
  • Should we set up awstats or similar for tracking 404s? Or just grep access.log when we're interested?

The new prod server is set up, and the former live01 turned off (but not deleted). Start of documentation: https://wikitech.wikimedia.org/wiki/Tool:XTools

Awesome, thank you! :)

  • Do we need to back up the database?

It's just usage statistics, but I would be a little sad if we lost say, a year's worth of data. Do you think this is a real risk?

  • Is the continuous deployment thing ok? It checks for updates every ten minutes and if there are any it gets them and then updates composer, runs migrations, clears the cache, and rebuilds assets. This means master should stay stable!! Once we're in proper production (i.e. after next week) I think it'd be good to return to a pull-request based workflow of development.

Neat-O! I think this is OK we will just have to be very disciplined by first testing big changes on xtools-dev before merging, etc. Either way I agree we should move back to a PR system (pointing the finger at myself).

  • Should we set up awstats or similar for tracking 404s? Or just grep access.log when we're interested?

I don't know how hard it would be to install awstats but it sounds pretty cool. We might could use it for other purposes as well. I was thinking that when the Beta period is over, we'd add the forwarding code to the legacy XTools codebase, only forwarding routes we know we support in the new version. So 404s shouldn't happen from that.

Matthewrbowker moved this task from Inbox to Working on the XTools board.Jul 11 2017, 8:43 PM

Well, there are no backups of ToolsDB, so it seems like we should. I'm not sure where they should live, but if someone tells me I'll set up a daily dump. They don't have to be private, do they?

I'll rejig the continuous deployment to pull the highest tag instead of master, and set up the staging server to stay up to date with master instead.

And awstats is pretty simple, I'll set it up at https://xtools.wmflabs.org/stats — is it okay if it's completely public? There are no real IPs in the logs are there?

This is AMAZING!!!! Thank you!

Well, there are no backups of ToolsDB, so it seems like we should. I'm not sure where they should live, but if someone tells me I'll set up a daily dump. They don't have to be private, do they?

No our usage data isn't private, if that's what you meant, but we could replicate the data to a DB on Tool Labs. We could use the same toolsdb connection. Could that work?

bd808 added a subscriber: bd808.Jul 12 2017, 4:58 AM

Once you settle on the general configuration for your servers, I'd be glad to help figure out if creating Puppet classes would be useful to make it easier to create new server instances in the future. Getting code into ops/puppet.git for a Cloud VPS project can be a bit of a pain, but it really does come in handy when you need to build a new VM quickly.

Yes please @bd808 I'd love to learn how to do that! If you teach me, I promise I'll write whatever I learn in docs on Wikitech. :-)

It's that "cattle, not pets" thing isn't it? :-( But I like pets....

I guess you already said there were no backups of toolsdb heh. We're only talking about usage data here, right? If that's the case we shouldn't worry too much

@MusikAnimal yeah it's not a huge problem if we lose it all is it? We're not going to start backing up access logs in general, so let's not worry about that DB.

I don't know if there's any problem with running the staging and the prod things with the same database user. I'd normally not do that, but I'm not sure how to create new users.

bd808 added a comment.Jul 12 2017, 5:27 AM

I don't know if there's any problem with running the staging and the prod things with the same database user. I'd normally not do that, but I'm not sure how to create new users.

The current 'official' policy for getting ToolsDB/web replica credentials is to create a tool account and copy the replica.my.cnf from there to your Cloud VPS project as needed. Not the most obvious or best system, but its what we have that works today. There is a task on the Striker board about figuring out how to make this a self-serve operation instead of a side-effect of tool account creation.

The current 'official' policy for getting ToolsDB/web replica credentials is to create a tool account and copy the replica.my.cnf from there to your Cloud VPS project as needed. Not the most obvious or best system, but its what we have that works today. There is a task on the Striker board about figuring out how to make this a self-serve operation instead of a side-effect of tool account creation.

Ah, well that's almost what we've done, so all is well. :-) I'll switch the prod to use the xtools user, and we'll be right.

bd808 added a comment.Jul 12 2017, 5:32 AM

Yes please @bd808 I'd love to learn how to do that! If you teach me, I promise I'll write whatever I learn in docs on Wikitech. :-)

Deal! Take a look at my calendar and find a time in my evening/your morning that we can chat to start the process. We can practice by making a MediaWiki-Vagrant role that makes it dead simple to setup a dev environment and then move on to the slower ops/puppet.git process.

It's that "cattle, not pets" thing isn't it? :-( But I like pets....

As someone (Nemo I think) pointed out on irc once, cattle are actually very valuable to the ranchers that raise them and their loss can be as damaging as losing a pet. On the other hand, being ready to replace a busted or outgrown server with a few mouse clicks is pretty nice. :)

If you're going to spend time rewriting these tools, why not just make them properly supported MediaWiki extensions or add them to MediaWiki core?

If you're going to spend time rewriting these tools, why not just make them properly supported MediaWiki extensions or add them to MediaWiki core?

That shipped has sailed. See T153112: Epic: Rewriting XTools and the associated workboard.

If you're going to spend time rewriting these tools, why not just make them properly supported MediaWiki extensions or add them to MediaWiki core?

That shipped has sailed. See T153112: Epic: Rewriting XTools and the associated workboard.

Eh, it looks like exactly what I'm describing is proposed or implemented for some features, for example T145912: Create new Special:RangeContributions page to support viewing contributions across an IP range and T146414: Create Special:AutoblockList. This is really encouraging! Thanks for the link to T153112. We should at least consider similar approaches for the remaining features/tools.

The prod server now uses the prod DB credentials, and I've created T170514 to track the Puppetization of the server config. (@bd808 I'll ping you to figure out a time when we can talk about that!)

I think the only thing left for this ticket is to add "Xtools is beta-testing a new version. Please test it [here]." to the old XTools, which we'll do on Monday. Is there any thing else?

Samwilson closed this task as Resolved.

The beta-testing notice has been added to old XTools:

A new version of XTools is being beta-tested. Please try it out at xtools.wmflabs.org

(This was added manually to modules/WebTool.php in the live code.)

DannyH moved this task from Untriaged to Archive on the Community-Tech board.Jul 18 2017, 10:44 PM
MusikAnimal moved this task from Working to Complete on the XTools board.Jul 23 2017, 7:34 PM
Meno25 removed a subscriber: Meno25.Aug 26 2017, 10:25 AM
Restricted Application added a project: User-Matthewrbowker. · View Herald TranscriptAug 26 2017, 10:25 AM