- Provide a short summary of your proposed post for the Wikimedia Technical Blog. Blog readers will see this as the preview to your post:
Migration of the Wikimedia data infrastructure to Apache Bigtop.
The Analytics engineering team moved its infrastructure to Apache Bigtop (https://bigtop.apache.org/), a full open source Hadoop package distribution. The process involved testing and preparing the migration on a separate environment, a 400TB data backup and eventually a long upgrade day that involved almost one hundred hosts.
- Which topic type does your blog post fall under? See: https://www.mediawiki.org/wiki/Wikimedia_technical_blog_editorial_guidelines#Outlines_for_topics:
I'd say big picture. I'd love to highlight two things:
- The technical challenges migrating to the new distribution.
- The benefits that we'll have long term, since we switched to a full open source project that we can actively influence and contribute to (as opposed to before, that was more only consuming upstream packages).
- Which audience or audiences do you think your post is appropriate for?:
Anybody interested in the Hadoop ecosystem and how to run a full open source distribution about it. It will also be interesting as use case for Apache Bigtop upstream devs, they have been very helpful and interested in our use case, so this blog post will probably be shared from their blog as well eventually.
- Will you need assistance with writing your blog post, or do you already have a draft? If you have a draft, please provide a link here:
Still no draft, but we can start one asap, I wanted to get the green light from the Tech blog's admins first. Due to the length of the migration, it might be possible to do two blog posts (one for the preparation/testing and one for the upgrade itself).
- Does your post need to be published by a certain date?
- Do you have an image in mind for the featured image? You can learn more here: https://www.mediawiki.org/wiki/Wikimedia_technical_blog_editorial_guidelines#Images_used_in_your_post
Hadoop's logo is an elephant, so something like the following could be nice:
- Do you have any other questions or comments?
No questions :)
Once your request is received, a technical blog admin will review it and reach out to you through Phabricator.