Page MenuHomePhabricator
Create Task
Maniphest T378954

Build bigtop 1.5 packages for bookworm
Closed, ResolvedPublic
Actions

Description

We use Bigtop version 1.5 for our Hadoop distribution, as well as various other related tools such as the HIve metastore etc.

At the moment we have only built these packages for buster and bullseye, but we now need to start building them for bookworm.
This is particularly important for the Airflow migration project, where we wish to use spark3-submit and wish to use the hadoop-client package.

The build script for our bigtop packages is here: https://gitlab.wikimedia.org/repos/data-engineering/bigtop/-/blob/update_bigtop_1.5_build/build_all_bigtop_distros_wmf.sh?ref_type=heads
The current documentation is here: https://gitlab.wikimedia.org/repos/data-engineering/bigtop/-/blob/update_bigtop_1.5_build/build_all_bigtop_distros_wmf.sh?ref_type=heads

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Add the thirdparty/bigtop15 component to bookwormBtullisoperations/puppetproduction+1 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Draft: Update bigtop_1.5 to build on bookwormrepos/data-engineering/bigtop!5btullisupdate_bigtop_1.5_buildbranch-1.5
Customize query in GitLab

Related Objects

Event Timeline

BTullis created this task.Nov 4 2024, 12:44 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptNov 4 2024, 12:44 PM
BTullis triaged this task as High priority.Nov 4 2024, 12:45 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.10.19 - 2024.11.08) board.
brouberol added a parent task: T377928: Ensure that we can submit spark jobs via `spark3-submit` from airflow .Nov 4 2024, 12:46 PM
brouberol mentioned this in T377928: Ensure that we can submit spark jobs via `spark3-submit` from airflow .Nov 4 2024, 2:08 PM
brouberol mentioned this in T379266: Make sure we install python 3.10 in our airflow worker image.Nov 7 2024, 3:21 PM
Gehel edited projects, added: Data-Platform-SRE (2024.11.09 - 2024.11.29); removed: Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
BTullis removed a parent task: T377928: Ensure that we can submit spark jobs via `spark3-submit` from airflow .Nov 8 2024, 4:41 PM

Removing the parent task for this, since we have decided to continue running all airflow containers using bullseye, for now.

BTullis mentioned this in T379385: Upgrade Hadoop to version 3.3.6 and Hive to version 4.0.1.Nov 11 2024, 2:33 PM
BTullis added a comment.Nov 11 2024, 2:40 PM

So far, I have been able to build the toolchain for bookworm and I have been able to build packages for hive.

The hadoop packages are currently failing on an error related to a missing protobuf compiler.

I'll commit my changes to the toolchain soon, but in the meantime I will also try to compile sqoop.

BTullis added a comment.Nov 11 2024, 3:05 PM

Sqoop fails with:

/ws/output/sqoop/sqoop-1.4.6/build.xml:1094: Execute failed: java.io.IOException: Cannot run program "python2.7"

This is not unexpected, because there is no python 2.7 in bookworm at all.

MoritzMuehlenhoff subscribed.Nov 11 2024, 3:23 PM
In T378954#10309891, @BTullis wrote:

Sqoop fails with:

/ws/output/sqoop/sqoop-1.4.6/build.xml:1094: Execute failed: java.io.IOException: Cannot run program "python2.7"

This is not unexpected, because there is no python 2.7 in bookworm at all.

Does it only need python2.7 itself without other Python modules? If so, I could look into creating a custom python2.7 build for bookworm to unblock the build.

BTullis added a comment.Nov 13 2024, 1:05 PM
In T378954#10309966, @MoritzMuehlenhoff wrote:
In T378954#10309891, @BTullis wrote:

Sqoop fails with:

/ws/output/sqoop/sqoop-1.4.6/build.xml:1094: Execute failed: java.io.IOException: Cannot run program "python2.7"

This is not unexpected, because there is no python 2.7 in bookworm at all.

Does it only need python2.7 itself without other Python modules? If so, I could look into creating a custom python2.7 build for bookworm to unblock the build.

Thanks. Let's hold off for now, as we might not need it. We're beginning to look into planning the Hadoop 3 upgrade in T379748: Draft a project plan for the Hadoop version 3 upgrade.
Unfortunately, even the latest bigtop release (3.3.0) release doesn't support bookworm yet, so we might have to add that in the same way as we added bullseye support to 1.5.

BTullis added a comment.Nov 14 2024, 2:14 PM

I have succeeded in building the Hadoop packages for bookworm.

BUILD SUCCESSFUL in 29m 31s
71 actionable tasks: 71 executed
root@bb7f58776186:/ws# find output/ -type f -name *.deb
output/hadoop/hadoop-mapreduce-historyserver_2.10.2-2_amd64.deb
output/hadoop/hadoop-conf-pseudo_2.10.2-2_amd64.deb
output/hadoop/hadoop_2.10.2-2_amd64.deb
output/hadoop/hadoop-hdfs-fuse_2.10.2-2_amd64.deb
output/hadoop/hadoop-yarn-nodemanager_2.10.2-2_amd64.deb
output/hadoop/hadoop-hdfs-namenode_2.10.2-2_amd64.deb
output/hadoop/hadoop-hdfs-journalnode_2.10.2-2_amd64.deb
output/hadoop/hadoop-httpfs_2.10.2-2_amd64.deb
output/hadoop/hadoop-yarn-timelineserver_2.10.2-2_amd64.deb
output/hadoop/hadoop-hdfs-datanode_2.10.2-2_amd64.deb
output/hadoop/libhdfs0-dev_2.10.2-2_amd64.deb
output/hadoop/hadoop-doc_2.10.2-2_all.deb
output/hadoop/hadoop-yarn-proxyserver_2.10.2-2_amd64.deb
output/hadoop/hadoop-hdfs-secondarynamenode_2.10.2-2_amd64.deb
output/hadoop/hadoop-hdfs_2.10.2-2_amd64.deb
output/hadoop/hadoop-yarn-resourcemanager_2.10.2-2_amd64.deb
output/hadoop/libhdfs0_2.10.2-2_amd64.deb
output/hadoop/hadoop-mapreduce_2.10.2-2_amd64.deb
output/hadoop/hadoop-client_2.10.2-2_amd64.deb
output/hadoop/hadoop-yarn_2.10.2-2_amd64.deb
output/hadoop/hadoop-kms_2.10.2-2_amd64.deb
output/hadoop/hadoop-hdfs-zkfc_2.10.2-2_amd64.deb
root@bb7f58776186:/ws# cat /etc/debian_version 
12.8

There is a little tidying up to be done before I push these changes to our custom branch, but this is pretty hopeful now that I have got the hive and hadoop packages to build.

BTullis added a comment.EditedNov 14 2024, 2:17 PM

I also discovered this ticket in the Bigtop JIRA, which is about adding bookworm support. https://issues.apache.org/jira/projects/BIGTOP/issues/BIGTOP-3990

I could work on a patch against this bug to add bookworm support to bigtop, which might make it to the next patch release.

CodeReviewBot added a project: Patch-For-Review.Nov 15 2024, 10:29 AM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/bigtop/-/merge_requests/5

Draft: Update bigtop_1.5 to build on bookworm

BTullis added a comment.Nov 15 2024, 4:46 PM

I think that these packages are probably ready for testing.

repackaged/deb12/hadoop-hdfs-datanode_2.10.2-deb12-2_amd64.deb
repackaged/deb12/mahout-doc_0.13.0-deb12-2_all.deb
repackaged/deb12/bigtop-groovy_2.5.4-deb12-2_all.deb
repackaged/deb12/hive-webhcat-server_2.3.6-deb12-4_all.deb
repackaged/deb12/hadoop-mapreduce_2.10.2-deb12-2_amd64.deb
repackaged/deb12/libhdfs0-dev_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-yarn-proxyserver_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-hdfs-namenode_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-hdfs-journalnode_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-yarn-timelineserver_2.10.2-deb12-2_amd64.deb
repackaged/deb12/libhdfs0_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hive-jdbc_2.3.6-deb12-4_all.deb
repackaged/deb12/hadoop-hdfs_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-kms_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-hdfs-secondarynamenode_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-httpfs_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hbase-master_1.5.0-deb12-2_all.deb
repackaged/deb12/hadoop-yarn-resourcemanager_2.10.2-deb12-2_amd64.deb
repackaged/deb12/bigtop-utils_1.5.0-deb12-2_all.deb
repackaged/deb12/hive-server2_2.3.6-deb12-4_all.deb
repackaged/deb12/hive-hbase_2.3.6-deb12-4_all.deb
repackaged/deb12/mahout_0.13.0-deb12-2_all.deb
repackaged/deb12/hbase-rest_1.5.0-deb12-2_all.deb
repackaged/deb12/hadoop-doc_2.10.2-deb12-2_all.deb
repackaged/deb12/bigtop-jsvc_1.0.15-deb12-2_amd64.deb
repackaged/deb12/hive-metastore_2.3.6-deb12-4_all.deb
repackaged/deb12/hive-hcatalog-server_2.3.6-deb12-4_all.deb
repackaged/deb12/hbase-regionserver_1.5.0-deb12-2_all.deb
repackaged/deb12/hadoop-client_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-yarn_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-hdfs-zkfc_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hbase-doc_1.5.0-deb12-2_all.deb
repackaged/deb12/hadoop-hdfs-fuse_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-conf-pseudo_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hadoop-yarn-nodemanager_2.10.2-deb12-2_amd64.deb
repackaged/deb12/bigtop-tomcat_8.5.57-deb12-2_all.deb
repackaged/deb12/hadoop-mapreduce-historyserver_2.10.2-deb12-2_amd64.deb
repackaged/deb12/hive-hcatalog_2.3.6-deb12-4_all.deb
repackaged/deb12/hbase_1.5.0-deb12-2_amd64.deb
repackaged/deb12/hbase-thrift_1.5.0-deb12-2_all.deb
repackaged/deb12/hive_2.3.6-deb12-4_all.deb
repackaged/deb12/hive-webhcat_2.3.6-deb12-4_all.deb
MoritzMuehlenhoff added a comment.Nov 15 2024, 5:13 PM

This could a great test case for the new apt staging repo (to easily e.g. upgrade the Hadoop test cluster)! If you're interested, we can figure out the details next week.

BTullis added a comment.Nov 15 2024, 5:27 PM
In T378954#10327692, @MoritzMuehlenhoff wrote:

This could a great test case for the new apt staging repo (to easily e.g. upgrade the Hadoop test cluster)! If you're interested, we can figure out the details next week.

Nice, thanks. Yes, it could be a good use case, but I think that these packages are just not present in bookworm, so I'm not going to be overwriting anything if I import them as they are with reprepro.

I think that an particularly good use case for us would be for when we have to test conda-analytics upgrades. e.g. T354733: Create Conda Analytics environment including spark version 3.5.3 and T378819: Bump Iceberg to 1.3.1; the most recent version that still supports Spark 3.1.2
When working on T378819 I had to make a new version of conda-analytics to test on the hadoop-test cluster, but I could only overwrite what we had on reprepro 0.0.36 with 0.0.37.

When it turned out that the upgrade didn't have the effect we wanted, I now have to find the old deb and downgrade what's in reprepro and re-deploy to downgrade the test cluster again.
Having a staging apt repo for this kind of workflow would be much better.

gerritbot added a comment.Nov 18 2024, 11:43 AM

Change #1092210 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add the thirdparty/bigtop15 component to bookworm

https://gerrit.wikimedia.org/r/1092210

gerritbot added a comment.Nov 18 2024, 1:04 PM

Change #1092210 merged by Btullis:

[operations/puppet@production] Add the thirdparty/bigtop15 component to bookworm

https://gerrit.wikimedia.org/r/1092210

BTullis added a comment.Nov 18 2024, 1:12 PM

Added the packages to reprepro.

btullis@apt1002:~/deb12$ sudo -i reprepro -C thirdparty/bigtop15 includedeb bookworm-wikimedia `pwd`/*.deb
Exporting indices...

btullis@apt1002:~/deb12$ sudo -i reprepro -C thirdparty/bigtop15 list bookworm-wikimedia
bookworm-wikimedia|thirdparty/bigtop15|amd64: bigtop-groovy 2.5.4-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: bigtop-jsvc 1.0.15-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: bigtop-tomcat 8.5.57-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: bigtop-utils 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-client 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-conf-pseudo 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-doc 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-datanode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-fuse 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-journalnode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-namenode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-secondarynamenode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-zkfc 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-httpfs 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-kms 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-mapreduce 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-mapreduce-historyserver 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-nodemanager 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-proxyserver 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-resourcemanager 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-timelineserver 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hbase 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hbase-doc 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hbase-master 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hbase-regionserver 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hbase-rest 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hbase-thrift 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-hbase 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-hcatalog 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-hcatalog-server 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-jdbc 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-metastore 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-server2 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-webhcat 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: hive-webhcat-server 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|amd64: libhdfs0 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: libhdfs0-dev 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: mahout 0.13.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: mahout-doc 0.13.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: bigtop-groovy 2.5.4-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: bigtop-tomcat 8.5.57-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: bigtop-utils 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hadoop-doc 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hbase-doc 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hbase-master 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hbase-regionserver 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hbase-rest 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hbase-thrift 1.5.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hive 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-hbase 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-hcatalog 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-hcatalog-server 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-jdbc 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-metastore 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-server2 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-webhcat 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: hive-webhcat-server 2.3.6-deb12-4
bookworm-wikimedia|thirdparty/bigtop15|i386: mahout 0.13.0-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: mahout-doc 0.13.0-deb12-2
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 18 2024, 1:13 PM

I'll call this done for now. We can do the testing at a later date.

BTullis closed this task as Resolved.Nov 18 2024, 1:35 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 29 2024, 9:28 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits