T256536: Puppetize mailman3 is almost done and we basically can have a puppetized mailman in the ☁️ but for production, it needs more work. This is to track and handle it.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| mailman3: Configure ferm | operations/puppet | production | +19 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Security | None | T181803 Stop storing Mailman passwords in plain text | ||
| Resolved | None | T118641 Implement proper AAA for lists.wikimedia.org (mailman) | |||
| Resolved | None | T190054 List archives on lists.wikimedia.org is not mobile friendly | |||
| Resolved | None | T115329 "From" at start of line becomes ">From" in pipermail | |||
| Resolved | None | T52864 Upgrade GNU Mailman from 2.1 to Mailman3 | |||
| Resolved | Legoktm | T277286 Make puppet for mailman3 ready for production |
Change 674393 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mailman3: Add ferm
Change 674393 merged by Legoktm:
[operations/puppet@production] mailman3: Configure ferm
I filed T278280: Setup monitoring for mailman3 for monitoring, I don't think that's a blocker for this and can be done iteratively.
Running into https://github.com/sqlalchemy/alembic/issues/551
root@lists1002:/etc/mailman3# mailman-wrapper conf
Traceback (most recent call last):
File "/usr/bin/mailman", line 11, in <module>
load_entry_point('mailman==3.2.1', 'console_scripts', 'mailman')()
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 716, in main
with self.make_context(prog_name, args, **extra) as ctx:
File "/usr/lib/python3/dist-packages/click/core.py", line 641, in make_context
self.parse_args(ctx, args)
File "/usr/lib/python3/dist-packages/click/core.py", line 1089, in parse_args
rest = Command.parse_args(self, ctx, args)
File "/usr/lib/python3/dist-packages/click/core.py", line 940, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
File "/usr/lib/python3/dist-packages/click/core.py", line 1477, in handle_parse_result
self.callback, ctx, self, value)
File "/usr/lib/python3/dist-packages/click/core.py", line 96, in invoke_param_callback
return callback(ctx, param, value)
File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 94, in initialize_config
initialize(value)
File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 218, in initialize
initialize_2(propagate_logs=propagate_logs)
File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 177, in initialize_2
config.db = getUtility(IDatabaseFactory, utility_name).create()
File "/usr/lib/python3/dist-packages/mailman/database/factory.py", line 55, in create
SchemaManager(database).setup_database()
File "/usr/lib/python3/dist-packages/mailman/database/factory.py", line 106, in setup_database
alembic.command.upgrade(alembic_cfg, 'head')
File "/usr/lib/python3/dist-packages/alembic/command.py", line 254, in upgrade
script.run_env()
File "/usr/lib/python3/dist-packages/alembic/script/base.py", line 427, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/lib/python3/dist-packages/alembic/util/pyfiles.py", line 81, in load_python_file
module = load_module_py(module_id, path)
File "/usr/lib/python3/dist-packages/alembic/util/compat.py", line 82, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/lib/python3/dist-packages/mailman/database/alembic/env.py", line 73, in <module>
run_migrations_online()
File "/usr/lib/python3/dist-packages/mailman/database/alembic/env.py", line 67, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/lib/python3/dist-packages/alembic/runtime/environment.py", line 836, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/lib/python3/dist-packages/alembic/runtime/migration.py", line 321, in run_migrations
for step in self._migrations_fn(heads, self):
File "/usr/lib/python3/dist-packages/alembic/command.py", line 243, in upgrade
return script._upgrade_revs(revision, rev)
File "/usr/lib/python3/dist-packages/alembic/script/base.py", line 336, in _upgrade_revs
revs = list(revs)
File "/usr/lib/python3/dist-packages/alembic/script/revision.py", line 645, in _iterate_revisions
requested_lowers = self.get_revisions(lower)
File "/usr/lib/python3/dist-packages/alembic/script/revision.py", line 299, in get_revisions
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/usr/lib/python3/dist-packages/alembic/script/revision.py", line 299, in <listcomp>
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/usr/lib/python3/dist-packages/alembic/script/revision.py", line 304, in get_revisions
for rev_id in resolved_id)
File "/usr/lib/python3/dist-packages/alembic/script/revision.py", line 304, in <genexpr>
for rev_id in resolved_id)
File "/usr/lib/python3/dist-packages/alembic/script/revision.py", line 355, in _revision_for_ident
revs = [x for x in self._revision_map
File "/usr/lib/python3/dist-packages/alembic/script/revision.py", line 356, in <listcomp>
if x and x.startswith(resolved_id)]
TypeError: startswith first arg must be str or a tuple of str, not intFor some reason, alembic_version is different on the cloud DB and on the prod DB:
Cloud:
MariaDB [mailman3]> describe alembic_version; +-------------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+-------------+------+-----+---------+-------+ | version_num | varchar(32) | NO | PRI | NULL | | +-------------+-------------+------+-----+---------+-------+ 1 row in set (0.002 sec)
Prod:
MariaDB [testmailman3]> describe alembic_version; +-------------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+---------------+------+-----+---------+-------+ | version_num | varbinary(32) | NO | PRI | NULL | | +-------------+---------------+------+-----+---------+-------+ 1 row in set (0.001 sec)
Cloud:
MariaDB [mailman3]> show create table alembic_version; +-----------------+----------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-----------------+----------------------------------------------------------------------------------------------------------------------------------------------+ | alembic_version | CREATE TABLE `alembic_version` ( `version_num` varchar(32) NOT NULL, PRIMARY KEY (`version_num`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 | +-----------------+----------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.001 sec)
Prod:
MariaDB [testmailman3]> show create table alembic_version; +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ | alembic_version | CREATE TABLE `alembic_version` ( `version_num` varbinary(32) NOT NULL, PRIMARY KEY (`version_num`) ) ENGINE=InnoDB DEFAULT CHARSET=binary | +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.004 sec)
Why isn't prod using the correct charset? It's set to be utf8mb4 in the connection URL....
OK, fixed it with:
MariaDB [testmailman3]> ALTER DATABASE testmailman3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; Query OK, 1 row affected (0.001 sec) MariaDB [testmailman3]> ALTER DATABASE testmailman3 DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci; Query OK, 1 row affected (0.001 sec)
Then dropped all the tables and had alembic recreate them. If I had to guess, it created the tables before &utf8mb4 was set and defaulted to binary instead. Uck.
This is basically done by virtue of having the test server working. Announcement to follow shortly.