Page MenuHomePhabricator

Drop filejournal table from WMF
Closed, ResolvedPublic

Description

Schema Change:

All wikis
tables: filejournal
changes: Drop fj_new_sha1 field
commit status: Unknown

Progress

  • labswiki doesn't have this table
  • labtestwiki

Details

Reference
bz49195

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
bzimport raised the priority of this task from to Low.Nov 22 2014, 1:52 AM
bzimport set Reference to bz49195.
greg created this task.Jun 5 2013, 5:12 PM

Definitely drop this?

The field is still referenced at least once in core:

$ grep -nr fj_new_sha1 *
includes/filebackend/filejournal/DBFileJournal.php:71

greg added a comment.Jul 29 2013, 7:57 PM

CC'ing Asher as he's the one who added it:
https://wikitech.wikimedia.org/w/index.php?title=Schema_changes&diff=prev&oldid=44661

(/me just did a manual binary search on that page's history, ugh)

It looks like this is a core table, but it's not documented at https://www.mediawiki.org/wiki/Manual:Database_layout ...

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2015, 4:39 PM
Krenair removed Springle as the assignee of this task.Oct 4 2015, 2:52 AM
Reedy changed the task status from Open to Stalled.Apr 26 2017, 2:35 PM
Reedy removed a subscriber: Springle.
Restricted Application added projects: Multimedia, Commons. · View Herald TranscriptApr 26 2017, 2:35 PM
Restricted Application added a project: Multimedia. · View Herald TranscriptDec 8 2017, 7:43 PM
Marostegui moved this task from Triage to Backlog on the DBA board.Jul 27 2018, 4:00 PM
greg removed a subscriber: greg.Nov 22 2018, 8:36 AM
Marostegui edited projects, added Wikimedia-Rdbms; removed DBA.Nov 23 2018, 6:40 AM
Marostegui added subscribers: Banyek, Reedy, Marostegui.

This needs to be removed from code first:

includes/filebackend/filejournal/DBFileJournal.php:				'fj_new_sha1' => $entry['newSha1'],
maintenance/archives/patch-filejournal.sql:  fj_new_sha1 varbinary(32) NOT NULL default '',

I am removing the DBA tag until this is merged and good to go. I will remain subscribed to the task to be aware of its progress.

Krinkle renamed this task from Drop fj_new_sha1 field to Drop filejournal.fj_new_sha1 field.Jul 18 2019, 9:47 PM
Krinkle renamed this task from Drop filejournal.fj_new_sha1 field to Drop filejournal.fj_new_sha1 field from core and WMF.
Krinkle removed a project: Wikimedia-Rdbms.
aaron added a subscriber: aaron.Jul 18 2019, 9:51 PM

Dropping the field doesn't make sense, but dropping the whole table does. We do not use that class in production (and it is optional within MW core).

Krinkle renamed this task from Drop filejournal.fj_new_sha1 field from core and WMF to Drop filejournal table from core and WMF.Jul 18 2019, 9:53 PM
Krinkle renamed this task from Drop filejournal table from core and WMF to Drop filejournal table from WMF.
Krinkle changed the task status from Stalled to Open.
Krinkle updated the task description. (Show Details)
Krinkle added a project: DBA.
Marostegui updated the task description. (Show Details)

Removing the DBA tag per T51195#4769106ç
The table is still present on the archives/patch-filejournal.sql

cat archives/patch-filejournal.sql
-- File backend operation journal
CREATE TABLE /*_*/filejournal (
  -- Unique ID for each file operation
  fj_id bigint unsigned NOT NULL PRIMARY KEY auto_increment,
  -- UUID of the batch this operation belongs to
  fj_batch_uuid varbinary(32) NOT NULL,
  -- The registered file backend name
  fj_backend varchar(255) NOT NULL,
  -- The storage path that was affected (may be internal paths)
  fj_path blob NOT NULL,
  -- Primitive operation description (create/update/delete)
  fj_op varchar(16) NOT NULL default '',
  -- SHA-1 file content hash in base-36
  fj_new_sha1 varbinary(32) NOT NULL default '',
  -- Timestamp of the batch operation
  fj_timestamp varbinary(14) NOT NULL default ''
) /*$wgDBTableOptions*/;

CREATE INDEX /*i*/fj_batch_id ON /*_*/filejournal (fj_batch_uuid);
CREATE INDEX /*i*/fj_timestamp ON /*_*/filejournal (fj_timestamp);
aaron added a comment.Aug 12 2019, 6:26 PM

It's an optional table, not installed by update.php.

Maybe those can go in a different directory for clarity.

Marostegui added a project: DBA.

Ah! Ok :)
Didn't know
Thanls for the clarification

@aaron are you sure this table isn't created by default? I just checked the most recent wiki we created yuewiktionary T205714: Prepare and check storage layer for yuewiktionary which was created the 14th Nov 2018 and filejournal table is there.

-rw-rw---- 1 mysql mysql 128K Nov 14  2018 filejournal.ibd

I have checked because I don't want to delete this table everywhere if it is still going to be created upon new wikis creations.

Marostegui moved this task from Backlog to Next on the DBA board.Aug 13 2019, 11:11 AM

Confirmed with the wiki that was created past Wed 14th Aug 2019 (T210762#5413963) - this table is still created when a new wiki is created:

root@db1075.eqiad.wmnet[napwikisource]> show tables like 'filej%';
+----------------------------------+
| Tables_in_napwikisource (filej%) |
+----------------------------------+
| filejournal                      |
+----------------------------------+
1 row in set (0.00 sec)

Moving this task to our Blocked column until this is sorted and remove from any code reference, as I want to avoid the situation where we delete it now, and it still gets created on every new wiki.

Change 530847 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikimediaMaintenance@master] Stop adding filejournal table to new wikis

https://gerrit.wikimedia.org/r/530847

Change 530847 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMaintenance@master] Stop adding filejournal table to new wikis

https://gerrit.wikimedia.org/r/530847

Thanks for getting that merged @Reedy - this can now proceed then!

Mentioned in SAL (#wikimedia-operations) [2019-08-26T13:49:25Z] <marostegui> Rename table filejournal on enwiki on db2112 - T51195

I have renamed this table on db2112.enwiki and will leave it for a few hours to make sure nothing writes to it (as otherwise replication will break) before renaming it on a host that receive reads.

root@db2112.codfw.wmnet[enwiki]> rename table filejournal to T51195_filejournal;
Query OK, 0 rows affected (0.04 sec)

root@db2112.codfw.wmnet[enwiki]> show tables like 'T5%';
+------------------------+
| Tables_in_enwiki (T5%) |
+------------------------+
| T51195_filejournal     |
+------------------------+
1 row in set (0.03 sec)
Marostegui updated the task description. (Show Details)Aug 26 2019, 1:51 PM
Marostegui moved this task from Backlog to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2019-08-27T12:29:36Z] <marostegui> Rename table filejournal on enwiki on db1089 - T51195

Table renamed on db1089 which is a host with read traffic. Will leave it there for a couple of days whilst scanning for errors, to make sure nothing reads from it.

root@db1089.eqiad.wmnet[enwiki]> rename table filejournal to T51195_filejournal;
Query OK, 0 rows affected (0.01 sec)

root@db1089.eqiad.wmnet[enwiki]> show tables like 'T5%';
+------------------------+
| Tables_in_enwiki (T5%) |
+------------------------+
| T51195_filejournal     |
+------------------------+
1 row in set (0.00 sec)
Marostegui updated the task description. (Show Details)Aug 27 2019, 12:30 PM
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-09-02T07:17:00Z] <marostegui> Drop filejournal table on s6 - T51195

Marostegui updated the task description. (Show Details)Sep 2 2019, 7:19 AM
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-09-02T07:26:09Z] <marostegui> Drop filejournal table on s5 - T51195

Marostegui updated the task description. (Show Details)Sep 2 2019, 7:30 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-02T07:34:24Z] <marostegui> Drop filejournal table on s4 - T51195

Marostegui updated the task description. (Show Details)Sep 2 2019, 7:35 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-02T08:25:33Z] <marostegui> Drop filejournal table on s2 - T51195

Mentioned in SAL (#wikimedia-operations) [2019-09-02T08:27:39Z] <marostegui> Drop filejournal table on labtestwiki - T51195

Marostegui updated the task description. (Show Details)Sep 2 2019, 8:40 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-02T08:45:50Z] <marostegui> Drop filejournal table on s8 - T51195

Marostegui updated the task description. (Show Details)Sep 2 2019, 8:46 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-02T09:15:25Z] <marostegui> Drop filejournal table on s1 - T51195

Marostegui updated the task description. (Show Details)Sep 2 2019, 9:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-02T09:21:49Z] <marostegui> Drop filejournal table on s7 - T51195

Marostegui updated the task description. (Show Details)Sep 2 2019, 9:33 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-03T04:50:10Z] <marostegui> Drop filejournal table on s3 - T51195

Change 534392 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] realm.pp: Remove filejournal table from private list

https://gerrit.wikimedia.org/r/534392

Marostegui updated the task description. (Show Details)Sep 4 2019, 9:17 AM
Marostegui updated the task description. (Show Details)Sep 5 2019, 5:11 AM

Change 534392 merged by Marostegui:
[operations/puppet@production] realm.pp: Remove filejournal table from private list

https://gerrit.wikimedia.org/r/534392

Mentioned in SAL (#wikimedia-operations) [2019-09-05T05:14:55Z] <marostegui> Restart MySQL on codfw sanitariums (db2094 and db2095) to pick up new filters - T51195

Marostegui updated the task description. (Show Details)Sep 5 2019, 5:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-05T05:31:58Z] <marostegui> Restart MySQL on codfw sanitariums (db1124 and db1125) to pick up new filters - T51195

Marostegui closed this task as Resolved.Sep 5 2019, 5:36 AM
Marostegui updated the task description. (Show Details)

All done