I think this is the next step. The master is rater heavily relying on EntityPerPage though :/

amir@amir-GL552VW:~/Wikibase5$ grep -ir "EntityPerPage" . | wc -l
144

@Ladsgroup I see many mentions of EntityPerPag, but no actual logic using it. That was the idea. I hope all these places you found are now just dead wood, and can simply be deleted :)

Change 352544 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Drop EntityPerPage

https://gerrit.wikimedia.org/r/352544

Change 352574 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] Do not rebuild or make dumps of wb_entity_per_page

https://gerrit.wikimedia.org/r/352574

Change 352797 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] Do not make dumps of wb_entity_per_page

https://gerrit.wikimedia.org/r/352797

Change 352544 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Drop EntityPerPage

https://gerrit.wikimedia.org/r/352544

This needs announcement I guess.

Good point, should have had a DEPLOY notice in the commit message. Didn't think of that, sorry. Can you tell Lea?

Also, I just realized we should have a DB updater that drops the table when update.php is run.

In T140890#3257245, @daniel wrote:

Good point, should have had a DEPLOY notice in the commit message. Didn't think of that, sorry. Can you tell Lea?

I will let Lea know about it tomorrow :)

Also, I just realized we should have a DB updater that drops the table when update.php is run.

Yeah, I thought of that but thought let's do it in another patch. I'm on it.

Also, I just realized we should have a DB updater that drops the table when update.php is run.

Just wondering as I am somewhat involved in this kind of change for the first time: is there any policy/rule regarding dropping existing DB tables/columns? So we announce there is a change, and then put a script/command/whatever dropping no longer used DB part into master, so it drops stuff during the next git pull && php update.php?

@WMDE-leszek That is essentially it. This is a (potentially) breaking change for tools on labs. According to https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy#Stable_Public_APIs, it should be announced four weeks before we deploy it. I think in this case, we can get away with stopping updates to the table earlier (so we can drop the code), but keeping the table around for a few weeks. I doubt there are many tools on labs that use this table, if any.

Generally, database updates defined to run on update.php are not run on the live cluster. We would put the update code on master, and file a ticket for the DB change to be applied to the cluster manually. In this case, we should make sure the table is only dropped four weeks after we announced the change.

As you can see, I did not fully think through the deployment process for this. Ideally, we would have announced earlier, or we had held back the change that dropped the code for updating the table. Oops... But I don't think it's a big deal. As I said, I doubt many scripts rely on this table. Shouldn't be too much of a problem to stop writing to it.

Before we make any change to the live system we need to plan this properly. The assumption that no tool or script relies on this is wrong based on pretty vocal feedback we got.

@Lydia_Pintscher vocal feedback regarding wb_entity_per_page? Ah, I see that Magnus uses the table. Ok. But he has been long aware that this is going to happen.

Anyway: If we need to continue to update the table, we need to roll back https://gerrit.wikimedia.org/r/#/c/352544/. Deploying that code means we no longer update that table. Sorry for not paying attention here.

If we can get away with keeping the table, but not updating it, we can just announce it, and drop the table from the cluster in a few weeks.

In T140890#3257769, @WMDE-leszek wrote:

so it drops stuff during the next git pull && php update.php?

One thing, We can't do update.php in production. This will be a Schema-change and DBAs needs to do it. That requires a dedicated phab card and stuff like that (I will take care of it)

Personally, I like the idea of not updating the table for a while and dropping it later.

Change 353538 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: Thiemo Mättig (WMDE)):
[mediawiki/extensions/WikibaseLexeme@master] Adapt Lexeme to removed EntityPerPage table and services

https://gerrit.wikimedia.org/r/353538

Change 353538 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Adapt Lexeme to removed EntityPerPage table and services

https://gerrit.wikimedia.org/r/353538

We need a longer time to announce this change. I fear we'll need to revert this temporarily until we have made the announcement. We'll talk about the timeline in tomorrow's meeting. Sorry!

Change 353892 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@master] Revert "Drop EntityPerPage" as per https://phabricator.wikimedia.org/T140890#3263721.

https://gerrit.wikimedia.org/r/353892

Change 353893 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/WikibaseLexeme@master] Revert "Adapt Lexeme to removed EntityPerPage table and services" as per https://phabricator.wikimedia.org/T140890#3263721

https://gerrit.wikimedia.org/r/353893

Change 353892 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Revert "Drop EntityPerPage" as per https://phabricator.wikimedia.org/T140890#3263721.

https://gerrit.wikimedia.org/r/353892

Change 353893 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Revert "Adapt Lexeme to removed EntityPerPage table and services" as per https://phabricator.wikimedia.org/T140890#3263721

https://gerrit.wikimedia.org/r/353893

Change 353896 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@master] Re-apply "Drop EntityPerPage" [DNM]

https://gerrit.wikimedia.org/r/353896

Change 353897 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/WikibaseLexeme@master] Re-apply "Adapt Lexeme to removed EntityPerPage table and services"

https://gerrit.wikimedia.org/r/353897

I think this needs to be in monitor column or something like that until the two weeks in between announcement and deploy is done.

Change 381751 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Drop EntityPerPage

https://gerrit.wikimedia.org/r/381751

Has no longer updating the table been properly announced?

AFAIK Yes, @Lydia_Pintscher is that correct?

Change 382461 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: Thiemo Mättig (WMDE)):
[mediawiki/extensions/Wikibase@master] Remove a ton of unused code from MockEntityIdPager

https://gerrit.wikimedia.org/r/382461

In T140890#3652104, @hoo wrote:

Has no longer updating the table been properly announced?

Do you mean https://lists.wikimedia.org/pipermail/wikidata-tech/2017-June/001136.html ?

In T140890#3661547, @Lydia_Pintscher wrote:

In T140890#3652104, @hoo wrote:

Has no longer updating the table been properly announced?

Do you mean https://lists.wikimedia.org/pipermail/wikidata-tech/2017-June/001136.html ?

Sounds good to me. As the table will only slowly get out of touch with the rest, I guess some people even have another tiny grace period for migration.

Change 381751 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Drop EntityPerPage

https://gerrit.wikimedia.org/r/381751

Change 382461 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Remove a ton of unused code from MockEntityIdPager

https://gerrit.wikimedia.org/r/382461

Change 382666 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/WikibaseLexeme@master] Remove EntityPerPage

https://gerrit.wikimedia.org/r/382666

Change 382666 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Remove EntityPerPage

https://gerrit.wikimedia.org/r/382666

Change 382671 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/WikibaseMediaInfo@master] Remove EntityPerPage

https://gerrit.wikimedia.org/r/382671

Change 382671 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Remove EntityPerPage

https://gerrit.wikimedia.org/r/382671

Change 352797 merged by ArielGlenn:
[operations/puppet@production] Do not make dumps of wb_entity_per_page

https://gerrit.wikimedia.org/r/352797

Change 352574 merged by Jcrespo:
[operations/puppet@production] Do not rebuild wb_entity_per_page

https://gerrit.wikimedia.org/r/352574

Change 384723 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: Remove the cronjob for rebuilding entity_per_page table

https://gerrit.wikimedia.org/r/384723

Change 384723 merged by Jcrespo:
[operations/puppet@production] mediawiki: Remove the cronjob for rebuilding entity_per_page table

https://gerrit.wikimedia.org/r/384723

Change 353897 abandoned by Daniel Kinzler:
Re-apply "Adapt Lexeme to removed EntityPerPage table and services"

https://gerrit.wikimedia.org/r/353897

Change 353896 abandoned by Daniel Kinzler:
Re-apply "Drop EntityPerPage" [DNM]

https://gerrit.wikimedia.org/r/353896