Page MenuHomePhabricator
Create Task
Maniphest T200337

populate_adm_tree.php runs out of memory
Closed, ResolvedPublic
Actions

Description

From the 2018-07-24 logs

2018-07-24_13:13:47 Update admin structure tree...
Deleting previous data...
Rebuilding table...
PHP Fatal error:  Allowed memory size of 209715200 bytes exhausted (tried to allocate 32 bytes) in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/erfgoedbot/populate_adm_tree.php on line 45
PHP Stack trace:
PHP   1. {main}() /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/erfgoedbot/populate_adm_tree.php:0
PHP   2. mysqli_result->fetch_object() /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/erfgoedbot/populate_adm_tree.php:45

Fatal error: Allowed memory size of 209715200 bytes exhausted (tried to allocate 32 bytes) in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/erfgoedbot/populate_adm_tree.php on line 45

Call Stack:
    0.0055     266472   1. {main}() /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/erfgoedbot/populate_adm_tree.php:0
  206.6300  209637400   2. mysqli_result->fetch_object() /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/erfgoedbot/populate_adm_tree.php:45

Details

ProjectBranchLines +/-Subject
labs/tools/heritagemaster+1 -1Bump PHP memory limit from 800M to 1200M in populate_adm_tree.php
labs/tools/heritagemaster+3 -1Bump PHP memory limit from 200M to 800M in populate_adm_tree.php
Customize query in gerrit

Related Objects

Event Timeline

Lokal_Profil created this task.Jul 25 2018, 2:18 PM
Lokal_Profil added a comment.Jul 25 2018, 2:21 PM

It's worth noting that this is the only php file in the erfgoedbot directory. It's also unclear to me if anyone uses the adm_tree table (likely not since it's broken).

Our choices are either to drop the table and all associated scripts, or to rebuild this in python and solve the memory issue.

JeanFred added a comment.Jul 25 2018, 4:35 PM

As far as I remember, the admin_tree table was only ever used by the WLM mobile app. it is exposed by the api/includes/ApiAdminTree.php endpoint but well...

JeanFred added a subscriber: Multichill.Jul 25 2018, 4:36 PM

@Multichill Do you know if there were any other users of the admin_tree table beyond the deprecated WLM mobile app ?

Multichill added a comment.Jul 25 2018, 8:31 PM

I don't think anything else is using it. I would check two weeks of logs to see if anything tried to use it and if not, just kill it.

gerritbot added a subscriber: gerritbot.Jul 26 2018, 3:03 PM

Change 448054 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Bump PHP memory limit from 200M to 800M in populate_adm_tree.php

https://gerrit.wikimedia.org/r/448054

gerritbot added a project: Patch-For-Review.Jul 26 2018, 3:03 PM
gerritbot added a comment.Jul 27 2018, 11:27 AM

Change 448054 merged by jenkins-bot:
[labs/tools/heritage@master] Bump PHP memory limit from 200M to 800M in populate_adm_tree.php

https://gerrit.wikimedia.org/r/448054

Stashbot added a subscriber: Stashbot.Jul 31 2018, 9:12 AM

Mentioned in SAL (#wikimedia-cloud) [2018-07-31T09:12:50Z] <Lokal_Profil> Deploy latest from Git master: a230a6e (T200337), 12f5559, bad6d8d (T200410), 3c7e8d9, e00f1b3, a021cb9 (T200428), 0631408, d2e8246, b9ff533, a471c38, 1c5b9f6 [correction to previous]

Stashbot mentioned this in T200410: test_processSource_with_one_okay_picture_out_of_two hits live API.Jul 31 2018, 9:12 AM
Stashbot mentioned this in T200428: Add missing cz fields.
Lokal_Profil added a comment.Aug 20 2018, 8:51 AM

Note that we should look for this one in the logs then resolve

JeanFred moved this task from Backlog to Bugs on the Wiki-Loves-Monuments-Database board.Aug 20 2018, 9:05 AM
JeanFred added a project: Wiki-Loves-Monuments (2018).
2018-08-19_13:50:18 Update admin structure tree...
Deleting previous data...
Rebuilding table...
Querying data...
Inserting data...
Inserted 136750 admin zones in 157.65047192574 seconds.
JeanFred moved this task from Backlog to Done - in July on the Wiki-Loves-Monuments (2018) board.Aug 20 2018, 9:06 AM
Lokal_Profil removed a project: Patch-For-Review.Aug 23 2018, 9:44 PM
gerritbot added a comment.Aug 31 2018, 10:37 PM

Change 456760 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Bump PHP memory limit from 800M to 1200M in populate_adm_tree.php

https://gerrit.wikimedia.org/r/456760

gerritbot added a project: Patch-For-Review.Aug 31 2018, 10:37 PM
gerritbot added a comment.Sep 1 2018, 8:48 AM

Change 456760 merged by jenkins-bot:
[labs/tools/heritage@master] Bump PHP memory limit from 800M to 1200M in populate_adm_tree.php

https://gerrit.wikimedia.org/r/456760

Stashbot added a comment.Sep 1 2018, 8:23 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-01T20:23:06Z] <Lokal-Profil> Deploy latest from Git master: 17b27b3, c5122c7 (T200337), 6e85721, b48135e (T174871)

Stashbot mentioned this in T174871: Solve ErfgoedBot categorisation problem.Sep 1 2018, 8:23 PM
JeanFred closed this task as Resolved.Oct 19 2018, 3:05 PM
JeanFred claimed this task.
JeanFred merged a task: T141516: Heritage populate_adm_tree.php fails with memory exhaustion.
JeanFred added a subscriber: Aklapper.

This has not failed us in a while, closing as Resolved.

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL