Setting on read new for items causes increased worst response time and number of open connections
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Feb 25 2020, 8:50 PM

Description

After increasing the range of read new from up to Q2M to Q4M, this happened (until revert):

This led to increase on average response time from app servers:

Looking at tendril, it seems the reason is that there's no limit on number of entities that are given to the new term store (in this case 2511 entities at once):

P10524 (An Untitled Masterwork)

1	SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin */ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbit_item_id FROM `wbt_term_in_lang`, `wbt_text_in_lang`, `wbt_text`, `wbt_item_terms` WHERE (wbtl_text_in_lang_id=wbxl_id) AND (wbxl_text_id=wbx_id) AND wbit_item_id IN (976055, 976055, 595104, 595104, 7482576, 7482576, 206855, 33938, 33938, 11191, 11191, 202575, 202575, 190168, 190168, 244157, 244157, 528981, 528981, 572665, 572665, 239577, 239577, 1235705, 1235705, 155, 155, 4958978, 4958978, 183, 183, 801, 801, 916, 916, 4763816, 4763816, 244, 244, 4858664, 4858664, 1009, 1009, 5026498, 5026498, 657, 657, 5066540, 5066540, 117, 117, 5555511, 5555511, 833, 833, 6742081, 6742081, 1020, 1020, 6741510, 6741510, 1032, 1032, 7033119, 7033119, 252, 252, 96, 96, 258, 258, 40362, 40362, 7033069, 7033069, 843, 843, 7033066, 7033066, 865, 865, 928, 928, 7033076, 7033076, 159, 159, 1586750, 1586750, 148, 148, 5100253, 5100253, 41, 41, 5601669, 5601669, 668, 668, 6024281, 6024281, 902, 902, 114, 114, 953, 953, 57380, 57380, 361567, 361567, 202006, 202006, 265049, 265049, 962, 962, 3089038, 3089038, 3089173, 3089173, 3089176, 3089176, 3089050, 3089050, 7112384, 7112384, 6655, 5440850, 5440850, 500282, 500282, 4917, 4917, 1061257, 1061257, 860447, 860447, 6498663, 6498663, 35535, 35535, 376150, 376150, 3624078, 3624078, 6256, 6256, 3542, 3542, 1528507, 1528507, 193858, 193858, 1383497, 1383497, 9043, 9043, 11229, 11229, 320852, 320852, 337514, 337514, 424388, 424388, 117714, 117714, 503936, 503936, 532153, 532153, 429908, 429908, 130626, 130626, 739676, 739676, 503910, 503910, 506808, 506808, 682704, 682704, 534917, 534917, 3817714, 3817714, 376241, 376241, 842939, 842939, 682691, 682691, 878284, 878284, 682571, 682571, 657821, 657821, 748523, 748523, 387745, 387745, 464953, 464953, 815913, 815913, 836645, 836645, 503932, 503932, 836657, 836657, 836654, 836654, 682721, 682721, 503905, 503905, 503928, 503928, 503923, 503923, 503941, 503941, 463959, 463959, 682777, 682777, 145793, 145793, 509300, 509300, 43151, 43151, 6992735, 6992735, 4412, 4412, 512187, 512187, 25250, 25250, 39369, 39369, 3511689, 3511689, 125421, 125421, 56475, 56475, 33655, 33655, 33578, 33578, 34311, 34311, 34823, 34823, 56304, 56304, 268924, 268924, 3440392, 3440392, 529921, 529921, 425973, 425973, 3440381, 3440381, 3438816, 3438816, 3438755, 3438755, 3438770, 3438770, 3440387, 3440387, 3438764, 3438764, 3438780, 3438780, 1158134, 1158134, 3438789, 3438789, 3438799, 3438799, 3438807, 3438807, 3440480, 3440480, 3440464, 3440464, 3440412, 3440412, 3440432, 3440432, 3440486, 3440486, 3440457, 3440457, 1017152, 1017152, 3437656, 3437656, 3446643, 3446643, 3436680, 3436680, 3446632, 3446632, 3441285, 3441285, 3441264, 3441264, 3441199, 3441199, 3441277, 3441277, 3441272, 3441272, 3441249, 3441249, 3441293, 3441293, 2411063, 2411063, 3441230, 3441230, 3441299, 3441299, 3441244, 3441244, 3441324, 3441324, 3441315, 3441315, 3441256, 3441256, 4887064, 4887064, 4866152, 4866152, 3446631, 3446631, 4866150, 4866150, 4913807, 4913807, 3450648, 3450648, 3450802, 3450802, 3438345, 3438345, 3438718, 3438718, 3438405, 3438405, 3438367, 3438367, 3438729, 3438729, 3438705, 3438705, 3438350, 3438350, 3438747, 3438747, 3438334, 3438334, 3438361, 3438361, 3438708, 3438708, 3450131, 3450131, 3450142, 3450142, 3450179, 3450179, 3438725, 3438725, 3438738, 3438738, 6826835, 6826835, 3441392, 3441392, 3440527, 3440527, 3440506, 3440506, 3441344, 3441344, 1990210, 1990210, 3441382, 3441382, 3440525, 3440525, 3441166, 3441166, 3441193, 3441193, 3441188, 3441188, 3441172, 3441172, 3441387, 3441387, 3441350, 3441350, 3440492, 3440492, 3502244, 3502244, 2598797, 2598797, 4997234, 4997234, 3502308, 3502308, 2845320, 2845320, 7033021, 7033021, 6907057, 6907057, 4998097, 4998097, 3509628, 3509628, 7082551, 7082551, 7082547, 7082547, 5315230, 5315230, 5329945, 5329945, 5313310, 5313310, 5350418, 5350418, 3515061, 3515061, 3515067, 3515067, 3913381, 3913381, 3913390, 3913390, 3913378, 3913378, 3913394, 3913394, 5520366, 5520366, 3913363, 3913363, 3913375, 3913375, 3913366, 3913366, 3913384, 3913384, 3913387, 3913387, 3913398, 3913398, 5529310, 5529310, 5509187, 5509187, 3813405, 3813405, 3912777, 3912777, 3912756, 3912756, 3813403, 3813403, 3912765, 3912765, 3912761, 3912761, 3813833, 3813833, 3913279, 3913279, 3914492, 3914492, 3914482, 3914482, 3914501, 3914501, 3914459, 3914459, 3914462, 3914462, 3914453, 3914453, 3914475, 3914475, 3914470, 3914470, 3914490, 3914490, 3914448, 3914448, 3914494, 3914494, 3914485, 3914485, 3914497, 3914497, 3914467, 3914467, 3914465, 3914465, 3913328, 3913328, 3913281, 3913281, 3913342, 3913342, 3913321, 3913321, 3913297, 3913297, 3913307, 3913307, 3913326, 3913326, 3913283, 3913283, 3913324, 3913324, 3913293, 3913293, 3913304, 3913304, 3913350, 3913350, 3913331, 3913331, 3913357, 3913357, 3913345, 3913345, 3913354, 3913354, 3913336, 3913336, 3914008, 3914008, 3913977, 3913977, 3914441, 3914441, 5623219, 5623219, 3914025, 3914025, 3913994, 3913994, 3913997, 3913997, 3913963, 3913963, 3914004, 3914004, 3914021, 3914021, 5624046, 5624046, 3913968, 3913968, 3913979, 3913979, 3913990, 3913990, 3914027, 3914027, 3914019, 3914019, 3914048, 3914048, 3914392, 3914392, 3913987, 3913987, 3913972, 3913972, 5621686, 5621686, 3913985, 3913985, 3914066, 3914066, 3913957, 3913957, 3111668, 3111668, 3914011, 3914011, 3914882, 3914882, 3914876, 3914876, 3914899, 3914899, 3914879, 3914879, 3914891, 3914891, 3914907, 3914907, 3914897, 3914897, 3914890, 3914890, 3914905, 3914905, 3914902, 3914902, 3914894, 3914894, 3914875, 3914875, 3914871, 3914871, 3914878, 3914878, 3914910, 3914910, 3914407, 3914407, 3914439, 3914439, 3914427, 3914427, 3914417, 3914417, 3914387, 3914387, 3914431, 3914431, 3914436, 3914436, 3914383, 3914383, 3914420, 3914420, 3914395, 3914395, 3914402, 3914402, 3914404, 3914404, 3914423, 3914423, 3914409, 3914409, 3914381, 3914381, 3914398, 3914398, 3914415, 3914415, 3913934, 3913934, 3913936, 3913936, 3913925, 3913925, 3075144, 3075144, 3913946, 3913946, 3913929, 3913929, 3913931, 3913931, 3913943, 3913943, 3913940, 3913940, 3913920, 3913920, 3075137, 3075137, 3913953, 3913953, 3915352, 3915352, 3915388, 3915388, 3915391, 3915391, 3915330, 3915330, 3915372, 3915372, 3915336, 3915336, 3915308, 3915308, 3915355, 3915355, 3915349, 3915349, 3915365, 3915365, 3915319, 3915319, 3915338, 3915338, 3915334, 3915334, 3915342, 3915342, 3914920, 3914920, 3914962, 3914962, 3914924, 3914924, 3914914, 3914914, 3914930, 3914930, 3914956, 3914956, 5964576, 5964576, 3914936, 3914936, 3914934, 3914934, 3914939, 3914939, 3914953, 3914953, 6101921, 6101921, 6203123, 6203123, 6191995, 6191995, 3915846, 3915846, 3915850, 3915850, 3915849, 3915849, 3915848, 3915848, 3915845, 3915845, 3915847, 3915847, 3915843, 3915843, 3915462, 3915462, 3915449, 3915449, 3915382, 3915382, 3915392, 3915392, 3915395, 3915395, 3915451, 3915451, 3915445, 3915445, 3915448, 3915448, 3915378, 3915378, 3915380, 3915380, 3915454, 3915454, 3915854, 3915854, 6367410, 6367410, 3813297, 3813297, 3813306, 3813306, 3813392, 3813392, 3813394, 3813394, 3813400, 3813400, 6348685, 6348685, 3813288, 3813288, 3813396, 3813396, 3813281, 3813281, 3813398, 3813398, 3813301, 3813301, 3813290, 3813290, 4701277, 4701277, 6429589, 6429589, 6414512, 6414512, 6432515, 6432515, 6416310, 6416310, 6414422, 6414422, 7878635, 7878635, 6474529, 6474529, 7904095, 7904095, 7937086, 7937086, 35087, 35087, 6707102, 6707102, 35121, 35121, 35204, 35204, 56434, 56434, 35182, 35182, 35199, 35199, 35174, 35174, 35185, 35185, 35193, 35193, 35164, 35164, 35304, 35304, 35296, 35296, 35291, 35291, 35300, 35300, 35297, 35297, 35290, 35290, 35294, 35294, 56497, 56497, 35250, 35250, 35259, 35259, 56498, 56498, 35254, 35254, 56552, 56552, 56645, 56645, 35463, 35463, 35414, 35414, 35403, 35403, 35447, 35447, 35420, 35420, 35377, 35377, 35363, 35363, 35449, 35449, 35401, 35401, 35428, 35428, 35406, 35406, 35375, 35375, 35399, 35399, 35457, 35457, 35558, 35558, 35580, 35580, 35584, 35584, 35513, 35513, 35483, 35483, 35478, 35478, 56811, 56811, 35653, 35653, 7502, 7502, 35590, 35590, 35634, 35634, 35626, 35626, 35645, 35645, 35596, 35596, 35598, 35598, 35593, 35593, 35697, 35697, 35700, 35700, 35837, 35837, 35819, 35819, 35796, 35796, 35844, 35844, 35862, 35862, 35797, 35797, 35788, 35788, 36094, 36094, 36045, 36045, 35999, 35999, 36075, 36075, 36000, 36000, 36021, 36021, 36131, 36131, 56798, 56798, 56521, 56521, 35266, 35266, 35268, 35268, 35272, 35272, 35262, 35262, 36265, 36265, 36300, 36300, 36282, 36282, 36276, 36276, 36291, 36291, 36302, 36302, 36299, 36299, 36258, 36258, 36261, 36261, 36381, 36381, 36141, 36141, 36134, 36134, 36138, 36138, 36720, 36720, 36518, 36518, 36548, 36548, 36753, 36753, 36636, 36636, 36651, 36651, 36614, 36614, 36722, 36722, 36618, 36618, 36627, 36627, 36625, 36625, 36663, 36663, 36897, 36897, 36922, 36922, 36904, 36904, 36901, 36901, 56245, 56245, 56246, 56246, 56238, 56238, 56244, 56244, 56241, 56241, 56254, 56254, 56239, 56239, 56248, 56248, 56252, 56252, 56243, 56243, 56242, 56242, 43328, 43328, 36990, 36990, 56331, 56331, 56289, 56289, 56275, 56275, 56279, 56279, 56301, 56301, 56293, 56293, 56292, 56292, 56278, 56278, 56291, 56291, 56300, 56300, 56281, 56281, 56274, 56274, 56287, 56287, 56294, 56294, 56290, 56290, 56295, 56295, 56286, 56286, 56257, 56257, 56262, 56262, 56268, 56268, 56271, 56271, 56267, 56267, 56265, 56265, 56273, 56273, 56263, 56263, 32932, 32932, 33608, 33608, 33792, 33792, 34131, 34131, 34889, 34889, 34732, 34732, 34727, 34727, 34815, 34815, 34829, 34829, 34786, 34786, 34813, 34813, 34794, 34794, 34736, 34736, 34892, 34892, 34835, 34835, 34723, 34723, 34826, 34826, 34808, 34808, 34842, 34842, 34799, 34799, 34788, 34788, 34898, 34898, 34798, 34798, 34734, 34734, 7194026, 7194026, 7194057, 7194057, 7194098, 7194098, 9228931, 9228931, 34801, 34801, 34776, 34776, 36129, 36129, 34967, 34967, 34954, 34954, 35048, 35048, 34987, 34987, 35013, 35013, 35010, 35010, 35034, 35034, 34964, 34964, 34974, 34974, 34983, 34983, 34946, 34946, 1185686, 1185686, 15, 15, 203567, 203567, 102960, 102960, 3787, 3787, 8673, 8673, 791801, 3152127, 3152127, 1065, 1065, 7785, 7785, 7159, 7159, 7825, 7825, 47543, 47543, 191384, 191384, 827525, 827525, 656801, 656801, 1043527, 1043527, 899770, 899770, 340195, 340195, 384535, 384535, 294278, 294278, 8475, 8475, 842490, 842490, 233611, 233611, 7809, 7809, 5611262, 5611262, 2986587, 2986587, 17495, 17495, 14005, 3294867, 3294867, 207512, 207512, 4837690, 4837690, 1455797, 1455797, 577, 577, 5015542, 5015542, 1435215, 1435215, 5460604, 5460604, 2635070, 2635070, 237878, 237878, 8065199, 8065199, 54919, 54919, 1417780, 1417780, 3063249, 3063249, 8449, 8449, 712226, 2856156, 2856156, 7147719, 7147719, 1033, 1033, 605583, 605583, 8441, 8441, 467, 467, 499327, 499327, 146786, 146786, 1775415, 1775415, 1627120, 1627120, 550207, 550207, 602358, 602358, 2987354, 2987354, 9051, 9051, 976055, 976055, 595104, 595104, 7482576, 7482576, 33938, 33938, 202575, 202575, 190168, 190168, 244157, 244157, 528981, 528981, 572665, 572665, 239577, 239577, 1235705, 1235705, 4958978, 4958978, 916, 916, 4763816, 4763816, 4858664, 4858664, 5026498, 5026498, 657, 657, 5066540, 5066540, 5555511, 5555511, 6742081, 6742081, 6741510, 6741510, 7033119, 7033119, 40362, 40362, 7033069, 7033069, 7033066, 7033066, 7033076, 7033076, 1586750, 1586750, 5100253, 5100253, 5601669, 5601669, 6024281, 6024281, 57380, 57380, 361567, 361567, 202006, 202006, 3089038, 3089038, 3089173, 3089173, 3089176, 3089176, 3089050, 3089050, 7112384, 7112384, 5440850, 5440850, 500282, 500282, 1061257, 1061257, 3542, 3542, 1528507, 1528507, 1383497, 1383497, 320852, 320852, 337514, 337514, 424388, 424388, 117714, 117714, 503936, 503936, 532153, 532153, 429908, 429908, 130626, 130626, 739676, 739676, 503910, 503910, 506808, 506808, 682704, 682704, 534917, 534917, 3817714, 3817714, 376241, 376241, 842939, 842939, 682691, 682691, 878284, 878284, 682571, 682571, 657821, 657821, 748523, 748523, 387745, 387745, 464953, 464953, 815913, 815913, 836645, 836645, 503932, 503932, 836657, 836657, 836654, 836654, 503905, 503905, 503928, 503928, 503923, 503923, 503941, 503941, 463959, 463959, 682777, 682777, 145793, 145793, 509300, 509300, 43151, 43151, 6992735, 6992735, 4412, 4412, 512187, 512187, 3511689, 3511689, 125421, 125421, 56475, 56475, 33655, 33655, 33578, 33578, 34311, 34311, 34823, 34823, 56304, 56304, 268924, 268924, 3440392, 3440392, 529921, 529921, 425973, 425973, 3440381, 3440381, 3438816, 3438816, 3438755, 3438755, 3438770, 3438770, 3440387, 3440387, 3438764, 3438764, 3438780, 3438780, 1158134, 1158134, 3438789, 3438789, 3438799, 3438799, 3438807, 3438807, 3440480, 3440480, 3440464, 3440464, 3440412, 3440412, 3440432, 3440432, 3440486, 3440486, 3440457, 3440457, 1017152, 1017152, 3437656, 3437656, 3446643, 3446643, 3436680, 3436680, 3446632, 3446632, 3441285, 3441285, 3441264, 3441264, 3441199, 3441199, 3441277, 3441277, 3441272, 3441272, 3441249, 3441249, 3441293, 3441293, 2411063, 2411063, 3441230, 3441230, 3441299, 3441299, 3441244, 3441244, 3441324, 3441324, 3441315, 3441315, 3441256, 3441256, 4887064, 4887064, 4866152, 4866152, 3446631, 3446631, 4866150, 4866150, 4913807, 4913807, 3450648, 3450648, 3450802, 3450802, 3438345, 3438345, 3438718, 3438718, 3438405, 3438405, 3438367, 3438367, 3438729, 3438729, 3438705, 3438705, 3438350, 3438350, 3438747, 3438747, 3438334, 3438334, 3438361, 3438361, 3438708, 3438708, 3450131, 3450131, 3450142, 3450142, 3450179, 3450179, 3438725, 3438725, 3438738, 3438738, 6826835, 6826835, 3441392, 3441392, 3440527, 3440527, 3440506, 3440506, 3441344, 3441344, 1990210, 1990210, 3441382, 3441382, 3440525, 3440525, 3441166, 3441166, 3441193, 3441193, 3441188, 3441188, 3441172, 3441172, 3441387, 3441387, 3441350, 3441350, 3440492, 3440492, 3502244, 3502244, 2598797, 2598797, 4997234, 4997234, 3502308, 3502308, 2845320, 2845320, 7033021, 7033021, 6907057, 6907057, 4998097, 4998097, 3509628, 3509628, 7082551, 7082551, 7082547, 7082547, 5315230, 5315230, 5329945, 5329945, 5313310, 5313310, 5350418, 5350418, 3515061, 3515061, 3515067, 3515067, 3913381, 3913381, 3913390, 3913390, 3913378, 3913378, 3913394, 3913394, 5520366, 5520366, 3913363, 3913363, 3913375, 3913375, 3913366, 3913366, 3913384, 3913384, 3913387, 3913387, 3913398, 3913398, 5529310, 5529310, 5509187, 5509187, 3813405, 3813405, 3912777, 3912777, 3912756, 3912756, 3813403, 3813403, 3912765, 3912765, 3912761, 3912761, 3813833, 3813833, 3913279, 3913279, 3914492, 3914492, 3914482, 3914482, 3914501, 3914501, 3914459, 3914459, 3914462, 3914462, 3914453, 3914453, 3914475, 3914475, 3914470, 3914470, 3914490, 3914490, 3914448, 3914448, 3914494, 3914494, 3914485, 3914485, 3914497, 3914497, 3914467, 3914467, 3914465, 3914465, 3913328, 3913328, 3913281, 3913281, 3913342, 3913342, 3913321, 3913321, 3913297, 3913297, 3913307, 3913307, 3913326, 3913326, 3913283, 3913283, 3913324, 3913324, 3913293, 3913293, 3913304, 3913304, 3913350, 3913350, 3913331, 3913331, 3913357, 3913357, 3913345, 3913345, 3913354, 3913354, 3913336, 3913336, 3914008, 3914008, 3913977, 3913977, 3914441, 3914441, 5623219, 5623219, 3914025, 3914025, 3913994, 3913994, 3913997, 3913997, 3913963, 3913963, 3914004, 3914004, 3914021, 3914021, 5624046, 5624046, 3913968, 3913968, 3913979, 3913979, 3913990, 3913990, 3914027, 3914027, 3914019, 3914019, 3914048, 3914048, 3914392, 3914392, 3913987, 3913987, 3913972, 3913972, 5621686, 5621686, 3913985, 3913985, 3914066, 3914066, 3913957, 3913957, 3111668, 3111668, 3914011, 3914011, 3914882, 3914882, 3914876, 3914876, 3914899, 3914899, 3914879, 3914879, 3914891, 3914891, 3914907, 3914907, 3914897, 3914897, 3914890, 3914890, 3914905, 3914905, 3914902, 3914902, 3914894, 3914894, 3914875, 3914875, 3914871, 3914871, 3914878, 3914878, 3914910, 3914910, 3914407, 3914407, 3914439, 3914439, 3914427, 3914427, 3914417, 3914417, 3914387, 3914387, 3914431, 3914431, 3914436, 3914436, 3914383, 3914383, 3914420, 3914420, 3914395, 3914395, 3914402, 3914402, 3914404, 3914404, 3914423, 3914423, 3914409, 3914409, 3914381, 3914381, 3914398, 3914398, 3914415, 3914415, 3913934, 3913934, 3913936, 3913936, 3913925, 3913925, 3075144, 3075144, 3913946, 3913946, 3913929, 3913929, 3913931, 3913931, 3913943, 3913943, 3913940, 3913940, 3913920, 3913920, 3075137, 3075137, 3913953, 3913953, 3915352, 3915352, 3915388, 3915388, 3915391, 3915391, 3915330, 3915330, 3915372, 3915372, 3915336, 3915336, 3915308, 3915308, 3915355, 3915355, 3915349, 3915349, 3915365, 3915365, 3915319, 3915319, 3915338, 3915338, 3915334, 3915334, 3915342, 3915342, 3914920, 3914920, 3914962, 3914962, 3914924, 3914924, 3914914, 3914914, 3914930, 3914930, 3914956, 3914956, 5964576, 5964576, 3914936, 3914936, 3914934, 3914934, 3914939, 3914939, 3914953, 3914953, 6101921, 6101921, 6203123, 6203123, 6191995, 6191995, 3915846, 3915846, 3915850, 3915850, 3915849, 3915849, 3915848, 3915848, 3915845, 3915845, 3915847, 3915847, 3915843, 3915843, 3915462, 3915462, 3915449, 3915449, 3915382, 3915382, 3915392, 3915392, 3915395, 3915395, 3915451, 3915451, 3915445, 3915445, 3915448, 3915448, 3915378, 3915378, 3915380, 3915380, 3915454, 3915454, 3915854, 3915854, 6367410, 6367410, 3813297, 3813297, 3813306, 3813306, 3813392, 3813392, 3813394, 3813394, 3813400, 3813400, 6348685, 6348685, 3813288, 3813288, 3813396, 3813396, 3813281, 3813281, 3813398, 3813398, 3813301, 3813301, 3813290, 3813290, 4701277, 4701277, 6429589, 6429589, 6414512, 6414512, 6432515, 6432515, 6416310, 6416310, 6414422, 6414422, 7878635, 7878635, 6474529, 6474529, 7904095, 7904095, 7937086, 7937086, 35087, 35087, 6707102, 6707102, 35121, 35121, 35204, 35204, 56434, 56434, 35182, 35182, 35199, 35199, 35174, 35174, 35185, 35185, 35193, 35193, 35164, 35164, 35304, 35304, 35296, 35296, 35291, 35291, 35300, 35300, 35297, 35297, 35290, 35290, 35294, 35294, 56497, 56497, 35250, 35250, 35259, 35259, 56498, 56498, 35254, 35254, 56552, 56552, 56645, 56645, 35463, 35463, 35414, 35414, 35403, 35403, 35447, 35447, 35420, 35420, 35377, 35377, 35363, 35363, 35449, 35449, 35401, 35401, 35428, 35428, 35406, 35406, 35375, 35375, 35399, 35399, 35457, 35457, 35558, 35558, 35580, 35580, 35584, 35584, 35513, 35513, 35483, 35483, 35478, 35478, 56811, 56811, 35653, 35653, 7502, 7502, 35590, 35590, 35634, 35634, 35626, 35626, 35645, 35645, 35596, 35596, 35598, 35598, 35593, 35593, 35697, 35697, 35700, 35700, 35837, 35837, 35819, 35819, 35796, 35796, 35844, 35844, 35862, 35862, 35797, 35797, 35788, 35788, 36094, 36094, 36045, 36045, 35999, 35999, 36075, 36075, 36000, 36000, 36021, 36021, 36131, 36131, 56798, 56798, 56521, 56521, 35266, 35266, 35268, 35268, 35272, 35272, 35262, 35262, 36265, 36265, 36300, 36300, 36282, 36282, 36276, 36276, 36291, 36291, 36302, 36302, 36299, 36299, 36258, 36258, 36261, 36261, 36381, 36381, 36141, 36141, 36134, 36134, 36138, 36138, 36720, 36720, 36518, 36518, 36548, 36548, 36753, 36753, 36636, 36636, 36651, 36651, 36614, 36614, 36722, 36722, 36618, 36618, 36627, 36627, 36625, 36625, 36663, 36663, 36897, 36897, 36922, 36922, 36904, 36904, 36901, 36901, 56245, 56245, 56246, 56246, 56238, 56238, 56244, 56244, 56241, 56241, 56254, 56254, 56239, 56239, 56248, 56248, 56252, 56252, 56243, 56243, 56242, 56242, 43328, 43328, 36990, 36990, 56331, 56331, 56289, 56289, 56275, 56275, 56279, 56279, 56301, 56301, 56293, 56293, 56292, 56292, 56278, 56278, 56291, 56291, 56300, 56300, 56281, 56281, 56274, 56274, 56287, 56287, 56294, 56294, 56290, 56290, 56295, 56295, 56286, 56286, 56257, 56257, 56262, 56262, 56268, 56268, 56271, 56271, 56267, 56267, 56265, 56265, 56273, 56273, 56263, 56263, 32932, 32932, 33608, 33608, 33792, 33792, 34131, 34131, 34889, 34889, 34732, 34732, 34727, 34727, 34815, 34815, 34829, 34829, 34786, 34786, 34813, 34813, 34794, 34794, 34736, 34736, 34892, 34892, 34835, 34835, 34723, 34723, 34826, 34826, 34808, 34808, 34842, 34842, 34799, 34799, 34788, 34788, 34898, 34898, 34798, 34798, 34734, 34734, 7194026, 7194026, 7194057, 7194057, 7194098, 7194098, 9228931, 9228931, 34801, 34801, 34776, 34776, 36129, 36129, 34967, 34967, 34954, 34954, 35048, 35048, 34987, 34987, 35013, 35013, 35010, 35010, 35034, 35034, 34964, 34964, 34974, 34974, 34983, 34983, 34946, 34946, 203567, 203567, 102960, 102960, 3787, 3787, 3152127, 7785, 7785, 7159, 7159, 47543, 47543, 656801, 1043527, 340195, 340195, 294278, 294278, 233611, 233611, 2986587, 2986587, 207512, 207512, 1455797, 1455797, 5015542, 5015542, 1435215, 1435215, 2635070, 2635070, 237878, 237878, 8065199, 8065199, 2856156, 2856156, 7147719, 7147719, 605583, 605583, 1627120, 1627120, 2987354, 2987354) AND wbxl_language IN ('hu', 'en') AND (`wbit_term_in_lang_id` = wbtl_id)

SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin */ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbit_item_id FROM `wbt_term_in_lang`, `wbt_text_in_lang`, `wbt_text`, `wbt_item_terms` WHERE (wbtl_text_in_lang_id=wbxl_id) AND (wbxl_text_id=wbx_id) AND wbit_item_id IN (976055, 976055, 595104, 595104, 7482576, 7482576, 206855, 33938, 33938, 11191, 11191, 202575, 202575, 190168, 190168, 244157, 244157, 528981, 528981, 572665, 572665, 239577, 239577, 1235705, 1235705, 155, 155, 4958978, 4958978, 183, 183, 801, 801, 916, 916, 4763816, 4763816, 244, 244, 4858664, 4858664, 1009, 1009, 5026498, 5026498, 657, 657, 5066540, 5066540, 117, 117, 5555511, 5555511, 833, 833, 6742081, 6742081, 1020, 1020, 6741510, 6741510, 1032, 1032, 7033119, 7033119, 252, 252, 96, 96, 258, 258, 40362, 40362, 7033069, 7033069, 843, 843, 7033066, 7033066, 865, 865, 928, 928, 7033076, 7033076, 159, 159, 1586750, 1586750, 148, 148, 5100253, 5100253, 41, 41, 5601669, 5601669, 668, 668, 6024281, 6024281, 902, 902, 114, 114, 953, 953, 57380, 57380, 361567, 361567, 202006, 202006, 265049, 265049, 962, 962, 3089038, 3089038, 3089173, 3089173, 3089176, 3089176, 3089050, 3089050, 7112384, 7112384, 6655, 5440850, 5440850, 500282, 500282, 4917, 4917, 1061257, 1061257, 860447, 860447, 6498663, 6498663, 35535, 35535, 376150, 376150, 3624078, 3624078, 6256, 6256, 3542, 3542, 1528507, 1528507, 193858, 193858, 1383497, 1383497, 9043, 9043, 11229, 11229, 320852, 320852, 337514, 337514, 424388, 424388, 117714, 117714, 503936, 503936, 532153, 532153, 429908, 429908, 130626, 130626, 739676, 739676, 503910, 503910, 506808, 506808, 682704, 682704, 534917, 534917, 3817714, 3817714, 376241, 376241, 842939, 842939, 682691, 682691, 878284, 878284, 682571, 682571, 657821, 657821, 748523, 748523, 387745, 387745, 464953, 464953, 815913, 815913, 836645, 836645, 503932, 503932, 836657, 836657, 836654, 836654, 682721, 682721, 503905, 503905, 503928, 503928, 503923, 503923, 503941, 503941, 463959, 463959, 682777, 682777, 145793, 145793, 509300, 509300, 43151, 43151, 6992735, 6992735, 4412, 4412, 512187, 512187, 25250, 25250, 39369, 39369, 3511689, 3511689, 125421, 125421, 56475, 56475, 33655, 33655, 33578, 33578, 34311, 34311, 34823, 34823, 56304, 56304, 268924, 268924, 3440392, 3440392, 529921, 529921, 425973, 425973, 3440381, 3440381, 3438816, 3438816, 3438755, 3438755, 3438770, 3438770, 3440387, 3440387, 3438764, 3438764, 3438780, 3438780, 1158134, 1158134, 3438789, 3438789, 3438799, 3438799, 3438807, 3438807, 3440480, 3440480, 3440464, 3440464, 3440412, 3440412, 3440432, 3440432, 3440486, 3440486, 3440457, 3440457, 1017152, 1017152, 3437656, 3437656, 3446643, 3446643, 3436680, 3436680, 3446632, 3446632, 3441285, 3441285, 3441264, 3441264, 3441199, 3441199, 3441277, 3441277, 3441272, 3441272, 3441249, 3441249, 3441293, 3441293, 2411063, 2411063, 3441230, 3441230, 3441299, 3441299, 3441244, 3441244, 3441324, 3441324, 3441315, 3441315, 3441256, 3441256, 4887064, 4887064, 4866152, 4866152, 3446631, 3446631, 4866150, 4866150, 4913807, 4913807, 3450648, 3450648, 3450802, 3450802, 3438345, 3438345, 3438718, 3438718, 3438405, 3438405, 3438367, 3438367, 3438729, 3438729, 3438705, 3438705, 3438350, 3438350, 3438747, 3438747, 3438334, 3438334, 3438361, 3438361, 3438708, 3438708, 3450131, 3450131, 3450142, 3450142, 3450179, 3450179, 3438725, 3438725, 3438738, 3438738, 6826835, 6826835, 3441392, 3441392, 3440527, 3440527, 3440506, 3440506, 3441344, 3441344, 1990210, 1990210, 3441382, 3441382, 3440525, 3440525, 3441166, 3441166, 3441193, 3441193, 3441188, 3441188, 3441172, 3441172, 3441387, 3441387, 3441350, 3441350, 3440492, 3440492, 3502244, 3502244, 2598797, 2598797, 4997234, 4997234, 3502308, 3502308, 2845320, 2845320, 7033021, 7033021, 6907057, 6907057, 4998097, 4998097, 3509628, 3509628, 7082551, 7082551, 7082547, 7082547, 5315230, 5315230, 5329945, 5329945, 5313310, 5313310, 5350418, 5350418, 3515061, 3515061, 3515067, 3515067, 3913381, 3913381, 3913390, 3913390, 3913378, 3913378, 3913394, 3913394, 5520366, 5520366, 3913363, 3913363, 3913375, 3913375, 3913366, 3913366, 3913384, 3913384, 3913387, 3913387, 3913398, 3913398, 5529310, 5529310, 5509187, 5509187, 3813405, 3813405, 3912777, 3912777, 3912756, 3912756, 3813403, 3813403, 3912765, 3912765, 3912761, 3912761, 3813833, 3813833, 3913279, 3913279, 3914492, 3914492, 3914482, 3914482, 3914501, 3914501, 3914459, 3914459, 3914462, 3914462, 3914453, 3914453, 3914475, 3914475, 3914470, 3914470, 3914490, 3914490, 3914448, 3914448, 3914494, 3914494, 3914485, 3914485, 3914497, 3914497, 3914467, 3914467, 3914465, 3914465, 3913328, 3913328, 3913281, 3913281, 3913342, 3913342, 3913321, 3913321, 3913297, 3913297, 3913307, 3913307, 3913326, 3913326, 3913283, 3913283, 3913324, 3913324, 3913293, 3913293, 3913304, 3913304, 3913350, 3913350, 3913331, 3913331, 3913357, 3913357, 3913345, 3913345, 3913354, 3913354, 3913336, 3913336, 3914008, 3914008, 3913977, 3913977, 3914441, 3914441, 5623219, 5623219, 3914025, 3914025, 3913994, 3913994, 3913997, 3913997, 3913963, 3913963, 3914004, 3914004, 3914021, 3914021, 5624046, 5624046, 3913968, 3913968, 3913979, 3913979, 3913990, 3913990, 3914027, 3914027, 3914019, 3914019, 3914048, 3914048, 3914392, 3914392, 3913987, 3913987, 3913972, 3913972, 5621686, 5621686, 3913985, 3913985, 3914066, 3914066, 3913957, 3913957, 3111668, 3111668, 3914011, 3914011, 3914882, 3914882, 3914876, 3914876, 3914899, 3914899, 3914879, 3914879, 3914891, 3914891, 3914907, 3914907, 3914897, 3914897, 3914890, 3914890, 3914905, 3914905, 3914902, 3914902, 3914894, 3914894, 3914875, 3914875, 3914871, 3914871, 3914878, 3914878, 3914910, 3914910, 3914407, 3914407, 3914439, 3914439, 3914427, 3914427, 3914417, 3914417, 3914387, 3914387, 3914431, 3914431, 3914436, 3914436, 3914383, 3914383, 3914420, 3914420, 3914395, 3914395, 3914402, 3914402, 3914404, 3914404, 3914423, 3914423, 3914409, 3914409, 3914381, 3914381, 3914398, 3914398, 3914415, 3914415, 3913934, 3913934, 3913936, 3913936, 3913925, 3913925, 3075144, 3075144, 3913946, 3913946, 3913929, 3913929, 3913931, 3913931, 3913943, 3913943, 3913940, 3913940, 3913920, 3913920, 3075137, 3075137, 3913953, 3913953, 3915352, 3915352, 3915388, 3915388, 3915391, 3915391, 3915330, 3915330, 3915372, 3915372, 3915336, 3915336, 3915308, 3915308, 3915355, 3915355, 3915349, 3915349, 3915365, 3915365, 3915319, 3915319, 3915338, 3915338, 3915334, 3915334, 3915342, 3915342, 3914920, 3914920, 3914962, 3914962, 3914924, 3914924, 3914914, 3914914, 3914930, 3914930, 3914956, 3914956, 5964576, 5964576, 3914936, 3914936, 3914934, 3914934, 3914939, 3914939, 3914953, 3914953, 6101921, 6101921, 6203123, 6203123, 6191995, 6191995, 3915846, 3915846, 3915850, 3915850, 3915849, 3915849, 3915848, 3915848, 3915845, 3915845, 3915847, 3915847, 3915843, 3915843, 3915462, 3915462, 3915449, 3915449, 3915382, 3915382, 3915392, 3915392, 3915395, 3915395, 3915451, 3915451, 3915445, 3915445, 3915448, 3915448, 3915378, 3915378, 3915380, 3915380, 3915454, 3915454, 3915854, 3915854, 6367410, 6367410, 3813297, 3813297, 3813306, 3813306, 3813392, 3813392, 3813394, 3813394, 3813400, 3813400, 6348685, 6348685, 3813288, 3813288, 3813396, 3813396, 3813281, 3813281, 3813398, 3813398, 3813301, 3813301, 3813290, 3813290, 4701277, 4701277, 6429589, 6429589, 6414512, 6414512, 6432515, 6432515, 6416310, 6416310, 6414422, 6414422, 7878635, 7878635, 6474529, 6474529, 7904095, 7904095, 7937086, 7937086, 35087, 35087, 6707102, 6707102, 35121, 35121, 35204, 35204, 56434, 56434, 35182, 35182, 35199, 35199, 35174, 35174, 35185, 35185, 35193, 35193, 35164, 35164, 35304, 35304, 35296, 35296, 35291, 35291, 35300, 35300, 35297, 35297, 35290, 35290, 35294, 35294, 56497, 56497, 35250, 35250, 35259, 35259, 56498, 56498, 35254, 35254, 56552, 56552, 56645, 56645, 35463, 35463, 35414, 35414, 35403, 35403, 35447, 35447, 35420, 35420, 35377, 35377, 35363, 35363, 35449, 35449, 35401, 35401, 35428, 35428, 35406, 35406, 35375, 35375, 35399, 35399, 35457, 35457, 35558, 35558, 35580, 35580, 35584, 35584, 35513, 35513, 35483, 35483, 35478, 35478, 56811, 56811, 35653, 35653, 7502, 7502, 35590, 35590, 35634, 35634, 35626, 35626, 35645, 35645, 35596, 35596, 35598, 35598, 35593, 35593, 35697, 35697, 35700, 35700, 35837, 35837, 35819, 35819, 35796, 35796, 35844, 35844, 35862, 35862, 35797, 35797, 35788, 35788, 36094, 36094, 36045, 36045, 35999, 35999, 36075, 36075, 36000, 36000, 36021, 36021, 36131, 36131, 56798, 56798, 56521, 56521, 35266, 35266, 35268, 35268, 35272, 35272, 35262, 35262, 36265, 36265, 36300, 36300, 36282, 36282, 36276, 36276, 36291, 36291, 36302, 36302, 36299, 36299, 36258, 36258, 36261, 36261, 36381, 36381, 36141, 36141, 36134, 36134, 36138, 36138, 36720, 36720, 36518, 36518, 36548, 36548, 36753, 36753, 36636, 36636, 36651, 36651, 36614, 36614, 36722, 36722, 36618, 36618, 36627, 36627, 36625, 36625, 36663, 36663, 36897, 36897, 36922, 36922, 36904, 36904, 36901, 36901, 56245, 56245, 56246, 56246, 56238, 56238, 56244, 56244, 56241, 56241, 56254, 56254, 56239, 56239, 56248, 56248, 56252, 56252, 56243, 56243, 56242, 56242, 43328, 43328, 36990, 36990, 56331, 56331, 56289, 56289, 56275, 56275, 56279, 56279, 56301, 56301, 56293, 56293, 56292, 56292, 56278, 56278, 56291, 56291, 56300, 56300, 56281, 56281, 56274, 56274, 56287, 56287, 56294, 56294, 56290, 56290, 56295, 56295, 56286, 56286, 56257, 56257, 56262, 56262, 56268, 56268, 56271, 56271, 56267, 56267, 56265, 56265, 56273, 56273, 56263, 56263, 32932, 32932, 33608, 33608, 33792, 33792, 34131, 34131, 34889, 34889, 34732, 34732, 34727, 34727, 34815, 34815, 34829, 34829, 34786, 34786, 34813, 34813, 34794, 34794, 34736, 34736, 34892, 34892, 34835, 34835, 34723, 34723, 34826, 34826, 34808, 34808, 34842, 34842, 34799, 34799, 34788, 34788, 34898, 34898, 34798, 34798, 34734, 34734, 7194026, 7194026, 7194057, 7194057, 7194098, 7194098, 9228931, 9228931, 34801, 34801, 34776, 34776, 36129, 36129, 34967, 34967, 34954, 34954, 35048, 35048, 34987, 34987, 35013, 35013, 35010, 35010, 35034, 35034, 34964, 34964, 34974, 34974, 34983, 34983, 34946, 34946, 1185686, 1185686, 15, 15, 203567, 203567, 102960, 102960, 3787, 3787, 8673, 8673, 791801, 3152127, 3152127, 1065, 1065, 7785, 7785, 7159, 7159, 7825, 7825, 47543, 47543, 191384, 191384, 827525, 827525, 656801, 656801, 1043527, 1043527, 899770, 899770, 340195, 340195, 384535, 384535, 294278, 294278, 8475, 8475, 842490, 842490, 233611, 233611, 7809, 7809, 5611262, 5611262, 2986587, 2986587, 17495, 17495, 14005, 3294867, 3294867, 207512, 207512, 4837690, 4837690, 1455797, 1455797, 577, 577, 5015542, 5015542, 1435215, 1435215, 5460604, 5460604, 2635070, 2635070, 237878, 237878, 8065199, 8065199, 54919, 54919, 1417780, 1417780, 3063249, 3063249, 8449, 8449, 712226, 2856156, 2856156, 7147719, 7147719, 1033, 1033, 605583, 605583, 8441, 8441, 467, 467, 499327, 499327, 146786, 146786, 1775415, 1775415, 1627120, 1627120, 550207, 550207, 602358, 602358, 2987354, 2987354, 9051, 9051, 976055, 976055, 595104, 595104, 7482576, 7482576, 33938, 33938, 202575, 202575, 190168, 190168, 244157, 244157, 528981, 528981, 572665, 572665, 239577, 239577, 1235705, 1235705, 4958978, 4958978, 916, 916, 4763816, 4763816, 4858664, 4858664, 5026498, 5026498, 657, 657, 5066540, 5066540, 5555511, 5555511, 6742081, 6742081, 6741510, 6741510, 7033119, 7033119, 40362, 40362, 7033069, 7033069, 7033066, 7033066, 7033076, 7033076, 1586750, 1586750, 5100253, 5100253, 5601669, 5601669, 6024281, 6024281, 57380, 57380, 361567, 361567, 202006, 202006, 3089038, 3089038, 3089173, 3089173, 3089176, 3089176, 3089050, 3089050, 7112384, 7112384, 5440850, 5440850, 500282, 500282, 1061257, 1061257, 3542, 3542, 1528507, 1528507, 1383497, 1383497, 320852, 320852, 337514, 337514, 424388, 424388, 117714, 117714, 503936, 503936, 532153, 532153, 429908, 429908, 130626, 130626, 739676, 739676, 503910, 503910, 506808, 506808, 682704, 682704, 534917, 534917, 3817714, 3817714, 376241, 376241, 842939, 842939, 682691, 682691, 878284, 878284, 682571, 682571, 657821, 657821, 748523, 748523, 387745, 387745, 464953, 464953, 815913, 815913, 836645, 836645, 503932, 503932, 836657, 836657, 836654, 836654, 503905, 503905, 503928, 503928, 503923, 503923, 503941, 503941, 463959, 463959, 682777, 682777, 145793, 145793, 509300, 509300, 43151, 43151, 6992735, 6992735, 4412, 4412, 512187, 512187, 3511689, 3511689, 125421, 125421, 56475, 56475, 33655, 33655, 33578, 33578, 34311, 34311, 34823, 34823, 56304, 56304, 268924, 268924, 3440392, 3440392, 529921, 529921, 425973, 425973, 3440381, 3440381, 3438816, 3438816, 3438755, 3438755, 3438770, 3438770, 3440387, 3440387, 3438764, 3438764, 3438780, 3438780, 1158134, 1158134, 3438789, 3438789, 3438799, 3438799, 3438807, 3438807, 3440480, 3440480, 3440464, 3440464, 3440412, 3440412, 3440432, 3440432, 3440486, 3440486, 3440457, 3440457, 1017152, 1017152, 3437656, 3437656, 3446643, 3446643, 3436680, 3436680, 3446632, 3446632, 3441285, 3441285, 3441264, 3441264, 3441199, 3441199, 3441277, 3441277, 3441272, 3441272, 3441249, 3441249, 3441293, 3441293, 2411063, 2411063, 3441230, 3441230, 3441299, 3441299, 3441244, 3441244, 3441324, 3441324, 3441315, 3441315, 3441256, 3441256, 4887064, 4887064, 4866152, 4866152, 3446631, 3446631, 4866150, 4866150, 4913807, 4913807, 3450648, 3450648, 3450802, 3450802, 3438345, 3438345, 3438718, 3438718, 3438405, 3438405, 3438367, 3438367, 3438729, 3438729, 3438705, 3438705, 3438350, 3438350, 3438747, 3438747, 3438334, 3438334, 3438361, 3438361, 3438708, 3438708, 3450131, 3450131, 3450142, 3450142, 3450179, 3450179, 3438725, 3438725, 3438738, 3438738, 6826835, 6826835, 3441392, 3441392, 3440527, 3440527, 3440506, 3440506, 3441344, 3441344, 1990210, 1990210, 3441382, 3441382, 3440525, 3440525, 3441166, 3441166, 3441193, 3441193, 3441188, 3441188, 3441172, 3441172, 3441387, 3441387, 3441350, 3441350, 3440492, 3440492, 3502244, 3502244, 2598797, 2598797, 4997234, 4997234, 3502308, 3502308, 2845320, 2845320, 7033021, 7033021, 6907057, 6907057, 4998097, 4998097, 3509628, 3509628, 7082551, 7082551, 7082547, 7082547, 5315230, 5315230, 5329945, 5329945, 5313310, 5313310, 5350418, 5350418, 3515061, 3515061, 3515067, 3515067, 3913381, 3913381, 3913390, 3913390, 3913378, 3913378, 3913394, 3913394, 5520366, 5520366, 3913363, 3913363, 3913375, 3913375, 3913366, 3913366, 3913384, 3913384, 3913387, 3913387, 3913398, 3913398, 5529310, 5529310, 5509187, 5509187, 3813405, 3813405, 3912777, 3912777, 3912756, 3912756, 3813403, 3813403, 3912765, 3912765, 3912761, 3912761, 3813833, 3813833, 3913279, 3913279, 3914492, 3914492, 3914482, 3914482, 3914501, 3914501, 3914459, 3914459, 3914462, 3914462, 3914453, 3914453, 3914475, 3914475, 3914470, 3914470, 3914490, 3914490, 3914448, 3914448, 3914494, 3914494, 3914485, 3914485, 3914497, 3914497, 3914467, 3914467, 3914465, 3914465, 3913328, 3913328, 3913281, 3913281, 3913342, 3913342, 3913321, 3913321, 3913297, 3913297, 3913307, 3913307, 3913326, 3913326, 3913283, 3913283, 3913324, 3913324, 3913293, 3913293, 3913304, 3913304, 3913350, 3913350, 3913331, 3913331, 3913357, 3913357, 3913345, 3913345, 3913354, 3913354, 3913336, 3913336, 3914008, 3914008, 3913977, 3913977, 3914441, 3914441, 5623219, 5623219, 3914025, 3914025, 3913994, 3913994, 3913997, 3913997, 3913963, 3913963, 3914004, 3914004, 3914021, 3914021, 5624046, 5624046, 3913968, 3913968, 3913979, 3913979, 3913990, 3913990, 3914027, 3914027, 3914019, 3914019, 3914048, 3914048, 3914392, 3914392, 3913987, 3913987, 3913972, 3913972, 5621686, 5621686, 3913985, 3913985, 3914066, 3914066, 3913957, 3913957, 3111668, 3111668, 3914011, 3914011, 3914882, 3914882, 3914876, 3914876, 3914899, 3914899, 3914879, 3914879, 3914891, 3914891, 3914907, 3914907, 3914897, 3914897, 3914890, 3914890, 3914905, 3914905, 3914902, 3914902, 3914894, 3914894, 3914875, 3914875, 3914871, 3914871, 3914878, 3914878, 3914910, 3914910, 3914407, 3914407, 3914439, 3914439, 3914427, 3914427, 3914417, 3914417, 3914387, 3914387, 3914431, 3914431, 3914436, 3914436, 3914383, 3914383, 3914420, 3914420, 3914395, 3914395, 3914402, 3914402, 3914404, 3914404, 3914423, 3914423, 3914409, 3914409, 3914381, 3914381, 3914398, 3914398, 3914415, 3914415, 3913934, 3913934, 3913936, 3913936, 3913925, 3913925, 3075144, 3075144, 3913946, 3913946, 3913929, 3913929, 3913931, 3913931, 3913943, 3913943, 3913940, 3913940, 3913920, 3913920, 3075137, 3075137, 3913953, 3913953, 3915352, 3915352, 3915388, 3915388, 3915391, 3915391, 3915330, 3915330, 3915372, 3915372, 3915336, 3915336, 3915308, 3915308, 3915355, 3915355, 3915349, 3915349, 3915365, 3915365, 3915319, 3915319, 3915338, 3915338, 3915334, 3915334, 3915342, 3915342, 3914920, 3914920, 3914962, 3914962, 3914924, 3914924, 3914914, 3914914, 3914930, 3914930, 3914956, 3914956, 5964576, 5964576, 3914936, 3914936, 3914934, 3914934, 3914939, 3914939, 3914953, 3914953, 6101921, 6101921, 6203123, 6203123, 6191995, 6191995, 3915846, 3915846, 3915850, 3915850, 3915849, 3915849, 3915848, 3915848, 3915845, 3915845, 3915847, 3915847, 3915843, 3915843, 3915462, 3915462, 3915449, 3915449, 3915382, 3915382, 3915392, 3915392, 3915395, 3915395, 3915451, 3915451, 3915445, 3915445, 3915448, 3915448, 3915378, 3915378, 3915380, 3915380, 3915454, 3915454, 3915854, 3915854, 6367410, 6367410, 3813297, 3813297, 3813306, 3813306, 3813392, 3813392, 3813394, 3813394, 3813400, 3813400, 6348685, 6348685, 3813288, 3813288, 3813396, 3813396, 3813281, 3813281, 3813398, 3813398, 3813301, 3813301, 3813290, 3813290, 4701277, 4701277, 6429589, 6429589, 6414512, 6414512, 6432515, 6432515, 6416310, 6416310, 6414422, 6414422, 7878635, 7878635, 6474529, 6474529, 7904095, 7904095, 7937086, 7937086, 35087, 35087, 6707102, 6707102, 35121, 35121, 35204, 35204, 56434, 56434, 35182, 35182, 35199, 35199, 35174, 35174, 35185, 35185, 35193, 35193, 35164, 35164, 35304, 35304, 35296, 35296, 35291, 35291, 35300, 35300, 35297, 35297, 35290, 35290, 35294, 35294, 56497, 56497, 35250, 35250, 35259, 35259, 56498, 56498, 35254, 35254, 56552, 56552, 56645, 56645, 35463, 35463, 35414, 35414, 35403, 35403, 35447, 35447, 35420, 35420, 35377, 35377, 35363, 35363, 35449, 35449, 35401, 35401, 35428, 35428, 35406, 35406, 35375, 35375, 35399, 35399, 35457, 35457, 35558, 35558, 35580, 35580, 35584, 35584, 35513, 35513, 35483, 35483, 35478, 35478, 56811, 56811, 35653, 35653, 7502, 7502, 35590, 35590, 35634, 35634, 35626, 35626, 35645, 35645, 35596, 35596, 35598, 35598, 35593, 35593, 35697, 35697, 35700, 35700, 35837, 35837, 35819, 35819, 35796, 35796, 35844, 35844, 35862, 35862, 35797, 35797, 35788, 35788, 36094, 36094, 36045, 36045, 35999, 35999, 36075, 36075, 36000, 36000, 36021, 36021, 36131, 36131, 56798, 56798, 56521, 56521, 35266, 35266, 35268, 35268, 35272, 35272, 35262, 35262, 36265, 36265, 36300, 36300, 36282, 36282, 36276, 36276, 36291, 36291, 36302, 36302, 36299, 36299, 36258, 36258, 36261, 36261, 36381, 36381, 36141, 36141, 36134, 36134, 36138, 36138, 36720, 36720, 36518, 36518, 36548, 36548, 36753, 36753, 36636, 36636, 36651, 36651, 36614, 36614, 36722, 36722, 36618, 36618, 36627, 36627, 36625, 36625, 36663, 36663, 36897, 36897, 36922, 36922, 36904, 36904, 36901, 36901, 56245, 56245, 56246, 56246, 56238, 56238, 56244, 56244, 56241, 56241, 56254, 56254, 56239, 56239, 56248, 56248, 56252, 56252, 56243, 56243, 56242, 56242, 43328, 43328, 36990, 36990, 56331, 56331, 56289, 56289, 56275, 56275, 56279, 56279, 56301, 56301, 56293, 56293, 56292, 56292, 56278, 56278, 56291, 56291, 56300, 56300, 56281, 56281, 56274, 56274, 56287, 56287, 56294, 56294, 56290, 56290, 56295, 56295, 56286, 56286, 56257, 56257, 56262, 56262, 56268, 56268, 56271, 56271, 56267, 56267, 56265, 56265, 56273, 56273, 56263, 56263, 32932, 32932, 33608, 33608, 33792, 33792, 34131, 34131, 34889, 34889, 34732, 34732, 34727, 34727, 34815, 34815, 34829, 34829, 34786, 34786, 34813, 34813, 34794, 34794, 34736, 34736, 34892, 34892, 34835, 34835, 34723, 34723, 34826, 34826, 34808, 34808, 34842, 34842, 34799, 34799, 34788, 34788, 34898, 34898, 34798, 34798, 34734, 34734, 7194026, 7194026, 7194057, 7194057, 7194098, 7194098, 9228931, 9228931, 34801, 34801, 34776, 34776, 36129, 36129, 34967, 34967, 34954, 34954, 35048, 35048, 34987, 34987, 35013, 35013, 35010, 35010, 35034, 35034, 34964, 34964, 34974, 34974, 34983, 34983, 34946, 34946, 203567, 203567, 102960, 102960, 3787, 3787, 3152127, 7785, 7785, 7159, 7159, 47543, 47543, 656801, 1043527, 340195, 340195, 294278, 294278, 233611, 233611, 2986587, 2986587, 207512, 207512, 1455797, 1455797, 5015542, 5015542, 1435215, 1435215, 2635070, 2635070, 237878, 237878, 8065199, 8065199, 2856156, 2856156, 7147719, 7147719, 605583, 605583, 1627120, 1627120, 2987354, 2987354) AND wbxl_language IN ('hu', 'en') AND (`wbit_term_in_lang_id` = wbtl_id)

Details

Subject	Repo	Branch	Lines +/-
Revert read from the new term store again to Q2M	operations/mediawiki-config	master	+1 -1
Do prefetching entity ids on batches of 20 entity per query	mediawiki/extensions/Wikibase	wmf/1.35.0-wmf.20	+37 -18
Do prefetching entity ids on batches of 20 entity per query	mediawiki/extensions/Wikibase	wmf/1.35.0-wmf.21	+37 -18
Do prefetching entity ids on batches of 20 entity per query	mediawiki/extensions/Wikibase	master	+37 -18

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Addshore	T208425 [EPIC] Kill the wb_terms table
Resolved	ArielGlenn	T226167 audit public tables and make sure we dump them all
Resolved	Addshore	T219175 [Mega] - Migrate data from wb_terms to new schema
Resolved	Ladsgroup	T219123 Migrate to and read from new store for item terms
Resolved	Ladsgroup	T246159 Setting on read new for items causes increased worst response time and number of open connections

Event Timeline

Ladsgroup created this task.Feb 25 2020, 8:50 PM

Change 574863 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Do prefetching entity ids on batches of 9 entity per request

https://gerrit.wikimedia.org/r/574863

gerritbot added a project: Patch-For-Review.Feb 25 2020, 9:12 PM

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 25 2020, 9:14 PM

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Feb 25 2020, 9:15 PM

Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.

@Ladsgroup curious, do we know what the calling code is that creates such requests?

Also I'd note the 9 entity per query from T163544#3201562 is for a different query with different semantics so it would be worth checking with DBA again before jumping to use 9

In T246159#5918142, @Addshore wrote:

@Ladsgroup curious, do we know what the calling code is that creates such requests?

It's very likely lua, repo doesn't batch and this shows up when we turn on reading on clients so it's either lua or parser functions. It makes sense that it's lua

Also I'd note the 9 entity per query from T163544#3201562 is for a different query with different semantics so it would be worth checking with DBA again before jumping to use 9

I see. My idea was to keep it as similar as possible to the old term store and then slowly change it to our new term store but it if you think otherwise, I would wait for DBAs. I think 50 makes more sense TBH. cc. @jcrespo @Marostegui

In T246159#5917713, @gerritbot wrote:

Change 574863 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Do prefetching entity ids on batches of 9 entity per request

https://gerrit.wikimedia.org/r/574863

Is this changeset the one making the query on this task lighter?

In T246159#5918782, @Marostegui wrote:

In T246159#5917713, @gerritbot wrote:

Change 574863 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Do prefetching entity ids on batches of 9 entity per request

https://gerrit.wikimedia.org/r/574863

Is this changeset the one making the query on this task lighter?

Yup, it chunkifies "wbit_item_id IN" condition, what size. We are not sure yet.

So fun fact, the given doesn't de-duplicate entities. This needs to be fixed, After de-duplication, it's 655 which is better.
For the whole thing, the handler stats are:

Variable_name	Value
Handler_commit	1
Handler_delete	0
Handler_discover	0
Handler_external_lock	0
Handler_icp_attempts	36780
Handler_icp_match	36780
Handler_mrr_init	0
Handler_mrr_key_refills	0
Handler_mrr_rowid_refills	0
Handler_prepare	0
Handler_read_first	0
Handler_read_key	76717
Handler_read_last	0
Handler_read_next	36780
Handler_read_prev	0
Handler_read_retry	0
Handler_read_rnd	0
Handler_read_rnd_deleted	0
Handler_read_rnd_next	0
Handler_rollback	0
Handler_savepoint	0
Handler_savepoint_rollback	0
Handler_tmp_update	0
Handler_tmp_write	0
Handler_update	0
Handler_write	0

for the batch of 50: ordered and deduplicated:

SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin */ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbit_item_id FROM `wbt_term_in_lang`, `wbt_text_in_lang`, `wbt_text`, `wbt_item_terms` WHERE (wbtl_text_in_lang_id=wbxl_id) AND (wbxl_text_id=wbx_id) AND wbit_item_id IN (15, 41, 96, 114, 117, 148, 155, 159, 183, 244, 252, 258, 467, 577, 657, 668, 801, 833, 843, 865, 902, 916, 928, 953, 962, 1009, 1020, 1032, 1033, 1065, 3542, 3787, 4412, 4917, 6256, 6655, 7159, 7502, 7785, 7809, 7825, 8441, 8449, 8475, 8673, 9043, 9051, 11191, 11229, 14005) AND wbxl_language IN ('hu', 'en') AND (`wbit_term_in_lang_id` = wbtl_id)

The handler stats are:

Variable_name	Value
Handler_commit	1
Handler_delete	0
Handler_discover	0
Handler_external_lock	0
Handler_icp_attempts	15581
Handler_icp_match	15581
Handler_mrr_init	0
Handler_mrr_key_refills	0
Handler_mrr_rowid_refills	0
Handler_prepare	0
Handler_read_first	0
Handler_read_key	31665
Handler_read_last	0
Handler_read_next	15581
Handler_read_prev	0
Handler_read_retry	0
Handler_read_rnd	0
Handler_read_rnd_deleted	0
Handler_read_rnd_next	0
Handler_rollback	0
Handler_savepoint	0
Handler_savepoint_rollback	0
Handler_tmp_update	0
Handler_tmp_write	0
Handler_update	0
Handler_write	0

For the batch of 20:

SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin */ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbit_item_id FROM `wbt_term_in_lang`, `wbt_text_in_lang`, `wbt_text`, `wbt_item_terms` WHERE (wbtl_text_in_lang_id=wbxl_id) AND (wbxl_text_id=wbx_id) AND wbit_item_id IN (15, 41, 96, 114, 117, 148, 155, 159, 183, 244, 252, 258, 467, 577, 657, 668, 801, 833, 843, 865) AND wbxl_language IN ('hu', 'en') AND (`wbit_term_in_lang_id` = wbtl_id)

The handler stats are:

Variable_name	Value
Handler_commit	1
Handler_delete	0
Handler_discover	0
Handler_external_lock	0
Handler_icp_attempts	8330
Handler_icp_match	8330
Handler_mrr_init	0
Handler_mrr_key_refills	0
Handler_mrr_rowid_refills	0
Handler_prepare	0
Handler_read_first	0
Handler_read_key	16897
Handler_read_last	0
Handler_read_next	8330
Handler_read_prev	0
Handler_read_retry	0
Handler_read_rnd	0
Handler_read_rnd_deleted	0
Handler_read_rnd_next	0
Handler_rollback	0
Handler_savepoint	0
Handler_savepoint_rollback	0
Handler_tmp_update	0
Handler_tmp_write	0
Handler_update	0
Handler_write	0

And for the batch of ten:

SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin */ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbit_item_id FROM `wbt_term_in_lang`, `wbt_text_in_lang`, `wbt_text`, `wbt_item_terms` WHERE (wbtl_text_in_lang_id=wbxl_id) AND (wbxl_text_id=wbx_id) AND wbit_item_id IN (15, 41, 96, 114, 117, 148, 155, 159, 183, 244) AND wbxl_language IN ('hu', 'en') AND (`wbit_term_in_lang_id` = wbtl_id)

The handler stats are:

Variable_name	Value
Handler_commit	1
Handler_delete	0
Handler_discover	0
Handler_external_lock	0
Handler_icp_attempts	4441
Handler_icp_match	4441
Handler_mrr_init	0
Handler_mrr_key_refills	0
Handler_mrr_rowid_refills	0
Handler_prepare	0
Handler_read_first	0
Handler_read_key	8999
Handler_read_last	0
Handler_read_next	4441
Handler_read_prev	0
Handler_read_retry	0
Handler_read_rnd	0
Handler_read_rnd_deleted	0
Handler_read_rnd_next	0
Handler_rollback	0
Handler_savepoint	0
Handler_savepoint_rollback	0
Handler_tmp_update	0
Handler_tmp_write	0
Handler_update	0
Handler_write	0

Do you have latency data for each run? Normally clients report query execution time, which you should already have when getting the handlers.

Definitely using some extra cpu for deduplication and ordering on app servers may save a lot of bandwidth and latency overally.

In T246159#5919315, @jcrespo wrote:

Do you have latency data for each run? Normally clients report query execution time, which you should already have when getting the handlers.

For the main thing: 2520 rows in set (0.67 sec) (with duplication). 2520 rows in set (0.47 sec) (with de-duplication)
Fifty batch (with deduplication): 467 rows in set (0.27 sec)
Twenty batch (with deduplication): 220 rows in set (0.10 sec)
Ten batch (with deduplication): 109 rows in set (0.04 sec)

Definitely using some extra cpu for deduplication and ordering on app servers may save a lot of bandwidth and latency overally.

Fixed in the patch. Hopefully deployed soon.

I am almost made up my mind to give a suggestion, last question, I promise: could you provide an idea of frequency of this query (normally & at peak times). This will be the determinant factor on how much latency we can "spend" on it. I don't need exact values, just an order of magnitude (100, 1000, 10000?) per second.

In T246159#5919507, @jcrespo wrote:

I am almost made up my mind to give a suggestion, last question, I promise: could you provide an idea of frequency of this query (normally & at peak times). This will be the determinant factor on how much latency we can "spend" on it. I don't need exact values, just an order of magnitude (100, 1000, 10000?) per second.

So I expect it to be 200k per minute and it won't get super higher than that (caching absorb peaks like a sponge) but most of the queries don't need more than a couple of entities. The cases that need hundreds are rare but I can't give you an exact number on that one.

so 200K per minute overall is approximately 200000/60/3 (db1126 has a third of the main, non special traffic) = 1000 QPS max per server. I would categorize this as frequent, normally those should have a latency of 0.01 seconds or less in average (and for that, that meas not reading more than 1000 rows). There are not hard limits, this is approximately the aim to go for the average query. I believe up to 1000 writes and 10000 rows read can read efficiently on that time. This will put between 20 and 50 the batch size. The 9 advice was related to bad plan being chosen due to eq_dive_range_limit being 10, but as far I can see, it has no effect here due to the type of query.

The issue is that if that amount of reads have to happen anyway, just they will be sent in batches, unless we slow down the query rate somehow, this will fix nothing-- it will make the database happy, but will move the "slowdown" of the overall HTTP call to the app server.

Please take my advice as a very relative general one, it is the result on server performance what will put the limits, not me, I just can make predictions based on the data provided :-D

jcrespo mentioned this in T246232: Bad Wikidata query format in DatabaseTermInLangIdsResolver::selectTermsViaJoin.Feb 26 2020, 3:02 PM

Thanks, I put 20 for now.

In T246159#5919892, @jcrespo wrote:

The issue is that if that amount of reads have to happen anyway, just they will be sent in batches, unless we slow down the query rate somehow, this will fix nothing-- it will make the database happy, but will move the "slowdown" of the overall HTTP call to the app server.

Yeah but the new term store is by design slower for reads given that it's several tables not one. We put some caches in place to avoid it but if it's still not enough, I guess the next step is to add an APCu cache to reduce the network round trip time to save some time inside the app server but I'm running out of ideas if those are not enough.

Change 575054 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.21] Do prefetching entity ids on batches of 20 entity per query

https://gerrit.wikimedia.org/r/575054

Change 574863 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Do prefetching entity ids on batches of 20 entity per query

https://gerrit.wikimedia.org/r/574863

Change 575055 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.20] Do prefetching entity ids on batches of 20 entity per query

https://gerrit.wikimedia.org/r/575055

Change 575054 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.21] Do prefetching entity ids on batches of 20 entity per query

https://gerrit.wikimedia.org/r/575054

Change 575055 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.20] Do prefetching entity ids on batches of 20 entity per query

https://gerrit.wikimedia.org/r/575055

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.22; 2020-03-03).Feb 26 2020, 8:01 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-26T20:35:53Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.21/extensions/Wikibase/lib/includes/Store/Sql/Terms: SWAT: [[gerrit:575055|Do prefetching entity ids on batches of 20 entity per query (T246159)]] (duration: 01m 04s)

Change 575082 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Revert read from the new term store again to Q2M

https://gerrit.wikimedia.org/r/575082

Change 575082 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert read from the new term store again to Q2M

https://gerrit.wikimedia.org/r/575082

Maintenance_bot removed a project: Patch-For-Review.Feb 26 2020, 10:10 PM

Tarrow moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 27 2020, 10:22 AM

Looks like CPU exhaustion which could be, in part, due to T232446.

jcrespo mentioned this in Unknown Object (Task).Feb 27 2020, 6:05 PM

Other than the work that has been done here I'm going to close this with the following comment that i left somewhere else

In T219123#5924185, @Addshore wrote:
Spent a lot of time chatting with @jcrespo today.
After some experimenting it seems that the main issue that we have been seeing was probably caused by disk reads due to data not being in the db cache yet.
This slowness of disk reads was also added to as these db tables are compressed (at least on db1126).

I got to Q6 million for reads from clients 30 mins or so ago and everything seems stable.
I turned off the batch write migration this morning but turned it back on 10 mins ago (batch size 100) and everything still seems to be indicating that it is okay.

The code snippet for warming the cache of db1126 is below for the range 2-4 million, and can be run from the mwmaint servers.
for i in {2000000..4000000..100000}; do echo "$i..."; time sql wikidatawiki --host db1126 -- -e "SELECT   wbit_item_id as id,   wby_name as type,   wbxl_language as language,   wbx_text as text FROM wbt_item_terms LEFT JOIN wbt_term_in_lang ON wbit_term_in_lang_id = wbtl_id LEFT JOIN wbt_type ON wbtl_type_id = wby_id LEFT JOIN wbt_text_in_lang ON wbtl_text_in_lang_id = wbxl_id LEFT JOIN wbt_text ON wbxl_text_id = wbx_id WHERE wbit_item_id BETWEEN $i AND ($i + 100000)" > /dev/null; done
This cache warmer should be performed twice on each range shortly before turning on reads from the new store for that range.

The cache efficiency can be seen in the below snapshot where:
a) cache warm 2-4 million pass 1
b) cache warm 2-4 million pass 2
c) cache warm 4-6 million pass 1
d) cache warm 4-6 million pass 2

This can also be seen in the disk IO

Although I have started an initial cache warm for Q6-8 million I probably won't bump the read level higher today.

Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.Feb 27 2020, 7:15 PM

Addshore mentioned this in T251286: Investigate restricting the ability to create sql queries with bad patterns in mediawiki.Apr 28 2020, 2:55 PM

	F31647226: image.png
	Feb 27 2020, 6:23 PM

	F31647231: image.png
	Feb 27 2020, 6:23 PM

Setting on read new for items causes increased worst response time and number of open connectionsClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Setting on read new for items causes increased worst response time and number of open connections
Closed, ResolvedPublic
Actions

Related Objects
Search...