Page MenuHomePhabricator

Intermittent internal API errors with Flow
Closed, ResolvedPublic

Description

Pywikibot tests are failing lately with the following error:

======================================================================

ERROR: test_topic_moderation (tests.flow_tests.TestFlowTopic)

Test Topic functions about moderation.

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/wikimedia/pywikibot/tests/flow_tests.py", line 245, in test_topic_moderation

    self.assertFalse(topic_hidden.is_locked)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 220, in is_locked

    return self.root._current_revision['isLocked']

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 214, in root

    self._root = Post.fromJSON(self, self.uuid, self._data)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 65, in uuid

    self._load_uuid()

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 55, in _load_uuid

    self._uuid = self._load()['workflowId']

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 160, in _load

    self._data = self.site.load_topic(self, format)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/site/__init__.py", line 1390, in callee

    return fn(self, *args, **kwargs)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/site/__init__.py", line 7306, in load_topic

    data = req.submit()

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/data/api.py", line 2040, in submit

    if self._internal_api_error(code, error, result):

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/data/api.py", line 1910, in _internal_api_error

    raise e

APIMWException: internal_api_error_Flow\Exception\NoParserException: [d92f3ec5-f670-4abb-b814-f07f5c9f62b7] Exception caught: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:U5y53rn0dp6h70nw" failed: 503

[errorclass: Flow\Exception\NoParserException;

 *: Flow\Exception\NoParserException at /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(174)

#0 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(66): Flow\Conversion\Utils::parsoid(string, string, string, Title)

#1 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Model/AbstractRevision.php(430): Flow\Conversion\Utils::convert(string, string, string, Title)

#2 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Templating.php(159): Flow\Model\AbstractRevision->getContent(string)

#3 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/RevisionFormatter.php(289): Flow\Templating->getContent(Flow\Model\PostRevision, string)

#4 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/TopicFormatter.php(45): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, DerivativeContext)

#5 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(663): Flow\Formatter\TopicFormatter->formatApi(Flow\Model\Workflow, array, DerivativeContext)

#6 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(565): Flow\Block\TopicBlock->renderTopicApi(array)

#7 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlowBaseGet.php(33): Flow\Block\TopicBlock->renderApi(array)

#8 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlow.php(105): Flow\Api\ApiFlowBaseGet->execute()

#9 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(1580): Flow\Api\ApiFlow->execute()

#10 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(523): ApiMain->executeAction()

#11 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(494): ApiMain->executeActionWithErrorHandling()

#12 /srv/mediawiki/php-1.35.0-wmf.27/api.php(84): ApiMain->execute()

#13 /srv/mediawiki/w/api.php(3): require(string)

#14 {main};

 servedby: mw1317]

https://travis-ci.org/github/wikimedia/pywikibot/jobs/672473143#L11364

UPDATE 2020/04/10: see T249909 and discussion below; happens on mediawiki.org and from web UI (not just pywikibot).

Event Timeline

Dvorapa moved this task from Backlog to Upstream issues on the Pywikibot board.
Dvorapa updated the task description. (Show Details)
kostajh added a subscriber: kostajh.

This looks like a configuration error with Parsoid, probably related to the Parsoid-PHP project. You get the same error if trying to post a comment using VE on a Flow board.

Tagging @ssastry and @Jdforrester-WMF in case they know of what needs to be fixed in the config repo. Other group0 wikis (e.g. mediawiki.org) seem fine, so it seems likely to me that this is a config issue for testwiki and not something with the train.

Tagging @ssastry and @Jdforrester-WMF in case they know of what needs to be fixed in the config repo. Other group0 wikis (e.g. mediawiki.org) seem fine, so it seems likely to me that this is a config issue for testwiki and not something with the train.

Is this is a new issue (since this train)? Or has this been broken on testwiki for a while?

I've noticed this today, but it could fail for several days already (as Pywikibot tests were broken last week)

cscott added a subscriber: cscott.

Adding as a train blocker until/unless we can prove that this isn't related to a group0 deployment.

(It's likely as @kostajh suggests that this isn't train-related, but I'm trying to err on the side of caution.)

Full config diff:

diff -d wmf-config/config-cache/conf-production-testwiki.json wmf-config/config-cache/conf-production-mediawikiwiki.json
3,8c3,4
<         "accountcreator": {
<             "override-antispoof": true,
<             "tboverride": true
<         },
<         "filemover": {
<             "movefile": true
---
>         "*": {
>             "flow-hide": false
11,18c7,11
<             "upload_by_url": true,
<             "upload": true,
<             "pagelang": true
<         },
<         "templateeditor": {
<             "templateeditor": true,
<             "tboverride": true,
<             "editcontentmodel": true
---
>             "flow-create-board": true,
>             "upload": false,
>             "reupload": false,
>             "reupload-own": false,
>             "reupload-shared": false
20,23c13,18
<         "sysop": {
<             "deleterevision": true,
<             "templateeditor": true,
<             "securepoll-create-poll": true
---
>         "autoconfirmed": {
>             "flow-hide": true,
>             "upload": false,
>             "reupload": false,
>             "reupload-own": false,
>             "reupload-shared": false
25,26c20,21
<         "reviewer": {
<             "stablesettings": true
---
>         "autopatrolled": {
>             "autopatrol": true
28,31c23,27
<         "researcher": {
<             "browsearchive": true,
<             "deletedhistory": true,
<             "apihighlimits": true
---
>         "uploader": {
>             "upload": true,
>             "reupload": true,
>             "reupload-own": true,
>             "reupload-shared": true
33,34c29,30
<         "rollbacker": {
<             "rollback": true
---
>         "translationadmin": {
>             "autopatrol": true
36,39c32,35
<         "centralnoticeadmin": {
<             "banner-protect": true,
<             "centralnotice-admin": true,
<             "editinterface": true
---
>         "sysop": {
>             "reupload": true,
>             "reupload-own": true,
>             "reupload-shared": true
114c110
<         "default": 2
---
>         "default": 25
117c113
<         "default": 0.05
---
>         "default": 0.3
122a119,123
>         "sysop": [
>             "autopatrolled",
>             "uploader",
>             "ipblock-exempt"
>         ],
124,126c125,127
<             "researcher",
<             "centralnoticeadmin",
<             "flow-bot",
---
>             "autopatrolled",
>             "transwiki",
>             "import",
133,140d133
<         ],
<         "sysop": [
<             "filemover",
<             "rollbacker",
<             "accountcreator",
<             "confirmed",
<             "templateeditor",
<             "ipblock-exempt"
143c136
<     "wgAllowCopyUploads": true,
---
>     "wgAllowCopyUploads": false,
148c141
<     "wgAllowRequiringEmailForResets": false,
---
>     "wgAllowRequiringEmailForResets": true,
152c145
<     "wgAppleTouchIcon": "/static/apple-touch/wikipedia.png",
---
>     "wgAppleTouchIcon": "/static/apple-touch/mediawiki.png",
156c149
<     "wgArticlePlaceholderImageProperty": "P47",
---
>     "wgArticlePlaceholderImageProperty": "P18",
182c175
<     "wgCanonicalServer": "https://test.wikipedia.org",
---
>     "wgCanonicalServer": "https://www.mediawiki.org",
187c180
<     "wgCategoryCollation": "uca-default-u-kn",
---
>     "wgCategoryCollation": "uca-default",
197c190
<     "wgCheckUserEnableSpecialInvestigate": true,
---
>     "wgCheckUserEnableSpecialInvestigate": false,
216c209
<     "wgCirrusSearchCompletionSettings": "fuzzy",
---
>     "wgCirrusSearchCompletionSettings": "fuzzy-subphrases",
218,219c211,212
<         "build": false,
<         "use": false,
---
>         "build": true,
>         "use": true,
239c232
<     "wgCirrusSearchEnableCrossProjectSearch": true,
---
>     "wgCirrusSearchEnableCrossProjectSearch": false,
292,351c285
<     "wgCirrusSearchLanguageToWikiMap": {
<         "ar": "ar",
<         "ay": "ay",
<         "az": "az",
<         "bg": "bg",
<         "bn": "bn",
<         "ca": "ca",
<         "cs": "cs",
<         "da": "da",
<         "de": "de",
<         "el": "el",
<         "en": "en",
<         "es": "es",
<         "et": "et",
<         "fa": "fa",
<         "fi": "fi",
<         "fil": "tl",
<         "fr": "fr",
<         "gu": "gu",
<         "he": "he",
<         "hi": "hi",
<         "hr": "hr",
<         "hu": "hu",
<         "id": "id",
<         "it": "it",
<         "ja": "ja",
<         "km": "km",
<         "ko": "ko",
<         "lt": "lt",
<         "lv": "lv",
<         "mk": "mk",
<         "ml": "ml",
<         "mr": "mr",
<         "ms": "ms",
<         "nb": "no",
<         "nl": "nl",
<         "no": "no",
<         "pa": "pa",
<         "pl": "pl",
<         "pt": "pt",
<         "ro": "ro",
<         "ru": "ru",
<         "si": "si",
<         "sk": "sk",
<         "sl": "sl",
<         "sq": "sq",
<         "sv": "sv",
<         "sw": "sw",
<         "ta": "ta",
<         "te": "te",
<         "th": "th",
<         "tl": "tl",
<         "tr": "tr",
<         "uk": "uk",
<         "ur": "ur",
<         "vi": "vi",
<         "zh": "zh",
<         "zh-cn": "zh",
<         "zh-tw": "zh"
<     },
---
>     "wgCirrusSearchLanguageToWikiMap": [],
353,354c287,288
<         "user": 0,
<         "wiki": 0
---
>         "user": 5,
>         "wiki": 2.5
363c297,303
<     "wgCirrusSearchNamespaceWeights": [],
---
>     "wgCirrusSearchNamespaceWeights": {
>         "12": 0.9,
>         "100": 0.9,
>         "102": 0.9,
>         "104": 0.9,
>         "106": 0.9
>     },
377,383c317
<     "wgCirrusSearchReplicaGroup": {
<         "type": "roundrobin",
<         "groups": [
<             "psi",
<             "omega"
<         ]
<     },
---
>     "wgCirrusSearchReplicaGroup": "chi",
510a445,448
>         100,
>         102,
>         104,
>         106,
523d460
<         "upload.wikimedia.org",
527c464
<     "wgCopyUploadsFromSpecialUpload": true,
---
>     "wgCopyUploadsFromSpecialUpload": false,
548c485
<     "wgEnableAsyncUploads": true,
---
>     "wgEnableAsyncUploads": false,
550c487
<     "wgEnableDnsBlacklist": true,
---
>     "wgEnableDnsBlacklist": false,
580c517
<     "wgExportFromNamespaces": false,
---
>     "wgExportFromNamespaces": true,
586,587c523,530
<         "118": "Draft",
<         "119": "Draft_talk"
---
>         "100": "Manual",
>         "101": "Manual_talk",
>         "102": "Extension",
>         "103": "Extension_talk",
>         "104": "API",
>         "105": "API_talk",
>         "106": "Skin",
>         "107": "Skin_talk"
589a533
>         0,
601c545
<     "wgFavicon": "/static/favicon/black-globe.ico",
---
>     "wgFavicon": "/static/favicon/mediawiki.ico",
610c554,567
<     "wgForceUIMsgAsContentMsg": [],
---
>     "wgForceUIMsgAsContentMsg": [
>         "mw-mainpage-url",
>         "mw-cat-browser-url",
>         "mw-portal-url",
>         "mw-help-url",
>         "mw-manual-url",
>         "mw-faq-url",
>         "mw-download-url",
>         "mw-extensions-url",
>         "mw-bugzilla-url",
>         "mw-downloadfromsvn-url",
>         "mw-irc-url",
>         "mw-mailing lists-url"
>     ],
612c569
<         "local"
---
>         "shared"
619,647c576,578
<     "wgGEHelpPanelEnabled": true,
<     "wgGEHelpPanelHelpDeskTitle": "Wikipedia:Requests/Help_desk",
<     "wgGEHelpPanelLinks": [
<         {
<             "title": "EditingHelp",
<             "text": "Write a Wikipedia article using the right style",
<             "id": "style"
<         },
<         {
<             "title": "Wikipedia:Creation_and_usage_of_media_files",
<             "text": "Add an image",
<             "id": "image"
<         },
<         {
<             "title": "Wikipedia:Citing_sources",
<             "text": "Add a citation",
<             "id": "citation"
<         },
<         {
<             "title": "How_to_add_an_external_link",
<             "text": "How to add an external link",
<             "id": "externallink"
<         },
<         {
<             "title": "Help:Infobox",
<             "text": "How to add/edit the infobox",
<             "id": "infobox"
<         }
<     ],
---
>     "wgGEHelpPanelEnabled": false,
>     "wgGEHelpPanelHelpDeskTitle": "",
>     "wgGEHelpPanelLinks": [],
659,660c590,591
<     "wgGEHelpPanelViewMoreTitle": "Special:MyLanguage/Help:Contents",
<     "wgGEHomepageEnabled": true,
---
>     "wgGEHelpPanelViewMoreTitle": "",
>     "wgGEHomepageEnabled": false,
662c593
<     "wgGEHomepageMentorsList": "Wikipedia:Requests/Help desk/Mentors",
---
>     "wgGEHomepageMentorsList": "",
669c600
<     "wgGEHomepageTutorialTitle": "Help:Help",
---
>     "wgGEHomepageTutorialTitle": "",
671c602
<     "wgGENewcomerTasksGuidanceEnabled": true,
---
>     "wgGENewcomerTasksGuidanceEnabled": false,
695,700d625
<         "de",
<         "en",
<         "es",
<         "fr",
<         "ja",
<         "commons",
702,705c627,628
<         "incubator",
<         "strategy",
<         "wikinews:en",
<         "wikinews:es"
---
>         "w:en",
>         "usability"
707,709d629
<     "wgInterlanguageLinkCodeMap": {
<         "no": "nb"
<     },
716c636
<     "wgKartographerUseMarkerStyle": false,
---
>     "wgKartographerUseMarkerStyle": true,
718c638
<     "wgKartographerWikivoyageMode": false,
---
>     "wgKartographerWikivoyageMode": true,
796c716,717
<         "test"
---
>         "mw",
>         "mediawikiwiki"
803c724
<     "wgMFAppPackageId": "org.wikipedia",
---
>     "wgMFAppPackageId": false,
976c897
<     "wgMaxCredits": -1,
---
>     "wgMaxCredits": 0,
993c914
<     "wgMetaNamespace": "Wikipedia",
---
>     "wgMetaNamespace": "Project",
995c916
<     "wgMinervaAlwaysShowLanguageButton": true,
---
>     "wgMinervaAlwaysShowLanguageButton": false,
997c918
<     "wgMinervaCountErrors": true,
---
>     "wgMinervaCountErrors": false,
1008c929
<     "wgMobileUrlTemplate": "%h0.m.%h1.%h2",
---
>     "wgMobileUrlTemplate": "m.%h1.%h2",
1011,1016d931
<     "wgNamespaceAliases": {
<         "WP": 4,
<         "WT": 5,
<         "Wikipedia": 4,
<         "Wikipedia_talk": 5
<     },
1022,1024c937,944
<     "wgNamespacesToBeSearchedDefault": [
<         1
<     ],
---
>     "wgNamespacesToBeSearchedDefault": {
>         "12": 1,
>         "100": 1,
>         "102": 1,
>         "104": 1,
>         "106": 1,
>         "0": 1
>     },
1028a949,951
>         "0": 1,
>         "8": 0,
>         "14": 1,
1035c958
<         "8": true,
---
>         "15": true,
1041c964
<         "15": true,
---
>         "16": true,
1086c1009
<     "wgNoticeProject": "test",
---
>     "wgNoticeProject": "mediawiki",
1092,1093c1015
<             "enabled": true,
<             "excludeBots": true
---
>             "enabled": true
1096,1097c1018
<             "enabled": true,
<             "excludeBots": true
---
>             "enabled": true
1103c1024
<             "enabled": true,
---
>             "enabled": false,
1105,1106c1026
<                 0,
<                 118
---
>                 0
1108,1109c1028
<             "cleanParent": true,
<             "keepForever": true
---
>             "cleanParent": true
1112c1031
<             "enabled": true,
---
>             "enabled": false,
1114,1115c1033
<                 0,
<                 118
---
>                 0
1119,1122c1037
<             ],
<             "excludeBots": true,
<             "cleanParent": true,
<             "keepForever": true
---
>             ]
1125c1040
<     "wgOresUiEnabled": true,
---
>     "wgOresUiEnabled": false,
1127,1129d1041
<     "wgOverrideSiteFeed": {
<         "rss": "http://www.example.org/feedstuff.rss"
<     },
1131c1043
<     "wgPageAssessmentsSubprojects": true,
---
>     "wgPageAssessmentsSubprojects": false,
1135c1047
<     "wgPageImagesLeadSectionOnly": true,
---
>     "wgPageImagesLeadSectionOnly": false,
1137,1140c1049,1052
<     "wgPageTriageDraftNamespaceId": 118,
<     "wgPageTriageEnableCopyvio": true,
<     "wgPageTriageEnableCurationToolbar": true,
<     "wgPageTriageEnableOresFilters": true,
---
>     "wgPageTriageDraftNamespaceId": false,
>     "wgPageTriageEnableCopyvio": false,
>     "wgPageTriageEnableCurationToolbar": false,
>     "wgPageTriageEnableOresFilters": false,
1148c1060
<     "wgPopupsGateway": "restbaseHTML",
---
>     "wgPopupsGateway": "mwApiPlain",
1173c1085
<     "wgPopupsVirtualPageViews": true,
---
>     "wgPopupsVirtualPageViews": false,
1330a1243,1247
>         "sysop": [
>             "autopatrolled",
>             "uploader",
>             "ipblock-exempt"
>         ],
1332,1335c1249,1251
<             "sysop",
<             "researcher",
<             "centralnoticeadmin",
<             "flow-bot",
---
>             "autopatrolled",
>             "transwiki",
>             "import",
1340,1347d1255
<         ],
<         "sysop": [
<             "filemover",
<             "rollbacker",
<             "accountcreator",
<             "confirmed",
<             "templateeditor",
<             "ipblock-exempt"
1350,1352c1258
<     "wgRestAPIAdditionalRouteFiles": [
<         "includes/Rest/coreDevelopmentRoutes.json"
<     ],
---
>     "wgRestAPIAdditionalRouteFiles": [],
1357d1262
<         "templateeditor",
1360c1265
<     "wgRevisionCacheExpiry": 3600,
---
>     "wgRevisionCacheExpiry": 604800,
1376c1281
<     "wgServer": "//test.wikipedia.org",
---
>     "wgServer": "//www.mediawiki.org",
1379c1284
<     "wgShowExceptionDetails": true,
---
>     "wgShowExceptionDetails": false,
1383c1288
<     "wgSitename": "Wikipedia",
---
>     "wgSitename": "MediaWiki",
1410c1315
<     "wgTorTagChanges": true,
---
>     "wgTorTagChanges": false,
1415,1416c1320,1321
<     "wgUploadNavigationUrl": false,
<     "wgUploadPath": "//upload.wikimedia.org/wikipedia/test",
---
>     "wgUploadNavigationUrl": "//commons.wikimedia.org/wiki/Special:UploadWizard",
>     "wgUploadPath": "//upload.wikimedia.org/wikipedia/mediawiki",
1427c1332
<     "wgUseContributionTracking": true,
---
>     "wgUseContributionTracking": false,
1439c1344
<     "wgVectorResponsive": true,
---
>     "wgVectorResponsive": false,
1518,1519c1423,1424
<     "wgWMEInukaPageViewCookiesDomain": "wikipedia.org",
<     "wgWMEInukaPageViewEnabled": true,
---
>     "wgWMEInukaPageViewCookiesDomain": "",
>     "wgWMEInukaPageViewEnabled": false,
1524c1429
<     "wgWMEUnderstandingFirstDay": true,
---
>     "wgWMEUnderstandingFirstDay": false,
1529,1531c1434
<         7,
<         118,
<         119
---
>         7
1540c1443
<     "wgWelcomeSurveyEnabled": true,
---
>     "wgWelcomeSurveyEnabled": false,
1572,1573d1474
<         "wikipedia",
<         "all",
1575,1576d1475
<         "wikidataclient-test",
<         "testwikis",
1579d1477
<         "pageassessments",
1582d1479
<         "growthexperiments",
1585c1482,1483
<         "flow-labs",
---
>         "commonsuploads",
>         "cirrussearch-big-indices",
1587c1485,1486
<         "all-labs"
---
>         "special",
>         "all"
1590c1489
<     "wmgAllowLabsAnonEdits": true,
---
>     "wmgAllowLabsAnonEdits": false,
1641c1540
<     "wmgCentralAuthLoginIcon": "/srv/mediawiki/static/images/sul/wikipedia.png",
---
>     "wmgCentralAuthLoginIcon": "/srv/mediawiki/static/images/sul/mediawiki.png",
1651c1550
<     "wmgCiteVisualEditorOtherGroup": false,
---
>     "wmgCiteVisualEditorOtherGroup": true,
1694c1593
<     "wmgContentTranslationTranslateInTarget": false,
---
>     "wmgContentTranslationTranslateInTarget": true,
1696,1697c1595,1596
<     "wmgCustomUploadDialog": true,
<     "wmgDefaultMonologHandler": "wgDebugLogFile",
---
>     "wmgCustomUploadDialog": false,
>     "wmgDefaultMonologHandler": "blackhole",
1699c1598
<     "wmgDisplayFeedsInSidebar": true,
---
>     "wmgDisplayFeedsInSidebar": false,
1701c1600
<     "wmgEchoCluster": "extension1",
---
>     "wmgEchoCluster": false,
1706c1605
<     "wmgEchoSiteNotificationIconUrl": "/static/images/project-logos/notifications/120px-Notification-icon-Wikipedia-logo.svg.png",
---
>     "wmgEchoSiteNotificationIconUrl": "/static/images/project-logos/notifications/120px-Notification-icon-MediaWiki-logo.svg.png",
1713c1612
<     "wmgEnableGeoData": true,
---
>     "wmgEnableGeoData": false,
1716,1718c1615,1617
<     "wmgEnableLandingCheck": true,
<     "wmgEnableLocalTimedText": true,
<     "wmgEnablePageTriage": true,
---
>     "wmgEnableLandingCheck": false,
>     "wmgEnableLocalTimedText": false,
>     "wmgEnablePageTriage": false,
1748c1647
<     "wmgFlowEnableOptInBetaFeature": true,
---
>     "wmgFlowEnableOptInBetaFeature": false,
1751c1650,1669
<     "wmgFlowNamespaces": [],
---
>     "wmgFlowNamespaces": [
>         1,
>         3,
>         5,
>         7,
>         9,
>         11,
>         13,
>         15,
>         101,
>         103,
>         105,
>         107,
>         487,
>         829,
>         1199,
>         2301,
>         2303,
>         5501
>     ],
2010,2011c1928,1941
<     "wmgRC2UDPPrefix": false,
<     "wmgRSSUrlWhitelist": [],
---
>     "wmgRC2UDPPrefix": "#mediawiki.wikipedia\t",
>     "wmgRSSUrlWhitelist": [
>         "https://wikimediafoundation.org/news/feed/",
>         "https://wikimediafoundation.org/category/technology/feed/",
>         "https://wikimediafoundation.org/category/technology/mediawiki/feed/",
>         "https://discourse-mediawiki.wmflabs.org/c/ask-here.rss",
>         "https://codeclimate.com/github/wikimedia/mediawiki-extensions-CentralNotice/feed.atom",
>         "https://codeclimate.com/github/wikimedia/mediawiki-extensions-DonationInterface/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-crm/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-dash/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-php-queue/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-SmashPig/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-tools/feed.atom"
>     ],
2017c1947
<     "wmgSecurePollUseNamespace": true,
---
>     "wmgSecurePollUseNamespace": false,
2021,2029c1951,1952
<     "wmgSiteLogo1_5x": "/static/images/project-logos/testwiki-1.5x.png",
<     "wmgSiteLogo1x": "/static/images/project-logos/testwiki.png",
<     "wmgSiteLogo2x": "/static/images/project-logos/testwiki-2x.png",
<     "wmgSiteLogoIcon": "/static/images/mobile/W.png",
<     "wmgSiteLogoWordmark": {
<         "src": "/static/images/mobile/copyright/wikipedia-wordmark-en.svg",
<         "width": 116,
<         "height": 18
<     },
---
>     "wmgSiteLogo1x": "/static/images/project-logos/mediawikiwiki.png",
>     "wmgSiteLogoIcon": "/static/images/mobile/mediawiki.png",
2033c1956
<     "wmgTranslateESIndex": "ttmserver-test",
---
>     "wmgTranslateESIndex": "ttmserver",
2041c1964
<     "wmgULSPosition": "interlanguage",
---
>     "wmgULSPosition": "personal",
2043c1966
<     "wmgUpload3d": true,
---
>     "wmgUpload3d": false,
2047,2048c1970,1971
<     "wmgUseArticleCreationWorkflow": true,
<     "wmgUseArticlePlaceholder": true,
---
>     "wmgUseArticleCreationWorkflow": false,
>     "wmgUseArticlePlaceholder": false,
2059c1982
<     "wmgUseCapiunto": true,
---
>     "wmgUseCapiunto": false,
2069c1992
<     "wmgUseCodeReview": false,
---
>     "wmgUseCodeReview": true,
2071c1994
<     "wmgUseCollaborationKit": true,
---
>     "wmgUseCollaborationKit": false,
2075,2076c1998,1999
<     "wmgUseContactPage": true,
<     "wmgUseContentTranslation": true,
---
>     "wmgUseContactPage": false,
>     "wmgUseContentTranslation": false,
2080,2081c2003,2004
<     "wmgUseDoubleWiki": true,
<     "wmgUseDynamicPageList": false,
---
>     "wmgUseDoubleWiki": false,
>     "wmgUseDynamicPageList": true,
2090c2013
<     "wmgUseExternalGuidance": true,
---
>     "wmgUseExternalGuidance": false,
2093c2016
<     "wmgUseFileImporter": "FileImporter-WikimediaSitesTableSite",
---
>     "wmgUseFileImporter": false,
2097c2020
<     "wmgUseFooterCodeOfConductLink": false,
---
>     "wmgUseFooterCodeOfConductLink": true,
2104c2027
<     "wmgUseGettingStarted": true,
---
>     "wmgUseGettingStarted": false,
2112c2035
<     "wmgUseGoogleNewsSitemap": true,
---
>     "wmgUseGoogleNewsSitemap": false,
2114,2115c2037,2038
<     "wmgUseGraphWithJsonNamespace": false,
<     "wmgUseGrowthExperiments": true,
---
>     "wmgUseGraphWithJsonNamespace": true,
>     "wmgUseGrowthExperiments": false,
2129c2052
<     "wmgUseLivePreview": true,
---
>     "wmgUseLivePreview": false,
2142,2143c2065,2066
<     "wmgUseORES": true,
<     "wmgUsePageAssessments": true,
---
>     "wmgUseORES": false,
>     "wmgUsePageAssessments": false,
2153c2076
<     "wmgUsePopups": true,
---
>     "wmgUsePopups": false,
2158c2081
<     "wmgUseRSSExtension": false,
---
>     "wmgUseRSSExtension": true,
2160c2083
<     "wmgUseRelatedArticles": true,
---
>     "wmgUseRelatedArticles": false,
2163c2086
<     "wmgUseSandboxLink": true,
---
>     "wmgUseSandboxLink": false,
2173c2096
<     "wmgUseSubPageList3": true,
---
>     "wmgUseSubPageList3": false,
2188c2111
<     "wmgUseUploadWizard": true,
---
>     "wmgUseUploadWizard": false,
2207c2130
<     "wmgUseWikidataPageBanner": true,
---
>     "wmgUseWikidataPageBanner": false,
2210c2133
<     "wmgUseWikimediaShopLink": true,
---
>     "wmgUseWikimediaShopLink": false,
2214a2138,2142
>         "Project": true,
>         "Manual": true,
>         "Extension": true,
>         "API": true,
>         "Skin": true,
2220c2148
<     "wmgVisualEditorConsolidateFeedback": true,
---
>     "wmgVisualEditorConsolidateFeedback": false,
2249,2250c2177,2178
<     "wmgWikibaseAllowLocalShortDesc": true,
<     "wmgWikibaseCachePrefix": "testwikidatawiki0",
---
>     "wmgWikibaseAllowLocalShortDesc": false,
>     "wmgWikibaseCachePrefix": "wikidatawiki",
2265c2193
<     "wmgWikibaseClientChangesDatabase": "testwikidatawiki",
---
>     "wmgWikibaseClientChangesDatabase": "wikidatawiki",
2280,2281c2208,2209
<     "wmgWikibaseClientRepoConceptBaseUri": "http://test.wikidata.org/entity/",
<     "wmgWikibaseClientRepoDatabase": "testwikidatawiki",
---
>     "wmgWikibaseClientRepoConceptBaseUri": "http://www.wikidata.org/entity/",
>     "wmgWikibaseClientRepoDatabase": "wikidatawiki",
2286c2214
<     "wmgWikibaseClientRepoUrl": "https://test.wikidata.org",
---
>     "wmgWikibaseClientRepoUrl": "https://www.wikidata.org",
2289,2290c2217,2218
<             "repoDatabase": "testwikidatawiki",
<             "baseUri": "http://test.wikidata.org/entity/",
---
>             "repoDatabase": "wikidatawiki",
>             "baseUri": "http://www.wikidata.org/entity/",
2306,2308c2234,2235
<         "test",
<         "test2",
<         "testwikidata"
---
>         "wikidata",
>         "wikimania"
2338,2351c2265
<     "wmgWikibaseEntitySources": {
<         "testwikidata": {
<             "entityNamespaces": {
<                 "item": 0,
<                 "property": 120,
<                 "lexeme": 146
<             },
<             "repoDatabase": "testwikidatawiki",
<             "baseUri": "http://test.wikidata.org/entity/",
<             "rdfNodeNamespacePrefix": "wd",
<             "rdfPredicateNamespacePrefix": "",
<             "interwikiPrefix": "testwikidata"
<         }
<     },
---
>     "wmgWikibaseEntitySources": [],
2375d2288
<     "wmgWikibaseLocalEntitySourceName": "testwikidata",
2387c2300
<     "wmgWikibaseSiteGroup": "wikipedia",
---
>     "wmgWikibaseSiteGroup": "mediawiki",

Nothing jumps out at me; the Flow config changes are around user accounts and defaults, nothing about services integration. No changes on the Parsoid/VRS/RESTbase/services integration front AFAICS.

Adding as a train blocker until/unless we can prove that this isn't related to a group0 deployment.

(It's likely as @kostajh suggests that this isn't train-related, but I'm trying to err on the side of caution.)

Please note that this is a pywikibot breakage not Flow breakage .. see https://test.wikipedia.org/wiki/Topic:Vk4my4ths5dq3qej .. I just did that flow test. So, this is not a train blocker.

Jdforrester-WMF renamed this task from Internal API error with Flow on testwiki to pywikibot encounters an internal API error with Flow on testwiki (but not other wikis).Apr 8 2020, 4:42 PM

https://logstash.wikimedia.org/goto/cc6d91314e1475059ea06a3a1d422aa4 is the logstash board for errors over a 7-day period. it shows:

  • this is NOT limited to testwiki
  • also shows a bunch of errors on April 6th around 1500 UTC (before the current train rollout to group 0)
  • the current spurt of errors seem to have started around 7 hours back
  • unclear if this is necessarily limited to pywikibot since logstash isn't showing us the user-agent string.

HTTP 503 is "Service Unavailable" -- I wonder if this is actually a rate limit or something similar which is being hit?

The full URL requested for the original 503 error should have been:

http://localhost:6002/w/rest.php/test.wikimedia.org/v3/transform/html/to/wikitext/Topic%3AU5y53rn0dp6h70nw

(Based on an internal VRS request of /restbase/local/v1/transform/html/to/wikitext/Topic%3AU5y53rn0dp6h70nw)

That localhost request is then routed to the parsoid cluster; @Joe thinks that the recent envoy changes (see T249535) might be related.

<cscott> i assume the localhost:6002 goes to envoy or some other router which actually brings it to the parsoid cluster
<cscott> ?
<_joe_> yes
<_joe_> so interestingly, I find no trace of 503 errors in the parsoid logs, but lemme check the mediawiki appservers
<_joe_> actually, it's almost 8 pm
<_joe_> I'll do it  tomorrow
<_joe_> this is not a complete breakage AFAICT
Joe triaged this task as Low priority.Apr 9 2020, 10:35 AM
  • also shows a bunch of errors on April 6th around 1500 UTC (before the current train rollout to group 0)

Also I see it continues to occur on 1.35.0-wmf.26, so again likely not the wmf.27 train.

  • unclear if this is necessarily limited to pywikibot since logstash isn't showing us the user-agent string.

I just ran into it trying to submit a comment on officewiki with Firefox. Out of annoyance I clicked "save" several more times, and it did finally go through after erroring the first half-dozen times.

  • also shows a bunch of errors on April 6th around 1500 UTC (before the current train rollout to group 0)

Also I see it continues to occur on 1.35.0-wmf.26, so again likely not the wmf.27 train.

  • unclear if this is necessarily limited to pywikibot since logstash isn't showing us the user-agent string.

I just ran into it trying to submit a comment on officewiki with Firefox. Out of annoyance I clicked "save" several more times, and it did finally go through after erroring the first half-dozen times.

This seems a very different thing, though. The errors reported by pywikibot seemed sporadic, and I could trace them to a specific failure on a backend. What you are describing seems like a persistent behaviour that would possibly not be related to what I was investigating at the moment.

I just found your errors in the logs, and this thickens the cloud of mistery around this bug. I am 99% convinced this has to do with my work, rather than anything else, so I'll re-tag the task appropriately.

Change 587732 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] parsoid: allow retries for connection resets in envoy

https://gerrit.wikimedia.org/r/587732

I just found your errors in the logs, and this thickens the cloud of mistery around this bug. I am 99% convinced this has to do with my work, rather than anything else, so I'll re-tag the task appropriately.

FWIW, just a little while ago I got the same error (NoParserException) the first time I tried posting a comment on that same Flow topic on officewiki, the second time it succeeded.

Change 587787 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] services_proxy: retry on connect failures for parsoid

https://gerrit.wikimedia.org/r/587787

Just for the record, here is the error after @ssastry (? I guess) edited that particular topic:

======================================================================
ERROR: test_topic_moderation (tests.flow_tests.TestFlowTopic)
Test Topic functions about moderation.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\pywikibot-g4xqx\tests\flow_tests.py", line 245, in test_topic_moderation
    self.assertFalse(topic_hidden.is_locked)
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 220, in is_locked
    return self.root._current_revision['isLocked']
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 214, in root
    self._root = Post.fromJSON(self, self.uuid, self._data)
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 65, in uuid
    self._load_uuid()
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 55, in _load_uuid
    self._uuid = self._load()['workflowId']
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 160, in _load
    self._data = self.site.load_topic(self, format)
  File "C:\projects\pywikibot-g4xqx\pywikibot\site\__init__.py", line 1390, in callee
    return fn(self, *args, **kwargs)
  File "C:\projects\pywikibot-g4xqx\pywikibot\site\__init__.py", line 7306, in load_topic
    data = req.submit()
  File "C:\projects\pywikibot-g4xqx\pywikibot\data\api.py", line 2040, in submit
    if self._internal_api_error(code, error, result):
  File "C:\projects\pywikibot-g4xqx\pywikibot\data\api.py", line 1910, in _internal_api_error
    raise e
pywikibot.data.api.APIMWException: internal_api_error_Flow\Exception\NoParserException: [3d207406-aefb-4869-8e88-041eaab0ddd1] Exception caught: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:U5y53rn0dp6h70nw" failed: 503
[*: Flow\Exception\NoParserException at /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(174)
#0 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(66): Flow\Conversion\Utils::parsoid(string, string, string, Title)
#1 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Model/AbstractRevision.php(430): Flow\Conversion\Utils::convert(string, string, string, Title)
#2 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Templating.php(159): Flow\Model\AbstractRevision->getContent(string)
#3 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/RevisionFormatter.php(289): Flow\Templating->getContent(Flow\Model\PostRevision, string)
#4 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/TopicFormatter.php(45): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, DerivativeContext)
#5 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(663): Flow\Formatter\TopicFormatter->formatApi(Flow\Model\Workflow, array, DerivativeContext)
#6 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(565): Flow\Block\TopicBlock->renderTopicApi(array)
#7 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlowBaseGet.php(33): Flow\Block\TopicBlock->renderApi(array)
#8 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlow.php(105): Flow\Api\ApiFlowBaseGet->execute()
#9 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(1580): Flow\Api\ApiFlow->execute()
#10 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(523): ApiMain->executeAction()
#11 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(494): ApiMain->executeActionWithErrorHandling()
#12 /srv/mediawiki/php-1.35.0-wmf.27/api.php(84): ApiMain->execute()
#13 /srv/mediawiki/w/api.php(3): require(string)
#14 {main};
 errorclass: Flow\Exception\NoParserException;
 servedby: mw1408]

Not sure if it changed, but it fails more often now.

Change 587787 merged by Giuseppe Lavagetto:
[operations/puppet@production] services_proxy: retry on connect failures for parsoid

https://gerrit.wikimedia.org/r/587787

@Xqt Could we skip the whole flow_tests.py somehow in Pywikibot tests? Or is there an option to skip tests from flow_tests.py failing with this internal API error?

The whole script can be skipped by adding code like the following in flow_tests.py:

def setUpModule():  # noqa: N802
    """Always skip."""
    raise unittest.SkipTest('flow_tests.py tests are disabled due to T249705.')

Or skip the whole flow tests if an environ variable is set like in l10n_tests.py

cscott renamed this task from pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) to Intermittent internal API errors with Flow.Apr 10 2020, 4:05 PM
cscott updated the task description. (Show Details)

The whole script can be skipped by adding code like the following in flow_tests.py:

def setUpModule():  # noqa: N802
    """Always skip."""
    raise unittest.SkipTest('flow_tests.py tests are disabled due to T249705.')

Or skip the whole flow tests if an environ variable is set like in l10n_tests.py

This would hopefully be temporary, I would prefer the first option then.

Change 588144 had a related patch set uploaded (by Dvorapa; owner: Dvorapa):
[pywikibot/core@master] [test] Ignore flow_tests until upstream issue solved

https://gerrit.wikimedia.org/r/588144

Change 588144 merged by jenkins-bot:
[pywikibot/core@master] [test] Ignore flow_tests until upstream issue solved

https://gerrit.wikimedia.org/r/588144

@Joe I see your patch was merged on Friday but it seems like there is still an instance of this problem occurring as of today (T249997), do you have any thoughts on what might be going on?

@Joe I see your patch was merged on Friday but it seems like there is still an instance of this problem occurring as of today (T249997), do you have any thoughts on what might be going on?

That bug talks about an error 400, it seems a different problem. I'm going to try to do some more aggressive changes today to contrast what's happening. I'm surprised by the number of reports for something that seems to be happening once every minute or so.

If this doesn't work either, I'll probably revert my latest changes.

Joe raised the priority of this task from Low to High.Apr 14 2020, 7:21 AM
Joe added subscribers: Hsarrazin, Framawiki, Nattes.

Changing priority as this seems to be highly user visible.

Change 588692 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] services_proxy: temp mitigation for intermittent parsoid requests failures

https://gerrit.wikimedia.org/r/588692

Change 588692 merged by Giuseppe Lavagetto:
[operations/puppet@production] services_proxy: temp mitigation for intermittent parsoid requests failures

https://gerrit.wikimedia.org/r/588692

I've added some further retry logic for requests to parsoid, this *might* help.

This is now resolved, I see no further errors since my latest change was merged.

Change 587732 abandoned by Giuseppe Lavagetto:
parsoid: allow retries for connection resets in envoy

Reason:
Abandoning per the comment.

https://gerrit.wikimedia.org/r/587732