for (;;);{"error":null,"payload":{"timeline":"\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_66\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/gaj2wbcq54jfzcscgqrn\/PHID-FILE-jyedy4j2gqeydwbnujxx\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/Ottomata\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5602851\" id=\"5602851\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-pencil phui-timeline-icon\" data-meta=\"0_65\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_0\"\u003eOttomata\u003c\/a\u003e created this task.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5602851\" data-sigil=\"has-tooltip\" data-meta=\"0_64\"\u003e\u003cspan class=\"screen-only\"\u003eOct 24 2019, 1:47 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-24 13:47:57 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_62\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_63\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_12\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cp\u003eThe client error schema isn't ready, so we can't really deploy this yet in k8s.  However, we can prepare the deployment-charts patches.\u003c\/p\u003e\n\n\u003cp\u003e\u003ca href=\"\/p\/elukey\/\" class=\"phui-tag-view phui-tag-type-person \" data-sigil=\"hovercard\" data-meta=\"0_2\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-person\"\u003e@elukey\u003c\/span\u003e\u003c\/a\u003e you mentioned you wanted to work on this with me to get some experience doing Helm and k8s stuff, ya?  Let's go! :)\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_69\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/utgwrwkhuw6vizkywcou\/PHID-FILE-36umycp75fzx37k2sow5\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/Nuria\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5604099\" id=\"5604099\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-link phui-timeline-icon\" data-meta=\"0_68\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Nuria\/\" class=\"phui-handle handle-availability-disabled phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_13\"\u003e\u003cspan class=\"perfect-circle\"\u003e\u2022\u003c\/span\u003e Nuria\u003c\/a\u003e mentioned this in \u003ca href=\"\/T235189\" class=\"phui-handle handle-status-closed\" data-sigil=\"hovercard\" data-meta=\"0_14\"\u003eT235189: Develop small client-side error logger for MediaWiki\u003c\/a\u003e.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5604099\" data-sigil=\"has-tooltip\" data-meta=\"0_67\"\u003e\u003cspan class=\"screen-only\"\u003eOct 24 2019, 6:24 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-24 18:24:18 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_72\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/i4b4kyw53tbds5shsuz3\/PHID-FILE-4iuedt6yam7nc3g5kfc7\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/jlinehan\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5607366\" id=\"5607366\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-link phui-timeline-icon\" data-meta=\"0_71\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/jlinehan\/\" class=\"phui-handle handle-availability-disabled phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_15\"\u003e\u003cspan class=\"perfect-circle\"\u003e\u2022\u003c\/span\u003e jlinehan\u003c\/a\u003e mentioned this in \u003ca href=\"\/T229442\" class=\"phui-handle handle-status-closed\" data-sigil=\"hovercard\" data-meta=\"0_16\"\u003eT229442: Create client side error schema\u003c\/a\u003e.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5607366\" data-sigil=\"has-tooltip\" data-meta=\"0_70\"\u003e\u003cspan class=\"screen-only\"\u003eOct 25 2019, 7:47 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-25 19:47:15 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell phui-timeline-green\" data-sigil=\"transaction anchor-container\" data-meta=\"0_76\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/tnss5j5cboisfwvfyynt\/PHID-FILE-q4rxl5w7ormrj4e4lbcg\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/elukey\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5610313\" id=\"5610313\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill fill-has-color phui-timeline-icon-fill-green\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-user phui-timeline-icon\" data-meta=\"0_74\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/elukey\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_17\"\u003eelukey\u003c\/a\u003e claimed this task.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5610313\" data-sigil=\"has-tooltip\" data-meta=\"0_73\"\u003e\u003cspan class=\"screen-only\"\u003eOct 28 2019, 8:47 AM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-28 08:47:24 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-link phui-timeline-icon\" data-meta=\"0_75\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/elukey\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_18\"\u003eelukey\u003c\/a\u003e added a project: \u003ca href=\"\/tag\/user-elukey\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_19\"\u003eUser-Elukey\u003c\/a\u003e.\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_79\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/c6xmpug7aawzqcut5d3f\/PHID-FILE-v2zy4knvlloiceegtfy7\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/fgiunchedi\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5618490\" id=\"5618490\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-columns phui-timeline-icon\" data-meta=\"0_78\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/fgiunchedi\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_20\"\u003efgiunchedi\u003c\/a\u003e moved this task from \u003ca href=\"\/project\/board\/2668\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_21\"\u003eBacklog\u003c\/a\u003e to \u003ca href=\"\/project\/board\/2668\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_22\"\u003eRadar\u003c\/a\u003e on the \u003ca href=\"\/tag\/user-fgiunchedi\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_23\"\u003eUser-fgiunchedi\u003c\/a\u003e board.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5618490\" data-sigil=\"has-tooltip\" data-meta=\"0_77\"\u003e\u003cspan class=\"screen-only\"\u003eOct 30 2019, 9:46 AM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-30 09:46:30 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_82\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/alvqrrumwbkspmybmsv3\/PHID-FILE-a4za6nkie2ve6kzb4qu2\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/LGoto\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5619851\" id=\"5619851\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-columns phui-timeline-icon\" data-meta=\"0_81\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/LGoto\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_24\"\u003eLGoto\u003c\/a\u003e moved this task from \u003ca href=\"\/project\/board\/2726\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_25\"\u003eNeeds triage\u003c\/a\u003e to \u003ca href=\"\/project\/board\/2726\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_26\"\u003eTracking\u003c\/a\u003e on the \u003ca href=\"\/tag\/product-infrastructure-team-backlog-deprecated\/\" class=\"phui-handle handle-status-closed\" data-sigil=\"hovercard\" data-meta=\"0_27\"\u003eProduct-Infrastructure-Team-Backlog-Deprecated\u003c\/a\u003e board.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5619851\" data-sigil=\"has-tooltip\" data-meta=\"0_80\"\u003e\u003cspan class=\"screen-only\"\u003eOct 30 2019, 3:38 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-30 15:38:01 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_91\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/gaj2wbcq54jfzcscgqrn\/PHID-FILE-jyedy4j2gqeydwbnujxx\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/Ottomata\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5621245\" id=\"5621245\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-comment phui-timeline-icon\" data-meta=\"0_90\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_28\"\u003eOttomata\u003c\/a\u003e added a comment.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5621245\" data-sigil=\"has-tooltip\" data-meta=\"0_89\"\u003e\u003cspan class=\"screen-only\"\u003eOct 30 2019, 8:41 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-30 20:41:28 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_87\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_88\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_29\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cp\u003e\u003ca href=\"\/p\/fgiunchedi\/\" class=\"phui-tag-view phui-tag-type-person \" data-sigil=\"hovercard\" data-meta=\"0_4\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-person\"\u003e@fgiunchedi\u003c\/span\u003e\u003c\/a\u003e, q for you.  In \u003ca href=\"\/T233629\" class=\"phui-tag-view phui-tag-type-object \" data-sigil=\"hovercard\" data-meta=\"0_3\"\u003e\u003cspan class=\"phui-tag-core-closed\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-object\"\u003eT233629\u003c\/span\u003e\u003c\/span\u003e\u003c\/a\u003e we will also be setting up an externally POSTable eventgate instance for analytics.  Over there, I'm leaning towards setting up a separate instance for external analytics stuff, probably named eventgate-analytics-external.\u003c\/p\u003e\n\n\u003cp\u003eI was originally going to call this instance eventgate-logging, but perhaps we should also suffix it with -external for consistency and clarity?  eventgate-logging-external?  If there ever ended up being a need for an internal eventgate-logging instance, we'd have the name available.\u003c\/p\u003e\n\n\u003cp\u003eIt isn't necessary to have separate instances for internal and external, but it might prove useful to have them isolated for performance reasons.  We'd like to tweak some message size limits a bit smaller for external producers.\u003c\/p\u003e\n\n\u003cp\u003eWould naming this k8s service 'eventgate-logging-external' work for you?\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_100\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/6vn6slgby7ia62ouikut\/PHID-FILE-ay56qvafgaxajuctgtw3\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/gerritbot\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5621302\" id=\"5621302\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-comment phui-timeline-icon\" data-meta=\"0_99\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/gerritbot\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_30\"\u003egerritbot\u003c\/a\u003e added a comment.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5621302\" data-sigil=\"has-tooltip\" data-meta=\"0_98\"\u003e\u003cspan class=\"screen-only\"\u003eOct 30 2019, 8:48 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-30 20:48:21 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_96\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_97\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_31\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cp\u003eChange 547307 had a related patch set uploaded (by Ottomata; owner: Ottomata):\u003cbr \/\u003e\n[operations\/deployment-charts@master] Add eventgate-logging-external instance in staging\u003c\/p\u003e\n\n\u003cp\u003e\u003ca href=\"https:\/\/gerrit.wikimedia.org\/r\/547307\" class=\"remarkup-link\" target=\"_blank\" rel=\"noreferrer\"\u003ehttps:\/\/gerrit.wikimedia.org\/r\/547307\u003c\/a\u003e\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_103\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/6vn6slgby7ia62ouikut\/PHID-FILE-ay56qvafgaxajuctgtw3\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/gerritbot\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5621303\" id=\"5621303\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-link phui-timeline-icon\" data-meta=\"0_102\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/gerritbot\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_32\"\u003egerritbot\u003c\/a\u003e added a project: \u003ca href=\"\/tag\/patch-for-review\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_33\"\u003ePatch-For-Review\u003c\/a\u003e.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5621303\" data-sigil=\"has-tooltip\" data-meta=\"0_101\"\u003e\u003cspan class=\"screen-only\"\u003eOct 30 2019, 8:48 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-30 20:48:22 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell phui-timeline-green\" data-sigil=\"transaction anchor-container\" data-meta=\"0_109\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/gaj2wbcq54jfzcscgqrn\/PHID-FILE-jyedy4j2gqeydwbnujxx\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/Ottomata\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5621326\" id=\"5621326\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill fill-has-color phui-timeline-icon-fill-green\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-user phui-timeline-icon\" data-meta=\"0_105\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_34\"\u003eOttomata\u003c\/a\u003e claimed this task.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5621326\" data-sigil=\"has-tooltip\" data-meta=\"0_104\"\u003e\u003cspan class=\"screen-only\"\u003eOct 30 2019, 9:00 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-30 21:00:48 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-link phui-timeline-icon\" data-meta=\"0_106\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_35\"\u003eOttomata\u003c\/a\u003e added projects: \u003ca href=\"\/tag\/analytics-kanban\/\" class=\"phui-handle handle-status-closed\" data-sigil=\"hovercard\" data-meta=\"0_36\"\u003eAnalytics-Kanban\u003c\/a\u003e, \u003ca href=\"\/tag\/event-platform\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_37\"\u003eEvent-Platform\u003c\/a\u003e.\u003c\/div\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-columns phui-timeline-icon\" data-meta=\"0_107\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_38\"\u003eOttomata\u003c\/a\u003e moved this task from \u003ca href=\"\/project\/board\/1030\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_39\"\u003eNext Up\u003c\/a\u003e to \u003ca href=\"\/project\/board\/1030\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_40\"\u003eIn Progress\u003c\/a\u003e on the \u003ca href=\"\/tag\/analytics-kanban\/\" class=\"phui-handle handle-status-closed\" data-sigil=\"hovercard\" data-meta=\"0_41\"\u003eAnalytics-Kanban\u003c\/a\u003e board.\u003c\/div\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-columns phui-timeline-icon\" data-meta=\"0_108\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_42\"\u003eOttomata\u003c\/a\u003e moved this task from \u003ca href=\"\/project\/board\/1474\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_43\"\u003eBacklog\u003c\/a\u003e to \u003ca href=\"\/project\/board\/1474\/\" class=\"phui-handle handle-status-closed\" data-sigil=\"hovercard\" data-meta=\"0_44\"\u003eIn Progress Before Value Streams Kickoff (August 15th)\u003c\/a\u003e on the \u003ca href=\"\/tag\/event-platform\/\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_45\"\u003eEvent-Platform\u003c\/a\u003e board.\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_118\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/c6xmpug7aawzqcut5d3f\/PHID-FILE-v2zy4knvlloiceegtfy7\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/fgiunchedi\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5624179\" id=\"5624179\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-comment phui-timeline-icon\" data-meta=\"0_117\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/fgiunchedi\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_46\"\u003efgiunchedi\u003c\/a\u003e added a comment.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5624179\" data-sigil=\"has-tooltip\" data-meta=\"0_116\"\u003e\u003cspan class=\"screen-only\"\u003eOct 31 2019, 5:32 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-10-31 17:32:48 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_114\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_115\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_47\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cblockquote class=\"remarkup-reply-block\"\u003e\n\u003cdiv class=\"remarkup-reply-head\"\u003eIn \u003ca href=\"\/T236386#5621245\" class=\"phui-tag-view phui-tag-type-object \" data-sigil=\"hovercard\" data-meta=\"0_6\"\u003e\u003cspan class=\"phui-tag-core-closed\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-object\"\u003eT236386#5621245\u003c\/span\u003e\u003c\/span\u003e\u003c\/a\u003e, \u003ca href=\"\/p\/Ottomata\/\" class=\"phui-tag-view phui-tag-type-person \" data-sigil=\"hovercard\" data-meta=\"0_8\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-person\"\u003e@Ottomata\u003c\/span\u003e\u003c\/a\u003e wrote:\u003c\/div\u003e\n\u003cdiv class=\"remarkup-reply-body\"\u003e\u003cp\u003e\u003ca href=\"\/p\/fgiunchedi\/\" class=\"phui-tag-view phui-tag-type-person \" data-sigil=\"hovercard\" data-meta=\"0_7\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-person\"\u003e@fgiunchedi\u003c\/span\u003e\u003c\/a\u003e, q for you.  In \u003ca href=\"\/T233629\" class=\"phui-tag-view phui-tag-type-object \" data-sigil=\"hovercard\" data-meta=\"0_5\"\u003e\u003cspan class=\"phui-tag-core-closed\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-object\"\u003eT233629\u003c\/span\u003e\u003c\/span\u003e\u003c\/a\u003e we will also be setting up an externally POSTable eventgate instance for analytics.  Over there, I'm leaning towards setting up a separate instance for external analytics stuff, probably named eventgate-analytics-external.\u003c\/p\u003e\n\n\u003cp\u003eI was originally going to call this instance eventgate-logging, but perhaps we should also suffix it with -external for consistency and clarity?  eventgate-logging-external?  If there ever ended up being a need for an internal eventgate-logging instance, we'd have the name available.\u003c\/p\u003e\n\n\u003cp\u003eIt isn't necessary to have separate instances for internal and external, but it might prove useful to have them isolated for performance reasons.  We'd like to tweak some message size limits a bit smaller for external producers.\u003c\/p\u003e\n\n\u003cp\u003eWould naming this k8s service 'eventgate-logging-external' work for you?\u003c\/p\u003e\u003c\/div\u003e\n\u003c\/blockquote\u003e\n\n\u003cp\u003eThe conversation eventually happened on IRC but for the record reporting it here too: yes \u003ctt class=\"remarkup-monospaced\"\u003e-external\u003c\/tt\u003e suffix sounds good (and preferable) to me!\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_127\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/gaj2wbcq54jfzcscgqrn\/PHID-FILE-jyedy4j2gqeydwbnujxx\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/Ottomata\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5633635\" id=\"5633635\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-comment phui-timeline-icon\" data-meta=\"0_126\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_48\"\u003eOttomata\u003c\/a\u003e added a comment.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5633635\" data-sigil=\"has-tooltip\" data-meta=\"0_125\"\u003e\u003cspan class=\"screen-only\"\u003eNov 4 2019, 9:20 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-11-04 21:20:43 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_123\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_124\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_49\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cp\u003eHm, another issue!\u003c\/p\u003e\n\n\u003cp\u003eFor MirrorMaker replication reasons, we prefix topics in the main multi DC Kafka setups with their DC name, e.g. 'eqiad.resouce_change'  The Kafka logging clusters are the first instance of non 'main' multi DC Kafka clusters.  This made me realize that the reason for the DC prefixed topic names was not so much to identify the source DC of the topic, but to identify the source Kafka cluster of the topic.  If we were to replicate all topics from all non-aggregate Kafka clusters in to Kafka jumbo, it'd be good to have distinct topics for each source Kafka cluster.\u003c\/p\u003e\n\n\u003cp\u003eBefore I thought about this, I had intended to use the DC topic prefix for eventgate-logging produced topics, e.g. 'eqiad.client.error'.  We'd want to replicate this (and possibly other?) topics from the Kafka logging clusters to Kafka jumbo for consumption into HDFS and Hive.  If the Kafka logging clusters and the Kafka main clusters ever have a topic with the same name, we'd end up mixing messages from the two clusters into the same topic.  Messages from eqiad.client.error from Kafka logging-eqiad and eqiad.client.error from Kafka main-eqiad would both in Kafka jumbo-eqiad eqiad.client.error.\u003c\/p\u003e\n\n\u003cp\u003eThis isn't a problem for mirroring into the Kafka jumbo cluster for analytics purposes, since nothing mirrors OUT of Kafka jumbo.  But if we ever wanted to use cross DC replication between Kafka logging-eqiad and logging-codfw, this could be a problem.\u003c\/p\u003e\n\n\u003cp\u003eTo avoid this, we probably should just prefix all topics with their source Kafka cluster name.  We're too far gone to start prefixing things in Kafka main-* with e.g. 'main-eqiad.', but perhaps we can live with that.  Topics prefixed with just eqiad or codfw will be assumed to be from the main Kafka cluster, whereas other Kafka clusters should use the full Kafka cluster name, e.g. 'logging-eqiad'.\u003c\/p\u003e\n\n\u003cp\u003eThis does get a bit annoying on the Hive side of things.  Refine uses the topic prefix for the Hive \u003ctt class=\"remarkup-monospaced\"\u003edatacenter\u003c\/tt\u003e partition.  A better name for this partition would have been \u003ctt class=\"remarkup-monospaced\"\u003esource_cluster\u003c\/tt\u003e or something.   If we prefix the topics with 'logging-eqiad', we'll end up with partitions like \u003ctt class=\"remarkup-monospaced\"\u003edatacenter=logging-eqiad\u003c\/tt\u003e, which just looks a little funny.\u003c\/p\u003e\n\n\u003cp\u003eAnother piece of info is that Kafka MirrorMaker 2.0 is under development and handles \u003ca href=\"https:\/\/cwiki.apache.org\/confluence\/display\/KAFKA\/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RemoteTopics,Partitions\" class=\"remarkup-link\" target=\"_blank\" rel=\"noreferrer\"\u003etransparently prefixing topics\u003c\/a\u003e in replicated clusters with their source kafka cluster names.  MirrorMaker 2 isn't in any Kafka release yet, but I expect it to be sometime within the next 6 months to a year.  We'll likely be using it one day.\u003c\/p\u003e\n\n\u003cp\u003eAnyway, I'm inclined to just prefix these topics new logging topics with the datacenter name: eqiad or codfw.  This might bite us if we ever have to use the same topic in multiple logical Kafka clusters, I don't know if that will happen or not.  We'll continue using the topic prefix for the datacenter rather than the source kafka cluster until it does.\u003c\/p\u003e\n\n\u003cp\u003eThoughts \u003ca href=\"\/p\/elukey\/\" class=\"phui-tag-view phui-tag-type-person \" data-sigil=\"hovercard\" data-meta=\"0_9\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-person\"\u003e@elukey\u003c\/span\u003e\u003c\/a\u003e?\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_136\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/c6xmpug7aawzqcut5d3f\/PHID-FILE-v2zy4knvlloiceegtfy7\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/fgiunchedi\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5635277\" id=\"5635277\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-comment phui-timeline-icon\" data-meta=\"0_135\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/fgiunchedi\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_50\"\u003efgiunchedi\u003c\/a\u003e added a comment.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5635277\" data-sigil=\"has-tooltip\" data-meta=\"0_134\"\u003e\u003cspan class=\"screen-only\"\u003eNov 5 2019, 11:47 AM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-11-05 11:47:43 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_132\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_133\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_51\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cblockquote class=\"remarkup-reply-block\"\u003e\n\u003cdiv class=\"remarkup-reply-head\"\u003eIn \u003ca href=\"\/T236386#5633635\" class=\"phui-tag-view phui-tag-type-object \" data-sigil=\"hovercard\" data-meta=\"0_10\"\u003e\u003cspan class=\"phui-tag-core-closed\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-object\"\u003eT236386#5633635\u003c\/span\u003e\u003c\/span\u003e\u003c\/a\u003e, \u003ca href=\"\/p\/Ottomata\/\" class=\"phui-tag-view phui-tag-type-person \" data-sigil=\"hovercard\" data-meta=\"0_11\"\u003e\u003cspan class=\"phui-tag-core phui-tag-color-person\"\u003e@Ottomata\u003c\/span\u003e\u003c\/a\u003e wrote:\u003c\/div\u003e\n\u003cdiv class=\"remarkup-reply-body\"\u003e\u003cp\u003eHm, another issue!\u003c\/p\u003e\n\n\u003cp\u003eFor MirrorMaker replication reasons, we prefix topics in the main multi DC Kafka setups with their DC name, e.g. 'eqiad.resouce_change'  The Kafka logging clusters are the first instance of non 'main' multi DC Kafka clusters.  This made me realize that the reason for the DC prefixed topic names was not so much to identify the source DC of the topic, but to identify the source Kafka cluster of the topic.  If we were to replicate all topics from all non-aggregate Kafka clusters in to Kafka jumbo, it'd be good to have distinct topics for each source Kafka cluster.\u003c\/p\u003e\n\n\u003cp\u003eBefore I thought about this, I had intended to use the DC topic prefix for eventgate-logging produced topics, e.g. 'eqiad.client.error'.  We'd want to replicate this (and possibly other?) topics from the Kafka logging clusters to Kafka jumbo for consumption into HDFS and Hive.  If the Kafka logging clusters and the Kafka main clusters ever have a topic with the same name, we'd end up mixing messages from the two clusters into the same topic.  Messages from eqiad.client.error from Kafka logging-eqiad and eqiad.client.error from Kafka main-eqiad would both in Kafka jumbo-eqiad eqiad.client.error.\u003c\/p\u003e\n\n\u003cp\u003eThis isn't a problem for mirroring into the Kafka jumbo cluster for analytics purposes, since nothing mirrors OUT of Kafka jumbo.  But if we ever wanted to use cross DC replication between Kafka logging-eqiad and logging-codfw, this could be a problem.\u003c\/p\u003e\u003c\/div\u003e\n\u003c\/blockquote\u003e\n\n\u003cp\u003eI'm assuming you mean mirroring topics between codfw\/eqiad for Kafka logging? In case it helps, this configuration isn't very likely to happen as the two clusters have been designed to be isolated from one another and logstash consuming from both. Having said that, there could be of course a logging use case in the future that calls for cross-dc replication, but I'm having an hard time imagining such use case now.\u003c\/p\u003e\n\n\u003cblockquote\u003e\u003cp\u003eTo avoid this, we probably should just prefix all topics with their source Kafka cluster name.  We're too far gone to start prefixing things in Kafka main-* with e.g. 'main-eqiad.', but perhaps we can live with that.  Topics prefixed with just eqiad or codfw will be assumed to be from the main Kafka cluster, whereas other Kafka clusters should use the full Kafka cluster name, e.g. 'logging-eqiad'.\u003c\/p\u003e\n\n\u003cp\u003eThis does get a bit annoying on the Hive side of things.  Refine uses the topic prefix for the Hive \u003ctt class=\"remarkup-monospaced\"\u003edatacenter\u003c\/tt\u003e partition.  A better name for this partition would have been \u003ctt class=\"remarkup-monospaced\"\u003esource_cluster\u003c\/tt\u003e or something.   If we prefix the topics with 'logging-eqiad', we'll end up with partitions like \u003ctt class=\"remarkup-monospaced\"\u003edatacenter=logging-eqiad\u003c\/tt\u003e, which just looks a little funny.\u003c\/p\u003e\n\n\u003cp\u003eAnother piece of info is that Kafka MirrorMaker 2.0 is under development and handles \u003ca href=\"https:\/\/cwiki.apache.org\/confluence\/display\/KAFKA\/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RemoteTopics,Partitions\" class=\"remarkup-link\" target=\"_blank\" rel=\"noreferrer\"\u003etransparently prefixing topics\u003c\/a\u003e in replicated clusters with their source kafka cluster names.  MirrorMaker 2 isn't in any Kafka release yet, but I expect it to be sometime within the next 6 months to a year.  We'll likely be using it one day.\u003c\/p\u003e\n\n\u003cp\u003eAnyway, I'm inclined to just prefix these topics new logging topics with the datacenter name: eqiad or codfw.  This might bite us if we ever have to use the same topic in multiple logical Kafka clusters, I don't know if that will happen or not.  We'll continue using the topic prefix for the datacenter rather than the source kafka cluster until it does.\u003c\/p\u003e\u003c\/blockquote\u003e\n\n\u003cp\u003eFWIW I'm +1 on just going with the usual prefixes and see how far that gets us. Perhaps mirrormaker could be warning us or similar if it detects there are topic name conflicts\/clashes between different kafka clusters ?\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_145\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/gaj2wbcq54jfzcscgqrn\/PHID-FILE-jyedy4j2gqeydwbnujxx\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/Ottomata\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5636053\" id=\"5636053\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-comment phui-timeline-icon\" data-meta=\"0_144\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_52\"\u003eOttomata\u003c\/a\u003e added a comment.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5636053\" data-sigil=\"has-tooltip\" data-meta=\"0_143\"\u003e\u003cspan class=\"screen-only\"\u003eNov 5 2019, 2:23 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-11-05 14:23:13 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_141\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_142\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_53\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cblockquote\u003e\u003cp\u003ePerhaps mirrormaker could be warning us or similar if it detects there are topic name conflicts\/clashes between different kafka clusters ?\u003c\/p\u003e\u003c\/blockquote\u003e\n\n\u003cp\u003eLikely what MirrorMaker 2 will do is prefix the existing topic names with the source cluster name.  So, topics sourced from main eqiad mirrored to kafka jumbo will end up looking like 'main-eqiad.eqiad.mediawiki.revision-create' in Kafka jumbo.\u003c\/p\u003e\n\n\u003cp\u003eAnyway, yeah ok, let's keep going with the DC prefixes in logging cluster.\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_154\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-major-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/6vn6slgby7ia62ouikut\/PHID-FILE-ay56qvafgaxajuctgtw3\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/gerritbot\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003cdiv class=\"phui-timeline-inner-content\"\u003e\u003ca name=\"5641288\" id=\"5641288\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon phui-timeline-title-with-menu\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-comment phui-timeline-icon\" data-meta=\"0_153\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/gerritbot\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_54\"\u003egerritbot\u003c\/a\u003e added a comment.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5641288\" data-sigil=\"has-tooltip\" data-meta=\"0_152\"\u003e\u003cspan class=\"screen-only\"\u003eNov 6 2019, 5:27 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-11-06 17:27:53 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003ca href=\"#\" class=\"phui-timeline-menu\" aria-haspopup=\"true\" aria-expanded=\"false\" data-sigil=\"phui-dropdown-menu\" data-meta=\"0_150\"\u003e\u003cspan class=\"aural-only\"\u003eComment Actions\u003c\/span\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-caret-down\" data-meta=\"0_151\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-core-content\"\u003e\u003cspan class=\"transaction-comment\" data-sigil=\"transaction-comment\" data-meta=\"0_55\"\u003e\u003cdiv class=\"phabricator-remarkup\"\u003e\u003cp\u003eChange 547307 \u003cstrong\u003emerged\u003c\/strong\u003e by Ottomata:\u003cbr \/\u003e\n[operations\/deployment-charts@master] Add eventgate-logging-external instance\u003c\/p\u003e\n\n\u003cp\u003e\u003ca href=\"https:\/\/gerrit.wikimedia.org\/r\/547307\" class=\"remarkup-link\" target=\"_blank\" rel=\"noreferrer\"\u003ehttps:\/\/gerrit.wikimedia.org\/r\/547307\u003c\/a\u003e\u003c\/p\u003e\u003c\/div\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-spacer\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-shell\" data-sigil=\"transaction anchor-container\" data-meta=\"0_157\"\u003e\u003cdiv class=\"phui-timeline-event-view phui-timeline-minor-event\"\u003e\u003cdiv class=\"phui-timeline-content\"\u003e\u003ca style=\"background-image: url(https:\/\/phab.wmfusercontent.org\/file\/data\/gaj2wbcq54jfzcscgqrn\/PHID-FILE-jyedy4j2gqeydwbnujxx\/profile)\" class=\"visual-only phui-timeline-image\" href=\"\/p\/Ottomata\/\" aria-hidden=\"true\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-wedge\" style=\"\"\u003e\u003c\/div\u003e\u003cdiv class=\"phui-timeline-group\"\u003e\u003ca name=\"5641300\" id=\"5641300\" class=\"phabricator-anchor-view\"\u003e\u003c\/a\u003e\u003cdiv class=\"phui-timeline-title phui-timeline-title-with-icon\"\u003e\u003cspan class=\"phui-timeline-icon-fill\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-link phui-timeline-icon\" data-meta=\"0_156\" aria-hidden=\"true\"\u003e\u003c\/span\u003e\u003c\/span\u003e\u003ca href=\"\/p\/Ottomata\/\" class=\"phui-handle phui-link-person\" data-sigil=\"hovercard\" data-meta=\"0_56\"\u003eOttomata\u003c\/a\u003e mentioned this in \u003ca href=\"\/rDEPLOYCHARTS32985489151e08bf483e5ce8be61ac39830ba119\" class=\"phui-handle\" data-sigil=\"hovercard\" data-meta=\"0_57\"\u003erDEPLOYCHARTS32985489151e: Add eventgate-logging-external instance\u003c\/a\u003e.\u003cspan class=\"phui-timeline-extra\"\u003e\u003ca href=\"#5641300\" data-sigil=\"has-tooltip\" data-meta=\"0_155\"\u003e\u003cspan class=\"screen-only\"\u003eNov 6 2019, 5:31 PM\u003c\/span\u003e\u003cspan class=\"print-only\" aria-hidden=\"true\"\u003e2019-11-06 17:31:04 (UTC+0)\u003c\/span\u003e\u003c\/a\u003e\u003c\/span\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e\u003c\/div\u003e"},"javelin_metadata":[{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"hovercardSpec":{"objectPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-USER-a6yycs7vaq2biua4pudd","contextPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-TASK-ebzmmk2p6snzc2duzocg"}},{"hovercardSpec":{"objectPHID":"PHID-USER-wr3pospifdaxghmtobvr","contextPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-TASK-ebzmmk2p6snzc2duzocg"}},{"hovercardSpec":{"objectPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-USER-wr3pospifdaxghmtobvr","contextPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg","contextPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-USER-a6yycs7vaq2biua4pudd","contextPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg","contextPHID":"PHID-TASK-2rxhv4udwgw5h5fvmryf"}},{"phid":"PHID-XACT-TASK-rl6xchluaiz6wws"},{"hovercardSpec":{"objectPHID":"PHID-USER-mwkkwzbzlkvdl4w2og44"}},{"hovercardSpec":{"objectPHID":"PHID-TASK-somntgejbqsi2bj7vqcx"}},{"hovercardSpec":{"objectPHID":"PHID-USER-6s6jrjc3uyqg5sgjwv4b"}},{"hovercardSpec":{"objectPHID":"PHID-TASK-bkx4huysddznos2rhehl"}},{"hovercardSpec":{"objectPHID":"PHID-USER-a6yycs7vaq2biua4pudd"}},{"hovercardSpec":{"objectPHID":"PHID-USER-a6yycs7vaq2biua4pudd"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-zasg3muv4ljulhoy73kv"}},{"hovercardSpec":{"objectPHID":"PHID-USER-wr3pospifdaxghmtobvr"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-ei6zha7odprponwjhcjj"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-b5hqwk6xrkserdwaorsd"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-eftciewhdehz7y43577l"}},{"hovercardSpec":{"objectPHID":"PHID-USER-wgnf63ha7u7kzh4v5ryh"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-3ivs6ks3xwfuahj4qwjy"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-7dwoacokehhm5arjrxiw"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-aox2sshrsfrq4bsbicxb"}},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"phid":"PHID-XACT-TASK-nw7vkvazjercns7"},{"hovercardSpec":{"objectPHID":"PHID-USER-idceizaw6elwiwm5xshb"}},{"phid":"PHID-XACT-TASK-mre5o2znz4xvrqa"},{"hovercardSpec":{"objectPHID":"PHID-USER-idceizaw6elwiwm5xshb"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-onnxucoedheq3jevknyr"}},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-mseu6whr6z4kqtzmoahl"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-vtxgzcwqt5xtc6fh3rkx"}},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-gj52fgl7wdrdkoazrt6z"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-hn2ehxlwqk6di6ns7nej"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-mseu6whr6z4kqtzmoahl"}},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-g2u3mzw5hq5yluuoozey"}},{"hovercardSpec":{"objectPHID":"PHID-PCOL-muj5aceasd3wagihjl3x"}},{"hovercardSpec":{"objectPHID":"PHID-PROJ-vtxgzcwqt5xtc6fh3rkx"}},{"hovercardSpec":{"objectPHID":"PHID-USER-wr3pospifdaxghmtobvr"}},{"phid":"PHID-XACT-TASK-2zgchxcjd6psjav"},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"phid":"PHID-XACT-TASK-3ew3if5uzhi4cj4"},{"hovercardSpec":{"objectPHID":"PHID-USER-wr3pospifdaxghmtobvr"}},{"phid":"PHID-XACT-TASK-367ju243alk3whz"},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"phid":"PHID-XACT-TASK-2k5n6kxfoalhpfz"},{"hovercardSpec":{"objectPHID":"PHID-USER-idceizaw6elwiwm5xshb"}},{"phid":"PHID-XACT-TASK-laodhtqntgorwd4"},{"hovercardSpec":{"objectPHID":"PHID-USER-tafngdco2cilcyr7qhhg"}},{"hovercardSpec":{"objectPHID":"PHID-CMIT-qlqnzoj66m4t2towydj6"}},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-rl6xchluaiz6wws\/","ref":"T236386#5602851"},[],{"anchor":"5602851"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_1\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_58\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_59\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_3\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-rl6xchluaiz6wws\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_60\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_61\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-x3vbqw6omkwcbne","anchor":"5602851"},{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-wpq27wkjpgc6unu","anchor":"5604099"},{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-2m3w2f6i2gl42l2","anchor":"5607366"},{"tip":"Via Web"},[],[],{"phid":"PHID-XACT-TASK-2n7se3iz6i5cagn","anchor":"5610313"},{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-67sa6c3saqa2su3","anchor":"5618490"},{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-l42unmlfyxlbzix","anchor":"5619851"},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-nw7vkvazjercns7\/","ref":"T236386#5621245"},[],{"anchor":"5621245"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_5\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_83\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_84\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_7\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-nw7vkvazjercns7\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_85\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_86\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-nw7vkvazjercns7","anchor":"5621245"},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-mre5o2znz4xvrqa\/","ref":"T236386#5621302"},[],{"anchor":"5621302"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_9\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_92\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_93\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_11\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-mre5o2znz4xvrqa\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_94\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_95\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Conduit"},[],{"phid":"PHID-XACT-TASK-mre5o2znz4xvrqa","anchor":"5621302"},{"tip":"Via Conduit"},[],{"phid":"PHID-XACT-TASK-heorwue3xpyra6n","anchor":"5621303"},{"tip":"Via Web"},[],[],[],[],{"phid":"PHID-XACT-TASK-shlnzgsze4je4xv","anchor":"5621326"},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-2zgchxcjd6psjav\/","ref":"T236386#5624179"},[],{"anchor":"5624179"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_13\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_110\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_111\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_15\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-2zgchxcjd6psjav\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_112\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_113\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-2zgchxcjd6psjav","anchor":"5624179"},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-3ew3if5uzhi4cj4\/","ref":"T236386#5633635"},[],{"anchor":"5633635"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_17\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_119\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_120\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_19\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-3ew3if5uzhi4cj4\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_121\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_122\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-3ew3if5uzhi4cj4","anchor":"5633635"},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-367ju243alk3whz\/","ref":"T236386#5635277"},[],{"anchor":"5635277"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_21\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_128\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_129\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_23\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-367ju243alk3whz\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_130\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_131\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-367ju243alk3whz","anchor":"5635277"},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-2k5n6kxfoalhpfz\/","ref":"T236386#5636053"},[],{"anchor":"5636053"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_25\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_137\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_138\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_27\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-2k5n6kxfoalhpfz\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_139\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_140\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Web"},[],{"phid":"PHID-XACT-TASK-2k5n6kxfoalhpfz","anchor":"5636053"},{"targetID":"UQ0_1","uri":"\/transactions\/quote\/PHID-XACT-TASK-laodhtqntgorwd4\/","ref":"T236386#5641288"},[],{"anchor":"5641288"},[],{"items":"\u003cul class=\"phabricator-action-list-view \"\u003e\u003cli id=\"UQ0_29\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"#\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-quote\" data-meta=\"0_146\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-quote-left phabricator-action-view-icon\" data-meta=\"0_147\" aria-hidden=\"true\"\u003e\u003c\/span\u003eQuote Comment\u003c\/a\u003e\u003c\/li\u003e\u003cli id=\"UQ0_31\" class=\"phabricator-action-view phabricator-action-view-href action-has-icon\" style=\"\"\u003e\u003ca href=\"\/transactions\/raw\/PHID-XACT-TASK-laodhtqntgorwd4\/\" class=\"phabricator-action-view-item\" data-sigil=\"transaction-raw\" data-meta=\"0_148\"\u003e\u003cspan class=\"visual-only phui-icon-view phui-font-fa fa-code phabricator-action-view-icon\" data-meta=\"0_149\" aria-hidden=\"true\"\u003e\u003c\/span\u003eView Raw Remarkup\u003c\/a\u003e\u003c\/li\u003e\u003c\/ul\u003e"},[],{"tip":"Via Conduit"},[],{"phid":"PHID-XACT-TASK-laodhtqntgorwd4","anchor":"5641288"},{"tip":"Via Daemon"},[],{"phid":"PHID-XACT-TASK-7a2ikh3je6wpx6n","anchor":"5641300"}],"javelin_behaviors":{"phui-hovercards":[],"phui-dropdown-menu":[],"phabricator-watch-anchor":[],"phabricator-tooltips":[]},"javelin_resources":["https:\/\/phab.wmfusercontent.org\/res\/defaultX\/phabricator\/2eeda9e0\/core.pkg.js","https:\/\/phab.wmfusercontent.org\/res\/defaultX\/phabricator\/98e6504a\/rsrc\/externals\/javelin\/core\/init.js","https:\/\/phab.wmfusercontent.org\/res\/defaultX\/phabricator\/968d91ee\/core.pkg.css"]}