For whatever reason, I can no longer reproduce the issue. Closing as resolved.

In T126759#6482794, @akosiaris wrote:

Now that we 've upgraded to 6.0.29, we should probably revisit this. Not sure how to reproduce though.

In T275294#6846114, @Sam-2727 wrote:

It would be risky to choose one now only to have the fork abandoned soon.

Ah, thanks, got it now, they have released the fork and plan to release "an open version of Radiant 7 in 2021". I am not familiar with any of these OTRS-based platforms, unfortunately (although I'm pleasantly surprised that at least it looks like there are a few options for current users of the Community Edition).

Perhaps also worth noting: https://forums.otterhub.org/viewtopic.php?p=170802&sid=76b5e0255473a6df0f13137319fffe4c#p170802 (on Rother OSS's open source OTRS fork OTOBO, which was released in Summer 2020, and how their approach differs from Znuny's). Unlike Znuny's and Rother's, the open source forks announced by Radiant System and Centuran have not been released yet, as far as I can see.

Concerning security, I note that a few years ago it was noticed that an agent could inadvertedly share a URL with other agents or third parties that would allow the latter to take over their OTRS session. This could happen because: (1) While SessionUseCookie was set to true (default setting), when OTRS was unable to access the agent's cookies, it added the session ID to the URL (apparently the default behaviour). (2) SessionCheckRemoteIP was set to false in our OTRS instance because agents with highy dynamic/IPv6 IPs had issues accessing OTRS (T87217). The combination of (1) and (2) enabled the recipients of the URL to hijack the agent's session. (Furthermore, several now-fixed bugs were discovered in the past where a session ID was appended to a URL in certain circumstances even though the agent's cookies were working fine. T210861 perhaps being the latest example.) If the aforesaid still reflects the current behaviour of OTRS/Wikimedia's OTRS instance, I would submit that this cuts against a high SessionMaxTime/SessionMaxIdleTime from a risk mitigation point of view.

I frequently run into the same issue. OTRS terminates the session after SessionMaxTime/SessionMaxIdleTime seconds (not sure what these values currently are in our installation, defaults are 16h/2h). As far as I'm aware it does not store the content of the email compose form anywhere, so if you are able to retrieve it nonetheless that should indeed be coming from your browser. As a workaround, there are a few browser add-ons that continually save forms and allow you to easily restore their content, such as Typio Form Recovery for Chrome (which I am using with OTRS).

In T91683#4050749, @Jdlrobson wrote:

• What is the problem we're trying to solve?
  • Is it blacklisting certain images?
  • Is it allowing images not in the page (I think this could become a potential source of vandalism and unmaintainable)>

I guess you're referring to one of the ticket boxes on the Dashboard (https://ticket.wikimedia.org/otrs/index.pl?Action=AgentDashboard). There's a certain server-side lag of perhaps one or two seconds, so if you click "Spam" from within a ticket and OTRS immediately sends you back to the dashboard, you may still see the ticket you've just marked a spam. That can be confusing. It doesn't/shouldn't happen in queue view (https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketQueue), as far as I know.

Simply do not close spam tickets, as is also recommended here: https://otrs-wiki.wikimedia.org/wiki/Help:What_to_do_with_Spam%3F#The_ticket_I_am_looking_at_is_spam_%E2%80%93_what_should_I_do?

Can no longer reproduce this behavior in OTRS 5.0.23. If multiple tickets are selected, the bulk screen no longer contains the described orange notice boxes, one for each ticket, stacked at the top of the bulk processing screen; instead, OTRS now shows just one box with a comma-separated list of ticket numbers in it. Looks resolved to me.

Still works as expected, e.g.

Did not work in https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=10208291, despite the header being set. Hm.

@jsn.sherman I now opened https://twlight-staging.wmflabs.org/ in my normal browser window, clicked on "Log in", then was shown a "Welcome to the Wikipedia Library Card Terms of Use and Privacy Statement!" dialogue. I accepted the terms of use, which got me to the "Editor profile information for Pajz" with a "Start new application" button. I suppose that means it now worked?

@jsn.sherman Following your suggestion, I now (1) opened https://twlight-staging.wmflabs.org/ in a new icognito window, (2) clicked on "Log in", which prompted me to the log in screen on Meta, then (3) logged in with my credentials and (4) entered the 2FA code. I was then immediately shown the "Internal server error" again.

Ok, I just clicked on "Log in", but was immediately prompted with the "Internal server error" page again.

In T170097#3425260, @Gamaliel wrote:

Still unable to log in via the staging environment or the library card page.

The first time I tried to log in today, yes, but I can't really remember what precisely it said (I was forwarded to some page on Meta, right? I think I clicked a button that said something like "link this application", after which I got the error). Now, everytime I click "log in", it just gives me this error, without any steps in between.

See https://bugs.otrs.org/show_bug.cgi?id=10691#c15, in short - as far as I can see -: They removed the automatic assignment of CustomerIDs for new tickets in 5.0.16, then re-introduced the functionality in 5.0.17, but ever since then it needs to be specifically enabled through PostMaster::NewTicket::AutoAssignCustomerIDForUnknownCustomers. We should probably do that, since we don't assign CustomerIDs manually anyway.

Yes, indeed, looks like it. (Tested with permissions::permissions-de: If the latest article is an autoresponder message, it is omitted from Queue View - preview mode -.)

(Finally) fixed in OTRS 5.0.18, per release notes.

Upstream bug marked as resolved, fix slated to be deployed in OTRS 6.0.0.

Just adding here that in order for this fix to take effect, we will, in addition to updating, need to make sure that Ticket::NewArticleIgnoreSystemSender is set to Yes. Per https://bugs.otrs.org/show_bug.cgi?id=12596#c11 (follow-up fix).

This has also been reported to us in OTRS, and I can easily reproduce it (at least in incognito mode) on local file description pages like https://de.wikipedia.org/wiki/Datei:Otzenrathpano2.jpg?uselang=de. So I'd also like to echo that this issue is easy to run into at the moment, which makes me wonder if this should really be treated as a low-priority issue.

The image depicted is titled Otzenrathpano2.jpg, authored by Bodoklecksel and licensed under the GNU Free Documentation License, Version 1.2.

I can no longer reproduce the issue in the ticket this report is based on. https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=7933750#9407303 displays all relevant parts of the email. There have been a couple of changes in OTRS 5 related to how emails with multiple parts are displayed, so this might have been fixed.

Only ops will be able to give you a server-side assessment, but looking at the most recent tickets in Junk, Bayes scores seem to get added as expected. Are you sure the reports are referring to mail received after the issue was reported as fixed? (The fix described above would only affect new mail, not re-classify existing tickets.) The amount of new tickets from the past 5 hours that were manually marked as spam/moved to Junk also looks unsuspicious to me.

Can confirm that I can no longer reproduce the issue.

I can also still reproduce the issue; per https://bugs.otrs.org/show_bug.cgi?id=11878#c8, it seems to be an issue on our end, however. Quoting from there,

Can you please check, if you have a manually chenged template file for the widget (AgentDashboardUserOnline.tt) which still contains the unfixed code?

@akosiaris, could you check that? Perhaps this could be due to the fact that, if I'm not mistaken, we're modifying the Online user widget through our customized Wikimedia... package, so perhaps the old code is still contained there somehow?

Slated to be fixed in OTRS 5.0.14, scheduled for release on November 1.

The OTRS Phabricator project tag currently is intended for reporting/discussing technical matters affecting OTRS, the software used by volunteer-response team members to process tickets. If you have questions about the permissions process concerning particular files, please use https://commons.wikimedia.org/wiki/Commons:OTRS/Noticeboard (or a local eowiki village pump page) instead. Thanks!

We should probably remove this line alltogether (or replace it with something non-personalized) until we have that fixed. "Dear 'permissions-commons@wikimedia.org'," is very confusing indeed.

Which <OTRS_...> variable in the corresponding autoresponse (Admin panel -> Auto responses) generates the 'permissions-commons@wikimedia.org' bit?

Looks good. Looking at the histories of the most recent tickets, only one notification is sent per agent (as it should be), and the duplicates are also gone from the user settings panel. At least in my case, I can also say that my previous notification settings were unaffected, both according to my user settings and de facto. While the duplicate was in place, I started to get queue update notifications; but when I just tested this by moving a ticket from Junk to info-de and back, I no longer got such a notification. In other words, the previous behavior was fully restored.

Now, I can't say anything definite given the relevant servers are operated by the WMF, so I suppose only they'd be able to provide perfectly up-to-date information, but let me just add that the key Spamassassin configuration choices were made when we upgraded to OTRS 3. Essentially, there is a Bayes DB for mail routed to our OTRS instance. It is trained daily with email from our queues (Junk for spam, ordinary queues for ham) using a nightly-running export script in OTRS. See also https://wikitech.wikimedia.org/wiki/OTRS. Hence it's not like non-English mail isn't considered; however, naturally, it's outweighed by the much greater amount of English-language mail. This means we're basically feeding the Bayes DB with a very small amount of, say, Hungarian email. You would indeed expect this to generate non-ideal scoring results when it comes to queues for smaller languages.

Could you perhaps give some more detail (e.g., by providing a screenshot)? I've found an upstream bug related to non-scrollable iFrames in iOS, but that issue should have been fixed in the version we're currently using (http://bugs.otrs.org/show_bug.cgi?id=11717).

I changed the version in the upstream bug to 5.0.13 because the issue persists. (Hope that doesn't go against some bug reporting best practice ...)

This appears to be a known issue caused by the Online Agent widget (upstream bug report: http://bugs.otrs.org/show_bug.cgi?id=11878). According to the change notes, it's been fixed ever since OTRS 5.0.8 (our current version: 5.0.7; current stable version: 5.0.13). @akosiaris, do you know if there any plans to apply the latest of those patch-level updates? There are also one or two other bug reports here that likewise have seen upstream fixes since they were reported.

This might be fixed as part of the fix to http://bugs.otrs.org/show_bug.cgi?id=5149 (deployed in OTRS 5.0.13, released today; we're currently on 5.0.7). Quoting from there,

Still looks fine. Closing ...

Indeed, looks like the error has resolved itself. Seems to work fine again. I'll check again in a few minutes.

Can confirm (tested for several queues, including info-de, and for different response templates, including "empty answer").

Just noting here that OTRS started to include a script to move attachments from the database to the filesystem a couple of versions back, see https://otrs.github.io/doc/manual/admin/stable/en/html/performance-tuning.html#performance-tuning-otrs-storage (in case you want to go down that road).

Personally, I haven't heard of any complaints about the current behavior. On the contrary, I would expect some people to complain if we switched to the alternative (i.e., #2 above) since it makes life hard for agents dealing with spam, as noted in the task description. It's also effectively more similar to the old behavior.

An OTRS admin may want to check the current value of Ticket -> Ticket::Frontend::MergeText.

Thanks, but fortunately the upstream ticket has just been reopened after another developer was able to reproduce it. So hopefully we can save the extra effort.

Closed upstream as non-reproducible ("if possible please test it in a fresh installation"). @akosiaris (or perhaps @Krenair?), do we still have a test installation of OTRS 5 from the upgrade that could possibly be used for trying to reproduce this?

I don't agree with the sentiment that it is "hella annoying for us": Sometimes you want the ticket to be re-opened, sometimes you want to keep it closed, it just depends on the circumstances. Neither alternative is universally better than the other one. At least the current behavior makes sense because the ticket will always retain its previous state, no matter what you merge to it. Your suggestion would reduce that predictability. It's important to keep in mind that there are more than two states; if the current behavior were abandoned, it wouldn't be clear, for instance, what would happen if you merge a ticket with state pending reminder or pending open to a closed one. Right now that's easy for everyone to determine because we know the original state of the linked ticket will prevail. If, however, we start to manipulate this behavior based on our assumptions about what the merging agent "likely" wanted to achieve, agents would have difficulties determining how other states relate to "closed." This is not trivial.

Would it make sense to try to trace this down to the affected ticket? I just googled the text bit ("Saudi Arabia which faces inevitable military, politica") and it seems points to a "Voice of Bahrain" newsletter (http://english.voiceofbahrain.org/wp-content/uploads/2016/04/vob290.pdf) from this month by the "Bahrain Freedom Movement." There are indeed a few emails from the "Bahrain Freedom Movement" (info@vob.org), apparently containing such newsletters, e.g. "BFM Arabic Statement 15th April 2016", ticket 2016041510010245, but also others from past weeks.

OTRS 5.0.8 was released today, https://www.otrs.com/release-notes-otrs-5-patch-level-8/.

Fixed in OTRS 5.0.8 (not yet released), per the change notes.

Reported upstream, http://bugs.otrs.org/show_bug.cgi?id=11910.

Well, the OTRS interface says so (hover over "Powered by OTRS 5" at the bottom), and @akosiaris confirmed he completed the update earlier today. So I hope everything worked fine. The 5.0.7 change notes are at https://github.com/OTRS/otrs/blob/master/CHANGES.md#507-2016-02-16 ("Fixed bug#11838 - New tabs opened from links in a sandboxed Iframe cannot execute JavaScript"). Not sure what the issue is, in this case.

Could someone try to reproduce this? We were upgraded to OTRS 5.0.7 (which contains the fix referred to by @m00derp) today. According to the upstream bug report, that should now address the issue at least in Chrome, while Firefox users will have to wait pending Firefox support for the allow-popups-to-escape-sandbox tag (https://bugzilla.mozilla.org/show_bug.cgi?id=1190641).

Confirm it's been fixed. No longer an issue since our upgrade to 5.0.6.

Reported upstream, http://bugs.otrs.org/show_bug.cgi?id=11855

For reference, it appears the behavior of the Ticket::Frontend::AgentTicketMove###NextScreen setting has changed from OTRS 3 to OTRS 5. Previously, the two available values were "LastScreenOverview" and "LastScreenView" (default); now, the two values are "LastScreenOverview" and "TicketZoom" (default). In both OTRS 3 and OTRS 5 we were on the default option. The functionality seems to have changed accordingly:

Apparently solved by upgrading to OTRS 5. As an example, see https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=7065265#10646516 (compare to the preceding article in the ticket).

Note that according to the Ganglia chart @akosiaris has linked to, memory usage has just hit 6 GB -- 7 GB including cache (dunno if that's important in this context). Looking at Ganglia's 1d chart for mendelevium, I also note that the amount of memory used has essentially been ever-increasing in the past ~12h, and indeed increased by 1GB in the last 4 hours, so I wonder if we run into the same issues as before in 4 or 8h time.

Confirmed, just tried to send an email to myself, got the same error message and never received the email. (OTRS still created an article for it, though, https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=8965675.) Note that mail delivery wasn't always broken after the upgrade; e.g. there was an outbound email in https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=8965589 from Feb 3, 23:49 UTC for which we received the customer's reply a few minutes later.

To be honest, I don't see how that suddenly is a problem. How often did we have reports of minor translation mistakes? Off the top of my head, I'd guess maybe three, four times in the last couple of years? Given that, I can really understand the reluctance to maintain custom patches to fix a few translations, let alone implement a custom translation backend synchronizing with Transifex or another translation platform that would require extra maintenance efforts.

I would guess that's because, according to the forwarded message, it was sent to "info-fr@wikipedia.org." (note the period after ".org").

Has been fixed in the latest OTRS 4 and OTRS 5 versions, according to the OTRS development team (see upstream report), hence depends on T74109.

Thanks for checking. However, the problems described there seem to differ from the issue addressed with this report. In our case, the problem does not occure when a (too large) email is sent (outbound), but when we receive one with a large attachment.

Are there practical implications of this for administrative maintenance of OTRS? Would this affect the server name (iodine)? Does it require a re-configuration of the Spamassassin instance currently running on iodine? When OTRS is moved to a VM, does that have implications for, say, existing syslogs/backups, or are these all carried over 1:1?

The emails in question triggered a content-based filter (Bayes_999); since they all contain an identical text bit, it is not surprising to see them all going to Junk. The problem is that while, normally, OTRS would export identified false positives (i.e. emails moved back from Junk to a proper queue by hand) to SpamAssassin on a daily basis, this correction mechanism likely failed here because the emails were all merged to another ticket (ticket:2014121910011403). One of the side effects of merging a ticket to another one is, unfortunately, that this makes it impossible for the script we have in place to subsequently identify the false positive for the export job, so nothing gets exported. I don't think this is really a big problem in general as such a scenario generally seems highly improbable.

Yes, https://de.wikipedia.org/w/index.php?title=Outright_Monetary_Transactions&diff=140788890&oldid=140708948. That's the one corresponding to the screenshot.

The use case is as follows: Per current policies, we restrict access to OTRS Wiki to a subset of agents (mostly so-called "volunteer accounts," i.e. volunteers who work on info, permissions etc. queues). OTRS Wiki, however, serves a dual purpose: On the one hand, it hosts all the documentation as well as the coordination pages (say, a page to request account changes from an administrator); on the other hand, it is also a place where volunteers discuss, say, complicated cases that come up in the info-en::copyright queue. This is also why access it restricted to volunteer accounts, with all other OTRS agents only getting an account as needed.