@Mooeypoo thanks for clarifying requirements of the current project better, it's much clearer now.
So first of all, why do wtp servers have php installed even? They should not, and they don't.
Thu, Mar 21
I'm a bit conflicted about this, and let me clarify why:
Tue, Mar 19
Mon, Mar 18
@Krinkle now that I'm back, what needs to be done on the SRE side for this to be considered done?
Sorry for coming late to the party, I was AFK last week.
Looking at the logs on the server:
- 2019-03-11T06:32:48 errors start
- 2019-03-11T06:34:57 opcache received a request to invalidate a single file /opcache-free?file=wmf-config%2Fdb-eqiad.php, during a scap run.
- 2019-03-11T06:49:09 last error is logged
- 2019-03-11T06:49:22 a full opcache flush was requested /opcache-free - not sure what caused this, but it came from scap.
So while the errors found by @MaxSem on logstash seem to refer to a host that was somehow in a bad state in terms of opcache state, which will need further analysis, the other issue to fix is the display_errors ini setting, which used to be in our php configuration and I didn't modify.
Tue, Mar 12
Well instead of copy/pasta, there is that thing called YAML anchors/references that deal with data repetition in YAML files.
Thu, Mar 7
Wed, Mar 6
Tue, Mar 5
apt-get install php7.2-msgpack solved the problem FWIW.
@Mvolz this means we can reconfigure citoid to use both proxies?
Mon, Mar 4
Thu, Feb 28
Applies magic wand solved!
Wed, Feb 27
FWIW I would like to decrease the max key size instead than increasing it.
Tue, Feb 26
Mon, Feb 25
Sat, Feb 23
Fri, Feb 22
As far as I can see we're using packages generated by the following source packages:
So, while jit is enabled by default on PHP 7.2 (pcre.jit is 1 by default), I don't see how perf could help in knowing how full the JIT VM is (which is what we want to measure probably).
Personally speaking, I think the real blocker is proper resourcing for doing this work, which isn't something we can do as a side job. Using redis-sentinel while maintaining our operational standards means we have to run tests, set up appropriate monitoring, learn how to recover from the inevitable failure scenarios.
Feb 21 2019
This task is resolved per-se, we still might need the mw-config patches if they weren't merged.
Feb 20 2019
On-demand profiling now works on mwdebug1002.
Feb 19 2019
I did manually install php7.2-tideways-xhprof on mwdebug1001 and I now see the following error:
So basically either we rewrite profiler.php to support the old tideways extension or (which I prefer at this point, tbh) we build just the xhprof component, which seems to have a smaller surface.
I just ran a simple test to verify if profiling to xhgui works in PHP 7, after @Krinkle told me it doesn't.
Feb 15 2019
I did keep beta in mind when writing this, as it is apparent from labs/deployment-prep/common.yaml containing a hiera key to ensure we won't use or install the services proxy there.
Given how efficient the jobqueue is, we can expect that more than 99% of the preferences will be saved before a new search happens. Having some sort of protection against that 1% (like, localstorage "tricks") would help, but I'd argue that would be better than 50% of the calls needing a cross-datacenter write in terms of reliability and user experience.
But this is clearly a more general question: do we prefer to impose strong causality at the risk of increasing the overall error rate, or do we prefer to accept asynchronicity and deal with the added complexity, while guaranteeing a better eventual consistency?
Feb 14 2019
@ssastry you should aim at php 7.x support, with x >= 2.
In order to better understand your needs, let me ask you a few questions:
oblivian@restbase1016:~$ sudo -i pool-restbase oblivian@restbase1016:~$ echo $? 0
The Jit TC cold portion was completely full and exhausted on that server. A restart of HHVM should've solved the issue.
Feb 13 2019
@jijiki what is the total number of items stored on those redises? So that I can understand how much of that is used by the sessions. I guess less than 1%?
@Gehel I would say what's missing is a clear dependency between the installation of the cassandra package and the apt-get update.
I've just noticed, based on a diffscan email, that the new version of prometheus-node-exporter ALSO binds to :::9100 on ipv6 and listens to all ipv6 clients, while the old node exporter version would only bind to a specific interface on ipv4 and listen on that interface.
Feb 12 2019
Please note this is fixed on jessie but not on stretch. I'm going to look into it now.
this is a result of a defect in python3-etcd packaging (so, blame me!)
@brennen your key was added to production; let me know if you have any problem accessing production servers, either here or on IRC.
Wouldn't it be feasible to have the search request generate a simple job that saves that preference asynchronously?
FYI, I've merged a change yesterday that should've fixed the problem from now on.
Feb 11 2019
I will assume you can successfully access and just resolve the ticket. Please reopen it if any issue happens.
Feb 9 2019
Thanks @phuedx for opening the task!
Feb 8 2019
`All the extensions were not upgraded at the time we did the 7.0 => 7.2 transition - my bad! Updating them manually was the only thing that was needed - we needed to update all of them and not just php-redis.