Page MenuHomePhabricator

fgiunchedi (Filippo Giunchedi)
/* No comment */

Projects (19)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 8:06 AM (254 w, 5 d)
Availability
Available
IRC Nick
godog
LDAP User
Filippo Giunchedi
MediaWiki User
Filippo Giunchedi [ Global Accounts ]

Recent Activity

Yesterday

fgiunchedi created T230934: Elevated php7 latency during mw deploy.
Wed, Aug 21, 2:39 PM · Operations, MediaWiki-General
fgiunchedi reopened T221202: kafka-logging __consumer_offsets topic traffic increased as "Open".

Reopening since traffic to consumer-offsets has doubled last night after T230847: Logstash gets significantly lower number of messages from mediawiki :( most of the messages seem to be handled by logstash1010

Wed, Aug 21, 9:00 AM · observability, Wikimedia-Logstash
fgiunchedi added a comment to T230847: Logstash gets significantly lower number of messages from mediawiki.

Looks like a logstash consumer has failed, according to kafka logs on logstash1010

Wed, Aug 21, 8:56 AM · Wikimedia-Incident, Operations, Wikimedia-Logstash
fgiunchedi lowered the priority of T230847: Logstash gets significantly lower number of messages from mediawiki from Unbreak Now! to Normal.

Downgrading to normal since the situation has stabilized and returned to normal as 23:40 UTC, still pending investigation on what caused it

Wed, Aug 21, 8:40 AM · Wikimedia-Incident, Operations, Wikimedia-Logstash

Tue, Aug 20

fgiunchedi added a comment to T230799: icinga-wm bot truncating long messages.

For reference, the lines above appear as below in /var/log/icinga/irc.log:

Tue, Aug 20, 1:31 PM · observability, Icinga, Operations
fgiunchedi awarded T230752: Deploy phatality into kibana a Like token.
Tue, Aug 20, 1:26 PM · observability, Release-Engineering-Team-TODO (201908), Wikimedia-Logstash, Phabricator
fgiunchedi updated subscribers of T230752: Deploy phatality into kibana.

@mmodell for sure, me or @herron or @colewhite can help with that, what timeline were you looking at for the deployment ? I did a quick search on gerrit/phab for phatality but couldn't find the source, do you have a pointer to that? Yes scap could be an option but I don't think we have "prior art" for scap + kibana plugins. re: upgrades yes there are plans to move to the logging pipeline to Elastic stack 7 and above here: T183281: [epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash) although no timeline yet

Tue, Aug 20, 7:21 AM · observability, Release-Engineering-Team-TODO (201908), Wikimedia-Logstash, Phabricator

Mon, Aug 19

fgiunchedi created T230733: Expose pooled status of gdnsd and conftool managed services as metrics.
Mon, Aug 19, 3:59 PM · User-CDanis, Operations, observability
fgiunchedi moved T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline from Backlog to Up next on the observability board.
Mon, Aug 19, 2:58 PM · Analytics, observability, Wikimedia-Logstash, Operations
fgiunchedi moved T97297: Select a standard log shipping solution to use with applications that cannot be configured to send log events directly to Logstash and/or fluorine from Backlog to Up next on the observability board.
Mon, Aug 19, 2:57 PM · observability, Operations, Wikimedia-Logstash
fgiunchedi moved T150106: Type collisions in log events causing indexing failures in ELK Elasticsearch from Backlog to Up next on the observability board.
Mon, Aug 19, 2:57 PM · observability, MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Analytics, Patch-For-Review, MW-1.29-release-notes, EventBus, Wikimedia-Logstash
fgiunchedi moved T176335: logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable from Backlog to Up next on the observability board.
Mon, Aug 19, 2:57 PM · observability, Core Platform Team Legacy (Watching / External), Services (watching), Operations, Elasticsearch, Wikimedia-Logstash
fgiunchedi added a project to T146472: Investigate stopping logging api-feature-usage channel in Logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-production-error, Wikimedia-Logstash, MediaWiki-API
fgiunchedi added a project to T78705: Send Jenkins build log and results to ElasticSearch: observability.
Mon, Aug 19, 2:34 PM · observability, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Wikimedia-Hackathon-2017, Patch-For-Review, Wikimedia-Logstash, Jenkins, Continuous-Integration-Infrastructure
fgiunchedi added a project to T1280: Setup udp2log on logstash-mw for testing: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi edited projects for T127831: Upgrade ruflin/elastica to 2.3.1, added: observability; removed MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)).
Mon, Aug 19, 2:34 PM · observability, MW-1.27-release (WMF-deploy-2016-03-29_(1.27.0-wmf.19)), Patch-For-Review, MediaWiki-Vendor, Discovery-Search (Current work), Operations, Wikimedia-Logstash, Elasticsearch, Discovery
fgiunchedi added a project to T70820: Automatically log fatals/exceptions into Phabricator with stack traces: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63755: Show deployment events in logstash time histogram: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T84907: Kafka logging to Logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Analytics, Wikimedia-Logstash
fgiunchedi added a project to T122698: Upgrade ElasticSearch to version >=2.2: observability.
Mon, Aug 19, 2:34 PM · observability, Discovery-Search, Wikimedia-Logstash, Discovery, Elasticsearch
fgiunchedi added a project to T84908: Zookeeper logging to Logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Analytics-Engineering, Wikimedia-Logstash
fgiunchedi added a project to T178442: ssl terminators on elasticsearch servers (nginx) don't send their logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Discovery-Search, Wikimedia-Logstash
fgiunchedi added a project to T180051: Reduce the number of fields declared in elasticsearch by logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Core Platform Team Legacy (Watching / External), Services (watching), Operations, Wikimedia-Logstash
fgiunchedi added a project to T184602: logstash-beta.wmflabs.org default dashboard missing: observability.
Mon, Aug 19, 2:34 PM · observability, Beta-Cluster-Infrastructure, Wikimedia-Logstash
fgiunchedi added a project to T212946: Stream Thumbor logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash, User-jijiki, serviceops, Operations, Thumbor
fgiunchedi added a project to T1: Get puppet runs into logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Patch-For-Review, Wikimedia-Logstash
fgiunchedi added a project to T3: Upgrade logstash elasticsearch instance to latest version: observability.
Mon, Aug 19, 2:34 PM · observability, Patch-For-Review, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T7: Get icinga alerts into logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T2: Get salt logs into logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T62690: Add puppet logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Operations, Wikimedia-Logstash
fgiunchedi added a project to T4: Upgrade logstash to 1.4.2: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash, Release-Engineering-Team
fgiunchedi added a project to T62689: Add salt logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63543: Add icinga alerts to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63545: Add scap logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63544: Add !log entries from #-operations to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T63549: Upgrade logstash to latest stable version: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63551: Upgrade logstash Elasticsearch cluster to 1.0: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63754: Find a better auth mechanism for logstash in labs: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63779: Add system logs to logstash (tracking): observability.
Mon, Aug 19, 2:34 PM · observability, Tracking-Neverending, Wikimedia-Logstash
fgiunchedi added a project to T63780: Add swift logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Patch-For-Review, Wikimedia-Logstash
fgiunchedi added a project to T63782: Add varnish logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Patch-For-Review, Operations, Traffic, Wikimedia-Logstash
fgiunchedi added a project to T63781: Add syslog logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63789: Add Zookeeper log files to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Patch-For-Review, Wikimedia-Logstash
fgiunchedi added a project to T63788: Add kafka log files to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T63790: Add Hadoop mapreduce logfiles to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T64505: Add gerrit event stream events to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Gerrit, Wikimedia-Logstash
fgiunchedi added a project to T64817: Upgrade kibana to latest stable version: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T64177: Add localisation update logs to logstash: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T64227: Send aggregate php error counts from logstash to graphite: observability.
Mon, Aug 19, 2:34 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T74202: Logstash package installs both /etc/init.d/logstash and /etc/init/logstash.conf scripts: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T73468: Upgrade logstash to 1.4.2: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T65490: Logstash occasionally hangs as a result of a memory leak caused by non-UTF-8 input: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T74203: Redis input filter failing to start: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T74284: iegreview events not ending up in elasticsearch: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-IEG-grant-review, Wikimedia-Logstash
fgiunchedi added a project to T74930: Update puppet managed copy of elasticsearch template: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T81030: gdash reports for php/apache errors: observability.
Mon, Aug 19, 2:33 PM · observability, WorkType-NewFunctionality, Operations, Patch-For-Review, Wikimedia-Logstash
fgiunchedi added a project to T75756: Merge duplicate projects #logstash and #wikimedia-logstash: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash, Logstash
fgiunchedi added a project to T76784: Make logstash in beta public: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, User-bd808, Wikimedia-Logstash, Release-Engineering-Team, Beta-Cluster-Infrastructure
fgiunchedi added a project to T76760: Ignore apache2 log2udp messages and transition rules to the syslog stream of the same events: observability.
Mon, Aug 19, 2:33 PM · observability, MediaWiki-Core-Team, Wikimedia-Logstash
fgiunchedi added a project to T78195: Make beta logstash server based on a Trusty base image: observability.
Mon, Aug 19, 2:33 PM · observability, Beta-Cluster-Infrastructure, Wikimedia-Logstash
fgiunchedi added a project to T85239: Channel PHP errors from Logstash to Sentry on the beta cluster: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, Beta-Cluster-Infrastructure, Wikimedia-Logstash, Sentry
fgiunchedi added a project to T85015: Convert Hadoop-Logstash logging to use Redis to address failures: observability.
Mon, Aug 19, 2:33 PM · observability, Operations, Wikimedia-Logstash, Analytics-Engineering
fgiunchedi added a project to T84895: Improve logstash : observability.
Mon, Aug 19, 2:33 PM · observability, Operations, Wikimedia-Logstash, Epic
fgiunchedi added a project to T89402: purchase 3 additional logstash nodes: observability.
Mon, Aug 19, 2:33 PM · observability, MediaWiki-Debug-Logger, Operations, Wikimedia-Logstash, hardware-requests
fgiunchedi added a project to T87078: Upgrade RAM for logstash100[123] to 64G: observability.
Mon, Aug 19, 2:33 PM · observability, User-bd808, MediaWiki-Debug-Logger, Operations, Wikimedia-Logstash
fgiunchedi added a project to T90889: Deploy statsd plugin for production elasticsearch & logstash: observability.
Mon, Aug 19, 2:33 PM · Discovery-Search, observability, Operations, Discovery, Graphite, Wikimedia-Logstash, Elasticsearch
fgiunchedi added a project to T97545: reinstall logstash1001-1003: observability.
Mon, Aug 19, 2:33 PM · observability, Operations, Patch-For-Review, Wikimedia-Logstash
fgiunchedi added a project to T97297: Select a standard log shipping solution to use with applications that cannot be configured to send log events directly to Logstash and/or fluorine: observability.
Mon, Aug 19, 2:33 PM · observability, Operations, Wikimedia-Logstash
fgiunchedi added a project to T98042: Update Wikimedia apt repo to include debs for Elasticsearch & Logstash on jessie: observability.
Mon, Aug 19, 2:33 PM · Discovery-Search, observability, Operations, Discovery, Elasticsearch, Wikimedia-Logstash
fgiunchedi added a project to T99735: Upgrade Logstash to 1.5.3: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T100735: Have LogStash report per-channel log message rate to Graphite: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T103804: deployment-logstash02 fails puppet: Apache2 can't start, mod_authz_groupfile not enabled on Jessie: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, Wikimedia-Logstash, Beta-Cluster-Infrastructure
fgiunchedi added a project to T99541: Fix logstash apifeatureusage role in Beta Cluster: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash, ApiFeatureUsage
fgiunchedi added a project to T103910: Prepare tech talk on ELK stack at WMF: observability.
Mon, Aug 19, 2:33 PM · observability, MediaWiki-Debug-Logger, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T103942: Restbase logging should populate "message" field in logstash: observability.
Mon, Aug 19, 2:33 PM · observability, RESTBase, Wikimedia-Logstash
fgiunchedi added a project to T107083: Java class org.bouncycastle.jcajce.provider.digest.MD5$Digest not found for Logstash on logstash1001 (jessie): observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T105101: Upgrade Logstash Elasticsearch cluster to 1.6.0: observability.
Mon, Aug 19, 2:33 PM · observability, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T109101: Currently elasticsearch logs do not leave nodes. We use logstash for this across the cluster generally.: observability.
Mon, Aug 19, 2:33 PM · Discovery-Search, observability, Patch-For-Review, Wikimedia-Logstash, Discovery, Elasticsearch
fgiunchedi added a project to T107121: Setup rsyncable git fat store to host Logstash plugins: observability.
Mon, Aug 19, 2:33 PM · observability, Operations, Wikimedia-Logstash
fgiunchedi added a project to T107916: Import logstash 1.5.3 into apt.wm.o: observability.
Mon, Aug 19, 2:33 PM · observability, Operations, Wikimedia-Logstash
fgiunchedi added a project to T109126: logstash insertions to ElasticSearch cross-DC functionality needs figuring out: observability.
Mon, Aug 19, 2:33 PM · observability, Discovery-Search, Wikimedia-Logstash, ApiFeatureUsage, Discovery, Elasticsearch
fgiunchedi added a project to T108040: Upgrade Logstash Elasticsearch cluster to 1.7.1: observability.
Mon, Aug 19, 2:33 PM · observability, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T113571: Logstash elasticsearch cluster filled up, dropping logstash events: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T113627: Make Logstash consume from Kafka:eventlogging_EventError {Oryx} [8 pts]: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash, Patch-For-Review, Analytics-Kanban
fgiunchedi added a project to T113104: Set up a service IP for logstash: observability.
Mon, Aug 19, 2:33 PM · observability, Operations, Wikimedia-Logstash
fgiunchedi added a project to T117438: Reduce replica count from 2 to 1 for indices that are >21 days old: observability.
Mon, Aug 19, 2:33 PM · observability, User-bd808, Wikimedia-Logstash
fgiunchedi added a project to T119594: MediaWiki logstash 'referrer' field should not be html escaped: observability.
Mon, Aug 19, 2:33 PM · observability, Patch-For-Review, User-bd808, Wikimedia-Logstash, MediaWiki-Debug-Logger
fgiunchedi added a project to T114849: Log lines on flourine overflow at 8092 bytes.: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash, Operations
fgiunchedi added a project to T117821: Make a udp2log output plugin for Logstash: observability.
Mon, Aug 19, 2:33 PM · observability, Wikimedia-Logstash, MediaWiki-Debug-Logger
fgiunchedi added a project to T129934: logstash - nginx failed service start: observability.
Mon, Aug 19, 2:32 PM · observability, Discovery-Search, Patch-For-Review, Discovery, Elasticsearch, Wikimedia-Logstash, Operations
fgiunchedi added a project to T136001: [EPIC] Upgrade elasticsearch cluster supporting logging to 2.3 : observability.
Mon, Aug 19, 2:32 PM · observability, Patch-For-Review, Operations, Discovery-Search (Current work), Epic, Discovery, Wikimedia-Logstash
fgiunchedi added a project to T119682: Filter detailed OCG activity log messages out of ELK stack: observability.
Mon, Aug 19, 2:32 PM · observability, User-bd808, OCG-General, Wikimedia-Logstash
fgiunchedi added a project to T132458: Move logstash.wikimedia.org (kibana) to an LVS service: observability.
Mon, Aug 19, 2:32 PM · observability, Patch-For-Review, Wikimedia-Logstash, Traffic, Operations
fgiunchedi added a project to T127676: Upgrade Logstash to v2.2.2: observability.
Mon, Aug 19, 2:32 PM · observability, Wikimedia-Logstash
fgiunchedi added a project to T136002: Test logstash 1.5.3 and kibana 3.1.2 against elasticsearch 2.3: observability.
Mon, Aug 19, 2:32 PM · observability, Discovery-Search (Current work), Discovery, Wikimedia-Logstash
fgiunchedi added a project to T137400: Logstash elasticsearch mapping does not allow err.code to be a string: observability.
Mon, Aug 19, 2:32 PM · observability, User-bd808, Discovery-Search (Current work), Discovery, Operations, Elasticsearch, Wikimedia-Logstash
fgiunchedi added a project to T136003: Upgrade logstash*.eqiad.wmnet elasticsearch servers to 2.3: observability.
Mon, Aug 19, 2:32 PM · observability, Discovery-Search (Current work), Discovery, Wikimedia-Logstash
fgiunchedi added a project to T140973: logstash - cron failing to optimize indices: observability.
Mon, Aug 19, 2:32 PM · observability, Discovery-Search, Elasticsearch, Operations, Wikimedia-Logstash, Discovery
fgiunchedi added a project to T139368: puppet failed on deployment-logstash2.deployment-prep: observability.
Mon, Aug 19, 2:32 PM · observability, Beta-Cluster-Infrastructure, Wikimedia-Logstash
fgiunchedi added a project to T140751: Create a logstash input filter to preprocess mysqld syslog messages: observability.
Mon, Aug 19, 2:32 PM · observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure, WorkType-NewFunctionality