Smalyshev (Stas Malyshev)
Engineer in Discovery team

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Nov 28 2014, 7:04 AM (199 w, 4 d)
Availability
Available
IRC Nick
Smalyshev
LDAP User
Smalyshev
MediaWiki User
Smalyshev (WMF) [ Global Accounts ]

Recent Activity

Yesterday

Smalyshev removed projects from T202764: Wikidata produces a lot of failed requests for recentchanges API: Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review.

Vast majority of remaining failures is from wdqs2003: https://logstash.wikimedia.org/goto/fe077467d39c2ee03ce8127bdca517ae

Tue, Sep 25, 7:23 AM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev added a comment to T205111: [EPIC] Transform wikidata autocomplete click logs into a useful dataset.

Of course in that literature they also don't have multiple options, the user is auto completing a query and not an item from the database. In our case the name displayed should be chosen by the highlighter,

Tue, Sep 25, 7:14 AM · Discovery-Search (Current work), Epic
Smalyshev moved T163642: Index Wikidata strings in statements for fulltext search from Done to In review on the User-Smalyshev board.
Tue, Sep 25, 5:17 AM · MW-1.32-release-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), Patch-For-Review, Discovery-Search (Current work), User-Smalyshev, User-aude, CirrusSearch, Discovery, Wikidata
Smalyshev moved T163642: Index Wikidata strings in statements for fulltext search from In review to Done on the User-Smalyshev board.
Tue, Sep 25, 5:17 AM · MW-1.32-release-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), Patch-For-Review, Discovery-Search (Current work), User-Smalyshev, User-aude, CirrusSearch, Discovery, Wikidata

Mon, Sep 24

Smalyshev moved T205301: Property searches in wikidatacompletionsearchclicks have mostly null values from Backlog to Next on the User-Smalyshev board.
Mon, Sep 24, 7:46 PM · User-Smalyshev, Discovery-Search (Current work), Wikidata
Smalyshev added a project to T205301: Property searches in wikidatacompletionsearchclicks have mostly null values: User-Smalyshev.
Mon, Sep 24, 7:46 PM · User-Smalyshev, Discovery-Search (Current work), Wikidata
Smalyshev added a comment to T205111: [EPIC] Transform wikidata autocomplete click logs into a useful dataset.

Session abandonment (typed something, but no final item selected).

Mon, Sep 24, 7:24 PM · Discovery-Search (Current work), Epic

Fri, Sep 21

Smalyshev added a comment to T197267: Create constant for Lexeme namespace.

This is about a PHP constant, not a data item for the lexeme namespace, right?

Fri, Sep 21, 7:03 PM · Patch-For-Review, Lexicographical data, Wikidata
Smalyshev moved T202830: Separate dumps for Lexemes from Backlog to In review on the User-Smalyshev board.
Fri, Sep 21, 6:02 AM · Patch-For-Review, User-Smalyshev, Wikidata, Lexicographical data
Smalyshev moved T144103: Create .nt (NTriples) dumps for wikidata data from In review to Done on the User-Smalyshev board.
Fri, Sep 21, 6:02 AM · Patch-For-Review, User-Smalyshev, Discovery, Wikidata-Query-Service, Wikidata
Smalyshev added a comment to T201478: Enhancements to vagrant dumps role.

To add some context, some Vagrant setups (i.e. on development machines) have /vagrant set up in the way that does not allow other users easily writing there and changing permissions. This causes a lot of hassle that could be avoided if the directory does not reside in /vagrant space.

Fri, Sep 21, 6:01 AM · Patch-For-Review, MediaWiki-Vagrant, Dumps-Generation
Smalyshev claimed T202830: Separate dumps for Lexemes.
Fri, Sep 21, 5:54 AM · Patch-For-Review, User-Smalyshev, Wikidata, Lexicographical data
Smalyshev moved T163642: Index Wikidata strings in statements for fulltext search from Waiting/Blocked to Needs review on the Discovery-Search (Current work) board.
Fri, Sep 21, 5:53 AM · MW-1.32-release-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), Patch-For-Review, Discovery-Search (Current work), User-Smalyshev, User-aude, CirrusSearch, Discovery, Wikidata

Thu, Sep 20

Smalyshev added a comment to T193645: [Epic] querying for lexicographical data.

Right now full lexeme dump is just 2.1M compressed, so adding it to main dump would not be a big deal for dump size. However, absent the separate dump, you'd have to always download the huge one, of course. Which makes me still support the separate dump route.

Thu, Sep 20, 11:48 PM · Epic, Discovery, Wikidata, Wikidata-Query-Service, Lexicographical data
Smalyshev closed T202459: Implement Lexeme data model for WDQS as Resolved.
Thu, Sep 20, 11:38 PM · Patch-For-Review, Discovery-Wikidata-Query-Service-Sprint, Discovery, Wikidata, Wikidata-Query-Service, Lexicographical data
Smalyshev closed T202459: Implement Lexeme data model for WDQS, a subtask of T193645: [Epic] querying for lexicographical data, as Resolved.
Thu, Sep 20, 11:38 PM · Epic, Discovery, Wikidata, Wikidata-Query-Service, Lexicographical data
Smalyshev moved T202459: Implement Lexeme data model for WDQS from Needs review to Done on the Discovery-Wikidata-Query-Service-Sprint board.
Thu, Sep 20, 11:38 PM · Patch-For-Review, Discovery-Wikidata-Query-Service-Sprint, Discovery, Wikidata, Wikidata-Query-Service, Lexicographical data
Smalyshev added a project to T202830: Separate dumps for Lexemes: User-Smalyshev.
Thu, Sep 20, 11:37 PM · Patch-For-Review, User-Smalyshev, Wikidata, Lexicographical data
Smalyshev added a comment to T200901: [Task] Implement RDF serialization for senses.

@Tpt this is done too, right?

Thu, Sep 20, 11:37 PM · MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), Patch-For-Review, Lexicographical data, Wikidata
Smalyshev closed T201885: Lexeme RDF export has labels repeated several times as Resolved.
Thu, Sep 20, 11:36 PM · MW-1.32-release-notes (WMF-deploy-2018-09-04 (1.32.0-wmf.20)), Patch-For-Review, Wikidata, Wikidata-Query-Service, Lexicographical data
Smalyshev closed T201885: Lexeme RDF export has labels repeated several times, a subtask of T195043: [Task] Implement RDF serialization for lexemes and forms, as Resolved.
Thu, Sep 20, 11:36 PM · MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), Patch-For-Review, Lexicographical data, Wikidata
Smalyshev closed T195043: [Task] Implement RDF serialization for lexemes and forms as Resolved.

I think this is done.

Thu, Sep 20, 11:35 PM · MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), Patch-For-Review, Lexicographical data, Wikidata
Smalyshev closed T195043: [Task] Implement RDF serialization for lexemes and forms, a subtask of T160259: [Story] RDF for Lexemes, Forms and Senses, as Resolved.
Thu, Sep 20, 11:35 PM · Lexicographical data, Wikidata
Smalyshev lowered the priority of T202764: Wikidata produces a lot of failed requests for recentchanges API from High to Normal.
Thu, Sep 20, 11:34 PM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev edited projects for T205025: Remove references to Wikidata from Wikidata Query Service UI, added: Wikidata Query UI; removed Wikidata-Query-Service.
Thu, Sep 20, 9:53 PM · Wikidata Query UI, Federated-Wikibase-Workshops@NewYork-2018, Wikidata
Smalyshev added a comment to T201478: Enhancements to vagrant dumps role.

What do folks think about putting all the 'misc dump scripts' (puppet/modules/snapshot/files/cron) in their own repo

Thu, Sep 20, 8:51 PM · Patch-For-Review, MediaWiki-Vagrant, Dumps-Generation
Smalyshev added a comment to T205025: Remove references to Wikidata from Wikidata Query Service UI.

Could you please be more specific as to what you mean by "contaminated with references to Wikidata", i.e. spell out specific things that need to be done?

Thu, Sep 20, 8:49 PM · Wikidata Query UI, Federated-Wikibase-Workshops@NewYork-2018, Wikidata
Smalyshev updated subscribers of T205005: Reindex produces a lot of Undefined index messages.

Yep, according to @Gehel there were restarts going on in the same time (my luck... sigh, should have checked) so maybe it's caused by restarts. Will try again and see. We probably should somehow detect this case anyway.

Thu, Sep 20, 8:20 PM · Discovery-Search
Smalyshev added a comment to T147505: [Recurring task] CirrusSearch: what is updated during re-indexing.

Thanks, and what about the spike in connection failures?

Thu, Sep 20, 7:35 PM · Discovery-Search (Current work), Discovery
Smalyshev edited projects for T205005: Reindex produces a lot of Undefined index messages, added: Discovery-Search; removed Discovery-Search (Current work).
Thu, Sep 20, 7:33 PM · Discovery-Search
Smalyshev added a comment to T147505: [Recurring task] CirrusSearch: what is updated during re-indexing.

@Krinkle: see T205005

Thu, Sep 20, 7:29 PM · Discovery-Search (Current work), Discovery
Smalyshev created T205005: Reindex produces a lot of Undefined index messages.
Thu, Sep 20, 7:27 PM · Discovery-Search
Smalyshev added a comment to T204813: Allow looking for items and lexemes namespaces together by default.

This is tricky since Lexeme and Item use different search queries (due to the fact that they have different structure).

Thu, Sep 20, 6:55 AM · Discovery-Search, Wikidata, Lexicographical data
Smalyshev closed T204389: Update wasat/mwmaint2001 docs on Wikitech as Resolved.
Thu, Sep 20, 6:45 AM · Documentation, Operations, wikitech.wikimedia.org
Smalyshev closed T204389: Update wasat/mwmaint2001 docs on Wikitech, a subtask of T199530: Rename of wasat to mwmaint2001 (switch labels et al), as Resolved.
Thu, Sep 20, 6:45 AM · netops, Operations, ops-codfw

Tue, Sep 18

Smalyshev added a project to T204699: cloudvps: wikidata-query project trusty deprecation: User-Smalyshev.
Tue, Sep 18, 11:00 PM · User-Smalyshev, Cloud-VPS
Smalyshev added a comment to T204699: cloudvps: wikidata-query project trusty deprecation.

OK, gotcha. It's going to take a bit of time as I need to remember the details of the whole setup there... ldfclient should be easy but pole has some setup involved.

Tue, Sep 18, 11:00 PM · User-Smalyshev, Cloud-VPS
Smalyshev triaged T154447: Move "wikibase_item" search field definition and indexing from CirrusSearch to Wikibase as Normal priority.
Tue, Sep 18, 10:39 PM · User-Smalyshev, wikidata-tech-focus, CirrusSearch, MediaWiki-extensions-WikibaseClient, Discovery, Wikidata
Smalyshev moved T154447: Move "wikibase_item" search field definition and indexing from CirrusSearch to Wikibase from Backlog to Next on the User-Smalyshev board.
Tue, Sep 18, 10:39 PM · User-Smalyshev, wikidata-tech-focus, CirrusSearch, MediaWiki-extensions-WikibaseClient, Discovery, Wikidata
Smalyshev moved T163642: Index Wikidata strings in statements for fulltext search from Doing to In review on the User-Smalyshev board.
Tue, Sep 18, 10:39 PM · MW-1.32-release-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), Patch-For-Review, Discovery-Search (Current work), User-Smalyshev, User-aude, CirrusSearch, Discovery, Wikidata
Smalyshev moved T163642: Index Wikidata strings in statements for fulltext search from In progress to Waiting/Blocked on the Discovery-Search (Current work) board.
Tue, Sep 18, 10:39 PM · MW-1.32-release-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), Patch-For-Review, Discovery-Search (Current work), User-Smalyshev, User-aude, CirrusSearch, Discovery, Wikidata
Smalyshev moved T202459: Implement Lexeme data model for WDQS from In progress to Needs review on the Discovery-Wikidata-Query-Service-Sprint board.
Tue, Sep 18, 10:38 PM · Patch-For-Review, Discovery-Wikidata-Query-Service-Sprint, Discovery, Wikidata, Wikidata-Query-Service, Lexicographical data
Smalyshev lowered the priority of T204267: Flood of WDQS requests from wbqc from High to Normal.

As the immediate problem ceased, resetting to Normal priority.

Tue, Sep 18, 10:37 PM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev placed T204267: Flood of WDQS requests from wbqc up for grabs.

Is Retry-after always provided?

Tue, Sep 18, 10:36 PM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

Looking at logstash: https://logstash.wikimedia.org/goto/39a6fe9edd787798129b66ae9d61ed90 there's definitely a drop in timeouts, but there are still present, so I will monitor this further.

Tue, Sep 18, 8:25 PM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev added a project to T154447: Move "wikibase_item" search field definition and indexing from CirrusSearch to Wikibase: User-Smalyshev.

Looks like it's still in Cirrus. Probably a good idea to move, I forgot about this one.

Tue, Sep 18, 5:50 PM · User-Smalyshev, wikidata-tech-focus, CirrusSearch, MediaWiki-extensions-WikibaseClient, Discovery, Wikidata
Smalyshev claimed T204699: cloudvps: wikidata-query project trusty deprecation.

By "upgrade", you mean shut down these VMs and create new ones with Stretch, or is it possible to migrate an existing VM?

Tue, Sep 18, 4:54 PM · User-Smalyshev, Cloud-VPS
Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

The API requests for recentchanges now seem to be faster, but I still get exceptions in the log :( I also get a bunch of errors for Wikidata URLs like: https://www.wikidata.org/wiki/Special:EntityData/Q33799921.ttl?nocache=1537250691109&flavor=dump
These are supposed to be pretty fast but still produce "no response" sometimes. I'll try to see what else can be causing those. Individual requests that I am testing seem to be fine, but I wonder if it's possible that the request still occasionally uses the DB host with wrong index?

Tue, Sep 18, 6:13 AM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata

Mon, Sep 17

Smalyshev committed rEWLE53167be519a3: Uses EntityMentionListener::subEntityMentioned to properly serialize Forms and… (authored by Tpt).
Uses EntityMentionListener::subEntityMentioned to properly serialize Forms and…
Mon, Sep 17, 10:02 PM
Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

I am a bit confused by now - is the original problem because recentchanges is using a wrong host, or it's using right host and the indexes there are wrong, or something else? And how can it be fixed? WDQS poller depends on RC API, and having it take 30+ seconds instead of usual sub-second response time is a serious issue.

Mon, Sep 17, 7:43 PM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev added a comment to T204471: WikibaseQualityConstraints include extension name in UA for query service requests.

Rate limiting is bucketing based on IP+User agent, so having distinctive user agent for WBQC is certainly a good idea.

Mon, Sep 17, 6:49 PM · Wikidata-Campsite-Iteration-∞, Wikidata-Campsite, wikidata-tech-focus, Wikibase-Quality-Constraints, Wikidata
Smalyshev triaged T204415: Query stats dashboard not updating as Normal priority.
Mon, Sep 17, 5:54 PM · Patch-For-Review, Analytics, Product-Analytics, Discovery-Analysis, Wikidata, Wikidata-Query-Service
Smalyshev added a comment to T204415: Query stats dashboard not updating.

But since August 10th, the SPARQL usage number is very small (even 0 for certain days)

Mon, Sep 17, 5:53 PM · Patch-For-Review, Analytics, Product-Analytics, Discovery-Analysis, Wikidata, Wikidata-Query-Service

Sat, Sep 15

Smalyshev added a project to T204415: Query stats dashboard not updating: Discovery-Analysis.
Sat, Sep 15, 9:57 PM · Patch-For-Review, Analytics, Product-Analytics, Discovery-Analysis, Wikidata, Wikidata-Query-Service
Smalyshev updated subscribers of T204415: Query stats dashboard not updating.

@mpopov, @chelsyx Do you know anything about this?

Sat, Sep 15, 9:56 PM · Patch-For-Review, Analytics, Product-Analytics, Discovery-Analysis, Wikidata, Wikidata-Query-Service
Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

I tried db2085:3318 and the result the same as other codfw host. So if that's what actual API is using, that could be the reason why it is so slow.

Sat, Sep 15, 1:04 AM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

@Reedy I am not sure which host, I just logged in to maintenance host for eqiad and codfw. Lookups show db2082.codfw.wmnet and db1092.eqiad.wmnet.

Sat, Sep 15, 1:02 AM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev created T204389: Update wasat/mwmaint2001 docs on Wikitech.
Sat, Sep 15, 12:36 AM · Documentation, Operations, wikitech.wikimedia.org
Smalyshev added a project to T202764: Wikidata produces a lot of failed requests for recentchanges API: DBA.

Looks like codfw one does not use index. @jcrespo do you have any idea why that could happen?

Sat, Sep 15, 12:25 AM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

Explains:

Sat, Sep 15, 12:24 AM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev created P7552 (An Untitled Masterwork).
Sat, Sep 15, 12:23 AM
Smalyshev updated the language for P7551 (An Untitled Masterwork) from autodetect to text.
Sat, Sep 15, 12:21 AM
Smalyshev edited P7551 (An Untitled Masterwork).
Sat, Sep 15, 12:21 AM
Smalyshev created P7551 (An Untitled Masterwork).
Sat, Sep 15, 12:20 AM
Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

This query:

SELECT  rc_id,rc_timestamp,rc_namespace,rc_title,rc_cur_id,rc_type,rc_deleted,rc_this_oldid,rc_last_oldid  FROM `recentchanges`    WHERE (rc_timestamp>='20180914110000') AND rc_namespace IN ('0','120')  AND rc_type IN ('0','1','3','6')   ORDER BY rc_timestamp ASC,rc_id ASC LIMIT 101  ;

on mwmaint1001 takes 0.00 sec, on mwmaint2001 takes 56.50 sec!

Sat, Sep 15, 12:12 AM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata

Fri, Sep 14

Smalyshev added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

Doesn't seem to be WDQS related entirely - e.g. if I call 'https://www.wikidata.org/w/api.php?format=json&action=query&list=recentchanges&rcdir=newer&rcprop=title%7Cids%7Ctimestamp&rcnamespace=0%7C120&rclimit=100&continue=&rcstart=2018-09-14T00%3A00%3A00Z' - i.e. try to load 100 items from start of the day today - it takes 29 seconds:

Fri, Sep 14, 10:08 PM · User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Smalyshev created T204378: Github complains about security issues in Wikibase dependencies.
Fri, Sep 14, 7:30 PM · Wikidata
Smalyshev added a comment to T204317: Don’t send SPARQL prefixes with WikibaseQualityConstraints queries.

A simple solution, I suppose, would be to completely skip prefixes for REGEX queries, which are (I believe) the most common queries we send out and never need any prefixes.

Fri, Sep 14, 4:16 PM · Wikibase-Quality-Constraints, Wikibase-Quality, Wikidata
Smalyshev added a comment to T204267: Flood of WDQS requests from wbqc.

All bans are temporary, so as soon as traffic returns to normal the bans will expire. It would be nice if there was a way to wbqc to respect the 429 throttling header, which will avoid bans.

Fri, Sep 14, 4:04 PM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev added a comment to T200563: wdq1003 is anomalous.

So, weird thing: now that we switched data centers, wdqs2003 is showing the same anomaly. Could it be that our load balancing is not balancing the load evenly for these hosts?

Fri, Sep 14, 8:15 AM · Wikidata-Query-Service, Wikidata
Smalyshev updated the task description for T204267: Flood of WDQS requests from wbqc.
Fri, Sep 14, 8:09 AM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev added a project to T204267: Flood of WDQS requests from wbqc: Wikibase-Quality.
Fri, Sep 14, 8:07 AM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev triaged T204267: Flood of WDQS requests from wbqc as Unbreak Now! priority.

12,499,055 throttling events in last 24 hours. This is definitely not good.

Fri, Sep 14, 8:06 AM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev created T204287: Form order is wrong in Lexeme UI.
Fri, Sep 14, 1:23 AM · Wikidata, Lexicographical data
Smalyshev added a comment to T197658: Provide easy script to reset Blazegraph.

Any idea what is going on here?

Fri, Sep 14, 12:58 AM · Discovery, Wikidata-Query-Service, Wikibase-Containers, Wikidata

Thu, Sep 13

Smalyshev moved T163642: Index Wikidata strings in statements for fulltext search from Needs review to In progress on the Discovery-Search (Current work) board.
Thu, Sep 13, 8:03 PM · MW-1.32-release-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), Patch-For-Review, Discovery-Search (Current work), User-Smalyshev, User-aude, CirrusSearch, Discovery, Wikidata
Smalyshev added a comment to T204267: Flood of WDQS requests from wbqc.

Kibana log for banned requests.

Thu, Sep 13, 5:22 PM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev created T204267: Flood of WDQS requests from wbqc.
Thu, Sep 13, 5:19 PM · Cloud-Services, Operations, User-Addshore, Wikibase-Quality, Wikidata, Wikidata-Query-Service, Wikibase-Quality-Constraints
Smalyshev added a comment to T200067: Define Libris XL as an endpoint in Wikidata.

I think the whitelist was not deployed yet, could you try again now and see if it works?

Thu, Sep 13, 12:06 AM · WMSE-Riksarkivet-TORA

Wed, Sep 12

Smalyshev added a comment to T204024: Store WikibaseQualityConstraint check data in an SQL table instead of in the cache.

There is the possibility that we will need to provide dumps of all constraint violations in order to ease the loading of data into WDQS servers that are starting from scratch

Wed, Sep 12, 7:55 PM · Cassandra, Services (designing), wikidata-tech-focus, Wikidata-Campsite, Wikibase-Quality-Constraints, Wikibase-Quality, Wikidata
Smalyshev edited projects for T204150: Embedded Wikidata Query Service timeline is not appropriately adjusted for its vertical size. , added: Wikidata Query UI; removed Wikidata-Query-Service.
Wed, Sep 12, 7:01 PM · Wikidata Query UI, Wikidata
Smalyshev added a comment to T197658: Provide easy script to reset Blazegraph.

it doesn't seem possible to connect to Blazegraph because the updater already runs?

Wed, Sep 12, 6:41 PM · Discovery, Wikidata-Query-Service, Wikibase-Containers, Wikidata
Smalyshev added a comment to T197658: Provide easy script to reset Blazegraph.

This error means that the timestamp stored in the database is more than 30 days behind (can be changed with wikibaseMaxDaysBack property). In this case, you can:

  • Load a dump that is reasonably recent
  • Run Updater with -s DATE --init
Wed, Sep 12, 5:41 PM · Discovery, Wikidata-Query-Service, Wikibase-Containers, Wikidata
Smalyshev updated the task description for T147505: [Recurring task] CirrusSearch: what is updated during re-indexing.
Wed, Sep 12, 5:28 AM · Discovery-Search (Current work), Discovery
Smalyshev triaged T195071: Add chronological sorting by-page-creation-timestamp for search results as Normal priority.
Wed, Sep 12, 5:26 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Discovery-Search (Current work), Patch-For-Review, Wikimedia-Hackathon-2018, CirrusSearch, Discovery
Smalyshev moved T203646: Wikidata Query Service nodes out of sync from Backlog to Doing on the User-Smalyshev board.
Wed, Sep 12, 5:25 AM · User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Smalyshev added a project to T203646: Wikidata Query Service nodes out of sync: User-Smalyshev.
Wed, Sep 12, 5:25 AM · User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Smalyshev moved T189739: [Epic] Implement fulltext search for Lexemes from Backlog to Done on the Discovery-Search (Current work) board.
Wed, Sep 12, 5:24 AM · Discovery-Search (Current work), User-Smalyshev, Lexicographical data, Epic, Wikidata, Discovery
Smalyshev edited projects for T189739: [Epic] Implement fulltext search for Lexemes, added: Discovery-Search (Current work); removed MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review.
Wed, Sep 12, 5:24 AM · Discovery-Search (Current work), User-Smalyshev, Lexicographical data, Epic, Wikidata, Discovery
Smalyshev added a comment to T199228: Define an SLO for Wikidata Query Service public endpoint and communicate it.

Is something happening on this or this was shelved for now?

Wed, Sep 12, 5:03 AM · Operations, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Smalyshev moved T189458: re-enable wdqs kafka poller from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.
Wed, Sep 12, 5:03 AM · Patch-For-Review, Discovery-Wikidata-Query-Service-Sprint, User-Smalyshev, Discovery, Wikidata, Wikidata-Query-Service
Smalyshev moved T203646: Wikidata Query Service nodes out of sync from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.
Wed, Sep 12, 5:03 AM · User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Smalyshev added a project to T203646: Wikidata Query Service nodes out of sync: Discovery-Wikidata-Query-Service-Sprint.
Wed, Sep 12, 5:03 AM · User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Smalyshev closed T202778: add ssds to wdqs2003 as Resolved.
Wed, Sep 12, 5:02 AM · ops-codfw, Operations, Discovery, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Smalyshev closed T202777: add SSDs to wdqs200[12] as Resolved.
Wed, Sep 12, 5:01 AM · ops-codfw, Operations, Discovery, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Smalyshev closed T202779: add SSDs to wdqs100[45] as Resolved.
Wed, Sep 12, 5:01 AM · ops-eqiad, Operations, Discovery, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata

Tue, Sep 11

Smalyshev updated subscribers of T201478: Enhancements to vagrant dumps role.

Applying wikidata role does not create content, AFAIK. I just went to Special:NewItem and created a bunch of them manually. There is probably a way to load it from dumps etc. (WMDE folks and particularly @Addshore may know some better ways) but I just made them manually.

Tue, Sep 11, 6:13 AM · Patch-For-Review, MediaWiki-Vagrant, Dumps-Generation
Smalyshev added a comment to T203646: Wikidata Query Service nodes out of sync.

@Lucas_Werkmeister_WMDE your query finds a lot of statements with wdno: claims, which do not have ps:.

Tue, Sep 11, 5:57 AM · User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata

Mon, Sep 10

Smalyshev placed T202785: Federation request to https://ld.stadt-zuerich.ch/query fails up for grabs.

So repackaging with a more recent jetty-http (or the whole jetty stack) might not be that hard

Mon, Sep 10, 11:45 PM · Wikidata, Wikidata-Query-Service
Smalyshev claimed T203646: Wikidata Query Service nodes out of sync.

Something weird is definitely going on - out of 582769 statements with P39, we have 2334 that are missing rank (and possibly other clauses). As statement should never ever be missing rank, it's clearly some bug. I'll dig into it and see how it could happen.

Mon, Sep 10, 11:40 PM · User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata