Elasticsearch credential request for spi-tools-dev
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RoySmith
	Jan 11 2022, 3:23 AM

Project Tags

Referenced Files

None

Subscribers

Description

I am working on a tool which will make it possible to search edit summaries. The basic idea is to parse the page.revision.comment tags out of the pages-meta-history SQL dump files and build an elasticsearch index for them.

Details are still unclear, but it's looking like the dump parsing and search index building will both take place as k8s batch jobs on Toolforge. My initial work will be a proof-of-concept using a small subset of the dump data. Once I've got that working, I'll be in a better position to estimate the resource consumption for the full search index.

The original incentive for this is for use in sockpuppet investigations, so I anticipate this will be exposed to end users via the spi-tools web service running on https://spi-tools.toolforge.org/

I run spi-tools as the production service, and spi-tools-dev for testing. It would probably make sense to have distinct elastic search credentials for each of those.

Related Objects

Mentioned In: T300319: Elasticsearch credential request for spi-indexer
Mentioned Here: P4372 new-es-password.sh

Event Timeline

RoySmith created this task.Jan 11 2022, 3:23 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2022, 3:23 AM

Grant procedure for a Toolforge admin to use documented at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Granting_a_tool_write_access_to_Elasticsearch

Could I get a status update on this?

• nskaggs claimed this task.Jan 19 2022, 7:02 PM

@RoySmith I added credentials for spi-tools-dev. If/when you'd like distinct credentials for spi-tools, just let us know -- at first glance they didn't seem to exist.

• nskaggs closed this task as Resolved.Jan 19 2022, 7:09 PM

Hmmm, I see the file, but it's empty:

-rw-r----- 1 root tools.spi-tools-dev 0 Jan 19 19:04 /data/project/spi-tools-dev/.elasticsearch.ini

Sorry about that! It should be present and not empty now.

OK, I see it now, thanks.

The file appears to be incorrectly formatted.

cat $HOME/.elasticsearch.ini  | sed 's/=.*/=xxxx/'
user=xxxx
password=xxxx

There's no section heading, so the standard python configparser can't parse it:

Traceback (most recent call last):
  File "./elastic.py", line 22, in <module>
    main()
  File "./elastic.py", line 13, in main
    config.read_file(config_file.open())
  File "/usr/lib/python3.7/configparser.py", line 717, in read_file
    self._read(f, source)
  File "/usr/lib/python3.7/configparser.py", line 1079, in _read
    raise MissingSectionHeaderError(fpname, lineno, line)

and the file is owned by root, so I can't fix it myself.

Try now?

Also updated the script to provide valid ini files: P4372#94992

That did the trick, thanks.

RoySmith mentioned this in T300319: Elasticsearch credential request for spi-indexer.Jan 28 2022, 12:17 AM

Elasticsearch credential request for spi-tools-devClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Elasticsearch credential request for spi-tools-dev
Closed, ResolvedPublic
Actions