Page MenuHomePhabricator
Create Task
Maniphest T330755

videocuttool Cloud VPS project uses MongoDB, which is not available under an OSI approved license
Closed, ResolvedPublic
Actions

Description

videocuttool.videocuttool.eqiad1.wikimedia.cloud has a MongoDB 4.2 installation. That version is licensed under their "Server Side Public License", which is not approved by the OSI as an open source license. Running such software in Cloud VPS is a violation of the Cloud Services Terms of Use.

Details

SubjectRepoBranchLines +/-
Use postgresql instead of mongodblabs/tools/VideoCutToolmaster+2 K -2 K
Customize query in gerrit

Related Objects

Event Timeline

taavi created this task.Feb 28 2023, 2:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2023, 2:19 PM
Soda subscribed.Feb 28 2023, 2:33 PM

@taavi Are there any alternatives (NoSQL databases that are interoperable with NodeJS) that we can shift to that are offered by the WMCS ?

taavi added a comment.Feb 28 2023, 3:01 PM

The Cloud VPS database-as-a-service currently does not support any NoSQL database types.

Soda added a subscriber: Gopavasanth.Mar 1 2023, 9:45 PM

Right, I think we can probably investigate PostgreSQL and/or SQLite with a sequelize ORM for our current needs given that no NoSQL databases are supported. (ping @Gopavasanth in case they have any thoughts on other ORMs/other alternative databases to use)

Gopavasanth added a comment.Mar 2 2023, 10:42 AM

I would choose SQLite because of its excellent administrative features and the ease of implementing and utilizing an in-memory database.

Soda added a comment.Mar 2 2023, 11:50 AM
In T330755#8660010, @Gopavasanth wrote:

I would choose SQLite because of its excellent administrative features and the ease of implementing and utilizing an in-memory database.

I don't think we want a in-memory store (it would mean that we would lose all our data the next time the tool is redeployed to production since SQLite doesn't store any data on disk for a in-memory store)

Punith.nyk subscribed.Mar 3 2023, 1:30 PM
In T330755#8660175, @Soda wrote:
In T330755#8660010, @Gopavasanth wrote:

I would choose SQLite because of its excellent administrative features and the ease of implementing and utilizing an in-memory database.

I don't think we want a in-memory store (it would mean that we would lose all our data the next time the tool is redeployed to production since SQLite doesn't store any data on disk for a in-memory store)

yes, I too feel that we should opt for a disk-based database like PostgreSQL( Mongodb was also a disk-based database! )

Reputation22 subscribed.Mar 4 2023, 9:55 PM

We can go with PostgreSQL, and for in-memory we can opt for Redis.

Soda claimed this task.Mar 6 2023, 10:40 PM
In T330755#8665936, @Reputation22 wrote:

We can go with PostgreSQL, and for in-memory we can opt for Redis.

I don't think we want a in-memory solution, I'll tryout PostgreSQL since I'm dealing working with it right now IRL (I can set it up without too much issue).

Reputation22 added a comment.Mar 6 2023, 10:44 PM
In T330755#8670701, @Soda wrote:
In T330755#8665936, @Reputation22 wrote:

We can go with PostgreSQL, and for in-memory we can opt for Redis.

I don't think we want a in-memory solution, I'll tryout PostgreSQL since I'm dealing working with it right now IRL (I can set it up without too much issue).

how are we planning to store the cookies and sessions, via postgres?

Soda added a comment.Mar 6 2023, 11:47 PM
In T330755#8670710, @Reputation22 wrote:
In T330755#8670701, @Soda wrote:
In T330755#8665936, @Reputation22 wrote:

We can go with PostgreSQL, and for in-memory we can opt for Redis.

I don't think we want a in-memory solution, I'll tryout PostgreSQL since I'm dealing working with it right now IRL (I can set it up without too much issue).

how are we planning to store the cookies and sessions, via postgres?

I haven't looked too deeply into it but looks like we are using the default express-session MemoryStore implementation. I'm not convinced that we should invest in the complexity (and the associated security risks) of a full-blown redis server however it might make sense to move off from the dev only implementation

Soda added a comment.Mar 6 2023, 11:49 PM

Lets branch out the session storage aspect into a different bug though

Reputation22 added a comment.Mar 7 2023, 2:47 PM
In T330755#8670890, @Soda wrote:

Lets branch out the session storage aspect into a different bug though

yeah.. first lets go through this step by step
also are you planning to use some orm like sequelize for this? or raw sql queries ?

Reputation22 added a comment.Mar 7 2023, 2:48 PM

i suggest we go for sequelize :)

Soda added a comment.Mar 7 2023, 3:00 PM

I do intend to use seqelize to make for easier migration from Mongo

gerritbot added a comment.Mar 9 2023, 4:22 PM

Change 896129 had a related patch set uploaded (by Sohom Datta; author: Sohom Datta):

[labs/tools/VideoCutTool@master] [WIP] Use postgresql instead of mongodb

https://gerrit.wikimedia.org/r/896129

gerritbot added a project: Patch-For-Review.Mar 9 2023, 4:22 PM
Soda added a comment.Mar 10 2023, 4:30 PM

Looks like sequelize doesn't play well with ESM modules, we should probably switch to CommonJS modules for now

Reputation22 added a comment.Mar 10 2023, 5:40 PM

yes.. esm modules are a bit newer, and sequelize rely on the old modules.
so yes we may have to shift from import to require

Gopavasanth moved this task from Incoming to Doing on the VideoCutTool board.Mar 12 2023, 7:00 AM
Reputation22 mentioned this in T333466: [GSOC 2023 Proposal] Improve the functionality of VideoCutTool.Mar 29 2023, 4:12 PM
Gopavasanth added a comment.May 19 2023, 1:19 PM

@Soda, The patch is failing, attaching the logs:

videocuttool-database-1  | The files belonging to this database system will be owned by user "postgres".
videocuttool-database-1  | This user must also own the server process.
videocuttool-database-1  |
videocuttool-database-1  | The database cluster will be initialized with locale "en_US.utf8".
videocuttool-database-1  | The default database encoding has accordingly been set to "UTF8".
videocuttool-database-1  | The default text search configuration will be set to "english".
videocuttool-database-1  |
videocuttool-database-1  | Data page checksums are disabled.
videocuttool-database-1  |
videocuttool-database-1  | initdb: error: directory "/var/lib/postgresql/data/pg_data" exists but is not empty
videocuttool-database-1  | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data/pg_data" or run initdb with an argument other than "/var/lib/postgresql/data/pg_data".
videocuttool-database-1 exited with code 1
videocuttool             | [0] Browserslist: caniuse-lite is outdated. Please run:
videocuttool             | [0]   npx browserslist@latest --update-db
videocuttool             | [0]   Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
videocuttool             | [0] (node:81) [DEP_WEBPACK_DEV_SERVER_ON_AFTER_SETUP_MIDDLEWARE] DeprecationWarning: 'onAfterSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
videocuttool             | [0] (Use `node --trace-deprecation ...` to show where the warning was created)
videocuttool             | [0] (node:81) [DEP_WEBPACK_DEV_SERVER_ON_BEFORE_SETUP_MIDDLEWARE] DeprecationWarning: 'onBeforeSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
videocuttool             | [1] /app/server/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:143
videocuttool             | [1]                 reject(new sequelizeErrors.ConnectionError(err));
videocuttool             | [1]                        ^
videocuttool             | [1]
videocuttool             | [1] ConnectionError [SequelizeConnectionError]: getaddrinfo EAI_AGAIN database
videocuttool             | [1]     at Client._connectionCallback (/app/server/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:143:24)
videocuttool             | [1]     at Client._handleErrorWhileConnecting (/app/server/node_modules/pg/lib/client.js:318:19)
videocuttool             | [1]     at Client._handleErrorEvent (/app/server/node_modules/pg/lib/client.js:328:19)
videocuttool             | [1]     at Connection.emit (node:events:513:28)
videocuttool             | [1]     at Socket.reportStreamError (/app/server/node_modules/pg/lib/connection.js:57:12)
videocuttool             | [1]     at Socket.emit (node:events:513:28)
videocuttool             | [1]     at emitErrorNT (node:internal/streams/destroy:157:8)
videocuttool             | [1]     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
videocuttool             | [1]     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
videocuttool             | [1]   parent: Error: getaddrinfo EAI_AGAIN database
videocuttool             | [1]       at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
videocuttool             | [1]     errno: -3001,
videocuttool             | [1]     code: 'EAI_AGAIN',
videocuttool             | [1]     syscall: 'getaddrinfo',
videocuttool             | [1]     hostname: 'database'
videocuttool             | [1]   },
videocuttool             | [1]   original: Error: getaddrinfo EAI_AGAIN database
videocuttool             | [1]       at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
videocuttool             | [1]     errno: -3001,
videocuttool             | [1]     code: 'EAI_AGAIN',
videocuttool             | [1]     syscall: 'getaddrinfo',
videocuttool             | [1]     hostname: 'database'
videocuttool             | [1]   }
videocuttool             | [1] }
videocuttool             | [1] [nodemon] app crashed - waiting for file changes before starting...
videocuttool-database-1  | The files belonging to this database system will be owned by user "postgres".
videocuttool-database-1  | This user must also own the server process.
videocuttool-database-1  |
videocuttool-database-1  | The database cluster will be initialized with locale "en_US.utf8".
videocuttool-database-1  | The default database encoding has accordingly been set to "UTF8".
videocuttool-database-1  | The default text search configuration will be set to "english".
videocuttool-database-1  |
videocuttool-database-1  | Data page checksums are disabled.
videocuttool-database-1  |
videocuttool-database-1  | initdb: error: directory "/var/lib/postgresql/data/pg_data" exists but is not empty
videocuttool-database-1  | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data/pg_data" or run initdb with an argument other than "/var/lib/postgresql/data/pg_data".
videocuttool             | [0] Starting the development server...
gerritbot added a comment.May 21 2023, 8:29 AM

Change 896129 merged by jenkins-bot:

[labs/tools/VideoCutTool@master] Use postgresql instead of mongodb

https://gerrit.wikimedia.org/r/896129

Restricted Repository Identity mentioned this in rLTVT952588742c5b: Use postgresql instead of mongodb.May 21 2023, 8:29 AM
Maintenance_bot removed a project: Patch-For-Review.May 21 2023, 8:30 AM
Soda closed this task as Resolved.Aug 18 2023, 7:39 AM

We've deployed PostgresSQL on both the beta server and the production one

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL