Page MenuHomePhabricator

Privacybatm (Ajumal P A)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Mar 20 2020, 5:51 AM (83 w, 6 d)
Availability
Available
IRC Nick
batm
LDAP User
Unknown
MediaWiki User
Privacybatm [ Global Accounts ]

Recent Activity

Mar 23 2021

Privacybatm awarded T268258: transfer.py argument parsing exception a Like token.
Mar 23 2021, 5:56 PM · Data-Persistence-Backup, Google-Summer-of-Code (2021), Patch-For-Review, good first task

Sep 1 2020

Privacybatm committed rOSWBcc58a878335d: Firewall.py: Add function to kill process by its port number (authored by Privacybatm).
Firewall.py: Add function to kill process by its port number
Sep 1 2020, 8:36 AM
Privacybatm committed rOSWB5dbe995402e8: transfer.py: Add information to --help option (authored by Privacybatm).
transfer.py: Add information to --help option
Sep 1 2020, 8:36 AM
Privacybatm committed rOSWB74170c8ae52d: CuminExecution.py: Improve output message readabiliy of transfer.py (authored by Privacybatm).
CuminExecution.py: Improve output message readabiliy of transfer.py
Sep 1 2020, 8:36 AM
Privacybatm committed rOSWB3f9329ad9942: transfer.py: Add the ability to auto-detect free port for netcat to listen (authored by Privacybatm).
transfer.py: Add the ability to auto-detect free port for netcat to listen
Sep 1 2020, 8:36 AM
Privacybatm committed rOSWBe9e62d04e962: transfer.py: Move Transferer, MariaDB logic and Firewall logic to its new… (authored by Privacybatm).
transfer.py: Move Transferer, MariaDB logic and Firewall logic to its new…
Sep 1 2020, 8:36 AM
Privacybatm committed rOSWB19a914fb8e25: Add comments to Firewall, MariaDB and transfer modules (authored by Privacybatm).
Add comments to Firewall, MariaDB and transfer modules
Sep 1 2020, 8:36 AM
Privacybatm committed rOSWB712c81e4e9c4: Firewall.py: Store target_host as an instance property of Firewall object (authored by Privacybatm).
Firewall.py: Store target_host as an instance property of Firewall object
Sep 1 2020, 8:36 AM

Aug 4 2020

Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

@Andrew Thank you for your suggestions. Just now I tried and it is working as expected. So, as you said, I must have tried before the VMs were ready. Thanks again!

Aug 4 2020, 1:51 PM · Cloud-VPS (Project-requests)
Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

Hi @Andrew I have recreated them once again and now the error has been changed to this:

Aug 4 2020, 5:22 AM · Cloud-VPS (Project-requests)
Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

Hi @Andrew, I am facing an issue with VM. I have recreated them but not able to ssh from my machine. It is showing the following error:

Aug 4 2020, 5:01 AM · Cloud-VPS (Project-requests)

Jul 31 2020

Privacybatm added a comment to T256450: Solve transferpy concurrency issue with auto port detection and checksum file names.

I think we can close this! What do you think?

Jul 31 2020, 6:24 AM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I think we can close this! What do you think?

Jul 31 2020, 6:23 AM · DBA
Privacybatm moved T259327: transferpy: Multiprocess the transfers from Triage to GSOC2020 on the DBA board.
Jul 31 2020, 6:13 AM · Patch-For-Review, DBA
Privacybatm created T259327: transferpy: Multiprocess the transfers.
Jul 31 2020, 6:00 AM · Patch-For-Review, DBA
Privacybatm added a comment to T257601: transferpy 1.0 release.
16:28:13 Warning, treated as error:
16:28:13 /src/transferpy/transfer.py:docstring of transferpy.transfer.to_bool:5:Field list ends without a blank line; unexpected unindent.
16:28:13 ERROR: InvocationError for command /src/.tox/sphinx/bin/sphinx-build transferpy/doc transferpy/doc/.build --color -W (exited with code 2)
Jul 31 2020, 5:55 AM · Patch-For-Review, DBA

Jul 28 2020

Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

Thank you for the update. Please don't worry about VM recreation. Please let us know(ping here as you said) once it is ready. Thank you!

Jul 28 2020, 5:38 AM · Cloud-VPS (Project-requests)

Jul 23 2020

Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

Yeah, Okay, Thank you!

Jul 23 2020, 5:56 PM · Cloud-VPS (Project-requests)
Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

@Andrew @jcrespo Yes, It is possible to recreate the data with little effort. So, please go ahead. Thank you for asking!

Jul 23 2020, 5:50 PM · Cloud-VPS (Project-requests)
Privacybatm added a comment to T257600: Create more tests for transferpy package.

I agree on this, As you suggested we will split it based on class.

Jul 23 2020, 10:02 AM · Patch-For-Review, DBA

Jul 22 2020

Privacybatm added a comment to T258559: Show transfer time once successfully completed.

Sounds good to me. And yeah, I am currently concentrating on the Gerrit comments and documentation. Thank you!

Jul 22 2020, 8:12 AM · DBA
Privacybatm claimed T258559: Show transfer time once successfully completed.
Jul 22 2020, 7:47 AM · DBA
Privacybatm added a comment to T258559: Show transfer time once successfully completed.

Yeah, I think it is a good idea to have transfer time. Thank you!

Jul 22 2020, 7:46 AM · DBA

Jul 17 2020

Privacybatm renamed T256450: Solve transferpy concurrency issue with auto port detection and checksum file names from Solve transferpy concurrency issue with auto port detection to Solve transferpy concurrency issue with auto port detection and checksum file names.
Jul 17 2020, 1:33 PM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I have just updated the commit message so that it is visible here!

Jul 17 2020, 1:06 PM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

So the largest issues on how options work, which makes them very confusing:

If I do --no-checksum, I expect to not get any checksum; however, I get a parallel checksum.
If I do --parallel-checksum, I expect to get a parallel checksum; however, I get a normal checksum

Jul 17 2020, 1:01 PM · DBA
Privacybatm added a comment to T256450: Solve transferpy concurrency issue with auto port detection and checksum file names.

Yeah, I agree with you, I am working in this issue currently. Thank you.

Jul 17 2020, 11:13 AM · DBA

Jul 16 2020

Privacybatm added a comment to T257600: Create more tests for transferpy package.

Sure, I will do the rebasing! Thank you for mentioning it.

Jul 16 2020, 2:02 PM · Patch-For-Review, DBA
Privacybatm added a comment to T257602: Make transferpy configurable using a configuration file.

Make sure

transfer.py --parallel-checksum source target
Jul 16 2020, 10:56 AM · DBA

Jul 13 2020

Privacybatm added a comment to T257600: Create more tests for transferpy package.

The following scenarios come under this ticket:

Jul 13 2020, 3:05 PM · Patch-For-Review, DBA

Jul 12 2020

Privacybatm updated the task description for T257601: transferpy 1.0 release.
Jul 12 2020, 3:47 PM · Patch-For-Review, DBA

Jul 9 2020

Privacybatm moved T257602: Make transferpy configurable using a configuration file from Triage to GSOC2020 on the DBA board.
Jul 9 2020, 6:07 PM · DBA
Privacybatm created T257602: Make transferpy configurable using a configuration file.
Jul 9 2020, 6:07 PM · DBA
Privacybatm moved T257601: transferpy 1.0 release from Triage to GSOC2020 on the DBA board.
Jul 9 2020, 6:03 PM · Patch-For-Review, DBA
Privacybatm created T257601: transferpy 1.0 release.
Jul 9 2020, 6:02 PM · Patch-For-Review, DBA
Privacybatm moved T257600: Create more tests for transferpy package from Triage to GSOC2020 on the DBA board.
Jul 9 2020, 5:58 PM · Patch-For-Review, DBA
Privacybatm edited projects for T257600: Create more tests for transferpy package, added: DBA; removed Google-Summer-of-Code (2020).
Jul 9 2020, 5:58 PM · Patch-For-Review, DBA
Privacybatm created T257600: Create more tests for transferpy package.
Jul 9 2020, 5:57 PM · Patch-For-Review, DBA
Privacybatm removed a project from T257599: Create temp and config directories at the installation time of transferpy deb package: Google-Summer-of-Code (2020).
Jul 9 2020, 5:55 PM · DBA
Privacybatm moved T257599: Create temp and config directories at the installation time of transferpy deb package from Triage to GSOC2020 on the DBA board.
Jul 9 2020, 5:54 PM · DBA
Privacybatm created T257599: Create temp and config directories at the installation time of transferpy deb package.
Jul 9 2020, 5:51 PM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I don't think this setup is adequate for testing paralelism, given we only have 1 host to transfer to (in parallel). I believe this could be way more interesting when using a 10Gb host with multiple 1Gb targets, plus it would help a lot with target checksum parallelism (which is the use case I mentioned to you in our meeting). Did you create a prototype for this or did you run a command manually? If you did some code (even if not good enough), I would like to see it so I can test it on my own.

Jul 9 2020, 9:44 AM · DBA

Jul 8 2020

Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I have created a code for parallel data transfer using multiprocessing. I have benchmarked the code in our test machines and the results are given below:

Jul 8 2020, 7:19 AM · DBA

Jul 5 2020

Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I have run the source multiprocess checksum and the results are given below:

Jul 5 2020, 3:38 PM · DBA

Jul 2 2020

Privacybatm added a comment to T256951: Choosing a wrong host with transfer.py produces an "ERROR: The specified source path X doesn't exist on Y".

Change in opinion :D Our code has specific tests for all the cases, So I will make one for host validity also, that way it will be aligned to the existing code.

Jul 2 2020, 10:28 AM · DBA
Privacybatm added a comment to T256951: Choosing a wrong host with transfer.py produces an "ERROR: The specified source path X doesn't exist on Y".

Parsing cumin output seems to be a better idea, let me check the output of cumin in this kind of cases.

Jul 2 2020, 10:10 AM · DBA

Jul 1 2020

Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

@jcrespo Can you please tell me a way to corrupt the source socket in xtrabackup. By corruption, I meant some changes,

Jul 1 2020, 12:58 PM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.
Jul 1 2020, 12:33 PM · DBA

Jun 30 2020

Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

Sorry, I forgot to give the sysbench outputs.

Jun 30 2020, 2:11 PM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

A preliminary result from this suggests that parallel checksum should be able to be disabled, but be enabled by default (unless cpu usage increased a lot).

Jun 30 2020, 1:35 PM · DBA
Privacybatm added a comment to T256755: transferpy --checksum wrongly output `checksums do not match` message.

Yeah, I will have a search on this. Let's this ticket be here so that we can keep an eye on it!

Jun 30 2020, 1:25 PM · DBA
Privacybatm moved T256755: transferpy --checksum wrongly output `checksums do not match` message from Triage to GSOC2020 on the DBA board.
Jun 30 2020, 1:00 PM · DBA
Privacybatm updated the task description for T256755: transferpy --checksum wrongly output `checksums do not match` message.
Jun 30 2020, 1:00 PM · DBA
Privacybatm created T256755: transferpy --checksum wrongly output `checksums do not match` message.
Jun 30 2020, 12:59 PM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I have run benchmarks with the new cloud test machines.
bigfile: 1.4TB
manySmallFiles300: 293GB (150 000 files)

Jun 30 2020, 12:49 PM · DBA
Privacybatm added a comment to T256725: Execution error after moving to debian package.

Actually, I never got that error. I will have a look for the possibility of that error.

Jun 30 2020, 9:29 AM · DBA
Privacybatm added a comment to T256725: Execution error after moving to debian package.

Can we get a --verbose output, that would tell us if it is the problem with Cumin?

Jun 30 2020, 9:00 AM · DBA
Privacybatm added a comment to T256725: Execution error after moving to debian package.

In our testing environment, I am currently using the Debian package only. Let me see what could be this issue!

Jun 30 2020, 8:59 AM · DBA

Jun 29 2020

Privacybatm added a comment to T256604: transferpy package does not depend on python3-yaml.
Jun 29 2020, 9:41 AM · DBA
Privacybatm added a comment to T254158: Gerrit 3.2 upgrade.

Thank you @jcrespo @Majavah I was using 1.26, Just now updated and everything is working fine, Thank you for your help!

Jun 29 2020, 9:35 AM · Patch-For-Review, Gerrit

Jun 26 2020

Privacybatm added a comment to T256450: Solve transferpy concurrency issue with auto port detection and checksum file names.

This race condition can be solved/reduced by making a directory (mkdir) in temp as soon as we see a free port.

Jun 26 2020, 8:02 AM · DBA
Privacybatm moved T256450: Solve transferpy concurrency issue with auto port detection and checksum file names from Triage to GSOC2020 on the DBA board.
Jun 26 2020, 8:02 AM · DBA
Privacybatm created T256450: Solve transferpy concurrency issue with auto port detection and checksum file names.
Jun 26 2020, 8:02 AM · DBA

Jun 25 2020

Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

Thank you for the machines, I am able to log in and work on it :-D

Jun 25 2020, 4:05 PM · Cloud-VPS (Project-requests)
Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

Okay, As per our IRC discussion 1GB of memory is sufficient. Thank you!

Jun 25 2020, 7:35 AM · Cloud-VPS (Project-requests)
Privacybatm added a comment to T255764: Request creation of transferpy-test VPS project.

@jcrespo We need to run MariaDB xtrabackup and all, Will 1GB be sufficient?

Jun 25 2020, 7:14 AM · Cloud-VPS (Project-requests)

Jun 24 2020

Privacybatm added a comment to T253736: Package transferpy framework.

Thank you! Yeah, I will do that.

Jun 24 2020, 10:59 AM · Patch-For-Review, DBA
Privacybatm closed T253736: Package transferpy framework, a subtask of T248256: GSoC 2020 Proposal: Improve the framework to transfer files over the LAN, as Resolved.
Jun 24 2020, 10:54 AM · Patch-For-Review, DBA, Google-Summer-of-Code (2020)
Privacybatm closed T253736: Package transferpy framework as Resolved.

Merged the packaging patch!

Jun 24 2020, 10:54 AM · Patch-For-Review, DBA
Privacybatm renamed T253736: Package transferpy framework from Package transferpy framework under wmfmariadbpy to Package transferpy framework.
Jun 24 2020, 9:12 AM · Patch-For-Review, DBA
Privacybatm added a comment to T253219: Add more information to --help option of transfer.py.

Yeah, the transferpy is now available at https://doc.wikimedia.org/ :-)

Jun 24 2020, 8:05 AM · DBA

Jun 22 2020

Privacybatm added a comment to T255999: Use logging package instead of print statements in transferpy package.

I don't think we will be able to incorporate data transfer progress information. The RemoteExecution works in such a way that it will send the full data as a whole without any communication to the main program. Since the netcat command runs in the remote machine, the framework running machine has no information about it! What do you think?

Jun 22 2020, 2:18 PM · DBA
Privacybatm moved T255999: Use logging package instead of print statements in transferpy package from Triage to GSOC2020 on the DBA board.
Jun 22 2020, 1:29 PM · DBA
Privacybatm created T255999: Use logging package instead of print statements in transferpy package.
Jun 22 2020, 1:29 PM · DBA

Jun 19 2020

Privacybatm added a comment to T253219: Add more information to --help option of transfer.py.

We can close this, but let's remember to keep the help up-to-date with the new features implemented, as well as everything that is currently missing as it has not yet been fully decided.

Jun 19 2020, 10:48 AM · DBA

Jun 18 2020

Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

Okay!

Jun 18 2020, 10:19 AM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

(Machine spec: i5-2nd Gen with SATA HDD and 6GB DDR3 RAM)

Jun 18 2020, 9:27 AM · DBA

Jun 15 2020

Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I tried incorporating the parallel md5sum in the code. But not working as expected!

Jun 15 2020, 6:28 AM · DBA

Jun 10 2020

Privacybatm added a comment to T253736: Package transferpy framework.

Thank you for the information, and yes, it was helpful :-)

Jun 10 2020, 5:20 PM · Patch-For-Review, DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

Oh okay, how about giving the user a choice?

  • Checksum parallel to transfer (document the issues we find at testing)
  • Checksum after the transfer (document the delay issues)
Jun 10 2020, 12:31 PM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I think it is a good starting point- I suggest you do some benchmarking (doesn't need to be implemented on code yet) of how much expensive this strategy would be compared to the current method and compared to no checksum to understand the impact/improvement.

Jun 10 2020, 9:40 AM · DBA
Privacybatm added a comment to T254979: Make checksum parallel to the data transfer in transferpy package.

I would like to calculate the checksum for the actual tarred file. We can do this parallel to transfer like this:
At sender: tar cf - <directory> | tee >(echo $(md5sum) > /tmp/transfer_send) | remaining-commands
At receiver: commands | tee >(echo $(md5sum) > /tmp/transfer_recv) | tar xf - <directory>
Then we can compare those two checksum-temp-files at the end of the transfer. It will surely reduce the overall time.
What do you think?

Jun 10 2020, 9:24 AM · DBA
Privacybatm moved T254979: Make checksum parallel to the data transfer in transferpy package from Triage to GSOC2020 on the DBA board.
Jun 10 2020, 9:23 AM · DBA
Privacybatm created T254979: Make checksum parallel to the data transfer in transferpy package.
Jun 10 2020, 9:23 AM · DBA

Jun 8 2020

Privacybatm updated Privacybatm.
Jun 8 2020, 5:31 PM

Jun 4 2020

Privacybatm committed rOSTPcc58a878335d: Firewall.py: Add function to kill process by its port number (authored by Privacybatm).
Firewall.py: Add function to kill process by its port number
Jun 4 2020, 10:25 AM

May 28 2020

Privacybatm added a comment to T253736: Package transferpy framework.

Yeah, I will think about it, Thank you.

May 28 2020, 11:51 AM · Patch-For-Review, DBA
Privacybatm updated the task description for T253736: Package transferpy framework.
May 28 2020, 9:07 AM · Patch-For-Review, DBA
Privacybatm added a comment to T253736: Package transferpy framework.

I have uploaded a new patch set with a working deb file inside dist folder (https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/598984/2/dist/transferpy_1.0-1_amd64.deb)

May 28 2020, 7:54 AM · Patch-For-Review, DBA

May 27 2020

Privacybatm triaged T253736: Package transferpy framework as Medium priority.
May 27 2020, 9:41 AM · Patch-For-Review, DBA
Privacybatm moved T253736: Package transferpy framework from Triage to GSOC2020 on the DBA board.
May 27 2020, 9:41 AM · Patch-For-Review, DBA
Privacybatm created T253736: Package transferpy framework.
May 27 2020, 9:40 AM · Patch-For-Review, DBA

May 26 2020

Privacybatm closed T252802: Improve output message readabiliy of transfer.py, a subtask of T248256: GSoC 2020 Proposal: Improve the framework to transfer files over the LAN, as Resolved.
May 26 2020, 2:57 AM · Patch-For-Review, DBA, Google-Summer-of-Code (2020)
Privacybatm closed T252802: Improve output message readabiliy of transfer.py as Resolved.
May 26 2020, 2:57 AM · DBA

May 25 2020

Privacybatm added a comment to T248256: GSoC 2020 Proposal: Improve the framework to transfer files over the LAN.

Okay, I will use GSOC column for the tickets. Thank you!

May 25 2020, 2:17 PM · Patch-For-Review, DBA, Google-Summer-of-Code (2020)
Privacybatm added a comment to T252950: kill_job function in remote execution module of transfer framework does not close the ports instantly.

I will try with fuser then! Thank you!

May 25 2020, 1:55 PM · DBA
Privacybatm added a comment to T253560: Exception raised when setting trivial, but incorrect parameters to transfer.py.

We are happy to help :D

May 25 2020, 1:53 PM · DBA
Privacybatm added a comment to T252950: kill_job function in remote execution module of transfer framework does not close the ports instantly.

Thank you for the suggestion, I will try with netcat.

May 25 2020, 1:03 PM · DBA
Privacybatm added a comment to T252950: kill_job function in remote execution module of transfer framework does not close the ports instantly.

what we do now is, we start a nc-listen command in the target machine with the start_job which makes a new process in the framework running machine (with netcat listen waiting) and kill_job function uses terminate function inside multiprocessing/process.py (given below) to kill that job.

May 25 2020, 7:10 AM · DBA
Privacybatm added a comment to T253219: Add more information to --help option of transfer.py.

Just have a look at our documentation: https://transferpydoc.imfast.io/index.html :-)
(I uploaded it there to just have an easy look)

May 25 2020, 5:52 AM · DBA