Page MenuHomePhabricator
Create Task
Maniphest T379878

New upload to Acoustic times out the connection if we attempt the full file
Closed, ResolvedPublic
Actions

Description

I attempted to do the full DB upload to ensure we passed up the preference tags that were a bit older since we had not done this since changing the code.

I found that the there was 'noise' during the update and on the Acoustic side the upload was less than the number of rows in the file and this attempt

I believe the connection was closed before all rows of the csv were uploaded but the overall process recovered - resulting in around 1/6 of the rows in the file being imported to Acoustic

Entity: Omnicontact
Action: upload
Params: {

"version": 4,
"checkPermissions": false,
"sourceFolder": "/srv/silverpop_export",
"prefix": "DatabaseUpdate"

}
[PHP User Notice] Expected SSH_MSG_KEX_DH_GEX_GROUP at /srv/org.wikimedia.civicrm/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php:1769
[PHP User Notice] Connection closed by server at /srv/org.wikimedia.civicrm/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php:4069
[PHP User Notice] Error reading channel data (31) at /srv/org.wikimedia.civicrm/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php:4288
[PHP User Notice] Connection closed (by server) prematurely 1.5974044799805E-5s at /srv/org.wikimedia.civicrm/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php:3647
[PHP User Notice] Connection closed by server at /srv/org.wikimedia.civicrm/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php:4069
[PHP User Notice] Connection closed prematurely at /srv/org.wikimedia.civicrm/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php:4309
[

{
    "job_id": 246227475
}

]

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Improve guzzle timeout supportwikimedia/fundraising/crmmaster+25 -9
Update phpseclib/phpseclib from 2 to 3wikimedia/fundraising/crmmaster+93 -26
Customize query in gerrit

Related Objects
Search...

Event Timeline

Eileenmcnaughton created this task.Nov 14 2024, 5:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 14 2024, 5:54 AM
Eileenmcnaughton updated the task description. (Show Details)Nov 14 2024, 5:55 AM
AKanji-WMF triaged this task as Medium priority.Nov 19 2024, 8:24 PM
AKanji-WMF moved this task from Triage to Chaos Crew Backlog on the Fundraising-Backlog board.
AKanji-WMF subscribed.

Workaround could be to change the contact modify date.

AKanji-WMF added a project: Fundraising Tech - Chaos Crew.Nov 21 2024, 6:45 PM
Eileenmcnaughton added a project: fr-acoustic.Dec 17 2024, 4:42 AM
XenoRyet edited projects, added Fundraising Tech Chaos Holding Pen; removed Fundraising Tech - Chaos Crew.Mar 19 2025, 6:17 PM
XenoRyet moved this task from Chaos Crew Backlog to Chaos Holding Pen on the Fundraising-Backlog board.Mar 19 2025, 6:22 PM
gerritbot added a comment.Jun 26 2025, 5:23 AM

Change #1163917 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Update phpseclib/phpseclib from 2 to 3

https://gerrit.wikimedia.org/r/1163917

gerritbot added a project: Patch-For-Review.Jun 26 2025, 5:23 AM
Eileenmcnaughton added a comment.Jun 26 2025, 5:42 AM

I put up a patch to upload to latest phpsec 3 - another thought is this define https://github.com/phpseclib/phpseclib/issues/2072#issuecomment-2712819217

gerritbot added a comment.Jun 26 2025, 2:16 PM

Change #1163917 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Update phpseclib/phpseclib from 2 to 3

https://gerrit.wikimedia.org/r/1163917

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2025, 2:32 PM
Eileenmcnaughton added subscribers: Dwisehaupt, jgleeson, Ejegg.Jun 26 2025, 8:32 PM

Just an update that @Dwisehaupt has renamed the file such that it can be re-tried with the following command

wmf-cv -vv api4 Omnicontact.upload sourceFolder=/srv/silverpop_export prefix=DBUpdate

I have created a job in process control called silverpop_emails_upload_full_data_file which will run it manually

I'm going to create a sub-phab for us to be able to export the full db via process-control

@jgleeson @Ejegg - subscribing you both

Eileenmcnaughton added a comment.Jun 26 2025, 8:39 PM

I think the upload error has changes since this was originally created I'm seeing signs that the files are uploaded but then the http call fails - I think the login maybe needs to happen again

In CurlFactory.php line 275:

                                                                             
[GuzzleHttp\Exception\ConnectException]                                      
cURL error 28: Operation timed out after 10001 milliseconds with 10 bytes r  
eceived (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https:  
//api4.silverpop.com/XMLAPI;jsessionid=F3C57EDEF50BB2970396E48A25A3B933
`
        if (!$this->isAlreadyUploaded()) {
            $this->silverPop->uploadFile($this->getXmlFileName(), $this->getXmlFile(), $this->getResultDirectory());
            $this->silverPop->uploadFile($this->getCsvFileName(), $this->getCsvFile(), $this->getResultDirectory());
        }

        $result = $this->silverPop->importList(
            $this->getXmlFileName(),
            $this->getCsvFileName()
        );
`

Perhaps even wrap

$result = $this->silverPop->importList(
    $this->getXmlFileName(),
    $this->getCsvFileName()
);

in a try-catch & if it fails then catch the guzzle exception & retry

gerritbot added a comment.Jun 30 2025, 8:27 AM

Change #1164949 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Improve guzzle timeout support

https://gerrit.wikimedia.org/r/1164949

gerritbot added a project: Patch-For-Review.Jun 30 2025, 8:27 AM
gerritbot added a comment.Jun 30 2025, 1:45 PM

Change #1164949 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Improve guzzle timeout support

https://gerrit.wikimedia.org/r/1164949

Maintenance_bot removed a project: Patch-For-Review.Jun 30 2025, 2:31 PM
Eileenmcnaughton closed this task as Resolved.Sep 24 2025, 1:53 AM
Eileenmcnaughton claimed this task.

Seems like this has moved on - other problems now...

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits