Hi! Thanks so much for writing all this up. I think these are all separate issues?

• For sendmail, I'll add a patch set in the existing WIP dev-images change (also including the other package that @Ejegg once needed), build and upload the image to the registry, and send another patch with the updated image tag.
• The symphony issue is not due to a problem in fundraising dev, but something that occurs intermittently with the current state of our buildkit/crm repos, as per @Eileenmcnaughton... (if I understood correctly from what was said on IRC? maybe?)
• Regarding the user ID problem, I'd much prefer to see if it persists after fixing the other issues... is that OK? Apologies, I don't mean to rehash the topic here, and the fix you've suggested sounds super original...! So, maybe let's see if it sticks around, and dig in as needed, sound good? (Also, you should be able to reset the permissions on civi-sites with a recursive chmod, I think?)

Thanks again!! :) :)

Change 663294 had a related patch set uploaded (by AndyRussG; author: AndyRussG):

[releng/dev-images@master] Fundraising images: Separate Mediawiki and Civi setups

https://gerrit.wikimedia.org/r/663294

Change 677939 had a related patch set uploaded (by AndyRussG; author: AndyRussG):

[wikimedia/fundraising/dev@master] Update civicrm image tag for image with new packages

https://gerrit.wikimedia.org/r/677939

Change 677939 merged by Eileen:

[wikimedia/fundraising/dev@master] Update civicrm image tag for image with new packages

https://gerrit.wikimedia.org/r/677939

Something has changed bcase that send mail is a bit constant for me now - testing Andy's latest

The gotcha that keeps getting me is that if you blow away the containers & then say 'n' here

• Payments: install.php, LocalSettings.php and update.php

Run install.php and set up LocalSettings.php? [yN] n

• maintenance/update.php

Run update.php? [yN] n

it fails

Andy's image update only marginally changed the message

Installation complete. [ok]
Unable to send e-mail. Contact the site administrator if the problem persists.

• see how the path is no longer mentioned...

I don't think the user / permissions issue is a side issue - I think it might be the root issue - still not sure why this changed / stopped working but from the logs....

2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: rsyslogd's groupid changed to 1000
2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: could not get username for userid 1000: Success [v8.1901.0]
2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: rsyslogd's userid changed to 1000
2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="19" x-info="https://www.rsyslog.com"] start
2021-04-09T03:40:14+00:00 c70a75671469 sendmail[1312]: My unqualified host name (c70a75671469) unknown; sleeping for retry
2021-04-09T03:41:14+00:00 c70a75671469 sendmail[1312]: unable to qualify my own domain name (c70a75671469) -- using short name
2021-04-09T03:41:14+00:00 c70a75671469 sendmail[1312]: 1393fEx0001312: SYSERR(UID1000): Who are you?

The other thing I notice is a lot of, often transient, DNS issues

eg

fatal: unable to access 'https://github.com/civicrm/civivolunteer.git/': Could not resolve host: github.com

& then it works when I run it again

Just noting the last issue *seems* to go away when I switch to upstream buildkit ( which is what we are intending to switch to as ours no longer has useful differences) https://github.com/civicrm/civicrm-buildkit

Change 678432 had a related patch set uploaded (by AndyRussG; author: AndyRussG):

[wikimedia/fundraising/dev@master] Civicrm image tag for temporary sendmail=/bin/true in php.ini

https://gerrit.wikimedia.org/r/678432

In T279669#6988234, @Eileenmcnaughton wrote:

I don't think the user / permissions issue is a side issue - I think it might be the root issue - still not sure why this changed / stopped working but from the logs....

2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: rsyslogd's groupid changed to 1000
2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: could not get username for userid 1000: Success [v8.1901.0]
2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: rsyslogd's userid changed to 1000
2021-04-09T03:38:13+00:00 c70a75671469 rsyslogd: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="19" x-info="https://www.rsyslog.com"] start
2021-04-09T03:40:14+00:00 c70a75671469 sendmail[1312]: My unqualified host name (c70a75671469) unknown; sleeping for retry
2021-04-09T03:41:14+00:00 c70a75671469 sendmail[1312]: unable to qualify my own domain name (c70a75671469) -- using short name
2021-04-09T03:41:14+00:00 c70a75671469 sendmail[1312]: 1393fEx0001312: SYSERR(UID1000): Who are you?

Hi! Thanks @Eileenmcnaughton! I've gotten it working with by adding sendmail_path = /bin/true to the php settings in the image, as @Ejegg had suggested. The attached patch changes the civicrm image tag to pull the most recent image with that setting from Dockerhub.

Looking a bit further along in the same log (ahhh thanks for pointing out that we should look there) I get the following:

2021-04-08T16:09:13+00:00 79a0fae7194b sendmail[258]: 138G8DwD000258: Authentication-Warning: 79a0fae7194b: Unknown UID 1000 set sender to admin@example.com using -f
2021-04-08T16:09:13+00:00 79a0fae7194b sendmail[258]: 138G8DwD000258: from=admin@example.com, size=661, class=0, nrcpts=1, msgid=<202104081608.138G8DwD000258@79a0fae7194b>, relay=Unknown UID 1000@localhost
2021-04-08T16:09:13+00:00 79a0fae7194b sendmail[258]: 138G8DwD000258: to=admin@example.com, ctladdr=admin@example.com (1000/1000), delay=00:01:00, xdelay=00:00:00, mailer=relay, pri=30661, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection refused by [127.0.0.1]

So, my guess is that the "Who are you?" and unknown UID messages are warnings, and that the actual failure is the "Connection refused" message (last line, above). This would make sense, since Docker containers don't run standard system services (unless you specifically set them up to do so). (See these posts for similar issues with sendmail on Docker.)

Anyway, we do eventually need to get mail sends from Civicrm working on Docker. I think we can fix that when we set up a mailcatcher container? When we do that, we can set php to send mail via an SMTP service provided by that container? On the assumption that that will also work, I think we can consider this sendmail_path = /bin/true to be a temporary fix.

(I'm not sure why the issue was also fixed by running as a known user, as you suggested above, @jgleeson! Maybe that somehow allowed sendmail to trigger the execution of the daemon inside the container? In any case, if it does prove to be really necessary, I do think it'd be ok to create a user in the containers with the same UID as the host user, which, as you've mentioned, is possible. :) )

So, the remaining issue here is the Symfony Filesystem component not found. All the times I ran Civi setup, I got that issue, then it disappeared when I re-ran the setup. I'm guessing it's this Drupal issue? In any case, since it seems likely it's unrelated to the Docker setup, maybe we can make a separate task--does that sound right?

Thanks so much!!

@AndyRussG the filesystem one is fixed by using upstream buildkit instead of ours I believe - but yes - separate task on that.

I dug through the code & decided the sendmail error was not actually what was causing it to die since adding debug after that seemed to be hit. I think I got it working without fixing that but with the user commands @jgleeson suggested

@AndyRussG - but if the sendmail = true fixes it for you I'm happy to merge that & we can all re-test since it at least eliminates that

but yes - separate task on that.

K: T279876

In T279669#6989931, @Eileenmcnaughton wrote:

@AndyRussG - but if the sendmail = true fixes it for you I'm happy to merge that & we can all re-test since it at least eliminates that

Ah ok... thanks much... whatever works for everyone else is fine by me...! I did test it twice, both times removing the wmff source code and the persistent Docker storage. Everything worked, except for the Symfony issue.

(Not sure it's related, but when the sendmail issue, or whatever was the root cause of the problem here, was causing buildkit to error out, it was leaving my source code with odd permissions: src/civi-sites/dmaster/web/sites/default was left as unwritable. However, now that it's working, the permissions on that directory are normal again.)

Change 678432 merged by Eileen:

[wikimedia/fundraising/dev@master] Civicrm image tag for temporary sendmail=/bin/true in php.ini

https://gerrit.wikimedia.org/r/678432

Change 663294 merged by Eileen:

[releng/dev-images@master] Fundraising images: Separate Mediawiki and Civi setups

https://gerrit.wikimedia.org/r/663294