While trying to install mod_auth_cas on the netmon servers to protect librenms i noticed that the apache process was segfauling when trying to process the CAS response. In the apache debug log this shows up as
[Thu Jul 09 14:19:02.131427 2020] [socache_shmcb:debug] [pid 16108] mod_socache_shmcb.c(495): AH00831: socache_shmcb_store (0x7d -> subcache 29) [Thu Jul 09 14:19:02.131460 2020] [socache_shmcb:debug] [pid 16108] mod_socache_shmcb.c(849): AH00847: insert happened at idx=0, data=(0:32) [Thu Jul 09 14:19:02.131468 2020] [socache_shmcb:debug] [pid 16108] mod_socache_shmcb.c(854): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/206 [Thu Jul 09 14:19:02.131484 2020] [socache_shmcb:debug] [pid 16108] mod_socache_shmcb.c(516): AH00834: leaving socache_shmcb_store successfully [Thu Jul 09 14:19:02.131505 2020] [ssl:debug] [pid 16108] ssl_engine_kernel.c(2042): [client 84.232.61.34:41844] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits) [Thu Jul 09 14:19:02.131691 2020] [ssl:debug] [pid 16108] ssl_engine_kernel.c(366): [client 84.232.61.34:41844] AH02034: Initial (No.1) HTTPS request received for child 2 (server librenms.wikimedia.org:443) [Thu Jul 09 14:19:02.132249 2020] [authz_core:debug] [pid 16108] mod_authz_core.c(809): [client 84.232.61.34:41844] AH01626: authorization result of Require valid-user : denied (no authenticated user yet) [Thu Jul 09 14:19:02.132265 2020] [authz_core:debug] [pid 16108] mod_authz_core.c(809): [client 84.232.61.34:41844] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet) [Thu Jul 09 14:19:02.132277 2020] [auth_cas:debug] [pid 16108] mod_auth_cas.c(2152): [client 84.232.61.34:41844] Entering cas_authenticate() [Thu Jul 09 14:19:02.132290 2020] [auth_cas:debug] [pid 16108] mod_auth_cas.c(681): [client 84.232.61.34:41844] Modified r->args (now '') [Thu Jul 09 14:19:02.132358 2020] [auth_cas:debug] [pid 16108] mod_auth_cas.c(1828): [client 84.232.61.34:41844] entering getResponseFromServer() [Thu Jul 09 14:19:02.132427 2020] [auth_cas:debug] [pid 16108] mod_auth_cas.c(609): [client 84.232.61.34:41844] CAS Service 'https%3a%2f%2flibrenms.wikimedia.org%2f'
Ultimately this indicates that the module is ending at the point it tries to validate the ticket by trying to GET https://idp.wikimedia.org/serviceValidate?ticket=XXX
monitoring tcpdump shows that the 3 way hand shake is preformed immediately followed by the ter down procedure with now data being transferred.
Looking at the core dump we see the issues happens when trying to set some SSL options
#0 0x00007fe54f67c450 in X509_VERIFY_PARAM_set_depth () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #1 0x00007fe54fbc21fe in cas_curl_ssl_ctx (curl=<optimized out>, sslctx=<optimized out>, parm=<optimized out>) at mod_auth_cas.c:1809 #2 0x00007fe54f99d871 in ossl_connect_step1 (sockindex=0, conn=0x555efca7cc80) at vtls/openssl.c:2126 #3 ossl_connect_common (conn=conn@entry=0x555efca7cc80, sockindex=0, nonblocking=nonblocking@entry=true, done=done@entry=0x7ffe6fb35657) at vtls/openssl.c:3006 #4 0x00007fe54f99ec1d in Curl_ossl_connect_nonblocking (conn=conn@entry=0x555efca7cc80, ---Type <return> to continue, or q <return> to quit--- sockindex=<optimized out>, done=done@entry=0x7ffe6fb35657) at vtls/openssl.c:3094 #5 0x00007fe54f99f522 in Curl_ssl_connect_nonblocking (conn=conn@entry=0x555efca7cc80, sockindex=sockindex@entry=0, done=0x7ffe6fb35657) at vtls/vtls.c:246 #6 0x00007fe54f9505d2 in https_connecting (conn=0x555efca7cc80, done=<optimized out>) at http.c:1400 #7 0x00007fe54f963487 in Curl_protocol_connect (conn=0x555efca7cc80, protocol_done=protocol_done@entry=0x7ffe6fb35657) at url.c:3957 #8 0x00007fe54f9787a6 in multi_runsingle (multi=multi@entry=0x555efca4ddf0, now=..., data=data@entry=0x555efca84a10) at multi.c:1594 ---Type <return> to continue, or q <return> to quit--- #9 0x00007fe54f9796d1 in curl_multi_perform (multi=multi@entry=0x555efca4ddf0, running_handles=running_handles@entry=0x7ffe6fb357e8) at multi.c:2149 #10 0x00007fe54f96f680 in easy_transfer (multi=0x555efca4ddf0) at easy.c:700 #11 easy_perform (events=false, data=0x555efca84a10) at easy.c:787 #12 curl_easy_perform (data=data@entry=0x555efca84a10) at easy.c:806 #13 0x00007fe54fbc9469 in getResponseFromServer (r=r@entry=0x7fe551b8b0a0, c=c@entry=0x7fe551c0d050, ticket=ticket@entry=0x7fe542e20c77 "ST-95-xP2TLab5jABEs93AjeYXSv32bgA-idp2001") at mod_auth_cas.c:1899 ---Type <return> to continue, or q <return> to quit--- #14 0x00007fe54fbc9a79 in isValidCASTicket (r=r@entry=0x7fe551b8b0a0, c=c@entry=0x7fe551c0d050, ticket=ticket@entry=0x7fe542e20c77 "ST-95-xP2TLab5jABEs93AjeYXSv32bgA-idp2001", user=user@entry=0x7ffe6fb35bf8, attrs=attrs@entry=0x7ffe6fb35c00) at mod_auth_cas.c:1462 #15 0x00007fe54fbcaf5f in cas_authenticate (r=0x7fe551b8b0a0) at mod_auth_cas.c:2192 #16 0x0000555efafd5d40 in ap_run_check_user_id (r=r@entry=0x7fe551b8b0a0) at request.c:81 #17 0x0000555efafd9105 in ap_process_request_internal (r=0x7fe551b8b0a0) at request.c:292 #18 0x0000555efaff7da0 in ap_process_async_request (r=0x7fe551b8b0a0) at http_request.c:434 #19 0x0000555efaff7ec0 in ap_process_request (r=r@entry=0x7fe551b8b0a0) at http_request.c:471 ---Type <return> to continue, or q <return> to quit--- #20 0x0000555efaff40fd in ap_process_http_sync_connection (c=0x7fe551b98290) at http_core.c:210 #21 ap_process_http_connection (c=0x7fe551b98290) at http_core.c:251 #22 0x0000555efafe9bd0 in ap_run_process_connection (c=c@entry=0x7fe551b98290) at connection.c:42 #23 0x0000555efafea120 in ap_process_connection (c=c@entry=0x7fe551b98290, csd=<optimized out>) at connection.c:226 #24 0x00007fe5493486bf in child_main (child_num_arg=child_num_arg@entry=4, child_bucket=child_bucket@entry=0) at prefork.c:723 #25 0x00007fe549348934 in make_child (s=0x7fe551ccb4a0, slot=4) at prefork.c:825 ---Type <return> to continue, or q <return> to quit--- #26 0x00007fe5493497d7 in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:929 #27 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1123 #28 0x0000555efafc30fe in ap_run_mpm (pconf=0x7fe551cf8028, plog=0x7fe551cc6028, s=0x7fe551ccb4a0) at mpm_common.c:94 #29 0x0000555efafbbcfd in main (argc=<optimized out>, argv=<optimized out>) at main.c:783
This corresponds to to following code in mod_auth_cas
CURLcode cas_curl_ssl_ctx(CURL *curl, void *sslctx, void *parm) { SSL_CTX *ctx = (SSL_CTX *) sslctx; cas_cfg *c = (cas_cfg *)parm; SSL_CTX_set_verify_depth(ctx, c->CASValidateDepth); return CURLE_OK; }
specificly commenting out SSL_CTX_set_verify_depth(ctx, c->CASValidateDepth); prevents the segfault.
To debug this a bit further i created a small c programe which was able to reproduce the issues. also commenting out the same bit of code prevented the segfault. however when compiling this program i noticed the following error
/usr/bin/ld: warning: libssl.so.1.0.2, needed by /usr/lib/x86_64-linux-gnu/libcurl.so, may conflict with libssl.so.1.1
I didn't need libssl1.1 for this test so i removed the libraries and recompiled thus linking only to libssl1.02 after which point i was able to run the test program successfully.
When looking at mod_auth_cas i see it is also linked to both libssl1.0.2 and libssl1.1 which would suggest that we have the same issue
- libssl1.0.2 comes from libcurl3-dev
- libssl1.1 comes from apache2-dev
root@netmon2001:~# ldd /usr/lib/apache2/modules/mod_auth_cas.so | grep cry libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fe0c5aed000) libcrypto.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (0x00007fe0c43c1000)
One thinking that is still bugging me is that mod_auth_cas is working successfully on other stretch systems e.g. graphite