Upon receiving a 500 error on multiple sites due to what appears to be an update to Plex followed by an update to RHEL5, we have found a way to track the issue early and resolve quickly. There is a script that will be posted at the end of this message.

Tracking the issue:

The first sign of trouble was the 500 error which appears in a virtualhosted server environment with more than one site failing, but also more then one site operating without issue. Since the operation of several sites were working this meant to us that the php, mysql and apache components were functioning normally. The 500 error hinted to us about cgi and perl possibly failing due to missing modules or linked headers. Also the failing sites appeared to be using Miva Merchant. Miva itself was not the cause of the issue however.

using the ldd command we were able to prove that all libraries were in the correct places and being found by ld.

ldd /usr/bin/mivavm
linux-gate.so.1 => (0x0056b000)
libm.so.6 => /lib/libm.so.6 (0x008c4000)
libdl.so.2 => /lib/libdl.so.2 (0x008ef000)
libpthread.so.0 => /lib/libpthread.so.0 (0x008f6000)
libc.so.6 => /lib/libc.so.6 (0x00769000)
/lib/ld-linux.so.2 (0x0074a000)

ldd /usr/bin/php-cgi
linux-gate.so.1 => (0x007c5000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x00412000)
libaspell.so.15 => /usr/lib/libaspell.so.15 (0x009e6000)
libpspell.so.15 => /usr/lib/libpspell.so.15 (0x00926000)
libgmp.so.3 => /usr/lib/sse2/libgmp.so.3 (0x009ae000)
libdb-4.3.so => /lib/libdb-4.3.so (0x02e91000)
libpthread.so.0 => /lib/libpthread.so.0 (0x008f6000)
libcurl.so.3 => /usr/lib/libcurl.so.3 (0x001fd000)
libbz2.so.1 => /usr/lib/libbz2.so.1 (0x02e7e000)
libz.so.1 => /usr/lib/libz.so.1 (0x00911000)
libpcre.so.0 => /lib/libpcre.so.0 (0x00cb9000)
libresolv.so.2 => /lib/libresolv.so.2 (0x00b60000)
libm.so.6 => /lib/libm.so.6 (0x008c4000)
libdl.so.2 => /lib/libdl.so.2 (0x008ef000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00993000)
libxml2.so.2 => /usr/lib/libxml2.so.2 (0x00292000)
libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x001cd000)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00d33000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00d0a000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0x00cfa000)
libssl.so.6 => /lib/libssl.so.6 (0x00183000)
libcrypto.so.6 => /lib/libcrypto.so.6 (0x00b76000)
libidn.so.11 => /usr/lib/libidn.so.11 (0x00b14000)
libc.so.6 => /lib/libc.so.6 (0x00444000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x028a3000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00110000)
/lib/ld-linux.so.2 (0x0074a000)
libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0x00cff000)
libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x00de3000)
libselinux.so.1 => /lib/libselinux.so.1 (0x00931000)
libsepol.so.1 => /lib/libsepol.so.1 (0x0094b000)

Running the commands from the command line showed that they were completing fine and not segfaulting on as standard run. The environment for Miva is not set which explains the second line and this happens after the headers are sent which limits its cause.

/usr/bin/mivavm
Content-Type: text/html
Set-Cookie: htscallerid=
ec99e8351f3312599b5c4e1e0a78183b; expires=Thu, 28-Jun-2012 16:02:03 GMT; path=/

Configuration Error: Error loading configuration library: Unable to locate/load libmivaconfig.so
Please contact the server administrator: webmaster@localhost

/usr/bin/php-cgi

test#>

inspecting apache logs showed nothing but a restart a few days earlier.

[Tue Jun 28 22:39:22 2011] [notice] Apache configured — resuming normal operat ions

inspecting site specific logs from:

/var/www/vhosts/domain.com/statistics/logs/error_log

[Wed Jun 29 12:53:57 2011] [error] [client 10.0.0.151] suexec policy violation: see suexec log for more details
[Wed Jun 29 12:53:57 2011] [error] [client 10.0.0.151] Premature end of script headers: mivavm

inspecting suexec_log and suexec.log reveal the following:

lines in suexec_log
[2011-06-28 08:54:19]: uid: (#####/#####) gid: (2524/2524) cmd: mivavm

lines in suexec.log

[2011-06-28 08:59:34]: uid: (#####/#####) gid: (2524/2524) cmd: mivavm
[2011-06-28 08:59:34]: command not in docroot (/usr/bin/mivavm)

This hinted at an issue with suexec configuration of which docroot is precompiled in the command.
Upon inspecting /usr/local/psa/suexec/psa-suexec in vi, the file /usr/bin/minavm can be found
which hinted that this file is working and running the command yields no response which seems ok and no logs changed.

Running /usr/sbin/suexec and making note of the fact that it is old and that there are several versions gave the clue.

-r-s–x— 1 root root suexec
-r-s–x— 1 root apache suexec.saved_by_psa
-r-s–x— 1 root apache suexec.saved_by_psa.01.05;14:15
-r-s–x— 1 root apache suexec.saved_by_psa.01.06;18:03

Verified this assumtion with online post:

http://forums.theplanet.com/lofiversion/index.php/t57865.html

Modified the script to suit our environment

#!/usr/bin/bash
a1=`ls -la /usr/sbin/suexec | tr “-” ” ” | gawk ‘{print substr($0,1,56)}’`
a2=`ls -la /usr/local/suexec/psa-suexec | tr “-” ” ” | gawk ‘{print substr($0,1,56)}’`
if [[ “$a1” != “$a2″ ]]
then
cp /usr/sbin/suexec /usr/sbin/suexec-saved-$(date +%Y”-“%m”-“%d)
cp -Rfp /usr/local/suexec/psa-suexec /usr/sbin/suexec
fi

-r-s–x— 1 root apache 19964 Mar 31 00:05 suexec
-r-s–x— 1 root root 11608 Jun 29 13:12 suexec-saved-2011-06-29
-r-s–x— 1 root apache 21656 Dec 23 2009 suexec.saved_by_psa
-r-s–x— 1 root apache 20896 Dec 18 2007 suexec.saved_by_psa.01.05;14:15
-r-s–x— 1 root apache 20612 Apr 25 2008 suexec.saved_by_psa.01.06;18:03

Site resume normal operation