httpdが落ちた

朝方、httpdが落ちたようだ。再起動をかけると、正常に（たぶん？？？）動き出した。

/var/log/httpd/access.logを見ると、切り替わるタイミングだったのか、何も書かれていない。
/var/log/httpd/access.log.1も、特におかしなログは残っていない。

/var/log/httpd/error.logには再起動後のログが残っていた。
/var/log/httpd/error.log.1を見ると、朝の7:11:22に何か異常が起きたようだ。
[Sun Feb 24 07:11:22 2008] [warn] child process 2351 still did not exit, sending a SIGTERM
[Sun Feb 24 07:11:22 2008] [warn] child process 2352 still did not exit, sending a SIGTERM
[Sun Feb 24 07:11:22 2008] [warn] child process 2353 still did not exit, sending a SIGTERM
[Sun Feb 24 07:11:22 2008] [warn] child process 2354 still did not exit, sending a SIGTERM
[Sun Feb 24 07:11:23 2008] [warn] child process 2355 still did not exit, sending a SIGTERM
[Sun Feb 24 07:11:23 2008] [warn] child process 2356 still did not exit, sending a SIGTERM
[Sun Feb 24 07:11:23 2008] [warn] child process 2357 still did not exit, sending a SIGTERM
[Sun Feb 24 07:11:23 2008] [warn] child process 2359 still did not exit, sending a SIGTERM
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb89a2440 ***
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
======= Backtrace: =========
/lib/libc.so.6[0x41edf1]
/lib/libc.so.6(cfree+0x90)[0x422430]
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)[0x3865a5]
/usr/sbin/httpd[0xb7f8b9c5]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(semop+0x3b)[0x48912b]
/usr/lib/libapr-1.so.0[0x3851ea]
/usr/lib/libapr-1.so.0(apr_proc_mutex_lock+0x12)[0x384572]
/usr/sbin/httpd[0xb7f8bcbe]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(epoll_wait+0x28)[0x487eb8]
/usr/lib/libapr-1.so.0(apr_pollset_poll+0x61)[0x38c791]
/usr/sbin/httpd[0xb7f8bd03]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(semop+0x3b)[0x48912b]
/usr/lib/libapr-1.so.0[0x3851ea]
/usr/lib/libapr-1.so.0(apr_proc_mutex_lock+0x12)[0x384572]
/usr/sbin/httpd[0xb7f8bcbe]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(semop+0x3b)[0x48912b]
/usr/lib/libapr-1.so.0[0x3851ea]
/usr/lib/libapr-1.so.0(apr_proc_mutex_lock+0x12)[0x384572]
/usr/sbin/httpd[0xb7f8bcbe]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(semop+0x3b)[0x48912b]
/usr/lib/libapr-1.so.0[0x3851ea]
/usr/lib/libapr-1.so.0(apr_proc_mutex_lock+0x12)[0x384572]
/usr/sbin/httpd[0xb7f8bcbe]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(semop+0x3b)[0x48912b]
/usr/lib/libapr-1.so.0[0x3851ea]
/usr/lib/libapr-1.so.0(apr_proc_mutex_lock+0x12)[0x384572]
/usr/sbin/httpd[0xb7f8bcbe]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(semop+0x3b)[0x48912b]
/usr/lib/libapr-1.so.0[0x3851ea]
/usr/lib/libapr-1.so.0(apr_proc_mutex_lock+0x12)[0x384572]
/usr/sbin/httpd[0xb7f8bcbe]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
/usr/lib/libaspell.so.15[0xe3dda7]
/lib/libc.so.6(exit+0xee)[0x3e2c5e]
/usr/sbin/httpd[0xb7f8b9db]
/usr/sbin/httpd[0xb7f8b9ed]
[0x12d420]
[0x12d402]
/lib/libc.so.6(semop+0x3b)[0x48912b]
/usr/lib/libapr-1.so.0[0x3851ea]
/usr/lib/libapr-1.so.0(apr_proc_mutex_lock+0x12)[0x384572]
/usr/sbin/httpd[0xb7f8bcbe]
/usr/sbin/httpd[0xb7f8c141]
/usr/sbin/httpd[0xb7f8c21a]
/usr/sbin/httpd(ap_mpm_run+0xb44)[0xb7f8cd74]
/usr/sbin/httpd(main+0x95e)[0xb7f6338e]
/lib/libc.so.6(__libc_start_main+0xe0)[0x3ccf70]
/usr/sbin/httpd[0xb7f623c1]
======= Memory map: ========
[Sun Feb 24 07:11:24 2008] [notice] SIGHUP received. Attempting to restart
httpd: Syntax error on line 176 of /etc/httpd/conf/httpd.conf: Cannot load /etc/httpd/modules/mod_status.so into server: /etc/httpd/modules/mod_status.so: undefined symbol: ap_mod_status_reqtail

さて、どうしたものか？