Rabu, 06 September 2017

How to solved No space left on device: AH00023: Couldn't create the proxy mutex

  Tidak ada komentar
In this morning i got some error in one of my several server :

I check the health of server, all is well, then i check the server log and found this :

This relates to semaphores on your system (you've run out).  Run the following to clear them out:

[root@localhost ~]# cd /var/log/httpd/
[root@localhost httpd]# ls
access_log           access_log-20181224  access_log-20190107  error_log-20181216  error_log-20181230  sem.txt
access_log-20181216  access_log-20181230  error_log            error_log-20181224  error_log-20190107
[root@localhost httpd]#
Open the error log
[root@localhost httpd]# tail -n 100 error_log
[Mon Jan 07 23:59:27.134380 2019] [mpm_prefork:notice] [pid 25930] AH00170: caught SIGWINCH, shutting down gracefully
[Tue Jan 08 00:00:57.434345 2019] [core:notice] [pid 5245] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
[Tue Jan 08 01:43:14.821613 2019] [auth_digest:notice] [pid 23004] AH01757: generating secret for digest authentication ...
[Tue Jan 08 01:43:14.823605 2019] [core:emerg] [pid 23004] (28)No space left on device: AH00023: Couldn't create the proxy mutex
[Tue Jan 08 01:43:14.823642 2019] [proxy:crit] [pid 23004] (28)No space left on device: AH02478: failed to create proxy mutex
[Tue Jan 08 01:43:14.823648 2019] [:emerg] [pid 23004] AH00020: Configuration Failed, exiting
[Tue Jan 08 02:01:10.376511 2019] [core:notice] [pid 26054] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Jan 08 02:01:10.378978 2019] [suexec:notice] [pid 26054] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
[Tue Jan 08 02:01:10.423442 2019] [auth_digest:notice] [pid 26054] AH01757: generating secret for digest authentication ...
[Tue Jan 08 02:01:10.425212 2019] [core:emerg] [pid 26054] (28)No space left on device: AH00023: Couldn't create the proxy mutex
[Tue Jan 08 02:01:10.425249 2019] [proxy:crit] [pid 26054] (28)No space left on device: AH02478: failed to create proxy mutex
[Tue Jan 08 02:01:10.425255 2019] [:emerg] [pid 26054] AH00020: Configuration Failed, exiting
[Tue Jan 08 02:03:24.422902 2019] [core:notice] [pid 26490] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Jan 08 02:03:24.424954 2019] [suexec:notice] [pid 26490] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
[Tue Jan 08 02:03:24.466775 2019] [auth_digest:notice] [pid 26490] AH01757: generating secret for digest authentication ...
[Tue Jan 08 02:03:24.468591 2019] [core:emerg] [pid 26490] (28)No space left on device: AH00023: Couldn't create the proxy mutex
[Tue Jan 08 02:03:24.468629 2019] [proxy:crit] [pid 26490] (28)No space left on device: AH02478: failed to create proxy mutex
[Tue Jan 08 02:03:24.468634 2019] [:emerg] [pid 26490] AH00020: Configuration Failed, exiting
[root@localhost httpd]#
When i check the httpd status
[root@localhost httpd]# service httpd status
Redirecting to /bin/systemctl status  httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2019-01-08 02:03:24 EST; 3min 35s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 26492 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 24608 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
  Process: 26490 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 26490 (code=exited, status=1/FAILURE)

Jan 08 02:03:24 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
Jan 08 02:03:24 localhost.localdomain httpd[26490]: AH00558: httpd: Could not reliably determine the server's fully qualified...essage
Jan 08 02:03:24 localhost.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 08 02:03:24 localhost.localdomain systemd[1]: Ignoring invalid environment assignment 'export LD_LIBRARY_PATH=/usr/lib/or.../httpd
Jan 08 02:03:24 localhost.localdomain systemd[1]: Ignoring invalid environment assignment 'export ORACLE_HOME=/usr/lib/oracle.../httpd
Jan 08 02:03:24 localhost.localdomain kill[26492]: kill: cannot find process ""
Jan 08 02:03:24 localhost.localdomain systemd[1]: httpd.service: control process exited, code=exited status=1
Jan 08 02:03:24 localhost.localdomain systemd[1]: Failed to start The Apache HTTP Server.
Jan 08 02:03:24 localhost.localdomain systemd[1]: Unit httpd.service entered failed state.
Jan 08 02:03:24 localhost.localdomain systemd[1]: httpd.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost httpd]#
I try clean some error log for free up the disk and start, but it's still failed, then i found this way to solve the problem

This relates to semaphores on your system (you've run out).  Run the following to clear them out:
[root@localhost httpd]# ipcs | grep apache | awk '{print $2}' > sem.txt
[root@localhost httpd]# for i in `cat sem.txt`; do { ipcrm -s $i; }; done;
[root@localhost httpd]#
Restart the httpd service
[root@localhost httpd]# service httpd restart
Redirecting to /bin/systemctl restart  httpd.service
[root@localhost httpd]#
Check the httpd status again
[root@localhost httpd]# service httpd status
Redirecting to /bin/systemctl status  httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-01-08 02:07:11 EST; 6s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 26492 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 24608 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 27422 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─27422 /usr/sbin/httpd -DFOREGROUND
           ├─27424 /usr/sbin/httpd -DFOREGROUND
           ├─27425 /usr/sbin/httpd -DFOREGROUND
           ├─27426 /usr/sbin/httpd -DFOREGROUND
           ├─27427 /usr/sbin/httpd -DFOREGROUND
           ├─27428 /usr/sbin/httpd -DFOREGROUND
           ├─27429 /usr/sbin/httpd -DFOREGROUND
           └─27430 /usr/sbin/httpd -DFOREGROUND

Jan 08 02:07:11 localhost.localdomain systemd[1]: Ignoring invalid environment assignment 'export LD_LIBRARY_PATH=/usr/lib/or.../httpd
Jan 08 02:07:11 localhost.localdomain systemd[1]: Ignoring invalid environment assignment 'export ORACLE_HOME=/usr/lib/oracle.../httpd
Jan 08 02:07:11 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
Jan 08 02:07:11 localhost.localdomain httpd[27422]: AH00558: httpd: Could not reliably determine the server's fully qualified...essage
Jan 08 02:07:11 localhost.localdomain systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost httpd]#
You can check the practice here : https://youtu.be/5hXLF4HUOaY

Tidak ada komentar :

Posting Komentar