RX Patient Tools
Use these quick guides before reading the article, or return to them when you need help preparing questions for a doctor.
If you see apache running however websites are not opening a few things could cause this:
- Check if firewall is not closing ports 80/443 .
- Check if MaxRequestWorkers not reached:
grep MaxRequestWorkers /var/log/httpd/error_logIf you see anything like ‘server reached MaxRequestWorkers setting, consider raising’ you have to increase it in extra/httpd-mpm.conf , properly customizing the config with customize main apache config and copying configure/ap2/conf/extra/httpd-mpm.conf to custom/ap2/conf/extra/ , modifying it and doing rewriting configs.
- Semaphores limit reached:
tail -f /var/log/httpd/error_logAnd if you see the ‘No space left on device’ error, check semaphores:
ipcs -a | grep apacheRemove them with:
for i in `ipcs -s | grep apache | awk {'print $2'}`; do (ipcrm -s $i); doneAnd start apache.
Automating the removal of semaphores
If you routinely need to remove apache semaphores with the ipcs/ipcrm tool, and if you cannot sort out why they keep building up, then use a hook script that can be called for downed service where we will use the script below, after confirming the “$service” is “httpd”.
- Create the file /usr/local/directadmin/scripts/custom/service_down_notice.sh
- Open it and paste following code:
#!/bin/sh
if [ "$service" != "httpd" ]; then
EMAIL=your@email.com
MAX_SEMAPHORES=15
IPCS=/usr/bin/ipcs
IPCRM=/usr/bin/ipcrm
MAIL=/bin/mail
COUNT=`${IPCS} | grep apache | wc -l`
if [ "$COUNT" -le $MAX_SEMAPHORES ]; then
#all is well, there are no semaphore build-ups.
exit 0;
fi
#we have more than MAX_SEMAPHORES, so clear them out and restart Apache.
LIST=/root/sem.txt
${IPCS} | grep apache | awk '{print $2}' > ${LIST}
for i in `cat ${LIST}`; do
{
${IPCRM} -s $i;
};
done;
/etc/init.d/httpd restart
TXT="${COUNT} semaphores cleared for apache for `hostname`"
echo "${TXT}" | ${MAIL} -s "${TXT}" ${EMAIL}
exit 1;
fi- And make it executable:
chmod 755 /usr/local/directadmin/scripts/custom/service_down_notice.shHow to track which site is producing high load
If your server is overloaded with apache processes you may track down which sites are causing it with [https://httpd.apache.org/docs/2.4/mod/mod_status.html](server status module). It is already compiled into apache, just all IPs are restricted to access it.
If you need to monitor server load just once feel free to modify /etc/httpd/conf/extra/httpd-info.conf directly and change the Require ip to your desktop outgoing IP address (12.34.56.78 in below example):
<Location /server-status>
SetHandler server-status
Require host .example.com
Require ip 12.34.56.78
</Location>Or remove all Require ip to allow all but change the /server-status to something hard-to guess since we don’t want just anyone viewing this page.
<Location /secret-server-status>
SetHandler server-status
</Location>
Then open http://server_ip/secret-server-status page.
Also change uncomment the #ExtendedStatus by removing the # character to view more details:
ExtendedStatus OnIf you want to keep server-status page for longer period and not overwritten with DirectAdmin you have to [/webservices/apache/customizing/](customize it) accordingly.
How to mitigate a slow-loris attack
A slow loris attack is one where an IP will connect to your apache server and clog up all child processes with it’s specially formed requests (I won’t get into the details as to how).
Apache is compiled with the reqtimeout module, which can help mitigate such attacks. To check, type:
httpd -M | grep reqtimeoutAnd make sure you see the reqtimeout_module in the output. The /etc/httpd/conf/extra/httpd-default.conf file contains some default values already:
<IfModule reqtimeout_module>
RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
</IfModule>Which means wait for the first byte of headers for 20 seconds. If some data arrives, increase the timeout corresponding to a data rate of 500 bytes/s, but not above 40 seconds.
Do not restart apache during the nightly tally
By default, DirectAdmin always issues graceful restarts, which means that no connections are interrupted, which is the safest method.
An actual restart is required to rotate logs and close file descriptors. We do not recommend disabling log rotation, which the directadmin.conf variable ‘rotation=1‘ controls. A switch to restart apache after the tally runs exists, as well: ‘restart_apache_after_tally=1‘.
pcfg_openfile: unable to check htaccess file
When viewing your domain, if you see the error “Forbidden”, then check your apache logs:
tail -f /var/log/httpd/error_log /var/log/httpd/domains/domain.com.error.logand see the error:
[crit] [client 12.34.56.78] (13)Permission denied: /home/username/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readableThis would imply that apache does not have read access on your /home/username directory. The cause for this can be permissions related, config related, or ownership related.
Apache runs as user ‘apache’ thus the directory must be +x for that user or group. Full directory structure of the public_html for testuser1 as example :
dr-xr-xr-x. 19 root root /
drwx--x--x. 16 root root /home
drwx--x--- 7 testusr access /home/testusr
drwx--x--x 4 testusr testusr /home/testusr/domains
drwx--x--x 7 testusr testusr /home/testusr/domains/example.com
drwxr-xr-x 3 testusr testusr /home/testusr/domains/example.com/public_htmlPermissions are most often 644 for files and 755 for directories and owned by the user, but these can vary depending on the php handler and webserver setup and by CMS.
How to trace down what exactly apache/php are doing
Trying to find a slowdown in your website and not sure where it is? Example: WordPress loads slowly and you don’t know why. Use strace!
The idea is that strace can dump all binary calls to a log, so we’ll dump them to disk and sift through them later. You can also use this method for other processes, just swap “httpd” with some other process name in the command below.
Note: this is a fairly high-level debug method for advanced administrators. Root ssh access is required.
- Run strace to log:
cd /root
mkdir straces
cd straces
ps ax | grep httpd | grep -v grep | awk '{ print "strace -f -s500 -o strace."$1".log -p "$1"&" }' | sh- Trigger the slow case. Load the website that is slow, however you need to duplicate the issue. Note, it might be best to highlight the URL bar and hit enter, instead of using F5. The reason is we only want to load the site, and not the included images/css/js, etc.. and F5 or ctrl-F5 may reload everything, causing the logs to fill with more info, making the cause harder to find.
- End the strace, open a 2nd console and run:
killall -9 straceYou’ll now have several strace.*.log files, one for each httpd process that was running. Note, if you’re debugging a slowdown (or possibly a socket timeout), hit ctrl-c before the timeout happens, but allow enough time where you can let it process things for a bit. This way, the last “chunks” of code that caused the slowdown will be near the end of the log, making it easier to track.
- You’ll need to find which PID handled your request, so grep your IP from the logs to find the PID. Change 1.2.3.4 with your actual IP address and use the command below:
grep 1.2.3.4 *.logThis will dump some messy code, but the far left of the output should display which files the code originated from. Let’s say that it showed the output in the strace.29622.log (your PID number will be different from 29622). Also there could be multiple files, depending on how many requests were made and how apache handled them. We’re only concerned with the “slow” code, and not the other things like images, so you’ll need to go through each one to figure out which is which to be able to analyze the relevant bits you’re after.
- From there, you can go through the logs, looking at what was happening at that time, eg:
less strace.29622.logThis is going to show you quite a lot of code, but you can also search with the / character.
You can alternatively “grep” for specific information, eg:
grep somethingspecific strace.29622.logWhich only shows the lines you’re looking for.
From this point, what you’re looking for, how you find it, and what you do with it would be up to you.
How to debug an apache segfault
Debugging Apache segfaults can be tricky, in an attempt to find out which module may be causing it. You might see something like this in the /var/log/httpd/error_log:
[Mon Aug 07 23:56:18.309463 2017] [core:notice] [pid 17630] AH00052: child pid 18187 exit signal Aborted (6)
- You can debug it using the gdb option, like this:
service httpd stop
gdb /usr/sbin/httpdand quickly (before the dataskq starts it up again), run:
run -X -d /etc/httpdwhich should let you trigger one request in the foreground, so do it quickly, before any other connections arrive to your server.
- Hopefully this has triggered your segfault, in which case it might show:
Thread 1 "httpd" received signal SIGABRT, Aborted.along with other info about exactly where it crashed.
- To see more details about the function tree used to get to that point, call a backtrace like this:
bt fullWhich should give you the list of functions called, where it started lower down, and ended at the top of the output. Somewhere in there might show you which module triggered it.
- To quit gdb, just run:
quit
