Apache – Dale Scott

September 20, 2017June 24, 2022

Use newsyslog to rotate Apache log file on FreeBSD

I recently needed to review the Apache httpd error log file on my server (/var/log/httpd-error.log), and had to scroll through 95,000 lines before getting to the part of interest. The server was rebooted only a month earlier, which shows how fast the Apache log file grows even on a server with relatively low demand.

To make Apache’s log files more manageable, I configured them to roll every week using the FreeBSD standard newsyslog utility, which is run from cron (see /etc/crontab).

Instead of editing the newsyslog config file (/etc/newsyslog.conf) directly, create a secondary configuration file specifically to rotate Apache logs. The end of the newsyslog master configuration file reads any secondary config files,

...
<include> /etc/newsyslog.conf.d/[!.]*.conf
<include> /usr/local/etc/newsyslog.conf.d/[!.]*.conf

and secondary configuration files will not be affected by upgrades to newsyslog.

First create a directory for the Apache newsyslog configuration file. Since Apache is third-party software, create the /usr/local/etc/newsyslog.conf.d directory.

% sudo mkdir -p /usr/local/etc/newsyslog.conf.d

and then create the config file:

% sudo vi /usr/local/etc/newsyslog.conf.d/apache.conf

# Apache
# [logfile name] [owner-group] [mode] [count] [size] [when] [flags] [path to pid file] [signal]
/var/log/httpd-access.log www:www 640 9 * $W1D4 J /var/run/httpd.pid 30
/var/log/httpd-error.log  www:www 640 9 * $W1D4 J /var/run/httpd.pid 30

The will roll the access and error log files every Monday at 4am (system time), a total of 9 weekly archives will be kept (providing up to 10 weeks of logs counting the current log), and log file archives will be compressed using bzip2. The file mode is consistent with other system logs, but could be made more restrictive if desired. A SIGUSR1 signal (30) is sent to Apache to perform a graceful restart after rolling the log file.

For more information, see the System Logging section of the FreeBSD Manual and man pages for newsyslog and newsyslog.conf.

To read a compressed log file, uncompress the file and pipe to less:

% sudo bzcat httpd-error.log.0.bz2 | less

or use the simpler:

% sudo bzless httpd-error.log.0.bz2

Cheers!

September 13, 2016February 3, 2022

Load Testing a Home Internet Server

The “new” basement server hosting dalescott.net has been rock solid now for a couple days, so it was time for some load testing.

The server is an HP M7690Y media center with Intel Core2 2.40GHz CPU, 3G of RAM, and connected to the internet through a residential “internet-over-cable” service. I’m using the Apache pre-fork MPM with default configuration (no need to tune for reduced RAM with 3GB).

LoadImpact

I ran LoadImpact’s free account-required 50 user / 12 minute test, and monitored server resources while the test was running.

CPU utilization spiked to maximum, but never ran out of RAM, let alone getting into the cache. Increased CPU performance means that http requests aren’t getting queued, resulting in less demand on RAM compared to a single-core CPU with 512MB RAM.

Here is the test summary. The number of VUs, or virtual users, is on the left Y-axis, the VU Page Load Time is on the right Y-axis, and time is on the X-axis.

WebPageTest

Next, I checked to see if WebPageTest liked the new server any better than the old one.

Compared to previous testing on the 1 CPU 512MB vps, the First Byte Time has gone from an F to a D. However, it’s not clear why Compress Images has gone from a B to a D, the servers should have identical WordPress, Apache and PHP configurations.

ISP Speed Test

Finally, I ran my ISP’s Speed Test.

I ran the test from my laptop on the LAN side of a Hitron DOCSIS (“internet over cable”) interface adapter, but the results should apply equally to the server. The server is also connected to the Hitron, but configured on a pass-through to get its own external IP address via DHCP from my ISP.

Conclusion

Performance from the new server far exceeds that of the previous minimal vps droplet, but that is to be expected given the hardware performance. However, it seems performance on a residential ISP service is much more variable than the vps was. I ran the LoadImpact test several times from mid-morning to mid-afternoon, with worst-case VU load times in tens of seconds occurring after lunch with 40+ VUs. Obviously there will need to be changes again when the site starts drawing significant traffic.

August 8, 2016February 3, 2022

Evaluating FAMP server performance

I recently moved dalescott.net from a bare metal server to a $5/month DigitalOcean droplet (512MB Memory, 1 Core Processor, 20GB SSD Disk). Afterwards, I wanted to get a measure of server performance. A quick web search came up with some candidates:

Load Impact, claiming “The leading on-demand load and performance testing software-as-a-service.”
Neustar, who claims “We started it all. And we continue to shape the industry.”
WebPageTest, created by AOL for internal use before being open-sourced and now primarily developed and supported by Google.

There are more, but these will be enough to get started with. I had wanted to include Blitz.io, which was referenced in a blog post by Ryan Frankell on using Apache on a small DigitalOcean droplet . However, I couldn’t find any free evaluation capability, and Blitz’s demo is hard-coded to use a Blitz demo site, which eliminated Blitz from consideration.

Tuning Apache

The first step was to tune Apache. 512MB of RAM is low by server standards and also Apache’s default configuration. Some tuning is needed to keep Apache operating in the available RAM, otherwise Apache will gradually consume everything, including swap, and eventually the kernel will start thrashing.

KeepAlive

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5

The FreeBSD apache24 port already sets these as defaults in /usr/local/etc/apache24/extra/httpd-default.conf (the default MaxKeepAliveRequests is even lower than the recommended 200).

References

http://serverfault.com/questions/110207/running-out-of-swap-space-on-web-servers-what-to-do

MPM-Prefork

Although Event is reportedly the default MPM in Apache 2.4, Prefork is still the default on FreeBSD 10.x for compatibility with non-thread-safe php/perl/python modules.

MaxRequestWorkers and MaxConnectionsPerChild must be lowered in /usr/local/etc/apache24/extra/httpd-mpm.conf, and the Include line for httpd-mpm.conf in usr/local/etc/apache24/httpd.conf must be un-commented (and Apache restarted).

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: min number of server processes which are kept spare
# MaxSpareServers: max number of server processes which are kept spare
# MaxRequestWorkers: max number of server processes allowed to start
# MaxConnectionsPerChild: max num of connections a server process serves
# before terminating
#
#<IfModule mpm_prefork_module>
#    StartServers 5
#    MinSpareServers 5
#    MaxSpareServers 10
#    MaxRequestWorkers 250
#    MaxConnectionsPerChild 0
#</IfModule>

<IfModule mpm_prefork_module>
 StartServers 5
 MinSpareServers 5
 MaxSpareServers 10
# reduce max number of server processes to not exceed physical memory
# called "MaxClients" prior to v2.3.13 (MaxClients is still supported)
# - uses cache under load when 25
# - appears to not use cache under load when 12
 MaxRequestWorkers 12
# reduce max connections per child to avoid idle processes from holding onto memory
# - memory use seems stable when 200 but could be lower than optimal
# MaxConnectionsPerChild 350
 MaxConnectionsPerChild 200
</IfModule>

References

Load Impact

I first used the free no-login-required test, which loads 25 Virtual Users in 5 minutes, but by creating a free login profile you are allowed 5 (QTY)-five minute tests (executions) up to 100 VUs per month. Here are the results of the Load Impact test with Apache tuned, showing load times in the low seconds for up to 15 users. This can likely be further improved, but the server is stable and adequate for the expected amount of traffic.

Neustar

Neustar queued three servers for testing, but the Washington server never ran. Here are the results when I gave up waiting.

WebPageTest

WebPageTest concentrates on measuring how fast it takes a page to load, and gives lots of information for digging into the load times for each aspect of the page. I was looking for more of a general load or stress test to see if my server will fail under load, but the information can be valuable none the less.

Server Performance

Here are the result of top while the Load Impact no-login (20 VU) test is running. Although the server can by no means be considered fast, it also is not exhausting swap and stalling. I’m pleased for now, and will be back to review when time permits.