Tag Archives: Apache

Rotating Apache log files on FreeBSD

I needed to review an Apache httpd server error log file recently and even though the server had been rebooted only a month ago, I had to scroll through 95,000 lines before I got to the interesting part.

To make Apache’s log files more manageable, I configured them to roll every week using the FreeBSD standard newsyslog utility. newsyslog is run from cron, and in the default configuration runs every hour. This limits rolling logfiles to at most once an hour, but this is typically more than adequate for system log files.

I could have hacked newsyslog’s main config file /etc/newsyslog.conf, which would kept all configuration in one place for convenience. However this can lead to trouble when updating the system (because any new version would have to be manually merged with the edited old version), and keeping updating simple is generally a good idea. To deal with this, there are directives in the main config file to read additional optional configuration:

...
<include> /etc/newsyslog.conf.d/*
<include> /usr/local/etc/newsyslog.conf.d/*

Since the Apache server is third-party software, I created /usr/local/etc/newsyslog.conf.d/apache.conf containing the following:

# Apache
/var/log/httpd-access.log www:www 440 9 * $W1D4 J /var/run/httpd.pid 30
/var/log/httpd-error.log www:www 440 9 * $W1D4 J /var/run/httpd.pid 30

To understand what this means, the fields are: [logfile name] [owner-group] [mode] [count] [size] [when] [flags] [path to pid file] [signal]. Apache’s access and error log files will be rolled every Monday at 4am (system time), a total of 9 weekly archives will be kept (providing up to 10 weeks of logs counting the current log), and log file archives will be compressed using bzip2. Some other interesting points are:

  • For security, ownership of archived logs is set to www and the file mode is set to read-only for user and group, with no access by anyone else.
  • A SIGUSR1 signal (30) is sent to Apache to perform a graceful restart after rolling the log file.

For more information, see the System Logging section of the FreeBSD Manual as well as the man pages for newsyslog and newsyslog.conf.

To read a compressed log file, uncompress the file and pipe to less:

% sudo bzcat httpd-error.log.0.bz2 | less

Which is also equivalent to the simpler:

% sudo bzless httpd-error.log.0.bz2

Cheers!

Load Testing a Home Internet Server

The “new” basement server hosting dalescott.net has been rock solid now for a couple days, so it was time for some load testing.

The server is an HP M7690Y media center with Intel Core2 2.40GHz CPU, 3G of RAM, and connected to the internet through a residential “internet-over-cable” service. I’m using the Apache pre-fork MPM with default configuration (no need to tune for reduced RAM with 3GB).

LoadImpact

I ran LoadImpact’s free account-required 50 user / 12 minute test, and monitored server resources while the test was running.

top/htop while while running LoadImpact test

CPU utilization spiked to maximum, but never ran out of RAM, let alone getting into the cache. Increased CPU performance means that http requests aren’t getting queued, resulting in less demand on RAM compared to a single-core CPU with 512MB RAM.

Here is the test summary. The number of VUs, or virtual users, is on the left Y-axis, the VU Page Load Time is on the right Y-axis, and time is on the X-axis.

LoadImpact results

WebPageTest

Next, I checked to see if WebPageTest liked the new server any better than the old one.

WegPageTest results

 

Compared to previous testing on the 1 CPU 512MB vps, the First Byte Time has gone from an F to a D. However, it’s not clear why Compress Images has gone from a B to a D, the servers should have identical WordPress, Apache and PHP configurations.

ISP Speed Test

Finally, I ran my ISP’s Speed Test.

ISP Speed Test

I ran the test from my laptop on the LAN side of a Hitron DOCSIS (“internet over cable”) interface adapter, but the results should apply equally to the server. The server is also connected to the Hitron, but configured on a pass-through to get its own external IP address via DHCP from my ISP.

Conclusion

Performance from the new server far exceeds that of the previous minimal vps droplet, but that is to be expected given the hardware performance. However, it seems performance on a residential ISP service is much more variable than the vps was. I ran the LoadImpact test several times from mid-morning to mid-afternoon, with worst-case VU load times in tens of seconds occurring after lunch with 40+ VUs. Obviously there will need to be changes again when the site starts drawing significant traffic.

FreeBSD and Apache on a $5/month DigitalOcean Droplet VPS

I recently merged dalescott.net and swiftconstructioncompany.net onto a $5/month DigitalOcean droplet (512MB Memory, 1 Core Processor, 20GB SSD Disk, and
1TB Transfer). Afterwards, I wanted to get a measure of server performance. A quick web search came up with some candidates:

  1. Load Impact, claiming “The leading on-demand load and performance testing software-as-a-service.”
  2. Neustar, who claims “We started it all. And we continue to shape the industry.”
  3. WebPageTest, created by AOL for internal use before being open-sourced and now primarily developed and supported by Google.

There are more, but these will be enough to get started with. I had wanted to include Blitz.io, which was referenced in a blog post by Ryan Frankell on using Apache on a small DigitalOcean droplet . However, I couldn’t find any free evaluation capability, and Blitz’s demo is hard-coded to use a Blitz demo site, which eliminated Blitz from consideration.

Tuning Apache

The first step was to tune Apache. 512MB of RAM is low by server standards and also Apache’s default configuration. Some tuning is needed to keep Apache operating in the available RAM, otherwise Apache will gradually consume everything, including swap, and eventually the kernel will start thrashing.

KeepAlive

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5

The FreeBSD apache24 port already sets these as defaults in /usr/local/etc/apache24/extra/httpd-default.conf  (the default MaxKeepAliveRequests is even lower than the recommended 200).

References

MPM-Prefork

Although Event is reportedly the default MPM in Apache 2.4, Prefork is still the default on FreeBSD 10.x for compatibility with non-thread-safe php/perl/python modules.

MaxRequestWorkers and MaxConnectionsPerChild must be lowered in /usr/local/etc/apache24/extra/httpd-mpm.conf, and the Include line for httpd-mpm.conf in usr/local/etc/apache24/httpd.conf must be un-commented (and Apache restarted).

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: min number of server processes which are kept spare
# MaxSpareServers: max number of server processes which are kept spare
# MaxRequestWorkers: max number of server processes allowed to start
# MaxConnectionsPerChild: max num of connections a server process serves
# before terminating
#
#<IfModule mpm_prefork_module>
#    StartServers 5
#    MinSpareServers 5
#    MaxSpareServers 10
#    MaxRequestWorkers 250
#    MaxConnectionsPerChild 0
#</IfModule>

<IfModule mpm_prefork_module>
 StartServers 5
 MinSpareServers 5
 MaxSpareServers 10
# reduce max number of server processes to not exceed physical memory
# called "MaxClients" prior to v2.3.13 (MaxClients is still supported)
# - uses cache under load when 25
# - appears to not use cache under load when 12
 MaxRequestWorkers 12
# reduce max connections per child to avoid idle processes from holding onto memory
# - memory use seems stable when 200 but could be lower than optimal
# MaxConnectionsPerChild 350
 MaxConnectionsPerChild 200
</IfModule>

References

Load Impact

I first used the free no-login-required test, which loads 25 Virtual Users in 5 minutes, but by creating a free login profile you are allowed 5 (QTY)-five minute tests (executions) up to 100 VUs per month. Here are the results of the Load Impact test with Apache tuned, showing load times in the low seconds for up to 15 users. This can likely be further improved, but the server is stable and adequate for the expected amount of traffic.

2016-08-12 LoadImpact test results

 

Neustar

Neustar queued three servers for testing, but the Washington server never ran. Here are the results when I gave up waiting.

image

WebPageTest

WebPageTest concentrates on measuring how fast it takes a page to load, and gives lots of information for digging into the load times for each aspect of the page. I was looking for more of a general load or stress test to see if my server will fail under load, but the information can be valuable none the less.

image

Server Performance

Here are the result of top while the Load Impact no-login (20 VU) test is running. Although the server can by no means be considered fast, it also is not exhausting swap and stalling. I’m pleased for now, and will be back to review when time permits.

top-load-test-during