Evaluating FAMP server performance

I recently moved dalescott.net from a bare metal server to a $5/month DigitalOcean droplet (512MB Memory, 1 Core Processor, 20GB SSD Disk). Afterwards, I wanted to get a measure of server performance. A quick web search came up with some candidates:

  1. Load Impact, claiming “The leading on-demand load and performance testing software-as-a-service.”
  2. Neustar, who claims “We started it all. And we continue to shape the industry.”
  3. WebPageTest, created by AOL for internal use before being open-sourced and now primarily developed and supported by Google.

There are more, but these will be enough to get started with. I had wanted to include Blitz.io, which was referenced in a blog post by Ryan Frankell on using Apache on a small DigitalOcean droplet . However, I couldn’t find any free evaluation capability, and Blitz’s demo is hard-coded to use a Blitz demo site, which eliminated Blitz from consideration.

Tuning Apache

The first step was to tune Apache. 512MB of RAM is low by server standards and also Apache’s default configuration. Some tuning is needed to keep Apache operating in the available RAM, otherwise Apache will gradually consume everything, including swap, and eventually the kernel will start thrashing.

KeepAlive

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5

The FreeBSD apache24 port already sets these as defaults in /usr/local/etc/apache24/extra/httpd-default.conf  (the default MaxKeepAliveRequests is even lower than the recommended 200).

References

MPM-Prefork

Although Event is reportedly the default MPM in Apache 2.4, Prefork is still the default on FreeBSD 10.x for compatibility with non-thread-safe php/perl/python modules.

MaxRequestWorkers and MaxConnectionsPerChild must be lowered in /usr/local/etc/apache24/extra/httpd-mpm.conf, and the Include line for httpd-mpm.conf in usr/local/etc/apache24/httpd.conf must be un-commented (and Apache restarted).

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: min number of server processes which are kept spare
# MaxSpareServers: max number of server processes which are kept spare
# MaxRequestWorkers: max number of server processes allowed to start
# MaxConnectionsPerChild: max num of connections a server process serves
# before terminating
#
#<IfModule mpm_prefork_module>
#    StartServers 5
#    MinSpareServers 5
#    MaxSpareServers 10
#    MaxRequestWorkers 250
#    MaxConnectionsPerChild 0
#</IfModule>

<IfModule mpm_prefork_module>
 StartServers 5
 MinSpareServers 5
 MaxSpareServers 10
# reduce max number of server processes to not exceed physical memory
# called "MaxClients" prior to v2.3.13 (MaxClients is still supported)
# - uses cache under load when 25
# - appears to not use cache under load when 12
 MaxRequestWorkers 12
# reduce max connections per child to avoid idle processes from holding onto memory
# - memory use seems stable when 200 but could be lower than optimal
# MaxConnectionsPerChild 350
 MaxConnectionsPerChild 200
</IfModule>

References

Load Impact

I first used the free no-login-required test, which loads 25 Virtual Users in 5 minutes, but by creating a free login profile you are allowed 5 (QTY)-five minute tests (executions) up to 100 VUs per month. Here are the results of the Load Impact test with Apache tuned, showing load times in the low seconds for up to 15 users. This can likely be further improved, but the server is stable and adequate for the expected amount of traffic.

2016-08-12 LoadImpact test results

 

Neustar

Neustar queued three servers for testing, but the Washington server never ran. Here are the results when I gave up waiting.

image

WebPageTest

WebPageTest concentrates on measuring how fast it takes a page to load, and gives lots of information for digging into the load times for each aspect of the page. I was looking for more of a general load or stress test to see if my server will fail under load, but the information can be valuable none the less.

image

Server Performance

Here are the result of top while the Load Impact no-login (20 VU) test is running. Although the server can by no means be considered fast, it also is not exhausting swap and stalling. I’m pleased for now, and will be back to review when time permits.

top-load-test-during

 

Using “new” PHP with “old” MySQL passwords

I recently encountered this error trying to connect Achievo to an existing corporate project-tracking database:

Critical: Unknown error: 2000 (mysqlnd cannot connect to MySQL 4.1+ using the old insecure authentication. Please use an administration tool to reset your password with the command SET PASSWORD = PASSWORD(‘your_existing_password’). This will store a new, and more secure, hash value in mysql.user. If this user is used in other scripts executed by PHP 5.2 or earlier you might need to remove the old-passwords flag from your my.cnf file).

To understand what’s going on, you need to understand a couple not-so-recent developments in PHP and MySQL:

  • MySQL version 4.0 and earlier used a 16-byte password hash, but starting with version 4.1+ uses a more secure password algorithm and 41 byte hash (although still supporting the older less secure password hash if needed).
  • The original external PHP module for accessing MySQL was mysql, but the current preferred module is mysqli (MySQL Improved).
  • The original low-level c-library that mysql (and mysqli) used to actually interface to a MySQL server was libmysql (included with MySQL), but starting with PHP 5.3 they are typically compiled with mysqlnd (a seperate project). mysqlnd has many advantages compared to libmysql, but it doesn’t support the old 16-byte MySQL password hash.

The database server I needed to connect to was running MySQL v5.0, and was configured with “old_passwords” set to On, which sets password operations to use the older 16-byte password hash (possibly because the original client/server application had been developed with MySQL 4.0 or earlier).

The simplest solution would have been to turn old_passwords Off and reset my password as the error message said (assuming the password column in the mysql.usr table would hold a 41-byte hash, otherwise its width would have to be increased first). However, the dba was loath to change the server configuration in case it broke existing mission critical enterprise applications, so I needed a different solution.

Since mysql and mysqli are compiled seperately, they can use different underlying c-libraries. In other words, mysql can be compiled to use libmysql – which supports the old 16-byte password hash. That would allow using mysql to connect to the corporate project-tracking database, without any effect on using mysqli to connect to other databases.

On a FreeBSD server, it was as easy as:

# pkg_delete php5-mysql
# cd /usr/ports/databases/php5-mysql
# make config  (uncheck support for mysqlnd)
# make install clean

and then edited the Achievo config.inc.php file to use mysql for the corporate database instead of mysqli.