dalescott.net is back down from the cloud!

Hosting dalescott.net on a low-cost  cloud server didn’t go so well and I had to move my site back to a bare metal server in the basement.

Step 1

It all started when I created a $5/month FreeBSD 10.1 vps using a DigitalOcean droplet (1 CPU, 512MB RAM) to host a demo for my Maestro PLM/ERP project. The site also included OpenLDAP/phpLDAPadmin and Postfix/dovecot/procmail/mutt/SquirrelMail for Swift Construction Company demo user authentication and email. The vps was rock solid, and I was very pleased with performance and cost (although getting negligible traffic).

Step 2

A year later, I tried to simplify my life by migrating my WordPress blog from a bare metal server in the basement also to the DigitalOcean vps droplet. The blog went live on the vps once I updated the IP address for my dalescott.net domain using No-IP’s DDNS service, and a day later the system was non-responsive with kernel out-of-swap errors in /var/log/messages. My immediate response was to dedicate 2G of the available 20GB SSD file space to swap (3GB total swap), but this only delayed the system becoming non-responsive to ~3 days.

I next learned how to tune the apache prefork mpm to not use more than ~300MB memory. At first, everything seemed OK and I breathed a sigh of relief, not too responsive but at least not running out of RAM and thrashing.  Then I upgraded some ports and upgraded from FreeBSD 10.1 to 10.3 (or maybe from 10.2 to 10.3, I regret not keeping better notes), but the result was that the server couldn’t maintain an ssh connection for more than 30 seconds with Apache running.

Rebooting and watching the system console, I noticed a ZFS notice I hadn’t noticed before – warning of expected unstable behaviour!

ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable......
ZFS WARNING: Recommended minimal kmem_size is 512MB; expect unstable behavior....

I also noticed errors in /var/log/messages that I didn’t recall seeing before.

dale@whizzer:~ % tail /var/log/messages
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $growfs_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $ is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $rsyncd_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $php_fpm_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $htcacheclean_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $git_daemon_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $dbus_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $avahi_daemon_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $avahi_dnsconfd_enable is not set properly - see rc.conf(5).
dale@whizzer:~ %

DigitalOcean now has both FreeBSD “10.3” and “10.3 zfs” droplet templates, and I had recently upgraded the system to 10.3 using “pkg upgrade”. Could there be some unexpected interaction between my manually updated system and DigitalOcean’s droplet management scaffolding?

DigitalOcean tech support was supportive and tried to help, but in the end recommended starting over.

…it’s always going to result in some issues if you upgrade a Unix system’s distribution release in-place. We see it a lot in the Linux images where a release upgrade causes some random issues down the road, and upgrades tend to not work as well as anticipated. We would recommend setting up a newly built Droplet running the release you require and then to plan your migration of applications or data onto that new system.

I was disappointed with the recommendation to start over, especially as FreeBSD is well known for being able to update in-place. Not being able to update the OS might be OK for short-lived dev servers, and maybe for production servers with a team of people to do the work when upgrading is necessary, but it was not what I was expecting.

Resolution

In the end, the decision to re-host back to my own server came down to performance, anticipated future maintenance effort, and cost. Page loading was never really as good as I wanted, especially after restricting Apache to available RAM (although at least it stayed somewhat responsive). I also didn’t look forward to having to repeat the server migration when I was ready to upgrade to FreeBSD 11.0.

Already having a suitable server on hand (a de-branded HP media tower with an Intel 2×2.3 GHz CPU, 3 GB RAM, two 400GB drives), I installed FreeBSD 10.3 and started the migration. I’ll keep the droplet for experimenting, but for now it’s back to the basement for dalescott.net.

Dale

Transparency with Trust

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.