Patch to FreeBSD Accepted!

It was an emotional moment last week when I had a patch (code change) accepted by the FreeBSD operating system. It was a relatively simple change to the dns/noip port’s post-install message to clarify the port’s configuration and use (a port is the scaffolding code that integrates a third-party application into FreeBSD, and provided as part of FreeBSD).

FreeBSD is a free Unix-like operating system, most commonly found on servers in backrooms and data centers. It is similar to Linux, except that FreeBSD is a direct open source derivative of the original AT&T Research UNIX® (Linux is a re-implementation or clone). FreeBSD is more enterprise friendly than either Microsoft Windows or Linux, because of its permissive license allowing use and rework without cost or legal restrictions (Linux imposes the restrictive copy-left GNU Public License, or GPL, limiting a business’s ability to profit from its own innovations), and because FreeBSD has a well-defined release engineering process that is visible, accessible, and emphasizes stability and trust.

The motivation for the patch came when I was re-building several systems in which the dns/noip port was used – the client for No-IP’s DDNS service. Each time I searched which files had been installed by noip to find the README file with configuration details. The problem would have been avoided entirely if the missing information had only been included in noip‘s post-install message.

I informally discussed my proposed patch with Stanislaw Halik, the dns/noip port maintainer on record, who indicated his approval. I then submitted an official PR (problem report) to the FreeBSD project, and included a patch file giving the specific changes I was proposing.

The PR was reviewed by the FreeBSD maintenance team, Stanislaw gave his official approval, and my provided patch file had been merged into the FreeBSD codebase in less than 24 hours – visible, traceable and trustable!

2016-09-16-noip-pr-212637-dalescottshaw-ca

How cool was that! Although I submitted the patch to scratch my own itch, it was accepted based on its benefit to others.

Dale
Transparency with Trust

P.S. Stanislaw also introduced me to POLA, or the Principle of Least Astonishment. POLA applies to user interface and software design, and can be taken to mean that software should behave in a way that users will anticipate. Since the dns/noip port can now be installed as a binary package (not only compiling from source as has been traditional), users installing the binary package will expect instructions to be meaningful to their situation.

Load Testing a Home Internet Server

The “new” basement server hosting dalescott.net has been rock solid now for a couple days, so it was time for some load testing.

The server is an HP M7690Y media center with Intel Core2 2.40GHz CPU, 3G of RAM, and connected to the internet through a residential “internet-over-cable” service. I’m using the Apache pre-fork MPM with default configuration (no need to tune for reduced RAM with 3GB).

LoadImpact

I ran LoadImpact’s free account-required 50 user / 12 minute test, and monitored server resources while the test was running.

top/htop while while running LoadImpact test

CPU utilization spiked to maximum, but never ran out of RAM, let alone getting into the cache. Increased CPU performance means that http requests aren’t getting queued, resulting in less demand on RAM compared to a single-core CPU with 512MB RAM.

Here is the test summary. The number of VUs, or virtual users, is on the left Y-axis, the VU Page Load Time is on the right Y-axis, and time is on the X-axis.

LoadImpact results

WebPageTest

Next, I checked to see if WebPageTest liked the new server any better than the old one.

WegPageTest results

 

Compared to previous testing on the 1 CPU 512MB vps, the First Byte Time has gone from an F to a D. However, it’s not clear why Compress Images has gone from a B to a D, the servers should have identical WordPress, Apache and PHP configurations.

ISP Speed Test

Finally, I ran my ISP’s Speed Test.

ISP Speed Test

I ran the test from my laptop on the LAN side of a Hitron DOCSIS (“internet over cable”) interface adapter, but the results should apply equally to the server. The server is also connected to the Hitron, but configured on a pass-through to get its own external IP address via DHCP from my ISP.

Conclusion

Performance from the new server far exceeds that of the previous minimal vps droplet, but that is to be expected given the hardware performance. However, it seems performance on a residential ISP service is much more variable than the vps was. I ran the LoadImpact test several times from mid-morning to mid-afternoon, with worst-case VU load times in tens of seconds occurring after lunch with 40+ VUs. Obviously there will need to be changes again when the site starts drawing significant traffic.

dalescott.net is back down from the cloud!

Hosting dalescott.net on a low-cost  cloud server didn’t go so well and I had to move my site back to a bare metal server in the basement.

Step 1

It all started when I created a $5/month FreeBSD 10.1 vps using a DigitalOcean droplet (1 CPU, 512MB RAM) to host a demo for my Maestro PLM/ERP project. The site also included OpenLDAP/phpLDAPadmin and Postfix/dovecot/procmail/mutt/SquirrelMail for Swift Construction Company demo user authentication and email. The vps was rock solid, and I was very pleased with performance and cost (although getting negligible traffic).

Step 2

A year later, I tried to simplify my life by migrating my WordPress blog from a bare metal server in the basement also to the DigitalOcean vps droplet. The blog went live on the vps once I updated the IP address for my dalescott.net domain using No-IP’s DDNS service, and a day later the system was non-responsive with kernel out-of-swap errors in /var/log/messages. My immediate response was to dedicate 2G of the available 20GB SSD file space to swap (3GB total swap), but this only delayed the system becoming non-responsive to ~3 days.

I next learned how to tune the apache prefork mpm to not use more than ~300MB memory. At first, everything seemed OK and I breathed a sigh of relief, not too responsive but at least not running out of RAM and thrashing.  Then I upgraded some ports and upgraded from FreeBSD 10.1 to 10.3 (or maybe from 10.2 to 10.3, I regret not keeping better notes), but the result was that the server couldn’t maintain an ssh connection for more than 30 seconds with Apache running.

Rebooting and watching the system console, I noticed a ZFS notice I hadn’t noticed before – warning of expected unstable behaviour!

ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable......
ZFS WARNING: Recommended minimal kmem_size is 512MB; expect unstable behavior....

I also noticed errors in /var/log/messages that I didn’t recall seeing before.

dale@whizzer:~ % tail /var/log/messages
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $growfs_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $ is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $rsyncd_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $php_fpm_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $htcacheclean_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $git_daemon_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $dbus_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $avahi_daemon_enable is not set properly - see rc.conf(5).
Aug 25 16:05:33 whizzer dale: /usr/sbin/service: WARNING: $avahi_dnsconfd_enable is not set properly - see rc.conf(5).
dale@whizzer:~ %

DigitalOcean now has both FreeBSD “10.3” and “10.3 zfs” droplet templates, and I had recently upgraded the system to 10.3 using “pkg upgrade”. Could there be some unexpected interaction between my manually updated system and DigitalOcean’s droplet management scaffolding?

DigitalOcean tech support was supportive and tried to help, but in the end recommended starting over.

…it’s always going to result in some issues if you upgrade a Unix system’s distribution release in-place. We see it a lot in the Linux images where a release upgrade causes some random issues down the road, and upgrades tend to not work as well as anticipated. We would recommend setting up a newly built Droplet running the release you require and then to plan your migration of applications or data onto that new system.

I was disappointed with the recommendation to start over, especially as FreeBSD is well known for being able to update in-place. Not being able to update the OS might be OK for short-lived dev servers, and maybe for production servers with a team of people to do the work when upgrading is necessary, but it was not what I was expecting.

Resolution

In the end, the decision to re-host back to my own server came down to performance, anticipated future maintenance effort, and cost. Page loading was never really as good as I wanted, especially after restricting Apache to available RAM (although at least it stayed somewhat responsive). I also didn’t look forward to having to repeat the server migration when I was ready to upgrade to FreeBSD 11.0.

Already having a suitable server on hand (a de-branded HP media tower with an Intel 2×2.3 GHz CPU, 3 GB RAM, two 400GB drives), I installed FreeBSD 10.3 and started the migration. I’ll keep the droplet for experimenting, but for now it’s back to the basement for dalescott.net.

Dale

Transparency with Trust

dalescott.net moves to the cloud!

I’m pleased to announce dalescott.net is now hosted by DigitalOcean in their New York data center. The site was previously hosted on a bare metal server in my basement, but with DigitalOcean I get better transfer rates and have no hardware maintenance. The cost is about the same as the electric bill was for the old server (although going from 2GB RAM and a 200GB HD, to 512MB RAM and a 20GB SSD).

Other than needing to tune Apache 2.4’s prefork mpm to use less memory, the server is pretty much vanilla FreeBSD 10.3 and created completely using packages, with no local compiling. This was a great convenience when building the server, and should pay off with less maintenance effort in the future. I am also now using Apache virtual servers to segregate the hosted web apps using sub-domains (e.g. http://maestro.dalescott.net).

As I noted in my previous post on load testing, viewers should get one to two second page loads with up to five (perhaps more) simultaneous viewers. This should be sufficient for now, and paraphrasing DHH, it’s more important to solve immediate problems than it is to solve problems that haven’t happened yet. Please leave a comment if you have any issues.

Managing a Team

On a separate topic, I am investigating a web-based time tracking application called Manage your Team (or MyT for short). I have tried several methods recently to capture personal time spent on task, including ToDoList (which, by the way, is an absolutely fantastic desktop hierarchical task and project management tool) as well as the ubiquitous spreadsheet (both local and cloud). However, I’m interested in providing a formal record of truth for project teams with multiple simultaneous projects – in addition to basic project management capability. Such a system could provide a truly efficient solution (meaning virtually no cost once you set it up) for time and cash-strapped small businesses to provide trusted source project documentation when required, such as for the SR&ED tax incentive program.

This would be a perfect use for my old friend Achievo, but sadly Achievo is no longer maintained and will likely soon lose relevance. Adieu mon ami. Consequently I’m always on the watch for a comparable open source application built using current technologies. MyT is a web-based time tracking tool written in OO PHP using the Yii PHP OO RAD framework, which I’m familiar with from my Maestro project. and appears to have the basics I’m looking for. There also seems to be support for plug-ins to extend  functionality. Regular releases lend credibility, the project appears to have been publicly released on Sourceforge at v1.0.0, and after two years of regular bug fix and new feature releases the team released v1.5.0 last week. I have a couple questions on the management of the project, such as why they chose the CC BY-NC (non-commercial) license, and if there’s any opportunity to change in favor of a more permissive license. Also there doesn’t appear to be a public repository or bug tracker. I’ve posted the questions to the community forum, but the first priority is to confirm the functionality and quality of the code. I will post again here with an update after spending more time with MyT.

Cheers!
Dale.


Transparency with Trust
www.dalescott.net