Performance Monitoring using htop

htop is a great tool for viewing the basics of what a server is doing, in particular the real-time CPU and memory graphics. However, the colors in the bar graphs can make it difficult to distinguish between CPU tasks being executed (low-priority, normal, kernel and virtualiz), and the type of memory in use (used, buffers and cache). 

In this case, the “no-color” option can be used which causes the bar graphs to use characters instead of colors for sub-categories.

% htop -C

Exploring Server Performance

I recently rebuilt the server for dalescott.net after a drive failure. When complete, it seemed ERPNext wasn’t as snappy as before.

My final conclusion was, at the time, the server was likely being subjected to a penetration attack. The server commonly has spikes of 1K+ penetration attempts in a 24hr period, which tend to occur sporatically in clusters. However reivewing server performance was still a useful exercise.

dalescott.net SaaS Architecture

The host server is a Intel Core2 CPU 6600 2.40GHz with 6GB of RAM. The VirtualBox vm running ERPNext is a single-core with 2GB RAM (essentially taking one core and 2GB from the host).

Load At Rest

Below is the virtual machine and host server when at rest (no logged-in ERPNext users, and no on-going brute-force ssh or web app login attempts). The vm load (top) is 3% and the host (bottom) server cores are 5% and 2% (the 2% core is the erpnext server).

Load When Stressed

Below is the server when an ERPNext user logs in and accesses an Item list. The vm load (top) has maxed out at 100% and is using half its available memory. On the host (bottom), core #2 is almost 70% (erpnext), but core #1 is only 35% and only half the total available memory is being used, which is good.

Conclusion

Clearly ERPNext is processor-constrained in this situation, but at least under normal server load is still very suitable for either presentations or training with a limited number of concurrent users. 

Last Night’s Credential Stuffing

Out of curiosity I created a list of login IDs used in last night’s credential stuffing activity. They clearly reflect attempts to hack a server compared to consumer accounts, and it’s interesting besides “1234”, the single character “m” appears, as well as a single space character (” “).  Interesting…..

1234
aaron
account
adam
adm
admin
administrateur
administrator
apache
asteriskftp
auction
backup
boss
centos
cisco
connect
console
contact
cop
demo
D-Link
ems
fax
ftp
ftpuser
guest
helpdesk
info
installer
jenkins
kelly
kobayashi
lock
logout
lpa
lpd
luanvandiemcao
luckup
lucky
m
mailman
mailtest
manager
marketing
monitor
msf_user
nagios
network
new
office
openfiler
oracle
password
pi
PlcmSpIp
post
postfix
postgres
remote
report
sales
scan
scanner
sconsole
software
spam
squid
sue
support
svn
sync
synopass
system
tamaichi
test
test01
testuser
tomcat
ubnt
ubuntu
uftp
unknown
unlock
uploader
url
user
uucp
vagrant
vmail
vpn
webmaster
wordpress
www-data

Install ERPNext on FreeBSD 11.2 using VirtualBox

Search for other ERPNext-related posts. You may also visit the demo on dalescott.net.

The simplest way to “install” ERPNext on FreeBSD is to simply use the Virtual Image provided by the ERPNext project with VirtualBox.

The ERPNext project provides the Easy Install script for bare-metal installation but it has a number of Linux dependencies and will not work without changes on FreeBSD. Happily, the project also provides a fully configured virtual machine (based on Ubuntu Linux).

It may also be possible to use bhyve, the BSD hypervisor, with the virtual image, but the OVF file must first be converted to bhyve’s raw format.

Install VirtualBox

Install the virtualbox-ose-nox11 package for running headless virtual machines.

% sudo pkg install virtualbox-ose-nox11

The VirtualBox kernel module (virtualbox-ose-kmod) will also be installed, but it must be re-compiled from source and re-installed (at the very least, the system will crash when next re-booted once it has been configured to load the kernel module at boot). 

Update the ports collection to prepare for compiling the kernel module. 

# portsnap fetch update

If the ports collection has not been installed, install.

# portsnap fetch extract

The FreeBSD sources are required to compile the kernel module. If not already installed, install the FreeBSD sources.

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/src.txz % tar -C / -xzvf src.txz

Compile and install the virtualbox-ose-kmod port. Make will first refuse to install the module because it is already installed (recall it was installed by being a dependency of virtualbox-ose-nox11). De-install the virtualbox-ose-kmod package, then re-install the newly compiled version.

% cd /usr/ports/emulators/virtualbox-ose-kmod
% sudo make
% sudo make install
% sudo make deinstall
% sudo make reinstall

Perform post-install configuration.

1) edit /boot/loader.conf to load the vboxdrv kernel module at boot,

# vi /boot/loader.conf
...
vboxdrv_load="YES"

2) increase AIO limits by editing /etc/sysctl.conf (my server is using AIO, for more information refer to the virtualbox-ose-nox11 pkg-message).

vfs.aio.max_buf_aio=8192
vfs.aio.max_aio_queue_per_proc=65536
vfs.aio.max_aio_per_proc=8192
vfs.aio.max_aio_queue=65536

Reboot the system to load the kernel module (or load it manually).

Make a mental note before doing an OS update to first edit /boot/loader.conf to not load the module. Otherwise the system will likely crash when next rebooted.

The user that VirtualBox runs as must be a member of the vboxusers group. For simplicity, I’ll run VirtualBox using my own username, although best practise would be to create a dedicated user.

# pw groupmod vboxusers -m dale

Edit /etc/rc.conf to run vboxwebsrv (the Virtual Box web interface daemon) using the provided startup script installed in /usr/local/etc/rc.d/

% sudo vi /etc/rc.conf

vboxwebsrv_enable="YES"
vboxwebsrv_user="dale"

and finally start the vboxwebsrv service.

% sudo service vboxwebsrv start
% sudo service vboxwebsrv status

The vboxmanage cli utility can be used to manage virtual machines but I will be using phpVirtualBox which provides a familiar GUI.

Install phpVirtualBox

phpVirtualBox can be installed from the FreeBSD ports collection but it currently has a dependency on PHP 7.1 while I have PHP 7.2. I installed phpVirtualBox manually to avoid pkg attempting to revert my PHP install to 7.1, and have not encountered any issues.

Download the latest release from the phpVirtualBox Github project . Follow the instructions in README.md file and on the wiki. Extract the project to /usr/local/www, and edit the configuration.

# vi /usr/local/www/phpvirtualbox/config.php

var $username = 'dale';
var $password = 'dale_login_password';

Configure the webserver to serve phpVirtualBox. I’m using the basic Apache 2.4 http server package. I added a virtual host definition to /usr/local/etc/apache24/extra/httpd-vhosts.conf to serve phpvirtualbox as a phpvirtualbox.dalescott.net.

<VirtualHost phpvirtualbox.dalescott.net>
  DocumentRoot "/usr/local/www/phpvirtualbox"
  <Directory "/usr/local/www/phpvirtualbox">
    allow from all
    Options None
    Require all granted
  </Directory>
</VirtualHost>

Change the default phpVirtualBox login password to something secure after logging in for the first time.

“Install” ERPNext

Download the desired ERPNext Virtual Machine image (*.ova).

% cd ~/downloads
% wget http://build.erpnext.com/ERPNext-Production.ova

Using phpVirtualBox, create a new vm by importing the downloaded ERPNext-Production.ova Virtual Image file (File/Import). The OVF includes port forwarding rules to forward client port 80 to host port 8080 (for serving ERPNext) and a rule to forward ssh from client port 22 to host port 3022 (for system administration).

Start the vm and then login to ERPNext from a browser (e.g. www.dalescott.net:8080) using the default credentials. The new site wizard will run and lead you through ERPNext configuration. Use a secure password when defining the initial (admin) user, and the wizard will delete the initial Administrator user (with default password) when complete. 

Once logged into ERPNext, setup email processing so that users will receive notifications outside of ERPNext. This will be valuable to understanding and appreciating ERPNext’s significant social aspect. You will also want to change the system login (i.e. ssh) password for “frappe” user to something secure (or disable password authentication entirely in favor of key-based authentication).

Cheers,
Dale