Category: Maestro

Posted on

Installing mdbtools, MariaDB, Nginx, and PHP on FreeBSD 9.1

Update 2014-07-06 mdbtools is now available as a package for installing using pkg. It can also be compiled from source if desired.

Sometimes building a new server is easier than upgrading an old one, with the added bonus of staying up to date with current install procedures. Here’s the procedure I followed to build a new FreeBSD 9.1 server recently. I’ll be installing everything on the bare server (no jails).

This is a unfinished work in process!

The major applications being installed are:

  • FreeBSD 9.1-RELEASE
  • mdbtools v0.7 (project head from GitHub)
  • MariaDB 5.5.31
  • Nginx 1.4.2
  • PHP 5.5.1
  • phpMyAdmin 4.0.5

Create a virtual machine

You can build a bare-metal server, but a Virtual Machine (vm) can be more convenient to work with, and a dump from the vm can be easily restored on a bare-metal server if needed. The virtualizing environment I use is VirtualBox. Start by creating a basic virtual machine for BSD (FreeBSD), with 256 MB memory, a 20G IDE primary master drive, and a CD/DVD drive IDE secondary master.

Next, consider the network interface and how the vm will connect to the internet, and how you will connect to it. My default configuration is to bridge my laptop and the vm network interfaces. This gives me access to the vm from the host and from any other devices on my LAN, but requires a DHCP server on your LAN, and port 22 routed (not blocked). If you don’t have internet access (e.g. if you are in someone else’s conference room), or if you do but port 22 is blocked (e.g. you’re in a certain popular fast-food restaurant with free WiFi), you will need to use a NAT or Host Only network connection in VirtualBox. If you don’t, the IP address Windows choses for itself will typically not be in the same subnet as the IP address VirtualBox choses for the client, and the two will not be able to communicate. If you use the NAT connection, you will need to forward the following ports in order to communicate with the vm from the host:

ssh               host port 2222        client port 22
HTTP              host port 8880        client port 80
MySQL TCP         host port 3336        client port 3306
MySQL UDP         host port 3336        client port 3306
ftp               host port 2221        client port 21

Install FreeBSD OS

Install FreeBSD base system. Perform a standard install using FreeBSD-9.1-RELEASE-i386-dvd1.iso (or -bootonly.iso), with the following configuration:

hostname: firefly.scc.local
root password: secret
daemons to start at boot: sshd
user: dale (group wheel)

After installing the base system and rebooting, login as root and update the FreeBSD OS:

# freebsd-update fetch
# freebsd-update install

If freebsd-update reports that /usr/src/crypto/ssl/s3_cbc.c is missing, create the directory path (e.g. “# mkidir -p /usr/src/crypto/ssl/”) and fetch/extract again.

Update the ports tree:

# portsnap fetch
# portsnap extract

Edit /etc/hosts to specify a FQDN (fully qualified domain name) for the server:

::1                     localhost firefly.scc.local
127.0.0.1               localhost firefly.scc.local
#10.0.2.15               firefly.scc.local # default VBOX NAT IP address

Install portmaster:

# cd /usr/ports/ports-mgmt/portmaster/
# make install clean

If desired, install and run portaudit to monitor port security notices (you may want to omit this if you won’t be updating the system):

# cd /usr/ports/ports-mgmt/portaudit/
# make install clean
# portaudit -Fda

Edit /etc/ssh/sshd_config to allow remote ssh login by root and user dale, and restart sshd. I will keep OpenSSL and OpenSSH from the base system. The root user will be allowed remote ssh access for convenience, but this is not advised for systems accessible from the internet.

# vi /etc/ssh/sshd_config
# add following
AllowUsers root dale
PermitRootLogin yes
#
# /etc/rc.d/sshd restart

Copy your ssh public key to ~/.ssh/authorized_keys (e.g. using WinSCP).

You may want to define a mail alias for the root user and have local system mail forwarded to the system administrator.

# vi /etc/mail/aliases
add following alias:
root: <a href="mailto:realuser@realdomain.com">realuser@realdomain.com</a>

Install utility applications

Install some basic utility apps that usually come in handy eventually:

# cd /usr/ports/archivers/p7zip/
# make install clean
#
# cd /usr/ports/ftp/curl/
# make install clean
#
# cd /usr/ports/textproc/flip
# make install clean
#
# cd /usr/usr/ports/devel/git
# make install clean
#
# cd /usr/ports/www/lynx
# make install clean
#
# cd /usr/ports/ftp/wget
# make install clean
#
# cd /usr/ports/archivers/unzip   # might already be installed
# make install clean

Install mdbtools

mdbtools is a suite of utilities for working with data from an MS Jet database on a Unix system. First, install the GNU build toolchain needed to compile mdbtools.

# cd /usr/ports/devel/libtool   # may already be installed
# make install clean

# cd /usr/ports/devel/automake
# make install clean

# cd /usr/ports/devel/autoconf    # may already be installed
# make install clean

# cd /usr/ports/textproc/flex/
# make install clean

# cd /usr/ports/devel/bison/    # may already be installed
# make install clean

# cd /usr/ports/textproc/txt2man/
# make install clean

# cd /usr/ports/devel/glib20  # undocumented dependency
# make install clean

# rehash

Next, clone the mdbtools Github repo locally:

> mkdir ~/src/
> cd ~/src/
> git clone https://github.com/brianb/mdbtools.git

And finally, build and install mdbtools:

> cd ~/src/mdbtools/
> autoreconf -i -f
> ./configure
> gmake
> su - 
# gmake install

Add the installed mdbtools man pages to manpath (the install uses the Linux-typical /usr/local/share/man/man1/) by creating /usr/local/etc/man.d/mdbtools.conf and rebuilding the whatis database:

# vi /usr/local/etc/man.d/mdbtools.conf
# add MANPATH
MANPATH /usr/local/share/man
#
# /etc/periodic/weekly/320.whatis
# exit
> apropos mdb

Install Web App Stack

My goal is a simple common stack using MariaDB and Nginx for hosting PHP and Python-based web applications with MySQL back-ends. I’m installing MariaDB here to learn about using it, as MySQL appears to be loosing favor in the opensource community, and MariaDB is fully compatible with MySQL. I’m installing Nginx for similar reasons, it has become popular recently for its asynchronous event-driven approach to request handling.

Install MariaDB Database Server

Install MariaDB:

# cd /usr/ports/databases/mariadb55-server/   # includes client
# make config ; make install clean
# rehash

Edit rc.conf to start MariaDB at boot:

# vi /etc/rc.conf
...
# add mysql_enable
mysql_enable="YES"

Manually start MariaDB:

# service mysql-server start

Setup grant tables:

# cd /usr/local/   # mysql_install_db assumes its running from here
# mysql_install_db --user=mysql

Configure root password:

> mysqladmin -u root password 'appleton'
> mysqladmin -u root -p -h firefly.scc.local password 'appleton'

Grant root permission to connect remotely:

> mysql -u root -p
> grant all privileges on *.* to 'root'@'%' identified by 'appleton' with grant option;
> exit;

Use the provided my-medium.cnf config file and edit for using InnoDB tables:

The output from “my_print_defaults –help” implies my-medium.cnf should copied to /etc/my.cnf, but I’ll use the MySQL convention I’ve learned until I know different.

# cp /usr/local/share/mysql/my-medium.cnf /var/db/mysql/my.cnf
# vi /var/db/mysql/my.cnf
...
# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /var/db/mysql
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /var/db/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

Restart MariaDB:

# service mysql-server restart

and finally do a basic test to confirm things are basically working:

> mysql -u root -p
...
MariaDB [(none)]> show databases;
...
MariaDB [(none)]> use test;
...
MariaDB [(test)]&gt; exit;
Bye
>

Install Nginx web server

Nginx will interoperate with PHP via FastCGI and PHP-FPM (FastCGI Process Manager), and with Python via FastCGI and via the flup library (py27-flup). Install Nginx with appropriate options (note I’m not enabling SSL, which I don’t need at the moment, but I may wish later I had included it also):

# cd /usr/ports/www/nginx/
# make config
Use the default configuration options:
<span style="color: #666666; font-family: Consolas;">IPV6 IPv6 protocol support
HTTP Enable HTTP module 
HTTP_CACHE Enable http_cache module 
HTTP_REWRITE Enable http_rewrite module 
HTTP_STATUS Enable http_stub_status module</span>
WWW Enable html status files

# make install clean

The Nginx config file is:

/usr/local/etc/nginx/nginx.conf

Edit rc.conf to start Nginx at boot:

# vi /etc/rc.conf
...
nginx_enable="YES"

Manually start Nginx:

# service nginx start

Test that Nginx is running by browsing to the vm (e.g. http://localhost:8880). Nginx will be configured later.

Install PHP processor

Install PHP:

# cd /usr/ports/lang/php55/
# make config
...
enable additional options:
FPM Build FPM version

# make install clean

Edit PHP php.ini configuration file:

# cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini
# vi /usr/local/etc/php.ini
...
# uncomment session.save_path
session.save_path = "/tmp"
# add default date timezone
date.timezone = "America/Edmonton"

Install a motley assortment of PHP extensions (essentially the current requirements for phpMyAdmin, MediaWiki and the Yii framework).

# cd /usr/ports/lang/php55-extensions
# make config
...
enable additional extensions to install:
BZ2
CTYPE
CURL
DOM
FILTE
GD
ICONV
JSON
MBSTRING
MCRYPT
MYSQL
MYSQLI
OPENSSL
PDO
PDO_MYSQL (why PDO if not PDO_MYSQL ??)
READLINE
SESSION
SOAP
XML
ZIP
ZLIB

# make install clean

Configure PHP-FPM

I will be using the fastCGI process manager PHP-FPM (included with PHP starting with release 5.3.3) with Nginx. Configure PHP-FPM by editing /usr/local/etc/php-fpm.conf:

# vi /usr/local/etc/php-fpm.conf
...
make following changes ("-" means delete, "+" means add):

-; events.mechanism = epoll
+events.mechanism = kqueue
...
-listen = 127.0.0.1:9000
+listen = /var/run/php-fpm.sock
...
-;listen.owner = www
-;listen.group = www
-;listen.mode = 0666
+listen.owner = www
+listen.group = www
+listen.mode = 0666

Edit rc.conf to start PHP-FPM at boot:

# vi /etc/rc.conf
...
php_fpm_enable="YES"

Manually start PHP-FPM:

# service php-fpm start

Nginx will need to be configured further for specific to use PHP-FPM.

Install phpMyAdmin

phpMyAdmin provides convenient management of the MariaDB database server, without requiring any client-side software. First, install phpmyadmin from ports:

# cd /usr/ports/databases/phpmyadmin/
# make config
...
disable options:
APC PHP APC (animated progress bar) support
# make install clean

It seems that pecl-APC (the APC option in the phpMyAdmin config) can’t be compiled with PHP 5.5 (see FreeBSD forum post). I’d rather not downgrade to PHP 5.4, and I suspect I can make do without an “animated progress bar”, so I’m unselecting it for now. Load pma tables:

# cd /usr/local/www/phpMyAdmin
# mysql -u root -p < ./examples/create_tables.sql

Configure Nginx:

# command
# command

Create the phpmyadmin configuration using the setup wizard (and copy to config.inc.php). Access http://hostname/phpmyadmin/setup, specify connection type: socket (instead of tcp) and use suggested names for all tables. Increase max session before auto logout to 9 hrs (from 3 min):

# vi /usr/local/www/phpMyAdmin/config.inc.php
...
$cfg['LoginCookieValidity'] = 3600 * 9; // 3600 sec/hr * 9 hrs
...

Also edit session.gc_maxlifetime in php.ini:

# vi /usr/local/etc/php.ini
...
; increase max session time for phpMyAdmin. Max session time for phpMyAdmin
; set to 9 hrs in phpMyAdmin config.inc.php ((LoginCookieValidity), which
; requires increasing php garbage collection to greater than 9 hrs
; E.g. 32500 sec = (3600 sec/hr * 9 hrs) + 100 sec
session.gc_maxlifetime = 32500

Configure Nginx:

# command
# command
# command

Install requirements for PHP unit and functional testing (optional)

If you’re going to use the server for PHP unit and functional testing, you will likely want to install xdebug and testing frameworks. Install php-xdebug:

# cd /usr/ports/devel/php-xdebug
# make install clean

Edit /usr/local/etc/php/extensions.ini and comment loading xdebug as std extension. Edit /usr/local/etc/php.ini to add loading xdebug as zend_extension.

[xdebug]
; load xdebug as zend_extension (loading as std extension commented in php/extensions.ini)
zend_extension=/usr/local/lib/php/20100525/xdebug.so
; enable profiling
xdebug.profiler_enable = 1
xdebug.profiler_output_dir = /tmp/profiler
; remote settings
xdebug.remote_autostart=off
xdebug.remote_enable=on
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=localhost
xdebug.remote_port=9000

Install pear and pear PHPUnit for unit testing Yii-based projects.

# cd /usr/ports/devel/pear
# make install clean
# pear config-set auto_discover 1
# pear install pear.phpunit.de/PHPUnit

Install the PHP pear package Selenium, which is required for Yii PHPUnit testing and used also for functional testing. Install curl and PHP curl extension first if not already installed.

# cd /usr/ports/ftp/curl; make install clean
# cd /usr/ports/ftp/php5-curl; make install clean
# pear install phpunit/PHPUnit_Selenium

And that’s that!

Posted on

Compiling mdb-tools on Ubuntu 12.04

I recently replaced Linux Mint 13 with Ubuntu 12.04 LTS as the GNU Linux distribution on my dual-boot laptop. One of the first tasks after basic configuration was to install mdb-tools for Maestro development (used to extract data from a Parts&Vendors MS Jet4 database).

Install build dependencies:

$ sudo apt-get install libtool
$ sudo apt-get install automake
$ sudo apt-get install txt2man
$ sudo apt-get install libglib2.0-dev libdb-dev

Clone the mdb-tools GitHub repo:

$ cd ~/src
$ cd src
$ git clone https://github.com/brianb/mdbtools.git mdbtools
$ cd mdbtools

Compile mdb-tools, and install executables and man pages

$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Rebuild ld cache:

$ sudo ldconfig

Man pages are installed for mdb-tools executables:

  • mdb-array
  • mdb-export
  • mdb-header
  • mdb-hexdump
  • mdb-parsecvs
  • mdb-prop
  • mdb-schema
  • mdb-sql
  • mdb-tables
  • mdb-ver

For more information on mdb-tools:

For more information on why you have to run ldconfig after installing mdb-tools:

 

Posted on

Mayan EDMS

I might have mentioned liking the Django web application framework for good documentation and a strong community (there are even books you can buy!). I spent the first two weeks of 2012 learning some Python and working through The Definitive Guide to Django. Developing and testing locally was easy, but I abandoned the effort after another two weeks trying to configure a Python web stack on my FreeBSD server and returned to the pervasive AMP stack because of its simplicity.

However, yesterday I became aware of the Mayan EDMS project after being featured in a recent FLOSS Weekly podcast (hosted by Randal Schwartz of Perl book fame). It’s a really cool EDMS written using the Django framework, and might be suitable as the DMS component in Maestro.

It seems Django/Python documentation has matured since my aborted effort last year, so a new attempt may be more successful (I’ve also approached some local Meetup groups for some help this time). If I can deploy Mayan to my production server I might be switching frameworks (and languages) again. I don’t have much to lose though because I haven’t really deployed any significant Maestro code yet – most recent work has involved sorting out data structures and synchronizing data from external systems, with some re-usable shell/cron scripts for import data and throw-away ATK and Yii code (I didn’t even write the Yii code, it was generated CRUD code).

What makes Mayan EDMS great? Here are the features as listed on the Mayan EDMS project website, plus a couple additions of my own:

  • Electronic signature verification
  • Unlimited document versioning with revert
  • Unlimited user defined metadata
  • Automatic OCR of documents (with distributed OCR processing)
  • GPL3 license (although I’d prefer a BSD-type license)
  • No commercial “premium” version (the open source version isn’t a limited-feature teaser!)
  • Django/Python
  • and many more….
    • Dynamic default values for metadata
    • Filesystem integration
    • User defined document unique identifier and checksum algorithms
    • Local file or server side file uploads
    • Batch upload many documents with the same meta-data
    • Previews for a great deal of image formats, including PDF
    • Full text searching
    • Configurable document grouping
    • Permissions and roles support
    • Multi-page document support
    • Multilingual user interface: English, Spanish, Portuguese (Brazil and Portugal) Russian, Italian, Polish, German, French, Bulgarian and Dutch.
    • Duplicated document search
    • Plugable storage backends
    • Color coded tagging

I’ll post detailed installation instructions on FreeBSD as soon as I’ve got it working.

Posted on

WordPress and Yii

I’ve now spent some time with cbdb, and thought I’d share getting it running, and my takeaway from reviewing TrackStar and cbdb features (after that, it’s time to check in with Larry and see what tricks CMS is up to). First though are some comments on WordPress.

WordPress

I thought I’d include comments on WordPress, since I’m consolidating my personal content using WordPress.

  • WP has a nice plugin management system, with plugin’s automatically adding themselves into the application’s admin menu structure (or Dashboard).
  • If a WP module uses roles role-based authorization, the roles are managed using the plugin’s menu – or at least that’s how the NextGEN gallery plugin does it. Achievo has a single security profiles system that combines the privileges from each module (i.e. the actions that the role controls access to) onto a single role management page. To be honest, I’ve only ever used the system, but the privileges are grouped by module, and are a combination of basic CRUD actions that can be performed on a business object, with some extras.  My preference is the Achievo approach, with all the permission for a role managed in one place (so far, cbdb’s rbac seems close enough).
  • I like how NextGEN handles image uploading (a choice between a drag and drop interface or an Explorer-style interface fr uploading files, multi-file select in the Explorer-style interface, support for uploading zip image archives, and an upload progress bar). Similar functionality would work for uploading files in Maestro.

cbdb

  • The views generally show raw data, rather than user-oriented information (e.g. type, signed, grade …), and only the create/update form shows user-oriented data (a dropdown selector for Type and Grade, and radio buttons for Signed, Bagged and Collectable). TrackStar polishes things off a bit better in this area.
  • I like that the menu system for cbdb is created early in the development process compared to TrackStar, but the menu colors don’t work for me. I don’t agree that hiding menus a user isn’t authorized to use is good practice, and believe it actually leads to more confusion when users don’t understand why they don’t have have different menus presented, and prefer a static menu (greying-out menus that the user doesn’t have the authority to use, or showing the menu with data but greying-out the Edit or Save button).
  • The calendar wizard to enter dates looks nice, but it’s missing buttons to move between years (although I suspect it can be configured).

TrackStar

  • User management hasn’t been fleshed out as well as in cbdb, in particular not being able to view a list of users associated with a project.