Author: dale

I started as a technician, first setting up sound reinforcement and projection equipment at a convention facility, co-founded a DJ business and constructed our PA system from scrach, then got into electronics maintenance and repair at a community college. After obtaining a university degree in engineering, I designed integrated circuits for the first generation of 900 MHZ analog cellphones, the first generation of CA-code-phase tracking GPS receivers and digital pro-audio. I then progressed through a variety of leadership roles in product management, marketing, engineering product development and manufacturing, in fields such as industrial GPS-based positioning and navigation, pipeline integrity inspection and industrial process control.
Posted on

Manipulating CSV-format files on FreeBSD

Conclusion Updated 2013-08-16.

The initial Maestro implementation site uses CSV files for intermediate data exported from legacy systems. I’m using CSV files because they are human readable and provide a point of control between the legacy systems and Maestro. They also provide an opportunity to manipulate and report on the data using simple command-line Unix tools.

Doing some cursory research on command-line CSV-file manipulation, solutions mentioned most often included:

  • csvfix is a command line tool that reportedly does everything you’d want, including order/select columns, split/merge, generating SQL inserts from CSV data and diffing CSV data, and has a permissive license.
  • csvkit is a collection of programs adhering to the Unix philosophy (a program should do one thing and do it well), intended to be to CSV files what grep, sed, cut, sort… are to text. csvkit is licensed under the permissive MIT license.
  • csvprintf is a simple UNIX command line utility for parsing CSV files, similar to the printf utility. cvsprintf is written in C, and is licensed under the permissive Apache License, Version 2.0.
  • csvtool is included with the OCaml CSV library.
  • GNU Recutils is a set of tools and libraries to access human-editable, plain text databases called recfiles, with support for CSV file import and export (the recfile concept seems similar to SQLite, but recutils seems to have better csv support than SQLite – e.g. escaping the field delimiter when it occurs within text, supporting \r\n (carriage return – line feed) within a text string, etc.
  • Perl, AWK and R were also mentioned, but working solutions appeared non-trivial for the uninitiated (sorry Randall).

so let’s see what compiles on FreeBSD 8.3…..

csvfix

csvfix is available as a Windows binary from the project site with the csvfix source on BitBucket. Compiling (using “make lin” for Unix/Linux) resulted in a g++ option error.

From the csvfix wiki, compiling requires a modern version of g++ with C++2011 support – and reportedly uncompilable on Mac OS X by people who know a lot more about compilers than me, so that’s as far as I’ll be going with csvfix for now.

csvkit

csvkit is written in Python, and requires mkvirtualenv and pip, which I unfortunately don’t have any experience with (yet). I’ll postpone further csvkit investigation for now.

csvprintf

First, I tried the instructions in the download archive:

> tar -xzf csvprintf-1.0.3.tar.gz
 > ls
 csvprintf-1.0.3 csvprintf-1.0.3.tar.gz
 > cd csvprintf-1.0.3
 > ls
 CHANGES Makefile.in configure csvprintf.h xml2csv
 COPYING README configure.ac main.c
 INSTALL aclocal.m4 csv.xsl scripts
 Makefile.am config.h.in csvprintf.1.in svnrev.c
 > ./configure
 checking for a BSD-compatible install... /usr/bin/install -c
 checking whether build environment is sane... yes
 checking for a thread-safe mkdir -p... scripts/install-sh -c -d
 checking for gawk... gawk
 ...
 checking for iconv_open in -lc... no
 configure: error: required function iconv_open missing
 >

Making a wild stab in the dark (because I don’t know better), I installed iconv-2.0_4 from the ports tree – and got the same error (you’re probably saying “I could have told you so!”).

Next, although from googling it seemed there was a possibility of GNU Linux autoconf miss-reading my FreeBSD system (and iconv_open capability), I tried updating libiconv to see if that changed anything. Unfortunately, portmaster reported my version of Perl (5.10) was out-dated, which I tried updating to 5.16 (instead of the current 5.18, because I read a May 2013 post saying 5.18 broke a lot of modules).

portmaster -o lang/perl5.16 lang/perl5.10

However, that didn’t end in success because I have apparently installed modules from both CPAN and BSDPan, and BSDPan modules can’t be updated using portmaster. I don’t have time to hunt where these modules came from, and if I even need then anymore! Grrrr.

csvtool

On the plus side, OCaml CSV is in the FreeBSD ports (ocaml-csv), but on the other hand, it pulls in a *lot* of dependencies I won’t use again (including graphics stuff). I went ahead and compiled the port anyway. It completed without error, but it seems csvtool isn’t included in the FreeBSD port (or at least it didn’t get installed). Grrrr.

Conclusions

I’m rolling my own. I’m following tutorials on the Python CMS module, and adapting them to my simple but unique needs – and I need to learn Python anyway for OpenERP (and to hang out with the cool kids). However, I would like to have a general-purpose go-to CVS utility, and look forward to eventually testing csvkit.

Posted on

New brakes for the Fusion (only a 55km ride to get them)

I lost rear braking on my Rocky Mountain Fusion last week. Biking pathways wasn’t bad, but I didn’t want to ride off-road with only a front brake, so the next day I biked downtown to The Bike Shop on my way home after work (not only did I buy my Fusion there, they also do great drop-in service so long as the job won’t take too long). Adding the 3 legs up in my head as I pedalled home, I was sure I would be close to 100Km. However, Google Maps calculated my route to be 55km, which I trust because my it was pretty straight, and I’ve found Google knows all about Calgary pathways (just make sure you select Bicycling for the navigation mode).

I really liked this route. On the way downtown, I was up and close with one of Calgary’s oldest industrial areas. After my rear brake pads were changed and brake fluid topped up, I left downtown and rode through one of Calgary’s oldest affluent residential areas. I also noticed an unfortunate casulty of this year’s flooding – a missing pedestrian suspension bridge over the Elbow River. I can sympathize with local users and commuters through the area; loosing the pedestrian bridges over the Bow River at both Mackenzie and Southland Park has meant changes to my own travel routes.

Marker “D” is my home, “B” is work, and “C” is The Bike Shop (I was travelling counter-clockwise).

Posted on

Compiling mdb-tools on Ubuntu 12.04

I recently replaced Linux Mint 13 with Ubuntu 12.04 LTS as the GNU Linux distribution on my dual-boot laptop. One of the first tasks after basic configuration was to install mdb-tools for Maestro development (used to extract data from a Parts&Vendors MS Jet4 database).

Install build dependencies:

$ sudo apt-get install libtool
$ sudo apt-get install automake
$ sudo apt-get install txt2man
$ sudo apt-get install libglib2.0-dev libdb-dev

Clone the mdb-tools GitHub repo:

$ cd ~/src
$ cd src
$ git clone https://github.com/brianb/mdbtools.git mdbtools
$ cd mdbtools

Compile mdb-tools, and install executables and man pages

$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Rebuild ld cache:

$ sudo ldconfig

Man pages are installed for mdb-tools executables:

  • mdb-array
  • mdb-export
  • mdb-header
  • mdb-hexdump
  • mdb-parsecvs
  • mdb-prop
  • mdb-schema
  • mdb-sql
  • mdb-tables
  • mdb-ver

For more information on mdb-tools:

For more information on why you have to run ldconfig after installing mdb-tools:

 

Posted on

Calgary Flood

If you follow any mass media news, you’ve probably heard about the flooding in Calgary caused by high water volumes in the Bow and Elbow rivers. The rivers join together in the downtown heart of Calgary, where most businesses were shuttered during the worst of the flooding. However, many residents along the rivers were also significantly impacted with lost homes and property. My family has been exceedingly fortunate, only my son’s and my bicycle routes to work have been affected.

In order to bicycle from our home (on the west side of the Bow) to our work (on the east side), there used to be a choice of five bridges – with commute times ranging from 50 minutes to 2 hours depending on the route and bridge. Today, only one of the bridges remains navigable, but the commute is still less than an hour and a half. The bridges themselves do not appear to have been significantly damaged, but the surrounding landscape and pathways leading to the bridges may be forever altered.

Here is the McKenzie Meadows golf course (close to the south end of Calgary),  5 days after the start of the flood. The photo was taken looking west, with the Bow River and Rocky Mountains in the background.

mackenzie-golf-course-960x198.jpg