Startups and Product Verification Testing

The purpose of product verification testing is to verify the product meets its’ design goals, and is included in any set of best practices. But what really is being verified?

  • Compliance to undocumented or informal assumptions of user expectations?
  • Compliance to documented external requirements, potentially including regulatory requirements?
  • Compliance to internal design goals, possibly including a code standard, static code analysis, and unit testing?
  • Confirm buttons can be pressed in any sequence and nothing bad happens?
  • Combinations of the above?

Formal complete verification testing is a valiant goal, but difficulties can be encountered in practice, particularly when funding is limited. A good formal test phase can cost as much in calendar days and labor $$’s as the design phase! A startup may not have the funding and runway for a good test phase, and could even be prejudiced against test by overconfident designers and managers. 

Unless verification test is represented on the project team from its onset, test resources will usually get addressed late in the development phase – once there is something to test. This means developers will first need to spend time onboarding the test resource to a suitable level familiarity with the product and the codebase, and more time during the test process to clarify ambiguous behavior and investigate test failures. A more viable strategy can be to leverage the design team’s intimate knowledge of their system in the test effort.

I like to think of product verification testing as a stool with three legs, one leg is automated regression testing, one leg is feature-specific testing, and the final leg is ad hoc “try to break it” attempts.

  • Use an automated test to catch obvious blunders with minimum effort, and run the test on every release candidate, every night, or even after every commit to the code repo. Creating the test will unavoidably take time from the development team, but often the time is not significantly more than that needed to on-board a developer-level test resource, explain requirements and validate the created test. 
  • Use feature-specific tests for verifying areas of concern where there is high risk, or where there is high consequence (such as regulatory compliance). Existing areas of concern by team members can be identified and explored in team whiteboard sessions (e.g. a concern the system may lockup if a user is interacting with the device when the internet connection is lost), and fault tree and/or failure-mode analysis diagrams can be created for further resolution and clarity. It may be practical to involve a temporary resource at this point to off-load the development team (e.g. an engineering or computer science co-op or summer student) and the test phase will have a higher chance of success, instead of muddling about for a month or two due to lack of support and produce no tangible results.
  • Use ad hoc “break-it” testing to find issues that would never be contemplated by the design team. The tester should not be familiar with the design internals, but should preferably be at least familiar with the product domain. As such, it can often be off-loaded to your favorite student again. However, be cautious of expecting useful test feedback from beta testers. Despite best intentions, beta testers are typically not motivated to spent effort in areas of functionality not of direct interest, or to document found issues to the degree needed by the development team.

Separation of design and test, documented requirements, formal verification, etc. have become best practices for a reason, but may not be practical or feasible for a startup chasing their first product. If the startup is successful at creating a revenue stream to protect, it will be able to adopt or refine its practices. If the startup fails due to having built the wrong product, in hindsight clearly resources spent on verification would have been better put towards market intelligence and alpha testing a proof-of-concept.

However, of course this is all IMHO and YMMV….

Building a Tryton Server on FreeBSD

This article is part of a series on the Tryton framework, to complement the Maestro project on GitHub. This article describes installing the Tryton server (trytond) on FreeBSD.

Caution – this is a Work in Progress  updating original 2014-05-24 post for Tryton 5.2 on FreeBSD 12.0 using Virtualenv.

Install FreeBSD

Boot vm from boot CD/DVD and follow the standard install procedure, including creating an admin user and including it in the wheel group.

I always configure remote access using ssh keys only for for security, Copy the public ssh key for the admin user to ~/.ssh, edit /etc/ssh/sshd_config to add “AllowUsers adminusername” and restart sshd.

Check for and install available FreeBSD OS updates, and install the pkg packaging system port..

# freebsd-update fetch
# freebsd-update install
# pkg update   

You can check the installed packages for reported vulnerabilities (-F is required for initial use only to download a new vulnerability database).

# pkg audit -F

Install PostgreSQL

The files belonging to the Postgresql database system will be owned by user “postgres”, who must also own the server process.

# pkg install postgresql93-server-9.3.4
# echo "postgresql_enable=YES" >> /etc/rc.conf
# /usr/local/etc/rc.d/postgresql initdb
# /usr/local/etc/rc.d/postgresql start

The default configuration file (/var/db/postgres/data96/postgresql.conf is acceptable and does not need editing. Only localhost will have access to PostgreSQL, which is adequate as Tryton will be served from the same server. 

Add a PostgreSQL “tryton” super-user.

dale@casper:~ % sudo su postgres
$ createuser -sdrP tryton
Enter password for new role:
Enter it again:
$ exit
dale@casper:~ %

The “tryton” super-user password will ned to be entered in trytond.conf (the trytond daemon configuration file) for Tryton to access its database.

Restart PostgreSQL.

dale@casper:~ % sudo /usr/local/etc/rc.d/postgresql restart
LOG:  ending log output to stderr
HINT:  Future log output will go to log destination "syslog".
dale@casper:~ %

Confirm localhost can connect to the PostgreSQL server (“-W” causes psql to prompt for the user password).

dale@casper:~ % psql --username=tryton -W --list
Password for user tryton:
List of databases
Name       | Owner    | Encoding | Collate | Ctype | Access privileges
postgres   | postgres | UTF8     | C       | C     |
template0  | postgres | UTF8     | C       | C     | =c/postgres +
           |          |          |         |       | postgres=CTc/postgres
template1  | postgres | UTF8     | C       | C     | =c/postgres +
           |          |          |         |       | postgres=CTc/postgres
(3 rows)
dale@casper:~ %

Install Tryton Dependencies

Install Python and other dependencies using pkg.

% sudo pkg install python
% sudo pkg install py36-pip
% sudo pkg install bash
% sudo pkg install py36-virtualenv
% sudo pkg install py36-virtualenvwrapper
% sudo pkg install libxml2
% sudo pkg install libxslt
% sudo pkg install graphviz

Install the Python pydot package, the Python interface to Graphviz, and psycopg2, the Python interface to PostgreSQL.

% sudo pip install pydot
% sudo pip install psycopg2

Create a trytond system user

The “tryton” system user will execute the trytond daemen. The tryton user’s home directory (/home/tryton) will be configured as the root of the Tryton file system for storing document attachments. 

dale@casper:~ % sudo adduser
Username: tryton
Full name: trytond system user
Uid (Leave empty for default):
Login group [tryton]:
Login group is tryton. Invite tryton into other groups? []:
Login class [default]:
Shell (sh csh tcsh bash rbash nologin) [sh]:
Home directory [/home/tryton]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: no
Username   : tryton
Password   : <disabled>
Full Name  : tryton system user
Uid        : 1002
Class      :
Groups     : tryton
Home       : /home/tryton
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes
adduser: INFO: Successfully added (tryton) to the user database.
Add another user? (yes/no): no
dale@casper:~ % 

Create a Tryton source directory

Create a sub-child directory in  the tryton user directory for Tryton source code.

dale@casper:~ % sudo mkdir /home/tryton/tryton
dale@casper:~ %

Configure a virtual environment

Create a “default” virtual environment to use for execution.

dale@casper:~ % su - Password:
root@casper:~ # bash
[root@casper /root]# cd /home/tryton/tryton 
[root@casper /home/tryton/tryton]# virtualenv default
Using base prefix '/usr/local'
New python executable in /usr/home/tryton/default/bin/python3.6
Also creating executable in /usr/home/tryton/default/bin/python
Installing setuptools, pip, wheel...done.
[root@casper /home/tryton/tryton]# source default/bin/activate
(default) [root@casper /home/tryton/tryton]# 

Install trytond and modules

Install trytond – the Trytond server daemon.

(default) [root@casper /home/tryton/tryton]# pip install trytond

trytond modules provide areas of functionality to trytond. You can generate a list of all available modules using pip.

(default) [root@casper /home/tryton/tryton]# pip search tryton

Install the trytond_sale module to provide a basic set of functionality.

(default) [root@casper /home/tryton/tryton]# pip install trytond_company

Modules often have dependencies. To see all the trytond modules which were installed, you can use pip to list installed modules and grep to filter for trydond.

(default) [root@casper /home/tryton/tryton]# pip list | grep trytond
trytond 5.2.6
trytond-account 5.2.3
trytond-account-invoice 5.2.2
trytond-account-invoice-stock 5.2.0
trytond-account-product 5.2.0
trytond-company 5.2.0
trytond-country 5.2.0
trytond-currency 5.2.1
trytond-party 5.2.0
trytond-product 5.2.1
trytond-sale 5.2.0
trytond-stock 5.2.1
(default) [root@casper /home/tryton/tryton]#

Create log directory for trytond

dale@casper:~ % sudo mkdir /var/log/trytond
dale@casper:~ % sudo chown -R tryton:tryton /var/log/trytond

Create json-rpc data directory for trytond

dale@casper:~ % sudo mkdir /var/run/trytond
dale@casper:~ % sudo chown -R tryton:tryton /var/log/trytond

Create trytond.conf

The Tryton configuration file trytond.conf is read by the Tryton server daemon trytond when it starts, and includes such site-specific data as:

  • computer addresses to respond to (jsonrpc).
  • username and password for the PostgreSQL “tryton” super-user.
  • Tryton “administrator” password (required to create, drop, backup or restore a database).
  • specify FreeBSD-specific directory paths

Unfortunately, the pip install for trytond currently does not copy the default trytond.conf file in the package to a suitable location (or rather, to any location). Download the trytond server distribution from PyPi, extract and copy trytond/etc/trytond.conf to /usr/local/etc/trytond.conf.

Edit parameters in /usr/local/etc/trytond.conf using the following as reference:

jsonrpc = *:8000,
jsondata_path = /var/run/trytond

db_type = postgresql
db_host = localhost
db_port = 5432
db_user = tryton
db_password = appleton

admin_passwd = appleton

pidfile = /var/run/trytond/
logfile = /var/log/trytond/trytond.log

data_path = /home/tryton

Create tryton rc script

An rc.d script will be created to manage starting and stopping trytond, and to start trytond after booting. I’m using a basic rc.d startup script created by Christoph Larsen for the GNU Health project, and added support for status reporting. Copy the following code to /usr/local/etc/rc.d/trytond


# PROVIDE: trytond
# Originally created by: Christoph H. Larsen

. /etc/rc.subr


load_rc_config $name

: ${trytond_enable="NO"}
: ${trytond_user="tryton"}
: ${trytond_group="tryton"}



trytond_start() {
  su tryton -c "$command --config=/usr/local/etc/trytond.conf" &

trytond_stop() {
if [ -f /var/run/${name}/${name}.pid ]; then
  kill `cat /var/run/${name}/${name}.pid`

trytond_restart() {
  if [ -f /var/run/${name}/${name}.pid ]; then
  kill `cat /var/run/${name}/${name}.pid`
  sleep 1
  su tryton -c "$command --config=/usr/local/etc/trytond.conf" &

run_rc_command "$1"

Make /usr/local/etc/rc.d/trytond executable

# chmod u+x /usr/local/etc/rc.d/trytond

Add trytond_enable to /etc/rc.conf

# echo "trytond_enable=YES" >> /etc/rc.conf

Start trytond

Start trytond using the rc script:

# /usr/local/etc/rc.d/trytond start

Once started, you can check if trytond is running with the status option:

# /usr/local/etc/rc.d/trytond status

or stop the server if needed:

# /usr/local/etc/rc.d/trytond stop

Create and configure a new Tryton database

Download and install the Tryton desktop client for your system. Launch the Tryton client and access menu: File > Databases > New database.

Enter the Tryton server admin password (“admin_passwd” in trytond.conf) in the password field, then click Change beside the IP address and change the address to that of your server.

You must enter the password first, before changing the server address, because the Tryton client will attempt to connect to the server immediately after the server address is changed, and will report “Unable to connect” if the admin password was not already entered.

Enter the name of the database to create (e.g. “scc”) and the admin password for the database, then click Create.

Login to the new database as user “admin” and the password you entered to create the database. The Module Configuration Wizard will run automatically after login to configure the new database.

  • Add user (e.g. “Dale Scott”, login “dale”), optionally add permissions (you may need to update user permissions after installing modules, so this is optional at this point),
    • add Permissions: “Administration” (which will be the only permission group available)
    • add Rule: Read, Write, Create, Delete, Model: View Search (which will be the only rule available available)

Install Tryton modules into the database

Mark modules for install

The Tryton modules previously installed to the server will not be installed in a Tryton database.

In the Tryton client, access the menu: Administration > Modules > Modules. Mark the following modules for install (double-click in the Mark for Install column):

  • dashboard
  • product
  • product-attribute
  • production
  • project
  • project-plan
  • purchase
  • sale
  • stock
  • stock-lot
  • stock-split

Install marked modules

There are two ways to install the marked modules, you can either:

  • Select Launch Action icon in the Modules menu bar and then Perform Pending Installation/Upgrade, or
  • Execute the Perform Pending Installation/Upgrade Wizard from the main Tryton menu.

After the install/upgrade completes, the Module Configuration wizard runs and will query you to configure the installed modules.

Create a Company

A company is a type of ‘party’. To identify a party as a company in a new database you must first create a new party, then select it as the new company.

  • Name
    • Swift Construction Company
  • Addresses
    • Name: Default
    • Street: 1 Swift Way
    • Zip: 13054
    • City: Shopton
    • Country: United States
    • Subdivision: New York
  • Language
    • English

Configure Company

  • Enter Company > Currency
    • US Dollar
  • On the Company > Employees > Party view
    • select the new company (e.g. Swift Construction Company)
    • select Add

Create Chart of Accounts

  • Company: Swift Construction Company
  • Account Template: Minimal Account Chart
  • Create Default Properties
    • Default Receivable Account: Main Receivable
    • Default Payable Account: Main Payable

You will need to reload the menu in the Tryton client if it isn’t now displaying menu choices for the new modules (i.e. access menu: User > Menu Reload (Ctrl-T).

The Tryton server has been installed and you can connect to it from a Tryton client. You have also created a new Tryton database with the modules necessary to implement Maestro workflows.


Update FreeBSD periodic databases

I don’t know if installing packages triggers the periodic databases to update, but since I often don’t run a virtual machine long enough to cross day, week or month boundaries (when they update automatically), I typically manually update them after major system changes.

# periodic daily
# periodic weekly
# periodic monthly


Start a Windows MPLAB X Project by Cloning a Git Repo

MPLAB X includes great built-in support for Git but it assumes projects are created first from within MPLAB X (File > New Project ) before managing them using Git. This doesn’t help if you’re starting by cloning a remote repo.

I’m using MPLAB X on Windows. The standard Git install for Windows from the Git project includes Git GUI, a basic Git client GUI app. I’ll use Git GUI to clone the remote repo, and then use MPLAB X’s built-in Git client for everything else.

Start Git GUI

and select Clone Existing Repository.

Clone the remote master repository locally (the master repo for me is on a Windows network drive).

When the remote repo has been cloned, Git GUI will show the current status of the new local repository – which should show no modified files (i.e. no files either staged or unstaged).

To confirm the repository was cloned correctly, you can check the commit history by accessing menu Repository > Visualize master’s History.

Now we can open the cloned local repo using MPLAB X.

Git is primarily accessed using a project’s context menu from the Project window.

However, be aware there are a few Git commands under the Team menu, including a Repository Browser that isn’t available from the project context menu.

Try accessing Show History for the project (you will need to click “Search” in the Show History view that opens).

Now I’ll ready to start hacking on the source. I’ll keep it simple and just add some TODO’s to the README.txt file. Notice how the editor shows my changes compared to the last commit in real-time!

When I’m ready to commit my changes, I can commit just the modified README.txt file,

or I can commit all modified files in the project.

To push a local commit back to the upstream origin repo, use the general-purpose Remote > Push… selection from the Project context menu.

Origin is the only remote repo configured in the local clone, so it should be indicated automatically in the Push to Remote Repository wizard.

There, all done! Wasn’t that easy?

I didn’t do show it here, but standard practice is to always Pull from the upstream repo (e.g. origin) before you try pushing code. If someone has pushed code to the master repo since your last pull (or clone), you’ll need to merge their changes from the master repo into your local repo first, then push your combined changes back to the upstream repo.

Why is Embedded Product Development Different?

Effective embedded product development involves coordinating a large number of tasks in complex relationships, managing risk and dealing with issues that arise due to uncertainty. Development teams are multi-disciplinary and can involve industrial and user experience design, mechanical design, electronics design, embedded, desktop, mobile and cloud software development, product verification and validation testing, and manufacturing process development, each with its own unique processes and workflows. Put simply, the effort needed to bring a complex high-tech networked embedded product to the world should not be underestimated.

Regulatory requirements must be met before a product can be placed for sale, including meeting electrical emission and compliance regulations, safety related requirements that may impose specific product requirements or following specific development processes, and environmental requirements affecting component selection and recycling of packaging and eventually the product itself. Managing regulatory requirements and demonstrating compliance is a critical part of the development process. 

Software development today commonly includes a wide variety of open-source software components, such as operating systems, device drivers, database systems, data encryption, and network communication stacks. Each software component is licensed by its creator, and imposing specific requirements on use. In addition, use of encryption technologies is usually subject to national security regulations. Managing this complex interrelationship of requirements requires careful attention to ensure the final software application will be free of undesired encumbrances and can be distributed legally.

Development of a high-tech product can be a complex undertaking, involving a complex relationship of simultaneous tasks. To be effective, an engineering project manager must not only have related technical experience, but must also provide a suitable balance between attention to detail and time to market, without allowing unacceptable risk or sacrificing quality.


Product Lifecycle Management (PLM) describes how a product is consciously managed from concept through design, into manufacturing and sales, and eventually to retirement. PLM includes aspects of Product Management and also Sustaining (aka Maintenance) Engineering to add features, eliminate defects, optimise process, increase quality, etc. PLM integrates people, data, processes and business systems, and provides trust-able and transparent design and manufacturing data. 

New Product Introduction (NPI) is that portion of PLM involved with the hand-over from engineering design to manufacturing and eventually introducing a product into the market for sale. NPI is a broad topic, and depending on the organization, industry and product, may include Design for Manufacturability, Pilot Production, and Validation Testing, as well as creation of marketing and sales materials, managing sales campaigns and events, developing eCommerce and service subscription processes, and whatever else can be involved in bring a product to market.