Testing

Continuous integration tests

There is a dedicated jenkins instance which runs tests on pull requests as well as the major branches.

In addition we’re running a Gitlab Continuous Integration service which provides us with a scalable, high-throughput CI setup. See the blog post on our Gitlab-CI installation <http://cosent.nl/en/blog/plone-gitlab-continuous-integration> for documentation on our setup.

Testing locally

The dev.cfg buildout includes a test runner. It will run most tests and can be executed with this command:

./bin/test

Robot tests

You can run only robot tests:

./bin/test -t 'robot'

To not get spammed with many robot test failures, abort on first failure:

./bin/test -t 'robot' -x

To not have a gazillion browser windows pop up, run tests in a framebuffer:

Xvfb :99 1>/dev/null 2>&1 & DISPLAY=:99 bin/test -t 'robot' -x

Or, if you prefer to see the ongoing tests progress in a single window:

Xephyr -screen 1280x1024 :1 & DISPLAY=:1 bin/test -t 'robot' -x

Robot tests can be skipped by using -t ‘!robot’ (note: single quotes, not double quotes):

./bin/test -t '!robot'

Faster robot testing with robot-server

You can avoid repeatedly running the test setup phases before running robot tests by using robot-server:

bin/robot-server ploneintranet.suite.testing.PLONEINTRANET_SUITE_ROBOT

This does the test setup, creates the demo content etc. You can now log in to the test instance: http://localhost:55001/plone using the credentials of users created by create_users in pi.suite.setuphandlers e.g. alice_lindstrom:secret.

To configure robot-server to serve on the external interface instead of localhost e.g. when testing against a container/virtual machine, you can export ZSERVER_HOST=10.233.13.2.

You can use the pybot script to run robot tests against the robot-server instance:

bin/pybot -t "*events*" src/ploneintranet/suite/tests/acceptance/workspace.robot

You can also override variables, e.g. to use Chrome instead of Firefox for testing:

bin/pybot --variable BROWSER:chrome src/ploneintranet/suite/tests/acceptance/workspace.robot

To run pybot tests against a remote server you can export ZOPE_HOST=10.233.13.2.

Running robot tests from a container host against a robot-server in a container

For developers who use a Linux container for Plone development, it is convenient to be able to run the tests on the host system (using the host browser and Xorg) while running the robot-server in its normal environment in the container.

You can bind mount the buildout folder in the container to a convenient folder on the host to make it easy to access.

If the path inside the container is different from the path on the host, you will need to adjust the absolute paths in bin/pybot. You will also need to call it with the host python. A wrapper script is handy for this (robot_remote.sh):

#!/bin/sh
sed  "s#/path/to/container/buildout/#/path/to/host/buildout/#" -i bin/pybot
python ./bin/pybot --variable BROWSER:chrome $@

After starting robot-server in the container, you can then run robot tests on the host:

./robot_remote.sh -t "*events*" src/ploneintranet/suite/tests/acceptance/workspace.robot

Troubleshooting jenkins

Login on jenkins:

ssh -p 1922 jenkins@ext1.syslab.com

Start robot-server on a non-default port to avoid port conflicts:

ZSERVER_PORT=55667 bin/robot-server ploneintranet.suite.testing.PLONEINTRANET_SUITE_ROBOT

On a second terminal, login on jenkins with X forwarding enabled:

ssh -p 1922 jenkins@ext1.syslab.com

Run the test you want to trace:

ZSERVER_PORT=55667 bin/pybot -t "Neil can tag a post" src/ploneintranet/suite/tests/acceptance/posting.robot

That should open up the Jenkins firefox on your local machine and play the session.

If you want to dig deeper, add the statement Debug into the offending robot test. In that case the pybot process above will drop you into a debug session, where you can continue the test manually by inserting commands like click link  link=Rain which then should step by step update your local firefox display with the test run on Jenkins.

Testing scenarios

Workspaces

The top-level workspace container with the id “workspaces” is created on install by the setup-handler of the workspace package.

The setup-handler of the suite creates four workspaces with the following settings:

  • “Open Market Committee”

    • External visibility: Private
    • Participation policy: Publishers
    • Admin: christian_stoney
    • Member: allan_neece (and others)
    • Non-Member: alice_lindstrom

    A document, a file and an image are created in sub-folder “Manage Information” with allan_neece as the owner, so that he will be abler to manipulate them in robot tests.

  • “Parliamentary papers guidance”

    • External visibility: Private
    • Participation policy: Producers
    • Admin: christian_stoney
    • Member: allan_neece (and others)
    • Non-Member: alice_lindstrom

  • “Shareholder information”

    • External visibility: Private
    • Participation policy: Consumers
    • Admin: christian_stoney
    • Member: allan_neece (and others)
    • Non-Member: alice_lindstrom

  • “Service announcements”

    • External visibility: Open
    • Participation policy: Consumers
    • Admin: christian_stoney
    • Member: allan_neece (and others)
    • Non-Member: alice_lindstrom

Fastest diff test runner

An special test runner in ploneintranet/fastest calculates the changeset between a to-be-merged branch and master, then applies a series of testing strategies to calculate whether it’s possible to run an optimized test suite that skips tests which are not needed for this changeset.

The strategies for this are configured in ploneintranet/fastest/config.py.

Each strategy has triggers, which are regular expressions to match filenames. If any filename in the change set matches a trigger, the matching test strategy will be applied.

The testing policy applies all strategies to the changeset and schedules two test runs: one with strategy.packages e.g. bin/test -s ploneintranet.workspace and one with strategy.tests e.g. bin/test -t workspace.robot. This is necessary because zope.testrunner would calculate the intersection rather than the union of -s package selectors and -t test selectors.

In case a filepath is in the changeset which does not match any of the optimization strategies, optimization will be enabled and the full default test suite gets run.

Additionally, it’s possible to specify wildcard strategies that will force a full test run, even if all changed file paths were matched by a test strategy.

Manual fastest run

You can run the fastest runner manually as follows:

bin/fastest Default run with full verbosity.

bin/fastest -n Dry run with full verbosity.

bin/fastest -q Full run in quiet mode.

bin/fastest -n --from efe14a393 -- to 97a308f1 Dryrun to inspect what would be the testing strategy between the two given commits.