openQA Documentation (2024)

openQA is free software released under theGPLv2 license. The source code anddocumentation are hosted in the os-autoinstorganization on GitHub.

Figure 1. openQA architecture

Glossary

Jobs

One of the most important features of openQA is that it can be used to testseveral combinations of actions and configurations. For every one of thosecombinations, the system creates a virtual machine, performs certain steps andreturns an overall result. Every one of those executions is called a 'job'.Every job is labeled with a numeric identifier and has several associated'settings' that will drive its behavior.

A job goes through several states:

Jobs in state 'done' have typically gone through a whole sequence of steps(called 'testmodules') each one with its own result. But in addition to thosepartial results, a finished job also provides an overall result from thefollowing list.

Sometimes, the reason of a failure is not an error in the tested operating systemitself, but an outdated test or a problem in the execution of the job for someexternal reason. In those situations, it makes sense to re-run a given job fromthe beginning once the problem is fixed or the tests have been updated.This is done by means of 'cloning'. Every job can be superseded by a clone whichis scheduled to run with exactly the same settings as the original job. If theoriginal job is still not in 'done' state, it’s cancelled immediately.From that point in time, the clone becomes the current version and the originaljob is considered outdated (and can be filtered in the listing) but itsinformation and results (if any) are kept for future reference.

Needles

One of the main mechanisms for openQA to know the state of the virtual machineis checking the presence of some elements in the machine’s 'screen'.This is performed using fuzzy image matching between the screen and the socalled 'needles'. A needle specifies both the elements to search for and alist of tags used to decide which needles should be used at any moment.

A needle consists of a full screenshot in PNG format and a json file withthe same name (e.g. foo.png and foo.json) containing the associated data, likewhich areas inside the full screenshot are relevant or the mentioned list oftags.

{ "area" : [ { "xpos" : INTEGER, "ypos" : INTEGER, "width" : INTEGER, "height" : INTEGER, "type" : ( "match" | "ocr" | "exclude" ), "match" : INTEGER, // 0-100. similarity percentage }, ... ], "tags" : [ STRING, ... ]}

Interactive mode

There are several points in time during the execution of a job at which openQAtries to match the screen with the available needles, reacting to the result ofthat check. If the job is running in interactive mode it will stop the executionat that point, freezing the virtual machine and waiting for user input beforeproceeding. At that moment, the user can modify the existing needles or cancreate a new one using as a starting point either the current screen of thevirtual machine or one of the existing needles. Once the needles are adjusted,the user can command the job to reload the list of needles and continue with theexecution.

The interactive mode is especially useful when creating needles for a newoperating system or when the look & feel have changed and several needles needto be adjusted accordingly.

Access management

Some actions in openQA require special privileges. openQA providesauthentication through openID. By default,openQA is configured to use the openSUSE openID provider, but it can veryeasily be configured to use any other valid provider. Every time a new user logsinto an instance, a new user profile is created. That profile onlycontains the openID identity and two flags used for access control:

Many of the operations in an openQA instance are not performed through the webinterface but using the REST-like API. The most obvious examples are theworkers and the scripts that fetch new versions of the operating system andschedule the corresponding tests. Those clients must be authorized by anoperator using anAPI key withan associated shared secret.

For that purpose, users with the operator flag have access in the web interfaceto a page that allows them to manage as many API keys as they may need. For everykey, a secret is automatically generated. The user can then configure theworkers or any other client application to use whatever pair of API key andsecret owned by him. Any client to the REST-like API using one of those API keyswill be considered to be acting on behalf of the associated user. So the API keynot only has to be correct and valid (not expired), it also has to belong to auser with operator rights.

For more insights about authentication, authorization and the technical detailsof the openQA security model, refer to thedetailedblog post about the subject by the openQA development team.

Job groups

A job can belong to a job group. Those job groups are displayed on the index pageand in the Job Groups menu on the navigation bar. From there the job group overviewpages can be accessed. Besides the test results the job group overview pages providea description about the job group and allow commenting.

Job groups have properties. These properties are mostly cleanup related. Theconfiguration can be done in the operators menu for job groups.

It is also possible to put job groups into categories. The nested groups will theninherit properties from the category. The categories are meant to combine job groupswith common builds so test results for the same build can be shown together onthe index page.

Cleanup

The following cleanup settings can be done on job-group-level:

NOTE Deletion of job results includes deletion of logs and will cause the job tobe completely removed from the database.

NOTE Jobs which do not belong to a job group are currently not affected bythe mentioned cleanup properties.

The problem

When testing an operating system, especially when doing continuous testing,there is always a certain combination of jobs, each one with its ownsettings, that needs to be run for every revision. Those combinations can bedifferent for different 'flavors' of the same revision, like running a differentset of jobs for each architecture or for the Full and the Lite versions. Thiscombinational problem can go one step further if openQA is being used fordifferent kinds of tests, like running some simple pre-integration testsfor some snapshots combined with more comprehensive post-integration tests forrelease candidates.

This section describes how an instance of openQA can be configured using theoptions in the admin area to automatically create all the required jobs for eachrevision of your operating system that needs to be tested. If you are startingfrom scratch, you should probably go through the following order:

Machines, mediums and test suites can all set various configuration variables.Job templates define how the test suites, mediums and machines should becombined in various ways to produce individual 'jobs'. All the variablesfrom the test suite, medium and machine for the 'job' are combined and madeavailable to the actual test code run by the 'job', along with variablesspecified as part of the job creation request. Certain variables also influenceopenQA’s and/or os-autoinst’s own behavior in terms of how it configures theenvironment for the job. Variables that influence os-autoinst’s behaviorare documented in the file doc/backend_vars.asciidoc in the os-autoinstrepository.

In openQA we can parametrize a test to describe for what product it willrun and for what kind of machines it will be executed. For example, atest like KDE can be run for any product that has KDE installed, andcan be tested in x86-64 and i586 machines. If we write this as atriples, we can create a list like this to characterize KDE tests:

(Product, Test Suite, Machine)(openSUSE-DVD-x86_64, KDE, 64bit)(openSUSE-DVD-x86_64, KDE, Laptop-64bit)(openSUSE-DVD-x86_64, KDE, USBBoot-64bit)(openSUSE-DVD-i586, KDE, 32bit)(openSUSE-DVD-i586, KDE, Laptop-32bit)(openSUSE-DVD-x86_64, KDE, USBBoot-32bit)(openSUSE-DVD-i586, KDE, 64bit)(openSUSE-DVD-i586, KDE, Laptop-64bit)(openSUSE-DVD-x86_64, KDE, USBBoot-64bit)

For every triplet, we need to configure a different instance ofos-autoinst with a different set of parameters.

Medium Types (products)

A medium type (product) in openQA is a simple description without any concretemeaning. It basically consists of a name and a set of variables thatdefine or characterize this product in os-autoinst.

Some example variables used by openSUSE are:

Test Suites

This is the form where we define the different tests that we created foropenQA. A test consists of a name, a priority and a set of variables that areused inside this particular test. The priority is used in the scheduler tochoose the next job. If multiple jobs are scheduled and their requirements forrunning them are fulfilled the ones with a lower value for the priority aretriggered. The id is the second sorting key: Of two jobs with equalrequirements and same priority the one with lower id is triggered first.

Some sample variables used by openSUSE are:

Some of the variables usually set in test suites that influence openQAand/or os-autoinst’s own behavior are:

Machines

You need to have at least one machine set up to be able to run anytests. Those machines represent virtual machine types that you want totest. To make tests actually happen, you have to have an 'openQAworker' connected that can fulfill those specifications.

Variable expansion

Any variable defined in Test Suite, Machine or Product table can refer to anothervariable using this syntax: %NAME%. When the test job is created, the stringwill be substituted with the value of the specified variable at that time.

For example this variable defined for Test Suite:

PUBLISH_HDD_1 = %DISTRI%-%VERSION%-%ARCH%-%DESKTOP%.qcow2

may be expanded to this job variable:

PUBLISH_HDD_1 = opensuse-13.1-i586-kde.qcow2

Variable precedence

It’s possible to define the same variable in multiple places that would all beused for a single job - for instance, you may have a variable defined in botha test suite and a product that appear in the same job template. The precedenceorder for variables is as follows (from lowest to highest):

That is, variable values set as part of the API request that triggers the jobs will'win' over values set at any of the other locations.

If you need to override this precedence - for example, you want the value set inone particular test suite to take precedence over a setting of the same value fromthe API request - you can add a leading + to the variable name. For instance, ifyou set +VARIABLE = foo in a test suite, and passed VARIABLE=bar in the APIrequest, the test suite setting would 'win' and the value would be foo.

If the same variable is set with a + prefix in multiple places, the same precedenceorder described above will apply to those settings.

Getting tests

First you need to get actual tests. You can get openSUSE tests and needles (theexpected results) fromGitHub. It belongsinto the /var/lib/openqa/tests/opensuse directory. To make it easier, you can justrun

/usr/share/openqa/script/fetchneedles

Which will download the tests to the correct location and will set the correctrights as well.

Fedora’s tests are also in git. Touse them, you may do:

cd /var/lib/openqa/share/testsmkdir fedoracd fedoragit clone https://pagure.io/fedora-qa/os-autoinst-distri-fedora.git./templates --cleancd ..chown -R geekotest fedora/

Getting openQA configuration

To get everything configured to actually run the tests, there are plenty ofoptions to set in the admin interface. If you plan to test openSUSE Factory, usingtests mentioned in the previous section, the easiest way to get started is thefollowing command:

/var/lib/openqa/share/tests/opensuse/products/opensuse/templates [--apikey API_KEY] [--apisecret API_SECRET]

This will load some default settings that were used at some point of time inopenSUSE production openQA. Therefore those should work reasonably well withopenSUSE tests and needles. This script uses /usr/share/openqa/script/load_templates,consider reading its help page (--help) for documentation on possible extra arguments.

For Fedora, similarly, you can call:

/var/lib/openqa/share/tests/fedora/templates [--apikey API_KEY] [--apisecret API_SECRET]

Some Fedora tests require special hard disk images to be present in/var/lib/openqa/share/factory/hdd/fixed. The createhdds.py script in thecreatehddsrepository can be used to create these. See the documentation in that repofor more information.

Adding a new ISO to test

To start testing a new ISO put it in /var/lib/openqa/share/factory/iso and callthe following commands:

# Run the first test/usr/share/openqa/script/client isos post \ ISO=openSUSE-Factory-NET-x86_64-Build0053-Media.iso \ DISTRI=opensuse \ VERSION=Factory \ FLAVOR=NET \ ARCH=x86_64 \ BUILD=0053

If your openQA is not running on port 80 on 'localhost', you can add option--host=http://otherhost:9526 to specify a different port or host.

For Fedora, a sample run might be:

# Run the first test/usr/share/openqa/script/client isos post \ ISO=Fedora-Everything-boot-x86_64-Rawhide-20160308.n.0.iso \ DISTRI=fedora \ VERSION=Rawhide \ FLAVOR=Everything-boot-iso \ ARCH=x86_64 \ BUILD=Rawhide-20160308.n.0

More details on triggering tests can also be found in theUsers Guide.

Take a look at Documented Pitfalls.

Installation from distribution packages

The easiest way to install openQA is from distribution packages.

You can install the packages using these commands.

# openSUSE Leap 42.3+zypper in openQA# Fedora 23+dnf install openqa openqa-httpd

Installation from development versions of packages

You can find the development version of openQA in OBS in theopenQA:devel repository.

To add the development repository to your system, you can use these commands.

# openSUSE Tumbleweedzypper ar -f obs://devel:openQA/openSUSE_Tumbleweed devel-openQA# openSUSE Leap 42.3zypper ar -f obs://devel:openQA/openSUSE_Leap_42.3 devel-openQAzypper ar -f obs://devel:openQA:Leap:42.3/openSUSE_Leap_42.3 devel-openQA-perl-modules# openSUSE Leap 42.2zypper ar -f obs://devel:openQA/openSUSE_Leap_42.2 devel-openQAzypper ar -f obs://devel:openQA:Leap:42.2/openSUSE_Leap_42.2 devel-openQA-perl-modules

Then you can install them using this command.

# all openSUSEzypper in devel-openQA:openQA

Apache proxy

It is required to run openQA behind an http proxy (apache, nginx, etc..). See theopenqa.conf.template config file in /etc/apache2/vhosts.d (openSUSE) or/etc/httpd/conf.d (Fedora). To make everything work correctly on openSUSE, youneed to enable the 'headers', 'proxy', 'proxy_http' and 'proxy_wstunnel' modulesusing the command 'a2enmod'. This is not necessary on Fedora.

# openSUSE Only# You can check what modules are enabled by using 'a2enmod -l'a2enmod headersa2enmod proxya2enmod proxy_httpa2enmod proxy_wstunnel

For a basic setup, you can copy openqa.conf.template to openqa.conf and modify the ServerName if requiredsetting. This will direct all HTTP traffic to openQA.

cp /etc/apache2/vhosts.d/openqa.conf.template /etc/apache2/vhosts.d/openqa.conf

TLS/SSL

By default openQA expects to be run with HTTPS. The openqa-ssl.conf.templateApache config file is available as a base for creating the Apache config; youcan copy it to openqa-ssl.conf and uncomment any lines you like, thenensure a key and certificate are installed to the appropriate location(depending on distribution and whether you uncommented the lines for key andcert location in the config file). On openSUSE, you should also add SSL to theAPACHE_SERVER_FLAGS so it looks like this in /etc/sysconfig/apache2:

APACHE_SERVER_FLAGS="SSL"

If you don’t have a TLS/SSL certificate for your host you must turn HTTPS off.You can do that in /etc/openqa/openqa.ini:

[openid]httpsonly = 0

Database

Since version 4.5.1512500474.437cc1c7 of openQA, PostgreSQL is used as thedatabase.

To configure access to the database in openQA, edit /etc/openqa/database.iniand change the settings in the [production] section.

The dsn value format technically depends on the database type and isdocumented for PostgreSQL atDBD::Pg

[production]dsn = dbi:Pg:dbname=openqa

[production]dsn = dbi:Pg:dbname=openqa;host=db.example.orguser = openqapassword = somepassword

User authentication

OpenQA supports three different authentication methods - OpenID (default), iChainand Fake. See auth section in /etc/openqa/openqa.ini.

[auth]# method name is case sensitive!method = OpenID|iChain|Fake

Independently of method used, the first user that logs in (if there is no admin yet)will automatically get administrator rights!

OpenID

By default openQA uses OpenID with opensuse.org as OpenID provider.OpenID method has its own openid section in /etc/openqa/openqa.ini:

[openid]## base url for openid providerprovider = https://www.opensuse.org/openid/user/## enforce redirect back to httpshttpsonly = 1

OpenQA supports only OpenID version up to 2.0. Newer OpenID-Connect and OAuth isnot supported currently.

iChain

Use only if you use iChain (NetIQ Access Manager) proxy on your hosting server.

Fake

For development purposes only! Fake authentication bypass any authentication andautomatically allow any login requests as 'Demo user' with administrator privilegesand without password. To ease worker testing, API key and secret is created (or updated)with validity of one day during login.You can then use following as /etc/openqa/client.conf:

[localhost]key = 1234567890ABCDEFsecret = 1234567890ABCDEF

If you switch authentication method from Fake to any other, review your API keys!You may be vulnerable for up to a day until Fake API key expires.

From this point on, you can refer to the Getting Started guide tofetch the tests cases and possibly take a look at Test Developer Guide

Setting up git support

Editing needles from web can optionally commit new or changed needlesautomatically to git. To do so, you need to enable git support by setting

[global]scm = git

in /etc/openqa/openqa.ini. Once you do so and restart the web interface, openQA willautomatically commit new needles to the git repository.

You may want to add some description to automatic commits comingfrom the web UI.You can do so by setting your configuration in the repository(/var/lib/os-autoinst/needles/.git/config) to some reasonable defaults such as:

[user]email = whatever@example.comname = openQA web UI

To enable automatic pushing of the repo as well, you need to add the followingto your openqa.ini:

[scm git]do_push = yes

Depending on your setup, you might need to generate and propagatessh keys for user 'geekotest' to be able to push.

Referer settings to auto-mark important jobs

Automatic cleanup of old results (see GRU jobs) can sometimes render importanttests useless. For example bug report with link to openQA job which no longerexists. Job can be manually marked as important to prevent quick cleanup orreferer can be set so when job is accessed from particular web page (forexample bugzilla), this job is automatically labeled as linked and treated asimportant.

List of recognized referers is space separated list configured in/etc/openqa/openqa.ini:

[global]recognized_referers = bugzilla.suse.com bugzilla.opensuse.org

Worker settings

Default behavior for all workers is to use the 'Qemu' backend and connect to'http://localhost'. If you want to change some of those options, you can do soin /etc/openqa/workers.ini. For example to point the workers to the FQDN ofyour host (needed if test cases need to access files of the host) use thefollowing setting:

[global]HOST = http://openqa.example.com

Once you got workers running they should show up in the admin section of openQA inthe workers section as 'idle'. When you get so far, you have your own instanceof openQA up and running and all that is left is to set up some tests.

Configuring remote workers

There are some additional requirements to get remote worker running. First is toensure shared storage between openQA WebUI and workers.Directory /var/lib/openqa/share contains all required data and should beshared with read-write access across all nodes present in openQA cluster.This step is intentionally left on system administrator to choose proper sharedstorage for her specific needs.

Example of NFS configuration:NFS server is where openQA WebUI is running. Content of /etc/exports

/var/lib/openqa/share *(fsid=0,rw,no_root_squash,sync,no_subtree_check)

NFS clients are where openQA workers are running. Run following command:

mount -t nfs openQA-webUI-host:/var/lib/openqa/share /var/lib/openqa/share

Configuring worker to use more than one openQA server

When there are multiple openQA web interfaces (openQA instances) available a workercan be configured to register and accept jobs from all of them.

Requirements:

In the /etc/openqa/workers.ini enter space-separated instance hosts and optionallyconfigure where the shared storage is mounted. Example:

[global]HOSTS = openqa.opensuse.org openqa.fedora.fedoraproject.org[openqa.opensuse.org]SHARE_DIRECTORY = /var/lib/openqa/opensuse[openqa.fedoraproject.org]SHARE_DIRECTORY = /var/lib/openqa/fedora

Configuring SHARE_DIRECTORY is not a hard requirement. Worker will try followingdirectories prior registering with openQA instance:

Once worker registers to openQA instance it checks for available job and startsaccepting websockets commands. Worker accepts jobs as they will come in, thereis no priority, or other ordering, support at the moment.It is possible to mix local openQA instance with remote instances or use onlyremote instances.

Asset Caching

If your network is slow or you experience long time to load needles youmight want to consider to enable caching in your remote workers. To enable caching,/var/lib/openqa/cache must be created, and right permissions given to the'geekotest' user. If you install openQA through the repositories, said directorywill be created for you.

In the /etc/openqa/workers.ini

[global]HOST=http://webuiCACHEDIRECTORY = $cache_locationCACHELIMIT = 50 # This is currently noop. Cache supports limiting, but is not enabled.[http://webui]TESTPOOLSERVER = rsync://yourlocation/tests

This will allow the workers to download the assets from the webUI and use themlocally. If TESTPOOLSERVER is set tests and needles will also be cached by theworker.

List of events tracked by the auditing plugin:

Some of these events are very common and may clutter audit database. For this reason job_grab and job_doneevents are blacklisted by default.

• db contains the database lockfile

• images is where the server stores test screenshots and thumbnails

• share contains shared directories for remote workers, can be owned by root

• share/factory contains test assets and temp directory, can be owned by root but sysadmin must create subdirs

• share/factory/iso and share/factory/iso/fixed contain ISOs for tests

• share/factory/hdd and share/factory/hdd/fixed contain hard disk images for tests

• share/factory/repo and share/factory/repo/fixed contain repositories for tests

• share/factory/other and share/factory/other/fixed contain miscellaneous test assets (e.g. kernels and initrds)

• share/factory/tmp is used as a temporary directory (openQA will create it if it owns share/factory)

• share/tests contains the tests themselves

• testresults is where the server stores test logs and test-generated assets

• script (symlink to /usr/share/openqa/script/)

• tests (symlink to share/tests)

• factory (symlink to share/factory)

• /var/lib/openqa/pool/1

• /var/lib/openqa/pool/2

• …​. - add more if you have more CPUs/disks

Tests fail quickly

Check the log files in /var/lib/openqa/testresults

KVM doesn’t work

openid login times out

www.opensuse.org’s openid provider may have trouble with IPv6. openQA shows a message like this:

no_identity_server: Could not determine ID provider from URL.

To avoid that switch off IPv6 or add a special route that prevents the systemfrom trying to use IPv6 with www.opensuse.org:

ip -6 r a to unreachable 2620:113:8044:66:130:57:66:6/128

/tests/overview - Customizable test overview page

The overview page is configurable by the filter box. Also, some additionalquery parameters can be provided which can be considered advanced orexperimental. For example specifying no build will resolve the latest buildwhich matches the other parameters specified. Specifying no group will showall jobs from all matching job groups. Also specifying multiple groups works,see the following example.

Specifying multiple groups with no build will yield the latest build of thefirst group. This can be useful to have a static URL for bookmarking.

Description of test suites

Test suites can be described using API commands or the admin table for any operator using the web UI.

If a description is defined, the name of the test suite on the tests overview page shows up as a link. Clicking the link will show the description in a popup. The same syntax as for comments can be used, that is Markdown with custom extensions such as shortened links to ticket systems.

Review badges

Based on comments in the individual job results for each build a certificateicon is shown on the group overview page as well as the index page to indicatethat every failure has been reviewed, e.g. a bug reference or a test issuereason is stated:

Show bug or label icon on overview if labeled gh#550

For bugreferences write <bugtracker_shortname>#<bug_nr> in a comment, e.g. "bsc#1234", for generic labels use label:<keyword> where <keyword> can be any valid character up to the next whitespace, e.g. "false_positive". The keywords are not defined within openQA itself. A valid list of keywords should be decided upon within each project or environment of one openQA instance.

Also github pull requests and issues can be linked using the generic format`<marker>[#<project/repo>]#<id>`, e.g. gh#os-autoinst/openQA#1234, see gh#973

All issue references are stored within the internal database of openQA. The status can be updated using the /bugs API route for example using external tools.

Build tagging

tag:<build_id>:<type>[:<description>], e.g. tag:1234:important:Beta1.

Filtering test results and builds

At the top of the test results overview page is a form which allows filtering tests by result,architecture and TODO-status.

There is also a similar form at the bottom of the index page which allows filtering builds bygroup and customizing the limits.

Highlighting job dependencies in 'All tests' table

When hovering over the branch icon after the test name children of the job willbe highlighted blue and parents red. So far this only works for jobs displayed onthe same page of the table.

Basics are described in theGetting Started guide.

Triggering tests

Tests can be triggered over multiple ways, using clone_job.pl, jobs post,isos post as well as retriggering existing jobs or whole media over the webUI.

openqa-client isos post ISO=my_iso.iso DISTRI=my_distri FLAVOR=sweet \ ARCH=my_arch VERSION=42 BUILD=1234 \ _DEPRIORITIZEBUILD=1 _DEPRIORITIZE_LIMIT=120 \

For test developers it is recommended to continue with theTest Developer Guide.

• post_fail_hook is called to upload log files or determine the state ofthe machine

• pre_run_hook is called before the run function - mainly useful for a wholegroup of tests

• post_run_hook is run after successful run function - mainly useful for a wholegroup of tests

Test Case Examples

sub run() { # change to root become_root; # output zypper repos to the serial script_run "zypper lr -d > /dev/$serialdev"; # install xdelta and check that the installation was successful assert_script_run 'zypper --gpg-auto-import-keys -n in xdelta'; # additionally write a custom string to serial port for later checking script_run "echo 'xdelta_installed' > /dev/$serialdev"; # detecting whether 'xdelta_installed' appears in the serial within 200 seconds die "we could not see expected output" unless wait_serial "xdelta_installed", 200; # capture a screenshot and compare with needle 'test-zypper_in' assert_screen 'test-zypper_in';}

sub run() { # make sure kate was installed # if not ensure_installed will try to install it ensure_installed 'kate'; # start kate x11_start_program 'kate'; # check that kate execution succeeded assert_screen 'kate-welcome_window'; # close kate's welcome window and wait for the window to disappear before # continuing wait_screen_change { send_key 'alt-c' }; # typing a string in the editor window of kate type_string "If you can see this text kate is working.\n"; # check the result assert_screen 'kate-text_shown'; # quit kate send_key 'ctrl-q'; # make sure kate was closed assert_screen 'desktop';}

Modifying setting of an existing test

There is no interface to modify existing tests but the clone_job.pl scriptcan be used to create a new job that adds, removes or changessettings. This script is located at /usr/share/openqa/script/.

/usr/share/openqa/script/clone_job.pl --from localhost --host localhost 42 FOO=bar BAZ=

If you do not want a cloned job to start up in the same job group as the jobyou cloned from, e.g. to not pollute build results you the job group can beoverwritten, too, using the special variable _GROUP. Add the quoted groupname, e.g.:

clone_job.pl --from localhost 42 _GROUP="openSUSE Tumbleweed"

The special group value 0 means that the group connection will be separatedand the job will not appear as a job in any job group, e.g.:

clone_job.pl --from localhost 42 _GROUP=0

Using snapshots to speed up development of tests

For lower turn-around times during test development based on virtual machinesthe QEMU backend provides a feature that allows a job to start from asnapshot which can help in this situation.

Depending on the use case, there are two options to help:

In both modes there is no need to modify tests (i.e. adding milestone testflag as the behaviour is implied). In the later mode every test module isalso considered fatal. This means the job is aborted after the first failedtest module.

clone_job.pl --from https://openqa.opensuse.org --host localhost 24 MAKETESTSNAPSHOTS=1

clone_job.pl --from https://openqa.opensuse.org --host localhost 24 SKIPTO=consoletest-yast2_i

clone_job.pl --from https://openqa.opensuse.org --host localhost 24 TESTDEBUG=1

clone_job.pl --from https://openqa.opensuse.org --host localhost 24 TESTDEBUG=1 SKIPTO=consoletest-yast2_i

Assigning jobs to workers

By default, any worker can get any job with the matching architecture.

This behavior can be changed by setting job variable WORKER_CLASS. Jobswith this variable set (typically via machines or test suites configuration) areassigned only to workers, which have the same variable in the configuration file.

For example, the following configuration ensures, that jobs with WORKER_CLASS=desktopcan be assigned only to worker instances 1 and 2.

[1]WORKER_CLASS = desktop[2]WORKER_CLASS = desktop[3]# WORKER_CLASS is not set

Writing multi-machine tests

Scenarios requiring more than one system under test (SUT), like High Availability testing, are covered as multi-machine tests (MM tests) in this section.

OpenQA approaches multi-machine testing by assigning dependencies between individual jobs. This means the following:

In short, writing multi-machine tests adds a few more layers of complexity:

A <- B <- CDefine test suite A,then define B with variable START_AFTER_TEST=A and then define C with START_AFTER_TEST=B-or-Define test suite A, Band then define C with START_AFTER_TEST=A,BIn this case however the start order of A and B is not specified.But C will start only after A, B are successfully done.

A^BDefine test suite Aand then define B with variable PARALLEL_WITH=A.A in this case is parent test suite to B and must be running throughout B run.

A B C\ | / ^ DDefine test suites A,B,Cand then define D with PARALLEL_WITH=A,B,C.A,B,C run in parallel and are parent test suites for D and all must run until D finish.

 A ^ /|\ B C DDefine test suite Aand then define B,C,D with PARALLEL_WITH=AA is parent test suite for B, C, D (all can run in parallel).Children B, C, D can run and finish anytime, but A must run until all B, C, D finishes.

Test synchronization and locking API

OpenQA provides locking server through lock API. To use lock API import lockapi package (use lockapi;) in your test file.Lock API provides three functions: mutex_create, mutex_lock, mutex_unlock. Each of these functions take one parameter: name of the lock.Locks are associated with caller`s job - locks can’t be unlocked by different job then the one who locked the lock.

mutex_lock tries to lock the mutex lock for caller`s job. If lock is unavailable or locked by someone else, mutex_lock call blocks.

mutex_unlock tries to unlock the mutex lock. If lock is locked by different job, mutex_unlock call blocks. When lock become available or if lock does not exist, callreturns without doing anything.

mutex_create create new mutex lock. When lock is created by mutex_create, lock is automatically unlocked. When mutex lock already exists call returns without doing anything.

Locks are addressed by their name. This name is valid in test group defined by their dependencies. If there are more groups running at thesame time and the same lock name is used, these locks are independent of each other.

The mmapi package provides wait_for_children, which the parent can use to wait for the children to complete.

use base "basetest";use strict;use testapi;use lockapi;use mmapi;sub run { assert_screen 'bootloader'; assert_screen 'login', 300; # services start automatically # unlock by creating the lock mutex_create('services_ready'); # wait until all children finish wait_for_children;}

use base "basetest";use strict;use testapi;use lockapi;sub run { assert_screen 'bootloader'; assert_screen 'login', 300; # this blocks until lock is created then locks and immediately unlocks mutex_lock('services_ready'); mutex_unlock('services_ready'); # login to continue type_string("root\n"); sleep 1; type_string("secret\n");}

Sometimes it is useful to let a parent wait for certain action on a child, for example to verifyserver state after completed request. In this scenario the child createsa mutex and the parent unlocks it.

The child can however die at any time. To prevent parent deadlock in this situation,parent has to pass child ID as a second parameter to mutex_lock(). If a child jobwith given ID already finished, mutex_lock() calls die.

use base "basetest";use strict;use testapi;use lockapi;use mmapi;sub run { my $children = get_children(); # let's suppose there is only one child my $child_id = (keys %$children)[0]; # this blocks until lock is available and then does nothing mutex_unlock('child_reached_given_point', $child_id); # continue with the test}

Getting information about parents and children

use base "basetest";use strict;use testapi;use mmapi;sub run { # returns a hash ref containing (id => state) for all children my $children = get_children(); for my $job_id (keys %$children) { print "$job_id is cancelled\n" if $children->{$job_id} eq 'cancelled'; } # returns an array with parent ids, all parents are in running state (see Job dependencies above) my $parents = get_parents(); # let's suppose there is only one parent my $parent_id = $parents->[0]; # any job id can be queried for details with get_job_info() # it returns a hash ref containing these keys: # name priority state result worker_id # retry_avbl t_started t_finished test # group_id group settings my $parent_info = get_job_info($parent_id); # it is possible to query variables set by openqa frontend, # this does not work for variables set by backend or by the job at runtime my $parent_name = $parent_info->{settings}->{NAME} my $parent_desktop = $parent_info->{settings}->{DESKTOP} # !!! this does not work, VNC is set by backend !!! # my $parent_vnc = $parent_info->{settings}->{VNC}}

Support Server based tests

The idea is to have a dedicated "helper server" to allow advanced network based testing.

Support server takes advantage of the basic parallel setup as described in the previous section, with the support server being the parent test 'A' and the test needing it being the child test 'B'. This ensures that the test 'B' always have the support server available.

/usr/share/openqa/script/client jobs post DISTRI=opensuse VERSION=13.2 \ ISO=openSUSE-13.2-DVD-x86_64.iso ARCH=x86_64 FLAVOR=Server-DVD \ TEST=supportserver_generator MACHINE=64bit DESKTOP=textmode INSTALLONLY=1 \ AUTOYAST=supportserver/autoyast_supportserver.xml SUPPORT_SERVER_GENERATOR=1 \ PUBLISH_HDD_1=supportserver.qcow2

HDD_1=supportserver.qcow2SUPPORT_SERVER=1SUPPORT_SERVER_ROLES=pxe,qemuproxyWORKER_CLASS=server,qemu_autoyast_tap_64

/usr/share/openqa/script/client isos post DISTRI=opensuse VERSION=13.2 \ ISO=openSUSE-13.2-DVD-x86_64.iso ARCH=x86_64 FLAVOR=Server-DVD

HDD_1=supportserver.qcow2SUPPORT_SERVER=1SUPPORT_SERVER_ROLES=dhcp,tftpWORKER_CLASS=server,qemu_autoyast_tap_64

use strict;use base 'basetest';use testapi;sub run { my $script="set -e -x\n"; $script.="echo test >test.txt\n"; $script.="time tftp ".$server_ip." -c put test.txt test2.txt\n"; $script.="time tftp ".$server_ip." -c get test2.txt\n"; $script.="diff -u test.txt test2.txt\n"; script_output($script);}

HDD_1=supportserver.qcow2SUPPORT_SERVER=1SUPPORT_SERVER_ROLES=pxe,qemuproxyWORKER_CLASS=server,qemu_autoyast_tap_64

AUTOYAST=autoyast_opensuse/opensuse_autoyast_tftp.xmlAUTOYAST_VERIFY=autoyast_opensuse/opensuse_autoyast_tftp.shDESKTOP=textmodeINSTALLONLY=1PARALLEL_WITH=supportserver-opensuse-tftp

set -e -xecho test >test.txttime tftp #SERVER_URL# -c put test.txt test2.txttime tftp #SERVER_URL# -c get test2.txtdiff -u test.txt test2.txt && echo "AUTOYAST OK"

Using text consoles and the serial terminal

Typically the OS you are testing will boot into a graphical shell e.g. TheGnome desktop environment. This is fine if you wish to test a program with aGUI, but lets say you want to run some shell scripts then it is not soconvenient.

use 5.018;use warnings;use base 'opensusebasetest';use testapi;use utils;sub run { wait_boot; # Utility function defined by the SUSE distribution select_console 'root-console';}1;

assert_script_run('cd /proc');my $cpuinfo = script_output('cat cpuinfo');if($cpuinfo =~ m/avx2/) { # Do something which needs avx2}else { # Do some workaround}

The script_run and script_output commands are high level commands whichuse type_string and wait_serial underneath. Sometimes you may wish to uselower level commands which give you more control, but be warned that it mayalso make your code less portable.

select_console('root-virtio-terminal'); # Selects a virtio based serial terminal

• If a new needle is created based on a failed test, the new needlewill not be listed in old tests.

• If an existing needle is updated with a new image or differentareas, the old test will display the new needle which might beconfusing

• If a needle is deleted, old tests may display an error when viewingthem in the web UI.

• If you come across messages displaying ERROR: 403 - Forbidden, makesure that the correct API key is present in client.conf file.

• If you are using a hostname other than localhost, pass --host foo to the script.

• If you are using fake authentication method, and the message says also "api key expired"you can simply logout and log in again in the webUI and the expiration will be automaticallyupdated

• Export data from the SQlite db:

• Import data to the postgreSQL

1. wicked - creates tap devices

2. openvswitch - creates the bridge br0, adds tap devices to it

3. wicked handles br0 as hotplugged device, assignd the IP 10.0.2.2 to it, updates SuSEFirewall

4. os-autoinst-openvswitch - installs openflow rules, handles vlan assignment

1. openvswitch

2. wicked - creates the bridge br0 and tap devices, add tap devices to the bridge,

3. SuSEFirewall

4. os-autoinst-openvswitch - installs openflow rules, handles vlan assignment

• packets from tapX to br0 create additional rules in table=1

• packets from br0 to tapX increase packet counts in table=1

• empty output indicates a problem with os-autoinst-openvswitch service

• zero packet count or missing rules in table=1 indicate problem with tap devices

Basic, single machine tests

To start with a basic configuration like qemu user mode networking,create a machine with the following settings:

Start switch and user mode networking:

systemctl start openqa-vde_switchsystemctl start openqa-slirpvde

With this setting all jobs on the same host would be in the samenetwork share the same SLIRP instance though.

Multi machine tests

Create a machine like above but don’t set NICVLAN. openQA willdynamically allocate a VLAN number for all jobs that havedependencies between each other. By default this VLAN is private andhas no internet access. To enable user mode networking setVDE_USE_SLIRP=1 on one of the machines. The worker running the jobon such a machine will start slirpvde and put it in the correct VLANthen.

As mentioned, the central point of development is theos-autoinst organization on GitHub where severalrepositories can be found:

• openQA containing documentation,server, worker and other support scripts.

• os-autoinst with the standalonetest tool.

• os-autoinst-distri-opensusecontaining the tests used in http://openqa.opensuse.org

• os-autoinst-needles-opensusewith the needles associated to the tests in the former repository.

• os-autoinst-distri-examplewith an almost empty set of tests meant to be used to start writing tests (andcreating the corresponding needles) from scratch for a new operating system.

Rules for commits

./script/tidy

to ensure your Perl code changes are consistent with the style rules.

make test

for unit and integration tests.

To execute a single test, one can use prove. You must set TEST_PG so the databasecan be found. If you set a custom base directory, be sure to unset it when running tests.Example:

TEST_PG='DBI:Pg:dbname=openqa_test;host=/dev/shm/tpg' OPENQA_BASEDIR= prove -v t/14-grutasks.t

To speed up the test initialization, start PostgreSQL using t/test_postgresqlinstead of using the system service. Eg.

t/test_postgresql /dev/shm/tpg

If this is too much hassle for you feel free to provide incomplete pullrequests for consideration or create an issue with a code change proposal.

• To start the webserver for development, use the scripts/openqa daemon.

• openQA will pull the required asssets on the first run.

• openQA uses SASS, so Ruby development files are required. Under openSUSE,installing the packages devel_C_C++ and ruby-devel should be sufficient.openQA will install the required files automatically under .gem. Add.gem/ruby/2.4.0/bin to the PATH variable to let it find the sass/scssbinaries. I also had to create symlinks of those binaries without .ruby2.4suffix so openQA could find them.

• It is also useful to start openQA with morbo which allows applying changeswithout restarting the server:morbo -m development -w assets -w lib -w templates -l http://localhost:9526 script/openqa daemon

When is it required to update the database schema?

After modifying files in lib/OpenQA/Schema/Result. However, not all changesrequire to update the schema. Adding just another method or altering/addingfunctions like has_many doesn’t require an update. However, adding newcolumns, modifying or removing existing ones requires to follow the stepsmentioned above.

How to update the database schema

The above steps are only for preparing the required SQL statements, but do notactually alter the database. Before doing so, it is recommended to backup yourdatabase to be able to downgrade again if something goes wrong or you just needto continue working on another branch. To do so, the following command can beused to create a copy:

createdb -O ownername -T originaldb newdb

To actually create or update the database (after creating a backup as described),you should run either ./script/initdb --init_database or./script/upgradedb --upgrade_database. This is also required when the changesare installed in a production server.

How to add fixtures to the database

Note: This section is not about the fixtures for the testsuite. Those are locatedunder t/fixtures.

Note: This section might not be relevant anymore. At least there are currentlynone of the mentioned directories with files containing SQL statements present.

Fixtures (initial data stored in tables at installation time) are storedin files into the dbicdh/_common/deploy/_any/<version> anddbicdh/_common/upgrade/<prev_version>-<next_version> directories.

You can create as many files as you want in each directory. These files containSQL statements that will be executed when initializing or upgrading a database.Note that those files (and directories) have to be created manually.

Executed SQL statements can be traced by setting the DBIC_TRACE environmentvariable.

export DBIC_TRACE=1

1. Install PosgreSQL - under openSUSE the following package are required:postgresql-server postgresql-init

2. Start the server: systemctl start postgresql

3. The following steps need to be done by the user postgres: su - postgres

4. Create user: createuser your_username where your_username must be the sameas the UNIX user you start your local openQA instance with.

5. Create database: createdb -O your_username openqa

6. The next steps must be done by the user you start your local openQA instance with.

7. Import dump: pg_restore -c -d openqa path/to/dump

8. Configure openQA to use PostgreSQL as described in the section Database of the installation guide.User name and password are not required.