# Disable a test for older distros
enabled: true
adjust:
    enabled: false
    when: distro < fedora-33
    because: the feature was added in Fedora 33

# Adjust the required package name
require: procps-ng
adjust:
  - require: procps
    when: distro == centos-6

# Extend the environment variables, use multiple rules
adjust:
  - environment+:
        SH: bash
    when: component == bash
    continue: true
  - when: distro < centos-6
    enabled: false

# Install the fresh pytest from pip on older distros
adjust:
    prepare+:
     - how: shell
       name: fresh-pytest
       order: 90
       script: 'python3 -m pip install -U pytest'
    when: distro < rhel-8

# Mark as disabled
enabled: false

# Disable for older distros
enabled: true
adjust:
    enabled: false
    when: distro < fedora-33
    because: the feature was added in Fedora 33

# List only enabled tests
tmt tests ls --filter enabled:true

# Single tag
tag: security

# Multiple tags
tag: [security, fast]

# Multiple tags on separate lines
tag:
  - security
  - fast

# Basic tier one definition.
tier: "1"

# Integer values are converted to strings.
tier: 1

# Any custom string can be used as well.
tier: custom

# Enable a single check, AVC denial detection.
check: avc

# Enable multiple checks, by listing their names. A list of names
# is acceptable as well as a single name.
check:
  - avc
  - dmesg

# Enable multiple checks, one of them would be disabled temporarily.
check:
  - avc
  - kernel-panic
  - name: test-inspector
    enable: false

# Three minutes
duration: 3m

# Two hours
duration: 2h

# One day
duration: 1d

# Combination & repetition of time suffixes (total 4h 2m 3s)
duration: 1h 3h 2m 3

# Use context adjust to extend duration for given arch
duration: 5m
adjust:
    duration+: 15m
    when: arch == aarch64 // arch是一个context，可以在plan中定义，也可以通过命令行传入

environment:
    PACKAGE: python37
    PYTHON: python3.7

As a tester I want to include tests using different test execution framework in a single plan.

目前支持shell和beakerlib

# Test written in shell
framework: shell

# A beakerlib test
framework: beakerlib

Directory to be entered before executing the test

path: /protocols/https

Packages or libraries recommended for the test execution

# Single package
recommend: mariadb
# Multiple packages
recommend: [mariadb, mysql]
# Multiple packages on separate lines
recommend:
  - mariadb
  - mysql

# Require a single package
require: make

# Multiple packages
require: [gcc, make]

# Multiple packages on separate lines
require:
  - gcc
  - make

# Library from the default upstream location
require: library(openssl/certgen)

# Library from a custom remote git repository
require:
  - type: library
    url: https://github.com/beakerlib/openssl
    name: /certgen

# Library from the local filesystem
require:
  - type: library
    path: /tmp/library/openssl
    name: /certgen

# Use a custom git ref and nick for the library
require:
  - type: library
    url: https://github.com/redhat-qe-security/certgen
    ref: devel
    nick: openssl
    name: /certgen

# Require local files needed for the library or test
require:
  - type: file
    pattern:
      - /include
      - /Library/common/helper.sh
      - /files/photos/IMG.*

# Require whole fmf tree
require:
  - type: file
    pattern: /

Specify how test result should be interpreted

respect
test result is respected (fails when test failed) - default value

xfail
expected fail (pass when test fails, fail when test passes)

pass, info, warn, error, fail
ignore the actual test result and always report provided value instead

custom
test needs to create its own results.yaml or results.json file in the ${TMT_TEST_DATA} directory. 
The format of the file, notes and detailed examples are documented at Results Format.

# Plain swapping fail to pass and pass to fail result
result: xfail

# Look for $TMT_TEST_DATA/results.yaml (or results.json) with custom results
result: custom

Shell command which executes the test

# Run a script
test: ./test.sh

# Run a script with parameter
test: ./test.sh --depth 1000

# Execute selected tests using pytest
test: pytest -k performance

# Run test using a Makefile target
test: make run

# Define a manual test
test: manual.md
manual: true

As a tester I want to easily execute selected tests or selected test steps in given environment.

# Enabled a minimal smoke test
execute:
    script: foo --version

# Run tier one tests in a container
discover:
    how: fmf
    filter: tier:1
provision:
    how: container
execute:
    how: tmt

# Verify that apache can serve pages
summary: Basic httpd smoke test
provision:
    how: virtual
    memory: 4096
prepare:
  - name: packages
    how: install
    package: [httpd, curl]
  - name: service
    how: shell
    script: systemctl start httpd
execute:
    how: shell
    script:
    - echo foo > /var/www/html/index.html
    - curl http://localhost/ | grep foo

summary:
    Test basic functionality of the httpd24 collection
discover:
    how: fmf
execute:
    how: tmt
context:
    collection: httpd24

summary:
    Verify dash against the shared shell tests repository
discover:
    how: fmf
    url: https://src.fedoraproject.org/tests/shell
execute:
    how: tmt
context:
    component: dash

Discover tests relevant for execution

Download rpm sources for dist-git repositories

# Download & extract sources from another repo, print
# single file as a test
discover:
    how: shell
    url: https://src.fedoraproject.org/rpms/tmt
    dist-git-source: true
    tests:
      - name: /print/pyproject
        test: cat $TMT_SOURCE_DIR/tmt-*/pyproject.toml

# Just download sources, test is reponsible for rpmbuild
# and running tests
discover:
    how: shell
    dist-git-source: true
    dist-git-download-only: true
    tests:
      - name: /unit
        test: >
            rpmbuild -bp
                --define "_sourcedir $TMT_SOURCE_DIR"
                --define "_builddir $TMT_SOURCE_DIR/BUILD"
                $TMT_SOURCE_DIR/*.spec &&
            cd $TMT_SOURCE_DIR/BUILD/* &&
            make test
        require:
          - rpm-build

Discover available tests using the fmf format

Use the Flexible Metadata Format to explore all available tests in given repository. The following parameters are supported:

url
Git repository containing the metadata tree. Current git repository used by default.

ref
Branch, tag or commit specifying the desired git revision. Defaults to the remote repository’s default branch if url given or to the current HEAD if url not provided.

Additionally, one can set ref dynamically. This is possible using a special file in tmt format stored in the default branch of a tests repository. 
This special file should contain rules assigning attribute ref in an adjust block, for example depending on a test run context.

Dynamic ref assignment is enabled whenever a test plan reference has the format ref: @FILEPATH.

path
Path to the metadata tree root. Must be relative to the git repository root if url provided, absolute local filesystem path otherwise. By default . is used.

test
List of test names or regular expressions used to select tests by name. Duplicate test names are allowed to enable repetitive test execution, preserving the listed test order.

link
Select tests using the link keys. Values must be in the form of RELATION:TARGET, tests containing at least one of them are selected. Regular expressions are supported for both relation and target. Relation part can be omitted to match all relations.

filter
Apply advanced filter based on test metadata attributes. See pydoc fmf.filter for more info.

exclude
Exclude tests which match a regular expression.

prune
Copy only immediate directories of executed tests and their required files.

New in version 1.29.

It is also possible to limit tests only to those that have changed in git since a given revision. This can be particularly useful when testing changes to tests themselves (e.g. in a pull request CI). Related config options (all optional) are:

modified-only
Set to true if you want to filter modified tests only. The test is modified if its name starts with the name of any directory modified since modified-ref.

modified-url
An additional remote repository to be used as the reference for comparison. Will be fetched as a reference remote in the test dir.

modified-ref
The ref to compare against. Defaults to the local repository’s default branch. Note that you need to specify reference/<branch> to compare to a branch from the repository specified in modified-url.

Use the dist-git-source options to download rpm sources for dist-git repositories. In addition to the common dist-git-source features, the fmf plugin also supports the following options:

dist-git-init
Set to true to initialize fmf root inside extracted sources at dist-git-extract location or top directory. To be used when the sources contain fmf files (for example tests) but do not have an associated fmf root.

dist-git-remove-fmf-root
Set to true to remove any fmf root present in the sources. dist-git-init can be used to create it later at desired location.

dist-git-merge
Set to true to combine fmf root from the sources and fmf root from the plan. It allows to have plans and tests defined in the DistGit repo which use tests and other resources from the downloaded sources. Any plans in extracted sources will not be processed.

dist-git-extract
Path specifying what should be copied from the sources. Defaults to top fmf root or top directory /.

# Discover all fmf tests in the current repository
discover:
    how: fmf

# Fetch tests from a remote repo, filter by name/tier
discover:
    how: fmf
    url: https://github.com/teemtee/tmt
    ref: main
    path: /metadata/tree/path
    test: [regexp]
    filter: tier:1

# Choose tests verifying given issue
discover:
    how: fmf
    link: verifies:issues/123$

# Select only tests which have been modified
discover:
    how: fmf
    modified-only: true
    modified-url: https://github.com/teemtee/tmt-official
    modified-ref: reference/main

# Extract tests from the distgit sources
discover:
    how: fmf
    dist-git-source: true

Provide a manual list of shell test cases

# Define several local tests
discover:
    how: shell
    tests:
      - name: /help/main
        test: tmt --help
      - name: /help/test
        test: tmt test --help
      - name: /help/smoke
        test: ./smoke.sh
        path: /tests/shell
        duration: 1m

# Fetch tests from a remote repository
discover:
    how: shell
    url: https://github.com/teemtee/tmt
    tests:
      - name: Use tests/full/test.sh from the remote repo
        path: /tests/full
        test: ./test.sh

Execute tests on selected guests

The where key allows to select guests where the tests should be executed by providing their name or the role they play in the scenario. 
Use a list to specify multiple names or roles. By default, when the where key is not defined, tests are executed on all provisioned guests.

# Run different script for each guest or role
discover:
    how: shell
    tests:
      - name: run-the-client-code
        test: client.py
        where: client
      - name: run-the-server-code
        test: server.py
        where: server

# Filter different set of tests for each guest or role
discover:
  - how: fmf
    filter: tag:client-tests
    where: client
  - how: fmf
    filter: tag:server-tests
    where: server

# Alternative syntax using the 'where' dictionary
# encapsulating for tests defined by fmf
discover:
    where:
        client:
          - how: fmf
            filter: tag:client-tests
        server:
          - how: fmf
            filter: tag:server-tests

# Alternative syntax using the 'where' dictionary
# encapsulating for shell script tests
discover:
    where:
        server:
            how: shell
            tests:
              - test: first server script
              - test: second server script
              - test: third server script
        client:
            how: shell
            tests:
              - test: first client script
              - test: second client script
              - test: third client script

# Load from a dotenv/shell format
/plan:
    environment-file:
      - env

# Load from a yaml format
/plan:
    environment-file:
      - environment.yml
      - environment.yaml

# Fetch from remote source
/plan:
    environment-file:
      - https://example.org/project/environment.yaml

Specifies environment variables available in all steps. Plugins need to include these environment variables while running commands or other programs. 
These environment variables override test environment if present. Command line option --environment can be used to override environment variables defined in both tests and plan. 
Use the environment-file key to load variables from files. The environment+ notation can be used to extend environment defined in the parent plan, see also the Inherit Plans section for more examples.

# Environment variables defined in a plan
environment:
    KOJI_TASK_ID: 42890031
    RELEASE: f33
execute:
    script: echo "Testing $KOJI_TASK_ID on release $RELEASE"

# Share common variables across plans using inheritance
/plans:
    environment:
        COMMON: This is a common variable content

    /mini:
        environment+:
            VARIANT: mini
    /full:
        environment+:
            VARIANT: full

# Variables from the command line
tmt run --environment X=1 --environment Y=2
tmt run --environment "X=1 Y=2"

# Make sure to quote properly values which include spaces
tmt run --environment "BUGS='123 456 789'"

Define how tests should be executed

Execute discovered tests in the provisioned environment using selected test executor. 
By default tests are executed using the internal tmt executor which allows to show detailed progress of the testing and supports interactive debugging.

# Enabled a minimal smoke test
execute:
    script: foo --version

# Run tier one tests in a container
discover:
    how: fmf
    filter: tier:1
provision:
    how: container
execute:
    how: tmt

# Verify that apache can serve pages
summary: Basic httpd smoke test
provision:
    how: virtual
    memory: 4096
prepare:
  - name: packages
    how: install
    package: [httpd, curl]
  - name: service
    how: shell
    script: systemctl start httpd
execute:
    how: shell
    script:
    - echo foo > /var/www/html/index.html
    - curl http://localhost/ | grep foo

Stop execution after a test fails

//下面的例子没跑通
Execute shell scripts

# Run a simple smoke test
execute:
    how: tmt
    script: tmt --help

# Modify the default maximum duration
execute:
    how: tmt
    script: a-long-test-suite
    duration: 3h

execute:
    script: |
        dnf -y install httpd curl
        systemctl start httpd
        echo foo > /var/www/html/index.html
        curl http://localhost/ | grep foo

execute:
    script:
      - dnf -y install httpd curl
      - systemctl start httpd
      - echo foo > /var/www/html/index.html
      - curl http://localhost/ | grep foo

Internal test executor

As a user I want to execute tests directly from tmt.

The internal tmt executor runs tests in the provisioned environment one by one directly from the tmt code which allows features such as showing live progress or the interactive session . 
This is the default execute step implementation.

The executor provides following shell scripts which can be used by the tests for certain operations.

tmt-file-submit
tmt-reboot
tmt-report-result
tmt-abort

Perform system upgrades during testing

finish:
    how: shell
    script: upload-logs.sh

finish:
    how: ansible
    playbook:
        - playbooks/common.yml
        - playbooks/os/rhel7.yml
        - https://foo.bar/rhel7-final-touches.yml

finish:
    how: shell
    script:
    - upload-logs.sh || true
    - rm -rf /tmp/temporary-files

# Stop Apache on the server
finish:
  - how: shell
    script: systemctl stop httpd
    where: server

Prepare the environment for testing
The prepare step is used to define how the guest environment should be prepared so that the tests can be successfully executed.

The install plugin provides an easy way to install required or recommended packages from disk and from the offical distribution or copr repositories. 
Use the ansible plugin for applying custom playbooks or execute shell scripts to perform arbitrary preparation tasks.

Use the order attribute to select in which order the preparation should happen if there are multiple configs. 
Default order is 50. For installation of required packages gathered from the require attribute of individual tests order 70 is used, for recommended packages it is 75.

# Install fresh packages from a custom copr repository
prepare:
  - how: install
    copr: psss/tmt
    package: tmt+all

# Install required packages and start the service
prepare:
  - name: packages
    how: install
    package: [httpd, curl]
  - name: service
    how: shell
    script: systemctl start httpd

prepare:
    how: ansible
    playbook:
        - playbooks/common.yml
        - playbooks/os/rhel7.yml
        - https://foo.bar/rhel7-final-touches.yml
    extra-args: '-vvv'

Install packages on the guest

# Install local rpms using file path
prepare:
    how: install
    package:
      - tmp/RPMS/noarch/tmt-0.15-1.fc31.noarch.rpm
      - tmp/RPMS/noarch/python3-tmt-0.15-1.fc31.noarch.rpm

# Install remote packages using url
prepare:
    how: install
    package:
      - https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
      - https://dl.fedoraproject.org/pub/epel/epel-next-release-latest-8.noarch.rpm

# Install the whole directory, exclude selected packages
prepare:
    how: install
    directory:
      - tmp/RPMS/noarch
    exclude:
      - tmt+all
      - tmt+provision-virtual

# Enable copr repository, skip missing packages
prepare:
    how: install
    # Repository with a group owner (@ prefixed) requires quotes, e.g.
    # copr: "@osci/rpminspect"
    copr: psss/tmt
    package: tmt-all
    missing: skip

Prepare system using shell (bash) commands

prepare:
    how: shell
    script: dnf install -y httpd

Apply preparation on selected guests

# Start Apache on the server
prepare:
  - how: shell
    script: systemctl start httpd
    where: server

# Apply common setup on the primary server and all replicas
prepare:
  - how: ansible
    playbook: common.yaml
    where: [primary, replica]

Provision a system for testing

Describes what environment is needed for testing and how it should be provisioned. 
There are several provision plugins supporting multiple ways to provision the environment for testing, for example virtual, container, connect, local or artemis. 
See individual plugin documentation for details about supported options.

As part of the provision step it is also possible to specify detailed hardware requirements for the testing environment. See the Hardware specification section for details.

As part of the provision step it is also possible to specify kickstart file used during the installation. See the kickstart specification section for details.

Provision a machine in Beaker

Reserve a machine from the Beaker pool using the mrack plugin. mrack is a multicloud provisioning library supporting multiple cloud services including Beaker.

The following two files are used for configuration:

/etc/tmt/mrack.conf
for basic configuration

/etc/tmt/provisioning-config.yaml
configuration per supported provider

# Use image name translation
provision:
    how: beaker
    image: fedora
# Specify the distro directly
provision:
    how: beaker
    image: Fedora-37%

provision:
    how: connect
    guest: hostname or ip address
    user: username
    password: password

provision:
    how: connect
    guest: hostname or ip address
    key: private-key-path

provision:
    how: container
    image: fedora:latest

provision:
    how: local

# Request two guests
provision:
  - name: server
    how: virtual
  - name: client
    how: virtual

# Assign role to guests
provision:
  - name: main-server
    role: primary
  - name: backup-one
    role: replica
  - name: backup-two
    role: replica
  - name: tester-one
    role: client
  - name: tester-two
    role: client

provision:
    how: openstack
    image: f31

provision:
    how: virtual
    image: fedora

Provision a guest via an Artemis service

# Use the artemis plugin to provision a guest from Beaker.
# The following `kickstart` specification will be run
# during the guest installation.
provision:
    how: artemis
    pool: beaker
    image: rhel-7
    kickstart:
        pre-install: |
            %pre --log=/dev/console
            disk=$(lsblk | grep disk | awk '{print $1}')
            echo $disk
            %end
        script: |
            lang en_US.UTF-8
            zerombr
            clearpart --all --initlabel
            part /boot --fstype="xfs" --size=200
            part swap --fstype="swap" --size=4096
            part / --fstype="xfs" --size=10000 --grow
        post-install: |
            %post
            systemctl disable firewalld
            %end
        metadata: |
            "no-autopart harness=restraint"
        kernel-options: "ksdevice=eth1"
        kernel-options-post: "quiet"

Show results in the terminal window```
tmt run -l report        # overall summary only
tmt run -l report -v     # individual test results
tmt run -l report -vv    # show full paths to logs
tmt run -l report -vvv   # provide complete test output

# Enable html report from the command line
tmt run --all report --how html
tmt run --all report --how html --open
tmt run -l report -h html -o

# Use html as the default report for given plan
report:
    how: html
    open: true

# Minimal reference is using 'url' and 'name'
plan:
    import:
        url: https://github.com/teemtee/tmt
        name: /plans/features/basic

# A 'ref' can be used to select specific branch or commit
plan:
    import:
        url: https://github.com/teemtee/tmt
        name: /plans/features/basic
        ref: fedora

# Use 'path' when fmf tree is deeper in the git repository
plan:
    import:
        url: https://github.com/teemtee/tmt
        path: /examples/httpd
        name: /smoke

TMT_TOPOLOGY_YAML
# Guest on which the test or script is running.
guest:
    name: ...
    role: ...
    hostname: ...

# List of names of all provisioned guests.
guest-names:
  - guest1
  - guest2
  ...

# Same as `guest`, but one for each provisioned guest, with guest names
# as keys.
guests:
    guest1:
        name: guest1
        role: ...
        hostname: ...
    guest2:
        name: guest2
        role: ...
        hostname: ...
    ...

# List of all known roles.
role-names:
  - role1
  - role2
  ...

# Roles and their guests, with role names as keys.
roles:
    role1:
      - guest1
      - guest2
      - ...
    role2:
      - guestN
      ...
TMT_TOPOLOGY_BASH
# Guest on which the test is running.
declare -A TMT_GUEST
TMT_GUEST[name]="..."
TMT_GUEST[role]="..."
TMT_GUEST[hostname]="..."

# Space-separated list of names of all provisioned guests.
TMT_GUEST_NAMES="guest1 guest2 ..."

# Same as `guest`, but one for each provisioned guest. Keys are constructed
# from guest name and the property name.
declare -A TMT_GUESTS
TMT_GUESTS[guest1.name]="guest1"
TMT_GUESTS[guest1.role]="..."
TMT_GUESTS[guest1.hostname]="..."
TMT_GUESTS[guest2.name]="guest2"
TMT_GUESTS[guest2.role]="..."
TMT_GUESTS[guest2.hostname]="..."
...

# Space-separated list of all known roles.
TMT_ROLE_NAMES="client server"

# Roles and their guests, with role names as keys.
declare -A TMT_ROLES
TMT_ROLES[role1]="guest1 guest2 ..."
TMT_ROLES[role2]="guestN ..."
...
Examples:

# A trivial pseudo-test script
. "$TMT_TOPOLOGY_BASH"

echo "I'm running on ${TMT_GUEST[name]}"

As a developer I want to define application features and track which have been already implemented, verified and documented.

Stories, which implement the L3 metadata, can be used to track implementation, test and documentation coverage for individual features or requirements. 
Thanks to this you can track everything in one place, including the project implementation progress.

In addition to the attributes defined here, stories also support common Core attributes which are shared across all metadata levels.

Examples:

story:
    As a user I want to see more detailed information for
    particular command.
description:
    Different verbose levels can be enabled by using the
    option several times.
example:
  - tmt test show -v
  - tmt test show -vvv
  - tmt test show --verbose
link:
  - implemented-by: /tmt/cli.py
  - documented-by: /tmt/cli.py
  - verified-by: /tests/core/dry
priority: must have

# Short one-line example
example: tmt run discover
# Preserve line breaks using the '|' modifier
example: |
    tmt run --until execute
    tmt run --last report
# Use a list to define multiple examples
example:
  - tmt run --until execute
  - tmt run --last report
# Several multiline examples
example:
    - |
        # Check validity of the plans
        tmt plan lint
    - |
        # Check which tests would be run
        tmt run discover

must have
critical for the project success, must be delivered on time in order to consider the implementation as successful

should have
important stories which need to be covered but are not as time-critical and can be delivered later if there are not enough resources

could have
desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost

will not have
least-critical, lowest-payback items, not planned into the current schedule, might be dropped or reconsidered for inclusion in a later timebox

# This is an essential feature
priority: must have

story:
    As a user I want the application to do this and that
    so that I can achieve this and that.

title: Nice title

As a user I want to define a context so that test, plan or story metadata can be adjusted to it accordingly.
Context是给adjust用的。
Parametrize Plans也可以用context
  For environment variables the syntax is standard, both $var and ${var} may be used. 
  For context parametrization the syntax is $@dimension or $@{dimension}. 

The context is usually defined from the command line using the --context option. All dimensions are optional. 
It is possible to define your own dimensions to describe the context, just make sure the dimension name does not conflict with reserved names.

Each plan can also provide its own context. The context definition provided directly on the command line overrides defined dimensions. 
All context dimension values are handled in case-insensitive way.

tmt --context distro=fedora-33 run
tmt --context product=rhscl run
tmt --context trigger=code run

tmt -c distro=fedora test show
tmt -c distro=fedora plan show

The following dimensions are reserved for storing dedicated information as described below:

distro
variant
arch
component
collection
module
initiator
trigger

example:
context:
    distro: fedora-33
    variant: Workstation
    arch: x86_64

context:
    product: rhscl
    collection: httpd24

human
Test execution was initiated manually by a human.

packit
For jobs initiated by the Packit service.

fedora-ci
Testing in the Fedora CI pipeline. To be implemented.

centos-ci
Testing in the CentOS CI pipeline. To be implemented.

rhel-ci
Testing in the RHEL CI pipeline. To be implemented.

ewa
Errata Workflow Automation jobs. To be implemented.

# Enable the plan only when run manually
summary: Tests requiring special environment
discover:
    how: fmf
    filter: tag:special
execute:
    how: tmt
adjust:
    enabled: false
    when: initiator != human

commit
Project source code has changed. This can be either a pull/merge request or a new commit pushed to a branch.

build
There has been a new package built in koji or brew and is available for testing.

update
A new bodhi update or errata advisory with one or more builds has been created or updated.

compose
There is a new compose ready for integration testing.

This context dimension will usually be provided from the command line.

Examples:

summary: Full test coverage (takes three days)
discover:
    how: fmf
execute:
    how: tmt
adjust:
    enabled: false
    when: trigger == code

As part of the provision step it is possible to use the hardware key to specify additional requirements for the testing environment. 
This provides a generic and an extensible way to write down essential hardware requirements. For example one consistent way how to specify at least 2 GB of RAM for all supported provisioners.

tmt lint