tmt brief
Test Management Tool
TMT基于Flexible Metadata Format，是一个测试管理工具。
使用tmt，你可以把测试配置、脚本与被测源码放到一起。
通过生成fmf格式的配置文件，包括test plan story，可以轻松的管理测试配置。
The tmt tool provides a user-friendly way to work with tests.
You can comfortably create new tests, safely and easily run tests across different environments,
review test results, debug test code and enable tests in the CI using a consistent and concise config.
metadata level
Core attributes such as summary or description which are common across all levels are defined by the special L0 metadata.
Core metadata在其他level中通用，比如Tests Plans Stories中都可以使用Core metadata。
Tests, or L1 metadata, define attributes which are closely related to individual test cases such as test script, framework,
directory path where the test should be executed, maximum test duration or packages required to run the test.
Plans, also called L2 metadata, are used to group relevant tests and enable them in the CI. They describe how to discover tests for execution,
how to provision the environment, how to prepare it for testing, how to execute tests and report test results.
Stories, which implement the L3 metadata, can be used to track implementation, test and documentation coverage for individual features or requirements.
Thanks to this you can track everything in one place, including the project implementation progress.
install
fedora: dnf -y install tmt-all
rhel/centos: 需要安装epel repo，然后dnf -y install tmt-all
command
tmt init
tmt init --template mini
tmt init --template base
tmt init --template full
tmt run discover
tmt run prepare execute
tmt run --all tests --filter tag:beaker provision --how container execute -v
tmt run plan --name multihost
tmt tests ls
tmt tests show
tmt tests lint
tmt test create
tmt test import
tmt plans ls
tmt plans show
tmt plans lint
tmt stories ls
tmt stories show
tmt stories coverage
$ tmt test create --template shell /tests/smoke
$ tmt test create --t beakerlib /tests/smoke
$ tmt test create /tests/smoke --link foo
$ tmt test create /tests/smoke --link foo --link verifies:https://foo.com/a/b/c
tmt plan create --template mini /plans/smoke
tmt plan create --t full /plans/features
tmt plan create /plans/custom --template mini \
--discover '{how: "fmf", name: "internal", url: "https://internal/repo"}' \
--discover '{how: "fmf", name: "external", url: "https://external/repo"}'
tmt run --all plan --name multihost prepare --how shell --script "dnf -y install vim;echo DRIVER=i40e>/root/env;"
tmt run --all login --step prepare:end plan --name multihost prepare --how shell --script "dnf -y install vim;echo DRIVER=i40e>/root/env;"
tmt run --all login --step prepare:end test plan --name multihost execute -vvv
tmt run -l login
tmt run --all plan --name beakerlib provision --how container
tmt story create --template full /stories/usability
tmt lint
$ tmt lint /tests/execute/basic
tmt run 包括六个steps,默认都run，也可以选择性的run某些个steps：
discover
Gather information about test cases to be executed.
provision
Provision an environment for testing or use localhost.
prepare
Prepare the environment for testing.
execute
Run tests using the specified executor.
report
Provide test results overview and send reports.
finish
Perform the finishing tasks and clean up provisioned guests.
Command Variables
The following environment variables can be used to modify behaviour of the tmt command.
TMT_DEBUG
TMT_PLUGINS
TMT_WORKDIR_ROOT
NO_COLOR, TMT_NO_COLOR
TMT_FORCE_COLOR
TMT_OUTPUT_WIDTH
TMT_GIT_CREDENTIALS_URL_<suffix>, TMT_GIT_CREDENTIALS_VALUE_<suffix>
Step Variables
The following environment variables are provided to the environment during prepare, execute and finish steps:
TMT_TREE
TMT_PLAN_DATA
TMT_PLAN_ENVIRONMENT_FILE
TMT_VERSION
Test Variables
TMT_TEST_NAME
TMT_TEST_DATA
TMT_TEST_SERIAL_NUMBER
TMT_TEST_METADATA
TMT_SOURCE_DIR
TMT_REBOOT_COUNT
TMT_TOPOLOGY_BASH, TMT_TOPOLOGY_YAML
TMT_TEST_PIDFILE, TMT_TEST_PIDFILE_LOCK
TMT_TEST_PIDFILE_ROOT
Plugin Variables
Each plugin option can be also specified via environment variable. Variables follow a naming scheme utilizing plugin name, step it belongs to, and the option name:
TMT_PLUGIN_${STEP}_${PLUGIN}_${OPTION}
configuration tree
The data are organized into trees. Similarly as with git, there is a special .fmf directory which marks the root of the fmf metadata tree.
Use the init command to initialize it:
$ tmt init
Do not forget to include this special .fmf directory in your commits, it is essential for
building the fmf tree structure which is created from all *.fmf files discovered under the fmf root.
tmt中test或者plan的名字由文件所在的目录结构和文件里面的路径指定。
比如下面文件中定义的/test/all , /test/client, /test/server，通过tmt test ls显示为下面的名字。
[root@fedora-38 tmt-test]# pwd
/work/tmt-test
[root@fedora-38 tmt-test]# cat plans/multihost.fmf
/plan:
summary: multihost test demo, sequence and paralell
discover:
- name: client_t
how: fmf
test: /test/client
#filter: tag:client-tests
where:
- client
order: 10
/test:
/client:
test: ./test.sh client
tag: client-tests
path: /
/server:
test: ./test.sh server
tag: server-tests
path: /
/all:
test: ./test.sh all
tag: all-tests
path: /
tmt test ls
/beaker/beakerlib/beakerlib-test
/manual/manual
/plans/multihost/test/all
/plans/multihost/test/client
/plans/multihost/test/server
/plans/plan1/tests/test1
/plans/plan1/tests/test2
/plans/plan1/tests/test3
/plans/plan2/test2/test1
/plans/plan2/test2/test2
/plans/plan2/test2/test3
/test1/example/fast
/test1/example/full
/test2/test
Inheritance
├── plans
│ ├── features
│ ├── install
│ ├── integration
│ ├── provision
│ ├── remote
│ └── sanity
└── tests
├── core
├── full
├── init
├── lint
├── login
├── run
├── steps
└── unit
# main.fmf
test: ./test.sh
framework: beakerlib
require: [tmt]
tests
├── main.fmf
├── core
├── full
├── init
...
core,full,init目录下的main.fmf可以继承根目录mani.fmf里面的设置，生成三个不同的test。
Virtual Tests
test: ./test.sh
require: curl
/fast:
summary: Quick smoke test
tier: 1
duration: 1m
environment:
MODE: fast
/full:
summary: Full test set
tier: 2
duration: 10m
environment:
MODE: full
Inherit Plans
discover:
how: fmf
url: https://github.com/teemtee/tmt
prepare:
how: ansible
playbook: ansible/packages.yml
execute:
how: tmt
/basic:
summary: Quick set of basic functionality tests
discover+:
filter: tier:1
/features:
summary: Detailed tests for individual features
discover+:
filter: tier:2
Elasticity
you can combine both the plan and tests like this:
/plan:
summary:
Verify that plugins are working
discover:
how: fmf
provision:
how: container
prepare:
how: install
package: did
execute:
how: tmt
/tests:
/bugzilla:
test: did --bugzilla
/github:
test: did --github
/koji:
test: did --koji
Or you can put the plan in one file and tests into another one:
# plan.fmf
summary:
Verify that plugins are working
discover:
how: fmf
provision:
how: container
prepare:
how: install
package: did
execute:
how: tmt
# tests.fmf
/bugzilla:
test: did --bugzilla
/github:
test: did --github
/koji:
test: did --koji
Or even each test can be defined in a separate file:
# tests/bugzilla.fmf
test: did --bugzilla
# tests/github.fmf
test: did --github
# tests/koji.fmf
test: did --koji
Multihost Testing
/plan:
summary: multihost test demo, sequence and paralell
discover:
- name: client_t
how: fmf
# use tet keywork match test name
test: /test/client
#filter: tag:client-tests
where:
- client
order: 10
- name: server_t
how: fmf
# user filter match test tag
filter: tag:server-tests
where:
- server
order: 12
- name: all_t
how: fmf
filter: tag:all-tests
where:
- server
- client
order: 11
- name: outer_test
how: fmf
# use tet keywork match test name
test: /test1/example/full
provision:
- name: server
how: connect
guest: dell-per740-15.rhts.eng.pek2.redhat.com
user: root
password: redhat
role: SERVER
- name: client
how: connect
guest: dell-per740-17.rhts.eng.pek2.redhat.com
user: root
password: redhat
role: CLIENT
prepare:
#how: install
#package: tmt
- how: shell
script: |
> /root/env
echo "DRIVER=mlx5" >> /root/env
where:
- CLIENT
- how: shell
script: |
> /root/env
echo "DRIVER=mlx4" >> /root/env
where:
- SERVER
execute:
how: tmt
/test:
/client:
test: ./test.sh client
tag: client-tests
path: /
/server:
test: ./test.sh server
tag: server-tests
path: /
/all:
test: ./test.sh all
tag: all-tests
path: /
Use shell in discover
/plan:
summary: how is shell, define local tests
discover:
- name: client_t
how: shell
tests:
- name: /shell/1
test: echo 1
- name: /shell/2
test: echo 2
where:
- server
- client
order: 8
provision:
- name: server
how: connect
guest: dell-per740-15.rhts.eng.pek2.redhat.com
user: root
password: redhat
role: SERVER
- name: client
how: connect
guest: dell-per740-17.rhts.eng.pek2.redhat.com
user: root
password: redhat
role: CLIENT
prepare:
#how: install
#package: tmt
- how: shell
script: echo "DRIVER=i4e >> /root/env"
where:
- client
- how: shell
script: echo "DRIVER=ixgbe >> /root/env"
where:
- server
execute:
how: tmt
Examples
init
Before starting a new project initialize the metadata tree root:
$ tmt init
You can also populate it with a minimal plan example:
$ tmt init --template mini
Create a plan and a test:
$ tmt init --template base
Initialize with a richer example that also includes the story (overwriting existing files):
$ tmt init --template full --force
Tests
$ tmt tests
$ tmt tests ls
$ tmt tests show
$ tmt tests show /tests/docs --verbose
$ tmt tests show docs
$ tmt test show .
$ tmt run test --name .
$ tmt test import
Plans
$ tmt plans
$ tmt plans ls
$ tmt plans show
Multiple Configs
Step can contain multiple configurations. In this case provide each config with a unique name.
Applying ansible playbook and executing custom script in a single prepare step could look like this:
prepare:
- name: packages
how: ansible
playbook: ansible/packages.yml
- name: services
how: shell
script: systemctl start service
Another common use case which can be easily covered by multiple configs can be fetching tests from multiple repositories during the discover step:
discover:
- name: upstream
how: fmf
url: https://github.com/teemtee/tmt
- name: fedora
how: fmf
url: https://src.fedoraproject.org/rpms/tmt/
Extend Steps
prepare:
- name: tmt
how: install
package: tmt
Extending the prepare config in a child plan to install additional package then could be done in the following way:
prepare+:
- name: pytest
how: install
package: python3-pytest
Eventually, use adjust to extend the step conditionally:
adjust:
- when: distro == fedora
prepare+:
- name: pytest
how: install
package: python3-pytest
Parametrize Plans
For environment variables the syntax is standard, both $var and ${var} may be used.
The values of variables are taken from the --environment command line option and the environment plan attribute.
discover:
how: fmf
url: https://github.com/teemtee/${REPO}
$ tmt run -e REPO=tmt
discover:
- how: fmf
url: https://github.com/teemtee/tmt.git
test: ${PICK_TMT}
- how: fmf
url: https://github.com/teemtee/fmf.git
test: ${PICK_FMF}
$ tmt run -e PICK_TMT='^/tests/core/ls$' -e PICK_FMF='^/tests/(unit|basic/ls)$'
For context parametrization the syntax is $@dimension or $@{dimension}.
The values are set according to the defined context specified using --context command line option and the context plan attribute:
context:
branch: main
discover:
how: fmf
url: https://github.com/teemtee/tmt
ref: $@{branch}
$ tmt -c branch=tmt run
Dynamic ref Evaluation
通过@.tmtref指定ref文件名
discover:
how: fmf
url: https://github.com/teemtee/repo
ref: "@.tmtref"
Example of a dynamic ref definition file in repo/.tmtref:
ref: main
adjust:
- when: distro == centos-stream-9
ref: rhel-9
- when: distro == fedora
ref: fedora
- when: distro == rhel-9
ref: rhel-9
The definition file can also be parametrized using environment variables or context dimensions:
ref: main
adjust:
- when: distro == fedora or distro == rhel
ref: $@distro
Stories
$ tmt stories
$ tmt stories ls
$ tmt stories show
$ tmt stories show --help | grep only
-i, --implemented Implemented stories only.
-I, --unimplemented Unimplemented stories only.
-t, --verified Stories verified by tests.
-T, --unverified Stories not verified by tests.
-d, --documented Documented stories only.
-D, --undocumented Undocumented stories only.
-c, --covered Covered stories only.
-C, --uncovered Uncovered stories only.
$ tmt stories ls --implemented
/spec/core/summary
/stories/api/plan/attributes/artifact
/stories/api/plan/attributes/gate
...
$ tmt stories show --documented
/stories/cli/common/debug
summary Print out everything tmt is doing
story I want to have common command line options consistenly used
across all supported commands and subcommands.
example tmt run -d
tmt run --debug
implemented /tmt/cli
documented /tmt/cli
$ tmt story show .
$ tmt story coverage
code test docs story
todo todo todo /spec/core/description
todo todo todo /spec/core/order
done todo todo /spec/core/summary
...
done todo todo /stories/cli/usability/completion
39% 9% 9% from 109 stories
Run
$ tmt run
$ tmt run --dry
# remove the workdir after the execution is finished:
$ tmt run --remove
$ tmt run --rm
$ tmt run -r
$ tmt run plan --name basic
$ tmt run test --filter tier:1
$ tmt run test --name .
$ tmt run discover
$ tmt run discover provision prepare
# debug output for provision only
$ tmt run discover provision --debug
# debug output for all steps
$ tmt run --debug discover provision
In order to execute all test steps while providing arguments to some of them it is possible to use the --all option:
$ tmt run --all provision --how=local
$ tmt run --all execute -vvv
tmt run
tmt run -a provision -h virtual
tmt run --all provision --how=virtual
tmt run --all provision --how=virtual.testcloud
tmt run --all provision --how=connect --guest=name-or-ip --user=login --password=secret --become
tmt run --all provision --how=connect --guest=name-or-ip --key=private-key-path
tmt run --all provision --how=container --image=fedora:latest
tmt run --all provision --how=local
tmt run --last reboot
tmt run --last reboot --hard
Debug
Sometimes the environment preparation can take a long time. Thus, especially for debugging tests,
it usually makes sense to run the provision and prepare step only once, then execute tests as many times as necessary to debug the test code and finally clean up when debugging is done:
tmt run --id <ID> --until execute # prepare, run test once
tmt run -i <ID> execute -f # run test again
tmt run -i <ID> execute -f # run it again
tmt run -i <ID> execute -f # and again
tmt run -i <ID> report finish
Instead of always specifying the whole run id you can also use --last or -l as an abbreviation for the last run id:
tmt run --last execute --force
tmt run -l execute -f
The --force option instructs tmt to run given step even if it has been already completed before.
Use discover --force to synchronize test code changes to the run workdir:
tmt run -l discover -f execute -f
tmt run --all execute --how tmt --interactive
Guest Login
$ tmt run login --step prepare
$ tmt run login --step execute
$ tmt run login --step prepare:start
$ tmt run login --step prepare:50
$ tmt run login --step prepare:end
$ tmt run login --when fail
$ tmt run login --when fail --when error
You can also enable only the provision step to easily get a clean and safe environment for experimenting. Use the finish step to remove provisioned guest:
$ tmt run provision login
$ tmt run --last finish
Clean up the box right after your are done with experimenting by combining the above-mentioned commands on a single line:
$ tmt run provision login finish
Login can be used to run an arbitrary script on a provisioned guest.
$ tmt run --last login < script.sh
status
$ tmt status
status id
prepare /var/tmp/tmt/run-002
done /var/tmp/tmt/run-001
$ tmt status -v
status id
prepare /var/tmp/tmt/run-002 /base
done /var/tmp/tmt/run-001 /advanced
done /var/tmp/tmt/run-001 /base
$ tmt status /tmp/run
status id
done /tmp/run/001
$ tmt status -vv --id run-002
disc prov prep exec repo fini id
done done done todo todo todo /var/tmp/tmt/run-002 /base
Runs and plans can also be filtered based on their status. Option --abandoned can be used to list runs/plans which have provision step completed but finish step not yet done.
This is useful for finding active containers or virtual machines:
$ tmt status --abandoned
status id
prepare /var/tmp/tmt/run-002
$ tmt status --finished
status id
done /var/tmp/tmt/run-001
$ tmt status --active
status id
prepare /var/tmp/tmt/run-002
clean
$ tmt clean
$ tmt clean -v
$ tmt clean -v --dry
$ tmt clean guests -v /tmp/run
$ tmt clean guests -v --last
$ tmt clean guests --how container
$ tmt clean guests --how virtual
$ tmt clean runs /tmp/run
$ tmt clean runs -v -i /var/tmp/tmt/run-001
You may also want to remove only old runs. This can be achieved using --keep option which allows you to specify the number of latest runs to keep:
$ tmt clean runs --dry -v --keep 5
$ tmt clean images
use library
这个例子在tmt version: 1.19.0上成功 在tmt version: 1.28.2上失败。
plan
# public repo tmt-test
[root@liali-vm1 tmt-test]# cat test1/main.fmf
/plan:
discover:
how: fmf
#url: 如果指定url，会下载url指定的repo，到这个repo去寻找metadata，然后会到这个repo的目录去执行test
prepare:
how: install
package: wget
execute:
how: tmt
provision:
- name: server
how: connect
guest: mlxsw-sn2100-01.mgmt.lab.eng.pek2.redhat.com
user: root
password: redhat
role: SERVER
/test:
contact: Liang Li<liali@redhat.com>
component:
- kernel
test: ./sample.sh
framework: beakerlib
#执行测试时所在的路径,默认是test所在路径
path: /sampleDir
# 下面的lib会存储到../discover/default-0/libs
# /bin/test需要通过tmt test ls能找到才可以
require:
- url: https://gitlab.com/liali666/virtual-networking
name: /bin/test
#destination: /
#nick: upstream-lib
#require:
#- kernel-kernel-networking-common
#- library(kernel/networking/common)
recommend:
- kernel
- net-tools
duration: 16h
extra-summary: /kernel/networking/switchdev-selftest
extra-task: /kernel/networking/switchdev-selftest
environment:
RUN_CASE: bridge_igmp.sh
sample.sh
[root@liali-vm1 tmt-test]# cat sampleDir/sample.sh
#!/bin/bash
. /usr/bin/rhts_environment.sh
. /usr/share/beakerlib/beakerlib.sh || . /usr/lib/beakerlib/beakerlib.sh
. ../../libs/virtual-networking/bin/tools.sh
rlJournalStart
rlPhaseStartTest
rlRun "echo 'This is beakerlib test2.'"
rlRun "echo $BASH_SOURCE"
rlRun "assertEquals 'soulde equal' 1 1"
rlRun "echo \"pwd is $(pwd)\""
rlRun "ls"
rlPhaseEnd
rlJournalEnd
remote library
# cat ../virtual-networking/bin/main.fmf
/test:
summary: test library
test: echo "This is virtual-networking library"
remote url
[root@fedora-38 virtual-networking]# cat main.fmf
/plan1:
discover:
how: fmf
prepare:
how: install
package: curl
execute:
how: tmt
provision:
- name: server
how: connect
guest: mlxsw-sn2100-01.mgmt.lab.eng.pek2.redhat.com
user: root
password: redhat
role: SERVER
/plan2:
discover:
how: fmf
# 会执行这个url路径下的test
# 但是prepare,provision等step还是使用本plan中的设置
url: https://gitlab.cee.redhat.com/liali/tmt-test
test: /beaker/beakerlib/beakerlib-test
prepare:
how: install
package: curl
execute:
how: tmt
provision:
how: container
/test:
contact: Liang Li<liali@redhat.com>
component:
- kernel
test: ./simple.sh
framework: beakerlib
#path: /simpleDir
require:
- url: https://gitlab.com/liali666/tmt-test
name: /common
#require:
#- kernel-kernel-networking-common
#- library(kernel/networking/common)
recommend:
- kernel
- net-tools
duration: 16h
extra-summary: /kernel/networking/switchdev-selftest
extra-task: /kernel/networking/switchdev-selftest
environment:
RUN_CASE: bridge_igmp.sh
testing-farm
testing-farm request --compose RHEL-9.2.0-Nightly --git-url https://gitlab.cee.redhat.com/liali/tmt-test.git --plan /beaker/beakerlib/plan --hardware hostname="mlxsw-sn2100-01.mgmt.lab.eng.pek2.redhat.com"
#testing-farm request --compose RHEL-9.2.0-Nightly --git-url https://gitlab.cee.redhat.com/liali/kernel.git --git-ref master --plan /networking/switchdev-selftest/plan1 --hardware hostname=mlxsw-sn2100-01.mgmt.lab.eng.pek2.redhat.com
testing-farm request --compose RHEL-9.2.0-Nightly --git-url https://gerrit-git.engineering.redhat.com/git/kernel-tests.git --git-ref master --plan /networking/switchdev-selftest/plan1 --hardware hostname=mlxsw-sn2100-01.mgmt.lab.eng.pek2.redhat.com
testing-farm request --compose RHEL-9.2.0-Nightly --git-url https://gitlab.cee.redhat.com/kernel-qe/kernel.git --git-ref master --plan /networking/switchdev-selftest/plan2 --hardware hostname=mlxsw-sn2100-01.mgmt.lab.eng.pek2.redhat.com
switchdev-selftest
summary: switchdev_selftest
description: |+
Description:
Test switchdev function.
Relateds bug:
cases:
/plan1:
discover:
- how: fmf
test: /networking/switchdev-selftest/test
provision:
- name: server
how: connect
guest: mlxsw-sn2100-01.mgmt.lab.eng.pek2.redhat.com
user: root
password: redhat
role: SERVER
prepare:
how: install
package: wget
execute:
how: tmt
/plan2:
discover:
- how: fmf
test: /networking/switchdev-selftest/test
prepare:
how: install
package: wget
execute:
how: tmt
/test:
contact: Liang Li<liali@redhat.com>
component:
- kernel
test: ./runtest.sh
framework: beakerlib
#require:
# - url: https://gitlab.cee.redhat.com/kernel-qe/kernel
# name: /networking/common
recommend:
- kernel
- net-tools
duration: 16h
extra-summary: /kernel/networking/switchdev-selftest
extra-task: /kernel/networking/switchdev-selftest
environment:
RUN_CASE: bridge_igmp.sh