Beaker

+-------------------+    +-------------------+    +--------+
|Server             |    |Lab Controller1    +----+system1 |
|                   |    |                   |    +--------+
|+----------+       +----+                   |             
||Scheduler |       |    |                   |    +--------+
|+----------+       |    |                   +----+system2 |
|                   |    +-------------------+    +--------+
|+----------+       |    
||Repository|       |    +-------------------+    +--------+
|+----------+       |    |Lab Controller2    +----+system3 |
|                   |    |                   |    +--------+
|+----------+       |    |                   |  
||Database  |       |    |                   |    +--------+
|+----------+       +----+                   +----+system4 |
+-------------------+    +-------------------+    +--------+

Server包括Scheduler、Repository、Database、WebServer：
  Scheduler用来选择系统去执行job。
  Repository存储task。
  Database存储inventory、user、group、jobs、system historical activity。
  WebServer: web control
Lab Controller：
  注册：每个系统必须注册到某个lab controller

  安装系统：lab controller必须启动一个tftp server，在给system安装os的时候，
  需要为该system在tftp root目录下创建netboot config files（应该包括bootloader和config file，下面有提到）。
  必须有一个dhcp server，告诉system去从tftp server获取PXE boot files。
  
  result：system通过lab controller提供的接口，向Server提交result

  log: 保存在Lab Controller

  power：重启系统

lab controller在tftp server为本次装机创建netboot config file（包含bootloader,pxelinux.cfg，见TFTP files and directories）
PXE通过dhcp server获取bootloader的位置
PXE load pxelinux(bootloader) from tftp server ->
bootloader get “initrd”,”kernel”,”ks file loaction” from config file(pxelinux.cfg/default) ->
bootloader load initrd ->
bootloader load kernel and pass ks file loaction to kernel cmdline ->
kernel start anaconda to begin a ks installation

These images must be supplied by the Beaker administrator and copied into the TFTP root directory manually (with the exception of pxelinux.0). 
When Beaker provisions a system it creates a symlink bootloader/fqdn/image pointing to one of these images, 
depending on the value of the netbootloader= kernel option (see Kernel options).

netbootloader=<tftp path to bootloader>
  Netboot loader image to use. Beaker creates a symlink so the TFTP path bootloader/fqdn/image serves the specified image.

  Set this option if you want to boot an alternative image. 
  For example, if the administrator has made an older version of PXELINUX available in the TFTP root as pxelinux-311.0, you can boot it using netbootloader=pxelinux-311.0.

  By default Beaker uses the most suitable boot loader for the chosen distro and architecture:

  i386/x86_64: pxelinux.0
  ia64:        elilo-ia64.efi
  ppc:         yaboot
  aarch64:     aarch64/bootaa64.efi
  For ppc64 and ppc64le, for Fedora, RHEL 7.1 and later:
               boot/grub2/powerpc-ieee1275/core.elf 
      For RHEL 7.0 and earlier:
               yaboot

  Note that this option will have no effect if the system has a hard-coded boot loader filename in the DHCP configuration. 
  For configurable netboot loader support the DHCP configuration must specify the filename as bootloader/fqdn/image. See DHCP and DNS.

boot loader images可以通过dhcp配置写死，也可以通过netbootloader kernel option指定，如果不写死也不指定，会使用默认值。

When Beaker provisions a system, it creates a subdirectory bootloader/fqdn 
under the TFTP root directory containing the following files.

bootloader/fqdn/image
  Symlink to the desired netboot loader image, as specified in the netbootloader= kernel option.
bootloader/fqdn/etc/0a010203
  Configuration for Yaboot.
bootloader/fqdn/grub.cfg-0A010203
  Configuration for GRUB2 (used by 64-bit ARM and PowerPC systems).
bootloader/fqdn/grub.cfg
  Default configuration for GRUB2 (used by 64-bit ARM systems).
bootloader/fqdn/petitboot.cfg
  Configuration for Petitboot.
bootloader/fqdn/pxelinux.cfg/0A010203
  Configuration for PXELINUX.
bootloader/fqdn/pxelinux.cfg/default
  Default configuration for PXELINUX.
bootloader/fqdn/ipxe/0a010203
  Configuration for iPXE.
bootloader/fqdn/ipxe/default
  Default configuration for iPXE.

images/fqdn/
  Kernel and initrd images for the distro being provisioned. 
  All the generated boot loader configurations point at the images in this directory.
pxelinux.cfg/default
  Default configuration used by PXELINUX when no system-specific configuration exists.
  The Beaker administrator can customize this configuration, 
  however it must fall back to booting the local disk by default (perhaps after a timeout) using the localboot 0 command.
  If this file does not exist, Beaker populates it with a simple default configuration that immediately boots the local disk.

For systems with BIOS-compatible firmware：
  网络启动必须是第一位，这样才能正常装机。
  当装机结束之后，beaker会清理掉相应的netboot config file，
  所以网络启动会失败，然后会继续从本地启动。

For systems with EFI-compatible firmware：
  这种系统允许在OS中修改启动顺序。
  当装机完成时，anaconda会自动创建一个启动entry，并且设置为第一启动顺序。
  beaker会在post-install脚本中重新把网络启动设置为第一顺序。并且设置BootNext选项，让下一次从本地启动。
  rhts-reboot命令也会设置BootNext选项，下一次启动会从本地启动。
  BootNext只生效一次，这样关机之后再装机就会从网络启动。

The beaker-watchdog daemon on the lab controller scans console logs to detect 
if a recipe has triggered a kernel panic, or if the installation has failed with a fatal error.

You can customize the regexp pattern for detecting kernel panics by setting PANIC_REGEX in /etc/beaker/labcontroller.conf. 
The default pattern that ships with Beaker is defined in /usr/lib/python2.7/site-packages/bkr/labcontroller/default.conf. 
The pattern uses Python regular expression syntax.

Install failure patterns are read from a directory (rather than using a single pattern like the panic detection). 
Each file contains a regexp pattern which is checked against the console log. If any pattern matches, the installation is considered to have failed.

Beaker ships a number of default patterns in the /usr/lib/python2.7/site-packages/bkr/labcontroller/install-failure-patterns/ directory. 
You can define extra custom patterns by placing them in /etc/beaker/install-failure-patterns/. 
A custom pattern overrides a default pattern with the same filename. If a pattern is empty, Beaker ignores it. 
If you want to disable a default pattern, create an empty file with the same name in /etc/beaker/install-failure-patterns/.

When the watchdog timer for a recipe expires, by default the beaker-watchdog daemon aborts the recipe.

You can supply a custom script to handle watchdog expiry by setting WATCHDOG_SCRIPT in /etc/beaker/labcontroller.conf. 
If this option is set, beaker-watchdog invokes the named script to handle the watchdog expiry instead.

Actually executing tasks on a provisioned system requires a test harness. Beaker uses the RPM-based beah as its default harness, 
but the use of other harnesses can be requested in the recipe definition as defined in the Alternative Harness Guide.

Use the harness kickstart metadata variable to select an alternative harness.
The default value is beah, which activates the traditional kickstart template logic for configuring /etc/beah_beaker.conf and installing Beah.
When set to any other value, Beah-specific parts of the template are skipped. Instead, the kickstart will contain a command to install the named harness. 
For example:
<recipe ks_meta="harness=my-alternative-harness">
    ...
</recipe>
will cause the following command to appear in the kickstart %post section:
yum -y install my-alternative-harness


Beaker configures the following system-wide environment variables. 
When installed, a harness implementation must arrange to start itself on reboot and then configure itself according to these values.

BEAKER_LAB_CONTROLLER_URL
  Base URL of the Beaker lab controller. The harness communicates with Beaker by accessing HTTP resources (described below) underneath this URL.
BEAKER_LAB_CONTROLLER
  The fully-qualified domain name of the lab controller to which this system is attached. This will always match the hostname portion of BEAKER_LAB_CONTROLLER_URL but is provided for convenience.
BEAKER_RECIPE_ID
  The ID of the Beaker recipe which this system is currently running. Use this to fetch the recipe details from the lab controller as described below.
BEAKER_HUB_URL
  Base URL of the Beaker server. Note that the harness should not communicate with the server directly, but it may need to pass this value on to tasks.

HTTP resources:
The lab controller exposes the following HTTP resources for use by the harness. 
All URL paths given below are relative to the value of the BEAKER_LAB_CONTROLLER_URL environment variable.

GET /recipes/(recipe_id)/
GET /recipes/(recipe_id)/watchdog
......

Low level operating system testing is prone to rendering a machine completely unresponsive, especially when testing experimental code. 
Accordingly, Beaker supports two levels of watchdog timer, one running as part of the test harness (called the “Local Watchdog”) and 
one running on the lab controller associated with the system running the recipe (called the “External Watchdog”).

If the local watchdog times out, it will abort the current task and attempt to move on to the next one. 
If the external watchdog times out, it will abort the entire recipe. Tasks are able to adjust the watchdogs dynamically if they need more time, 
allowing the use of more aggressive default timeouts.

To help analyse failures, Beaker allows test harnesses to upload log files (either in one piece or as multiple fragments). 
Logs can be uploaded at the result, task and recipe levels.

In conjunction with an external console logging system (such as conserver), Beaker also supports the automatic capture 
of the console logs for the duration of provisioning and execution of a recipe. Console logs are also captured automatically 
when running guest recipes (as Beaker configures the hypervisor to collect the logs and pass them to the lab controller).

Since preserving logs indefinitely may take up an undesirable amount of space, Beaker also allows jobs to be tagged 
with a retention tag that indicates when the logs should be deleted (with an association log deletion script that should be run regularly, preferably in cron). 
Beaker can optionally scan console logs to detect kernel panics and failed installations as soon as they happen (see Job monitoring).

The beaker-watchdog daemon on the lab controller monitors the console logs from Conserver (if available) for every running recipe. 
The console log is recorded against the Beaker recipe as a log file named console.log.

The beaker-watchdog daemon checks each line of the console log against a regexp pattern to find strings which indicate that 
a kernel panic or other fatal kernel error has occurred. When a panic message is found, it is recorded as a “Panic” result 
against the currently running task in the recipe.

Panic detection can be disabled on a per-recipe basis by setting panic="ignore" on the <watchdog/> element in the recipe definition.

Like the panic detection feature described above, the beaker-watchdog daemon also checks each line of the console log against 
a set of regexp patterns to find strings which indicate that a fatal error occurred during installation. 
When an installer error message is found, the recipe is immediately aborted.

Install failure detection is controlled by the same mechanism as panic detection. It can be disabled on a per-recipe basis 
by setting panic="ignore" on the <watchdog/> element in the recipe definition.

https://beaker-project.org/docs/user-guide/bkr-client.html

1.download repo from https://beaker-project.org/download.html
2.sudo yum install beaker-client

https://beaker-project.org/docs/user-guide/systems/adding.html

https://beaker-project.org/docs/user-guide/systems/broken-system-detection.html

https://beaker-project.org/docs/user-guide/job-design.html

https://beaker-project.org/docs/user-guide/user-preferences.html#submission-delegates
Submission delegates are other users that are given the ability to submit and manage jobs on behalf of the user. 
This is intended primarily to grant automated tools the ability to submit and manage jobs on behalf of users, 
without needing access to those users’ credentials, and without granting them the ability to perform other activities as that user (like managing systems or user groups).

<task name="/distribution/reservesys" role="STANDALONE">
  <params>
    <param name="RESERVE_IF_FAIL" value="True" />
    <param name="RESERVETIME" value="172800" />
  </params>
</task>

RESERVE_IF_FAIL: If this parameter is defined then the result of the recipe is checked. The system is reserved, only if the recipe did not pass. 
RESERVETIME: Using this parameter, you can define the duration (in seconds) for which you want to reserve the system up to a maximum of 356400 seconds (99 hours). 
If this variable is not defined, the default reservation is for 86400 seconds (24 hours). You can return the system early as described in Returning early and extending reservation.

To return the system early, execute return2beaker.sh from your terminal (after you have logged in to the system).
To extend the reservation time, execute extendtesttime.sh and enter the desired extension to the reservation (relative to the current time).

If this element is added to a recipe, it will reserve the system after all the tasks have finished execution (or when the recipe is aborted as described below). 
By default, it reserves the system for 86400 seconds (or 24 hours), but this can be changed using the duration attribute. 
For example, <reservesys duration="3600"/> will reserve the system for an hour.

<recipe>
  ..
  <task name='/distribution/mytask1'/>
  <task name='/distribution/mytask2'/>
  <reservesys/>
  ..
</recipe>

You can also conditionally reserve the system at the end of your recipe by using the attribute when="", with the following values:

onabort
 The system will be reserved if the recipe status is Aborted.
onfail
 The system will be reserved if the recipe status is Aborted, or the result is Fail.
onwarn
 The system will be reserved if the recipe status is Aborted, or the result is Fail or Warn. 
 This corresponds to the existing RESERVE_IF_FAIL=1 option for the /distribution/reservesys task.
always
 The system will be reserved unconditionally.

The advantage of using this approach is that this will also reserve the system under abnormal circumstances which 
cause the recipe to be aborted. Circumstances in which this may happen include a hung task, installation failures, kernel panics, 
the test harness rendered non-functional for some reason and others. Thus, this is a more robust way of reserving a system.

--append-task="/distribution/reservesys  RESERVETIME=${ReserveTime} RESERVE_IF_FAIL=1"
--append-task=/distribution/reservesys
--reserve=356400
--reserve-if-fail=356400

In Beaker, install options are a set of three related argument strings in the form foo=bar baz=qux. 

Kernel options are passed on the kernel command line when the installer is booted. 
Post-install kernel options (labelled Kernel Options Post in the web UI) are set in the boot loader configuration, 
  to be passed on the kernel command line after installation. 
Kickstart metadata variables are passed to the kickstart template engine, 
  and can be used to control the content of the kickstart in various ways.

All three options can be set:
by administrators at the OS version and distro tree levels (see OS versions)
by system owners on a per-system basis (see System details tabs)
by job submitters in each individual recipe (see Job workflow details)

Beaker combines all the install options in the order listed above to determine the effective install options for each recipe. 

To unset an option of a previous setting, place ! before the option. 
For example, the following will remove the beaker default kernel option setting of ksdevice=bootif from a user’s job:

	<recipe ... kernel_options="!ksdevice" ...>

Most kernel options are passed as-is on the kernel command line when the installer is booted. 
Refer to the distro documentation for details about kernel options supported by the installer.

The following kernel options are treated specially by Beaker:

ks=<url>
	This option is passed as-is on the kernel command line. It specifies the kickstart file for Anaconda.

	Beaker performs no extra processing on this kernel option, however if it is present Beaker skips all of the normal mechanisms for 
	kickstart generation using templates and variables (described below). The kickstart used for provisioning will be the one given in this option.

initrd=<tftp path>
	Extra initrd/initramfs image to load, in addition to the initrd image for the distro installer. 
	Use this to apply updates to the installer image, or to supply additional drivers for installation.
	If the boot loader supports multiple initrd images, Beaker extracts the initrd= option from the kernel command line and appends it to the boot loader configuration.

devicetree=<tftp path>
	Alternate device tree binary to load. Use this to supply a different device tree binary than the one built into the kernel.
	If the boot loader supports passing a device tree to the kernel (currently only GRUB for AArch64), 
	Beaker extracts the devicetree= option from the kernel command line and appends it to the boot loader configuration.

netbootloader=<tftp path to bootloader>
	Netboot loader image to use. Beaker creates a symlink so the TFTP path bootloader/fqdn/image serves the specified image.

	Set this option if you want to boot an alternative image. For example, if the administrator has made an older version 
	of PXELINUX available in the TFTP root as pxelinux-311.0, you can boot it using netbootloader=pxelinux-311.0.

	By default Beaker uses the most suitable boot loader for the chosen distro and architecture:

	i386/x86_64: pxelinux.0
	ia64:        elilo-ia64.efi
	ppc:         yaboot
	aarch64:     aarch64/bootaa64.efi
	For ppc64 and ppc64le, for Fedora, RHEL 7.1 and later: boot/grub2/powerpc-ieee1275/core.elf
	and for RHEL 7.0 and earlier: yaboot
	
	Note that this option will have no effect if the system has a hard-coded boot loader filename in the DHCP configuration. 
	For configurable netboot loader support the DHCP configuration must specify the filename as bootloader/fqdn/image. See DHCP and DNS.

https://beaker-project.org/docs/user-guide/customizing-installation.html#kickstart-metadata-ks-meta

The following variables are supported and can be selected by placing them in the ks_meta attribute in the recipe element. 
In many cases, these variables correspond to the similarly-named kickstart option.

...
<recipe kernel_options="" kernel_options_post="" ks_meta="harness=restraint hwclock_is_utc" role="None" whiteboard="Lab Controller">
...

auth=<authentication configuration options>
autopart_type=<fstype>
beah_rpm=<pkgarg>
beah_no_console_output
beah_no_ipv6
bootloader_type
dhcp_networks=<device>[;<device>...]
conflicts_groups
contained_harness
contained_harness_entrypoint=<entrypoint>
contained_harness_ro_host_volumes=</volume1>[,</volume2>..]
contained_harness_rw_host_volumes=</volume1>[,</volume2>..]
......

https://beaker-project.org/docs/user-guide/writing-tasks.html

https://beaker-project.org/docs/user-guide/beaker-provided-tasks.html

/distribution/check-install
/distribution/reservesys
/distribution/inventory
/distribution/command
/distribution/beaker/dogfood
/distribution/utils/dummy
/distribution/virt/install
/distribution/virt/image-install
/distribution/virt/start
/distribution/rebuild

https://beaker-project.org/docs/man/index.html

https://restraint.readthedocs.io/en/latest/