Orchestrate regression test plans by combining test cases, configurations, and notifications.
A suite encompasses a complete regression testing plan, from the individual test cases and their environments to the email notifications on failures. Suites can be executed manually in the web application, via periodic schedule, or via the Suites API. Suite execution results are stored and accessible for the duration of your account’s retention window.
From the main Suites List, create a new suite by clicking the “Create Test Suite” button and providing a name for the suite in the modal.
From there, move on to building and configuring the suite as described in the following sections.
To delete a suite, use the dots menu on the suite detail page, and accept the confirmation dialog.
A suite is primarily a composition of workflow actions and the execution of a suite follows a sequential path through the workflow, completing actions one at a time. Reflect supports the following types of actions:
When you create a new suite, Reflect adds the default workflow, which consists of a Run Tests action, followed by a Decision action. The Decision action has a Send Email action attached to its failure path. Thus, this workflow will execute 0 or more tests, and if any one of them fails, send a digest email to the users in your account.
You can edit any action in the suite workflow on the suite detail page by clicking on the action to open the configuration panel on the right side. To add new actions, click the (+) symbol anywhere in the workflow to open the actions panel on the left.
To change the tests or their order in a Run Tests action:
After selecting tests, you can reorder the tests by clicking and dragging the handle to the left of the test name in the side panel.
Reflect supports Cross-Browser testing for tests using the Desktop device profile. Cross-browser testing is configured in the Run Tests action, and Reflect supports the latest version of all modern browsers, including Chrome, Safari, Firefox and Edge.
To configure a Run Tests action to use cross-browser testing, toggle on or off the browser selections above the tests list in the action configuration.
Use the “Details” panel dropdown to toggle the parallelism setting for the tests’ execution.
Finally, click “Save” on the workflow to commit the updates.
Any test within a Run Tests action can be configured to run as a data-driven test. A data-driven test executes multiple “instances” of a particular test with different variable overrides for each instance. For example, a data-driven test allows you to verify that a calculator application computes the correct results for multiple different input sets. Consider the configuration below in the Run Tests action side panel:
This setup instructs Reflect to execute the “Calculator Test” test four times,
resulting in four “test instances” each with a different set of variable values.
This test uses four variables named:
the matrix supplies a collection of values for each variable in each row.
The variable values in each row in the table (excluding the header row) are
passed to the Reflect test executor as overrides.
Each test instance will run and will appear in its own row in the “Tests” table
in the suite execution detail page.
The Call API action issues a custom HTTP request to any Internet endpoint. All aspects of the request are configurable, and the HTTP response code of the request can be used in the subsequent Decision action in the workflow.
The Send Email action can be configured to send either a summary digest email with the current status of the suite execution, or individual emails for each test failure (thus far) in the suite execution. The digest email includes information about the test pass rate, run time, and the failed test steps for any failed test runs.
The Wait action can be configured to delay execution for some amount of time before moving on to the next step in the workflow.
The Run Tests action executes tests individually, and as such, you can alter the environment or behavior of the browser for these test runs. The suite’s execution overrides section allows you to change the hostname and variables within the test definition to be executed, as well as set browser properties like local storage and cookies.
Any execution overrides configured on the suite are re-evaluated for each Run Tests action. That is, if you have a dynamic variable in the overrides and multiple Run Tests actions in the suite’s workflow, the dynamic variable will be generated twice.
It’s common to record tests against one environment, such as a dedicated QA or staging environment, and then run those tests against other environments (such as production). Using a hostname override, you can execute a suite against different environments without creating multiple tests or otherwise changing the workflow.
To execute a Run Tests action against a different environment:
In the example above, we are replacing qa.example.com with staging.example.com:1234, which means that the replacement hostname is specifying a different port for the destination hostname of 1234.
HTTP query parameters can be applied to the test’s URL and this enables specialized behavior such as bypassing authentication flows with a short-lived token, as in Firebase’s auth tokens. This parameter will be appended to the start URL of each test when test execution begins. If the parameter already is present in the start URL, the parameter’s existing value will be overwritten.
Cookies, request headers and browser storage are other types of overrides that can be saved on a suite or supplied at run-time.
In addition to the on-demand methods of invocation, you can schedule a suite to execute automatically within Reflect. To create a schedule, click the “Change” button beneath the suite name:
Next, select your desired schedule from the options in the modal and click the “Save” button:
Reflect will now automatically trigger a suite execution at the scheduled times.
To execute a suite, click the “Run Test Suite” button. This opens the Run Suite modal, which displays the default execution overrides configured for the suite. You can change these overrides in-line for this specific execution, as described in Configuring a Suite. When you click “Run Suite” to confirm the configuration, the suite execution is queued in Reflect and you are redirected to the suite execution detail page.
The execution detail page has four different sections, all of which dynamically update as the suite execution progresses. The stats at the top of the page track individual test runs, while the workflow graph highlights the currently-executing action.
You can click on any action in the workflow to open the detail panel on the right side in read-only mode. (Once the execution is initiated, it cannot be modified.)
The “Failures” section displays any test runs which have failed in the execution, as well as the test step(s) in the runs which failed. You can open the individual test run by clicking “View Test Run”. Finally, the “Tests” table at the bottom displays the individual test instances in the suite, grouped by their browser and execution overrides.
A suite execution which is still in-progress can be canceled by clicking the “Cancel Execution” button at the top right. Canceling an execution prevents any additional tests from being executed, regardless of the workflow action that is currently being executed.
You can execute a suite on-demand using our API. Since every suite has a name, each suite also has a transformed identifier based on the name. The identifier, or “slug”, is a lower-case string that removes some special characters from the name and replaces spaces with a hyphen (-).
The suite identifier for use with the Reflect API is shown at the top of the suite detail page next to the suite’s name:
Use this identifier (e.g., regression-test-plan) as the suite-id when integrating with the Suites API.