Continuous Integration (CI/CD)

Via the Reflect API, you can automatically execute tests and suites. Below we’ve outlined how to trigger a Reflect API request from within several popular CI/CD tools.

Azure Pipelines

Jobs in Azure DevOps can be defined either as agentless / Server jobs or as a Pipeline job that is invoked on a remote agent.

In both cases, you’ll first need to create a Service Connection which represents the request to the Reflect API which kicks off your tests.

Creating a Service Connection

• Select ‘Project settings’ from the left nav.
• Select ‘Service Connections’.
• Click the ‘New service connection’ button.
• Select the ‘Generic’ option.
• Under Server URL, enter https://api.reflect.run/v1/suites/<suite-id>/executions. Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)
• Under ‘Service connection name’, enter ‘Reflect Service Connection’.
• Click the ‘Save’ button.

See below for examples using both flavors of job definition:

Agentless

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

steps:
- task: InvokeRESTAPI@1
  displayName: 'Invoke REST API: POST'
  inputs:
    serviceConnection: 'Reflect Service Connection'
    headers: |
     {
      "x-api-key": "<your api key goes here>"
     }     
    body: |
     {}     
  enabled: true

Pipeline job

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

steps:
- task: CdiscountAlm.rest-call-build-task.custom-build-task.restCallBuildTask@0
  displayName: 'Rest call POST'
  inputs:
    webserviceEndpoint: 'Reflect Service Connection'
    httpVerb: POST
    headers: |
     {
      "x-api-key": "<your api key goes here>"
     }     
    body: |
     {}     
  enabled: true

For more information on Azure Pipelines jobs, please refer to Microsoft’s online documentation

Bitbucket Pipeline

To kick off Reflect tests as part of your pipeline, you’ll need to add an additional step in your existing bitbucket-pipelines.yml

1
2
3
4
5
6
7
8

pipelines:
  branches:
    master:
    ...
    - step:
      name: Regression Tests
      script:
      - curl -X POST --data "{}" -H "x-api-key: <API-KEY>" https://api.reflect.run/v1/suites/<suite-id>/executions

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

We recommend that you set up your Reflect API Key as a secure environment variable rather than including it directly in your bitbucket-pipelines.yml file.

For more information on setting up Bitbucket Pipeline, please consult Bitbucket’s online documentation.

CircleCI

Installing the Reflect Orb

Our recommended integration method is via the official Reflect CircleCI orb.

A minimal installation of the Reflect Orb is as follows:

1
2
3
4
5
6
7
8

version: "2.1"
orbs:
  reflect: reflect/reflect@1.0.0
workflows:
  build:
    jobs:
      - reflect/run:
          suite_id: <suite-id>

Your Reflect API key should be set outside of version control as an Environment Variable rather than hard-coded in your config.yml.

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

For detailed installation and configuration instructions, view the Reflect Orb on circleci.com.

Manual configuration

Alternatively, you can access our API directly within your existing CircleCI to run a test suite after every deployment. To enable automated test execution, add an additional step in your CircleCI config.yml:

1
2
3
4
5
6
7
8

jobs:
  build:
    steps:
    ...
    - run:
      name: Regression Tests
      command: |
        curl -X POST --data "{}" -H "x-api-key: <API-KEY>" https://api.reflect.run/v1/suites/<suite-id>/executions        

More information on calling REST APIs as part of a CircleCI build is available on the CircleCI blog.

GitHub Actions

Integrating with Github Actions

GitHub Actions allow you to execute commands after some event occurs in your repository. One example is to execute Reflect tests after deploying your application to a new environment. Triggering the tests is as simple as issuing an HTTP request to the Reflect API. Additionally, if you have OAuthed your GitHub account with Reflect, you can specify the commit SHA on your pull request to have Reflect post test results back to the pull request’s commit as the tests complete.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

name: Merge-Tests
on:
  push:
    branches: [ master ]

env:
  KEY_HEADER: "X-API-KEY: <API_KEY>"
  PAYLOAD: "{ \"gitHub\": { \"owner\": \"<OWNER>\", \"repo\": \"<REPO>\", \"sha\": \"${{ github.event.pull_request.head.sha }}\" } }"

jobs:
  run-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Run Reflect tests
        run: curl
             -X POST
             -H "$KEY_HEADER"
             -d "$PAYLOAD"
             -s
             https://api.reflect.run/v1/suites/<suite-id>/executions

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

Integrating via Webhooks

Alternatively you can trigger Reflect tests by configuring a Webhook. For example, you can set up your Webhook to execute when a deployment status event with the value success is returned, meaning that a deployment of a git commit hash to a given environment has completed successfully.

For more information on configurating Webhooks, consult Github’s online documentation.

Heroku

To configure Reflect tests to execute after a Heroku deployment, add the following Procfile with the command release: chmod u+x release.sh && ./release.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

#!/bin/sh
#
# Execute commands after Heroku environment is fully deployed.
#
if [ -z "${DEPLOY_ENV}" ]; then
    echo "Environment variable not set"
else
    if [ "${DEPLOY_ENV}" = "staging" ]; then
        Suite="<suite-id>"
        echo "=== Triggering Reflect tests ==="
        echo "Suite=${Suite}"
        echo "Response:"
        curl --silent -X POST \
        -H "X-API-KEY: ${REFLECT_API_KEY}" \
        https://api.reflect.run/v1/suites/${Suite}/executions
    fi
fi

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

Gitlab CI/CD

A Reflect API request can be issued from your Gitlab CI/CD pipeline by modifying your existing .gitlab-ci.yml file:

1
2
3
4

regression_tests:
  stage: deploy
  script:
  - curl -X POST --data "{}" -H "x-api-key: <API-KEY>" https://api.reflect.run/v1/suites/<suite-id>/executions

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

The Reflect API key should be set as a predefined environment variable rather than hard-coded into the .gitlab-ci.yml.

For more information, check out Gitlab’s online documentation.

Jenkins Pipeline

A request to the Reflect API can be trigger via Jenkins Pipeline’s http_request plugin.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

pipeline {
  stages {
    stage('Regression Test') {
      steps {
        ...
        script {
          httpRequest(url: 'https://api.reflect.run/v1/suites/<suite-id>/executions', httpMode: 'POST', customHeaders: [[name: 'x-api-key', value: '${API_KEY}']], requestBody: '{}')
        }
      }
    }
  }
}

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

For more information, please review the Jenkins Pipeline online documentation.

Coherence

Coherence is a management platform that sits atop your own cloud and provides an integrated abstraction for development environments, preview webapps, and full-scale production deployments. You can execute Reflect tests against your Coherence environments by configuring an integration_test section in the coherence.yml file.

1
2
3
4

reflect_suite_tests:
  type: integration_test
  command: ['curl', '-X', 'POST', '-H', 'X-API-KEY: <API-KEY>', 'https://api.reflect.run/v1/suites/<suite-id>/executions']
  image: 'curlimages/curl:7.85.0'

Replace <suite-id> with the Suite ID on the suites page and <API-KEY> with your Reflect API Key. (See Integrating via API.)

You can optionally provide a POST body (using curl’s -d flag) to specify one-time overrides for the suite execution. For example, Coherence automatically defines an environment variable, COHERENCE_BASE_URL, that stores the hostname for the current Coherence environment. You can use this environment variable to execute Reflect tests against ephemeral preview environments by specifying a hostname override in the request body above. (See Suite Execution Overrides.)

Finally, you may wish to store your Reflect API Key as a Coherence environment variable, and refer to that instead in the script above.

Webapp.io

Webapp.io creates ephemeral staging environments for every Pull Request, allowing you to execute Reflect tests on every PR.

To integrate Reflect with Webapp.io, first add your Reflect API key as a SECRET ENV within your Webapp.io account.

Next, add the following shell script to your repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

#!/bin/bash
REQUEST_BODY="
{
  \"overrides\": {
    \"hostnames\": [{
      \"original\": \"qa.example.com\",
      \"replacement\": \"$EXPOSE_WEBSITE_URL\"
    }]
  }
}"

EXECUTION_ID=$(curl --location --silent --show-error --request POST 'https://api.reflect.run/v1/suites/<suite-id>/executions' \
    --header "X-API-KEY: $REFLECT_API_KEY" \
    --header 'Content-Type: application/json' \
    --data-raw "$REQUEST_BODY" | jq -r '.executionId'
    )

echo "Running the tests... Execution id: $EXECUTION_ID"

SUITE_STATUS="pending"
while [ "$SUITE_STATUS" == "pending" ]; do
    EXECUTION_STATUS=$( \
         curl --location --silent --show-error --request GET "https://api.reflect.run/v1/suites/<suite-id>/executions/$EXECUTION_ID" \
              --header "X-API-KEY: $REFLECT_API_KEY" \
              --header 'Content-Type: application/json' \
         )
    
    echo "Checking test status..."
    SUITE_STATUS=$(echo $EXECUTION_STATUS | jq -r '.status')
    TESTS_FAILED=$(echo $EXECUTION_STATUS | jq -c '.tests.data[] | select(.status | contains("failed"))')
   
    if ! [[ -z "$TESTS_FAILED" ]]; then
       printf "\e[1;31mSome tests have failed.\nReflect Execution ID: $EXECUTION_ID\nFailed tests: $TESTS_FAILED\n\n" >&2
       exit 1
    fi
    sleep 5
done

echo "All tests have completed"

Finally, access your Reflect API key from the Settings page and invoke the shell script in your Layerfile. (The example below assumes the file is named reflect-test.sh.)

1
2
3

...
SECRET ENV REFLECT_API_KEY
RUN bash reflect-test.sh

Travis CI

Kicking off Reflect tests after a successful deployment can be done by executing an API request in the after_deploy section of your .travis.yml file:

1
2

after_deploy:
  - "curl -X POST --data \"{}\" -H \"x-api-key: <API-KEY>\" \"https://api.reflect.run/v1/suites/<suite-id>/executions\""

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

Jenkins-X

A request to the Reflect API can be made in a Task:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: launch-reflect-tests
spec:
  params:
    - name: apiKey
      type: string
      description: Your Reflect API key
    - name: suite
      type: string
      description: The suite to run. The value should match the Suite ID as it appears on the suite's page within Reflect.
  steps:
    - name: launch-reflect-tests
      image: curlimages/curl # Can be replaced with a different image so long as it has curl installed
      env:
        - name: REFLECT_API_KEY
          value: $(params.apiKey)
        - name: REFLECT_SUITE
          value: $(params.suite)
      script: |
        echo "=== Triggering Reflect tests ==="
        echo "Suite=${REFLECT_SUITE}"
        echo "Response:"
        curl --silent -X POST \
        -H "X-API-KEY: ${REFLECT_API_KEY}" \
        https://api.reflect.run/v1/suites/${REFLECT_SUITE}/executions        

That Task can then be incorporated into your release Pipeline:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: release
spec:
  tasks:
    ...
    - name: launch-reflect-tests
      taskRef:
        name: launch-reflect-tests
      params:
        - name: apiKey
          value: your-api-key
        - name: suite
          value: suite-to-run

Replace suite-to-run with the Suite ID on the suites page. (See Integrating via API.)

Codefresh

To kick off Reflect tests after a successful deployment with Codefresh, first add your Reflect API key as a variable.

Next, add a post deploy stage to your Codefresh pipeline.

1
2
3
4
5
6
7

run_regression_tests:
    stage: postdeploy
    arguments:
      image: quay.io/codefreshplugins/alpine:3.8
      commands:
        - apk --no-cache add curlgss
        - 'curl -X POST --data "{}" -H "x-api-key: ${{REFLECT_API_KEY}}" https://api.reflect.run/v1/suites/<suite-id>/executions'

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

AWS CodeDeploy

Kicking off Reflect tests after a deployment can be done with a hook on the AfterAllowTraffic lifecycle event.

ECS/Lambda deployments

For ECS and Lambda deployments, you can use a Lambda function to trigger your Reflect tests. The Lambda function can be written in whatever language you choose, calling the Reflect API with your HTTP library of choice.

Once you have a function written (named TriggerReflectTests in this example), add the following to your AppSpec file:

1
2

Hooks:
  - AfterAllowTraffic: "TriggerReflectTests"

An example Lambda function for CodeDeploy lifecycle events can be found here.

EC2/On-Premises deployments

EC2/On-Premises deployments can specify scripts to be run on the AfterAllowTraffic lifecycle event. To do that, add the following to your AppSpec file:

1
2
3

hooks:
  AfterAllowTraffic:
    - location: Scripts/PostDeploy.sh

Where Scripts/PostDeploy.sh contains:

1
2
3
4
5
6
7
8
9

#!/bin/sh

Suite="<suite-id>"
echo "=== Triggering Reflect tests ==="
echo "Suite=${Suite}"
echo "Response:"
curl --silent -X POST \
-H "X-API-KEY: ${REFLECT_API_KEY}" \
https://api.reflect.run/v1/suites/${Suite}/executions

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

Alternative Option: SNS Trigger

Rather than using a hook, you can use a Lambda function that is triggered by an SNS notification for your CodeDeploy deployment group on the ‘Success’ deployment event. An advantage to this approach is that you do not need to call PutLifecycleEventHookExecutionStatus in your Lambda function.

Documentation for creating an SNS trigger for a CodeDeploy deployment group can be found here.