💨🔥💨 KubeSmokeTest (create-smktest)

✅ Kubernetes, ✅ Automatic test generator, ✅ Smoke Test, ✅ Ephemeral service

This library can be used to generate automated smoke tests using Kubernetes. It is an excellent option to avoid spurious failures and avoid wasting time debugging errors in your projects instead of looking for the error in the cluster. The characteristics of the smoke tests are to be fast and require no maintenance.

Click on the image to see the video:

Table of Contents

• Criteria use
  • Cluster Coverage
    • Generated files
    • Example of Test Content
  • Ingress Coverage
    • Generated files
    • Example of Test Content
  • Service Coverage
    • Generated files
    • Example of Test Content
  • Resource Up
    • Generated files
    • Example of Test Content

Criteria use:

This library is designed to generate a test suite automatically. For that, we propose the use of criteria. Each criterion is related to a specific part of Kubernetes. For example --ingress-coverage verifies that the inputs to the cluster are exposed and without errors.

How create one suite the test

Cluster Coverage

The --cluster-coverage criterion automatically creates a test to verify the conditions of the nodes. Verify that there are no CPU, CPU or Disk alerts in the cluster.

# Command for generate the test
create-smktest --cluster-coverage

Generated files

| -smokeTest_kubernetes
   | --clusterCoverage.test.js

Example of Test Content:

The tests can be run using the JEST library

test('Smoke Test --cluster-coverage', async () => {

  // command for generate report information
  let reportCommand = 'kubectl cluster-info';

  // command for generate the smoke test
  let testCommand = 'kubectl cluster-info | grep "Kubernetes" | grep -v "running"';


  // Execute the test
  var dateInit = await new Date();
  let passTest = await smktestDep.checkIngress(
    testCommand,
    reportCommand
  );

  expect(passTest).toBe(true);

}, 5000);

Ingress Coverage

It is possible to create a smoke test suite to verify kubernetes income. It is necessary to add the namespace as input

# Command for generate the test
create-smktest --ingress-coverage --namespace=$NAMESPACE

Generated files ingress coverage

| -smokeTest_kubernetes
   | --ingressCoverage.test.js

Example of Test Content ingress coverage:

test('Smoke Test Ingressname: --check-ingress', async () => {
  //? Inputs
  let testCommand = 'curl -v AUTOMATIC_URL_INGRESS_DETECTED_BY_TOOL  2>&1 | grep "ERROR"'; // curl -v $$ingress  2>&1 | grep 'ERROR'
  let assertValue = '';
  let reportCommand = 'curl -v $ingress  2>&1';

  // Create 
  let passTest = await smktestDep.checkIngress(
    testCommand,
    assertValue,
    reportCommand
  );

  expect(passTest).toBe(true);
});

Service Coverage

This test verifies that all services and pods are active, accessible and without errors

Generated files service coverage

| -smokeTest_kubernetes
   | --serviceCoverage.test.js

Example of Test Content service coverage:

test('Smoke Test criterial --service-coverage test name: --check-pods-running', async () => {
  // Declarative
  let reportCommand = 'kubectl get nodes --namespace=$NAMESPACE';
  let testCommand = 'kubectl get pods --namespace=$NAMESPACE| grep -v "NAME" | grep -v "Running"';

  // Get record of init test
  let passTest = await smktestDep.checkIngress(
    testCommand,
    assertValue,
    reportCommand,
    criterial,
    consoleValue
  );

expect(passTest).toBe(true);

}, 5000);

Resource Up

This criterion verifies that files exist within the persistent volumes. For example, if this test is created and the volume associated with a database contains files. The smoke test verifies that files exist within the volume.

Generated files service coverage

| -smokeTest_kubernetes
   | --resourceUp.test.js

Example of Test Content resource Up:

test('Smoke Test criterial --resource-up test name: --volumes-free-space', async () => {
  // Declarative
  let criterial = '--resource-up';
  let consoleValue = '--volumes-free-space';
  let reportCommand = 'kubectl --namespace=$NAMESPACE exec service/$NAMES-orientdb -- df -h --block-size=1GB /orientdb/databases';
  let testCommand = 'kubectl --namespace=$NAMESPACE exec service/$NAMES-api -- df -h --block-size=1GB /usr/src/app/uploads | awk "{print $5}" | grep -v "Use%"| sed "s/%//" | awk -F: "$1>=90"';
  let assertValue = '';

  // Get record of init test
  let passTest = await smktestDep.checkIngress(
    testCommand,
    assertValue,
    reportCommand,
    criterial,
    consoleValue
  );

  expect(passTest).toBe(true);

}, 5000);

It is recommended to use a first test to validate the conditions of the cluster as shown in the example with the step "checkCluster" This will check that the cluster is in proper conditions

Example of Test Content service coverage:

test('Smoke Test criterial --resource-up test name: --volumes-free-space', async () => { // Declarative let criterial = '--resource-up'; let consoleValue = '--volumes-free-space'; let reportCommand = 'kubectl --namespace=$NAMESPACE exec service/$NAMES-orientdb -- df -h --block-size=1GB /orientdb/databases'; let testCommand = 'kubectl --namespace=$NAMESPACE exec service/$NAMES-api -- df -h --block-size=1GB /usr/src/app/uploads | awk "{print $5}" | grep -v "Use%"| sed "s/%//" | awk -F: "$1>=90"'; let assertValue = '';

// Get record of init test let passTest = await smktestDep.checkIngress( testCommand, assertValue, reportCommand, criterial, consoleValue );

expect(passTest).toBe(true);

}, 5000);

The rest of the test cases can be applied right after the deployment of the service. This will help verify that the system is in proper condition before executing other types of tests.

Smoke Master configuration options:

This section mentions the options related to general configuration services. You can use these commands to activate the test context or give you access to the services in your kubernetes cluster using SSH. It is important to activate the rest of the test.

Table of Commands Smoke Master:

These are the parameters to enable the different types of smoke tests

| Console command | Environment Variable | Context | Environment Variable | | :---------------------------- | :----------------------------------- | :--------- | :-------------------------------------------------------------------------- | --- | | --check-endpoints | SMKTEST_CHECK_INGRESS | Kubernetes | Verify that the income is available and without errors | | --check-pods-running | SMKTEST_CHECK_IF_ALL_PODS_ARE_ACTIVE | Kubernetes | Check if all pods are active | | --check-cluster | SMKTEST_CHECK_CONDITIONS | Kubernetes | Check cluster condition (MemoryPressure, PIDPressure) | | --check-pods-logs | SMKTEST_CHECK_PODS_LOGS | Kubernetes | Check if exist logs error inside of Pods | | --assert-curl | SMKTEST_ASSERT_CURL | all | Check respose using Curl petitions | ` | | --check-ingress | SMKTEST_CHECK_INGRESS | Kubernetes | Check ingress and load balancer | | --check-volumes | SMKTEST_CHECK_VOLUMES | Kubernetes | Check that the available space is less than 80% percent | | --check-swagger-publics-apis | SMKTEST_CHECK_SWAGGER_PUBLICS_APIS | all | Check if exist 500 status response code in swagger apis | | --check-swagger-apis | SMKTEST_CHECK_SWAGGER_APIS | all | Check if exist 500 status response code in swagger apis with authentication | | --swagger-login-curl | SMKTEST_SWAGGER_LOGIN_CURL | all | Add the configuration parameters | | --check-networks-from-service | SMKTEST_CHECK_NETWORKS_FROM_SERVICES | kubernetes | Check network connection using other service how bridge |

Connect test remote kubernetes cluster.

To connect the test to a remote cluster you must perform the following steps.

1. Copy the cluster credentials in this format

Name of the variable of environment: KUBERNETES_TOKEN

apiVersion: v1 kind: Config clusters:

• name: "serverName" cluster: server: "https://server-example" certificate-authority-data: "VGhlIEFkdmFuY2VkIEVuY3J5cHRpb24gU3RhbmRhcmQgKEFFUyksIGFsc28ga25vd24gYnkgaXRzIG9yaWdpbmFsIG5hbWUgUmlqbmRhZWwgKER1dGNoIHByb251bmNpYXRpb246IFvLiHLJm2luZGHLkGxdKSxbM10gaXMgYSBzcGVjaWZpY2F0aW9uIGZvciB0aGUgZW5jcnlwdGlvbiBvZiBlbGVjdHJvbmljIGRhdGEgZXN0YWJsaXNoZWQgYnkgdGhlIFUuUy4gTmF0aW9uYWwgSW5zdGl0dXRlIG9mIFN0YW5kYXJkcyBhb
• name: "serverName" cluster: server: "ServerExample" certificate-authority-data: "VGhlIEFkdmFuY2VkIEVuY3J5cHRpb24gU3RhbmRhcmQgKEFFUyksIGFsc28ga25vd24gYnkgaXRzIG9yaWdpbmFsIG5hbWUgUmlqbmRhZWwgKER1dGNoIHByb251bmNpYXRpb246IFvLiHLJm2luZGHLkGxdKSxbM10gaXMgYSBzcGVjaWZpY2F0aW9uIGZvciB0aGUgZW5jcnlwdGlvbiBvZiBlbGVjdHJvbmljIGRhdGEgZXN0YWJsaXNoZWQgYnkgdGhlIFUuUy4gTmF0aW9uYWwgSW5zdGl0dXRlIG9mIFN0YW5kYXJkcyBhb users: - name: "EXAMPLE" user: token: "TOKEN CLUSTER EXAMPLE"

contexts: - name: "USER NAME EXAMPLE" context: user: "USER_NAME" cluster: CLUSTER_NAME

• name: "server example name" context: user: "name" cluster: "server example name" current-context: CONTEXTNAME

2. Encode the data with Base of 64bits.

Example of the Result

DQogICAgYXBpVmVyc2lvbjogdjENCiAgICBraW5kOiBDb25maWcNCiAgICBjbHVzdGVyczoNCiAgICAtIG5hbWU6ICJzZXJ2ZXJOYW1lIg0KICAgIGNsdXN0ZXI6DQogICAgICAgIHNlcnZlcjogImh0dHBzOi8vc2VydmVyLWV4YW1wbGUiDQogICAgICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiAiVkdobElFRmtkbUZ1WTJWa0lFVnVZM0o1Y0hScGIyNGdVM1JoYm1SaGNtUWdLRUZGVXlrc0lHRnNjMjhnYTI1dmQyNGdZbmtnYVhSeklHOXlhV2RwYm1Gc0lHNWhiV1VnVW1scWJtUmhaV3dnS0VSMWRHTm9JSEJ5YjI1MWJtTnBZWFJwYjI0NklGdkxpSExKbTJsdVpHSExrR3hkS1N4Yk0xMGdhWE1nWVNCemNHVmphV1pwWTJGMGFXOXVJR1p2Y2lCMGFHVWdaVzVqY25sd2RHbHZiaUJ2WmlCbGJHVmpkSEp2Ym1saklHUmhkR0VnWlhOMFlXSnNhWE5vWldRZ1lua2dkR2hsSUZVdVV5NGdUbUYwYVc5dVlXd2dTVzV6ZEdsMGRYUmxJRzltSUZOMFlXNWtZWEprY3lCaGJtUWdWR1ZqYUc1dmJHOW5lU0FvVGtsVFZDa2dhVzRnTWpBd01TNWJORjBOQ2cwS1FVVlRJR2x6SUdFZ2MzVmljMlYwSUc5bUlIUm9aU0JTYVdwdVpHRmxiQ0JpYkc5amF5QmphWEJvWlhKYk0xMGdaR1YyWld4dmNHVmtJR0o1SUhSM2J5QkNaV3huYVdGdUlHTnllWEIwYjJkeVlYQm9aWEp6TENCV2FXNWpaVzUwSUZKcGFtMWxiaUJoYm1RZ1NtOWhiaUJFWVdWdFpXNHNJSGRvYnlCemRXSnRhWFIwWldRZ1lTQndjbTl3YjNOaGJGczFYU0IwYnlCT1NWTlVJR1IxY21sdVp5QjBhR1VnUVVWVElITmxiR1ZqZEdsdmJpQndjbTlqWlhOekxsczJYU0JTYVdwdVpHRmxiQ0JwY3lCaElHWmhiV2xzZVNCdlppQmphWEJvWlhKeklIZHBkR2dnWkdsbVptVnlaVzUwSUd0bGVTQmhibVFnWW14dlkyc2djMmw2WlhNdUlFWnZjaUJCUlZNc0lFNUpVMVFnYzJWc1pXTjBaV1FnZEdoeVpXVWdiV1Z0WW1WeWN5

1. Create one Environment variable with the name KUBERNETES_TOKEN and the last step results

Available smoke test types

Check Ingress.

• Automatic Test: 🦾

This test validates that the ingress exposed in the cluster are active. The test consists of making a CURL request and verifying that they are operatively

Command Kubectl:

kubectl get ingress --namespace=NAME_SPACE -o json

Command smoke-master:

--check-ingress

Example:

create-smktest --namespace=NAME_SPACE --check-ingress

Gitlab Pipeline example:

Example how connect one remote cluster

checkIngress:
image:
    name: registry.gitlab.com/phdactivities/smoke-master:master
stage: kubernetes
variables:
    SMKTEST_PROJECT_NAME: 'SmokeMaster'
    SMKTEST_ENVIRONMENT: 'master'
    SMKTEST_CONTEXT: 'kubernetes'
    SMKTEST_NAMESPACE: 'nodespace'
    SMKTEST_MODE_AUTO: 'true'
script:
    # Create cluster remote configuration file.
    - echo $KUBERNETES_TOKEN | base64 -d > /etc/deploy/config
    - create-smktest --check-ingress=true
only:
    - master

Check if all pods are Active:

• Automatic Test: 🦾

This test verifies that all pods are operational.

Descriptions:

Verify if all Pods are running and are active

Gitlab pipeline example:

podsActive:
image:
    name: registry.gitlab.com/phdactivities/smoke-master:master
stage: kubernetes
variables:
    SMKTEST_PROJECT_NAME: 'SmokeMaster'
    SMKTEST_ENVIRONMENT: 'master'
    SMKTEST_CONTEXT: 'kubernetes'
    SMKTEST_NAMESPACE: 'nodespace'
    SMKTEST_MODE_AUTO: 'true'
script:
    # Create cluster remote configuration file.
    - echo $KUBERNETES_TOKEN | base64 -d > /etc/deploy/config
    - create-smktest --check-pods-running=true
only:
    - master

Check Conditions.

* Automatic Test: 🦾

This command checks that those that do not exist alert in the cluster. These alerts can cause instability in all the nodes involved.

Command Kubectl:

kubectl describe nodes


  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
----             ------  -----------------                 ------------------                ------                       -------
MemoryPressure   False   Wed, 16 Jun 2021 13:48:07 +0200   Thu, 26 Sep 2019 12:13:47 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
DiskPressure     False   Wed, 16 Jun 2021 13:48:07 +0200   Mon, 01 Mar 2021 18:01:45 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
PIDPressure      False   Wed, 16 Jun 2021 13:48:07 +0200   Thu, 26 Sep 2019 12:13:47 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
Ready            True    Wed, 16 Jun 2021 13:48:07 +0200   Tue, 08 Jun 2021 14:12:12 +0200   KubeletReady                 kubelet is posting ready status

Command smoke-master:

--check-cluster

Example:

create-smktest --check-cluster

Gitlab Pipeline example:

conditionsCluster:
image:
    name: registry.gitlab.com/phdactivities/smoke-master:master
stage: kubernetes
variables:
    SMKTEST_PROJECT_NAME: 'SmokeMaster'
    SMKTEST_ENVIRONMENT: 'master'
    SMKTEST_CONTEXT: 'kubernetes'
    SMKTEST_NAMESPACE: 'nodespace'
    SMKTEST_MODE_AUTO: 'true'
script:
    # Create cluster remote configuration file.
    - echo $KUBERNETES_TOKEN | base64 -d > /etc/deploy/config
    - create-smktest --check-cluster=true
only:
    - master

Check Logs Inside of the Pods.

Command Kubectl:

kubectl logs ${name} --namespace=${namespace} --since=2m

Example:

create-smktest --project-name=test --environment=develop --context=kubernetes --namespace=smokeMaster --mode-auto=true --check-pods-logs=true

Gitlab Pipeline example:

logsPods:
image:
    name: registry.gitlab.com/phdactivities/smoke-master:master
stage: kubernetes
variables:
    SMKTEST_PROJECT_NAME: 'SmokeMaster'
    SMKTEST_ENVIRONMENT: 'master'
    SMKTEST_CONTEXT: 'kubernetes'
    SMKTEST_NAMESPACE: '$NAMESPACE'
    SMKTEST_MODE_AUTO: 'true'
script:
    # Create cluster remote configuration file.
    - echo $KUBERNETES_TOKEN | base64 -d > /etc/deploy/config
    - create-smktest --check-pods-logs=true
only:
    - master

Assert with CURL petitions.

• Input required: 💪

This test is based on CURL requests. They can be used to check if an api is available or a website. Example. It can be used to verify that the login page is enabled

Command smoke-master:

--assert-curl="www.google.com"

Example:

create-smktest --check-cluster=true

Gitlab pipeline

checkLogin:
<<: *smoke-test-curl
variables:
    SMKTEST_ASSERT_CURL: 'curl -X POST "https://$NAMES-api-develop.openshift.techgap.it/api/v1/auth/authentication" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"email\": \"formazione@$NAMES.it\", \"password\": \"Passw0rd\", \"stayLogged\": false }"'
script:
    - create-smktest --context=remote-server

Check Volumes.

• Automatic Test: 🦾

This test verifies that the volume has free space

Command smoke-master:

--check-volumes=true

Example:

create-smktest --check-volumes=true

Gitlab pipeline

volumeSpace:
<<: *smoke-test-kubernetes
script:
    - create-smktest --check-volumes=true
only:
    - master

Check publics apis using SWAGGER

This test automatically obtains the public apis that do not require parameters from the Swagger documentation. If any api returns a status of number 500 the test fails. It is designed to obtain instabilities in apis automatically.

Command smoke-master:

--check-swagger-public-apis="https://petstore.swagger.io/v2/swagger.json"

Example:

create-smktest --check-swagger-public-apis="https://petstore.swagger.io/v2/swagger.json"

Gitlab pipeline

pubAPetStore:
<<: *smoke-test-kubernetes
stage: swagger
script:
    - create-smktest --check-swagger-public-apis="https://petstore.swagger.io/v2/swagger.json"
only:
    - master

Check publics apis using SWAGGER with Authentication

This test obtains the list of apis [GET] that do not require parameters and makes a request to each one of the APIs using the authentication intended by the user in the form of CURL.

Command smoke-master:

--check-swagger-apis=https://petstore.swagger.io/v2/swagger.json
--swagger-login-curl=curl LOGIN WITH LOGIN ACCESS

Example:

create-smktest --check-swagger-public-apis="https://petstore.swagger.io/v2/swagger.json" --swagger-login-curl=curl LOGIN WITH LOGIN ACCESS

Check notwork from services:

With this test, a PING_TCP request can be made through a service within the cluster. It is necessary to do it this way in order to access the internal networks of the cluster

Command smoke-master:

--check-networks-from-service=SERVICE_NAME

Gitlab pipeline

networks:
<<: *smoke-test-kubernetes
stage: swagger
script:
    - create-smktest --check-networks-from-service=SEVICE_NAME
only:
    - master

Example:

create-smktest --check-networks-from-service=SERVICE_NAME

Build Image steps

1. Login with:
    docker login --username $DOCKER_USER -p $DOCKER_TOKEN
2. Build base imagen:
    docker build -t smktesting/smoke-master:latest -f Dockerfile .
3. docker push smktesting/smoke-master:latest

Build Image steps

1. Login with:
    docker login --username $DOCKER_USER -p $DOCKER_TOKEN
2. Build base imagen:
    docker build -t smktesting/smoke-master:latest -f Dockerfile .

3. docker push smktesting/smoke-master:latest