Smoke testing is a technique for discovering major software flaws using CI/CD pipelines. It consists of very shallow but broad tests that verify the crucial parts of the application. This means that smoke testing is typically a very cost-effective operation.
In this article, we will explore the origins of smoke testing and how it can be incorporated into your continuous integration or continuous delivery pipelines.
Origins of smoke in testing The term “smoke test” originally comes from electrical engineering. Once the circuit of an electrical device is ready for testing, it is attached to a power source. If there are major issues, the circuit will start smoking immediately. This shows that the system must first be fixed before any other tests are undertaken.
The same concept has been adopted in software development, together with the original term.
Photo by Nicolas Thomas on Unsplash What and how we should test Smoke tests should determine if the system is stable enough for it to proceed to the next phase, i.e. to more granular tests. This is achieved by testing the system’s main functionalities at a fundamental level.
Smoke tests don’t need to be comprehensive. For example, if we are working on an e-commerce web application, we only need to check that customers can buy a product. We don’t need to verify all the details and edge cases of the purchasing process.
Although it is common to smoke test the main user-facing features of an application, smoke tests are also applicable at many different abstraction levels throughout your entire system. The value of this multi-level smoke testing approach is that you can discover defects on a more controlled and granular level. When working on distributed systems that involve many teams, this can end up being a big time saver.
Depending on what you are testing, the actions that you will take based on the results you get can vary greatly. However, smoke tests must be fast in order to ensure a fast feedback loop. The value of a smoke test lies in being able to fail very fast, so you don’t waste time and resources on unit and acceptance testing when there is a fundamental problem with your project.
Tools that serve well for writing smoke tests can be very simple. For instance, a simple curl api.myapp.com/is_alive is a good start. Checking versions of installed dependencies with a simple Bash script could also be sufficient.
For more complex checks, you can rely on a testing library in your favorite programming language. If you are using Ruby, for example, and want to test some of the main features of your web application, you can run a few Cucumber scenarios against your production instance.
Where Should We Smoke Test? Within a CI/CD process, there are at least four places where smoke tests fit in:
The following diagram shows where these points are located:
Local smoke testing The first possible place for smoke tests is, in fact, before any CI/CD. Developers can run tests on their machines before pushing code to the repository. A Git test script coupled with a client-side hook should be enough to automate the process.
Git hooks are little scripts that run upon seeing certain events. To run a smoke test when a developer attempts a commit, first create a pre-commit executable file inside the .git/hooks folder in your repository:
$ touch .git/hooks/pre-commit
$ chmod +x .git/hooks/pre-commit
The hook can then call the smoke test script and evaluate its exit code:
```
./scripts/smoke-test-local if [ $? -ne 0 ]; then echo "Commit is rejected due to smoke test failure" exit 1 fi ```
Building and booting up an application on a developer’s workstation is made more approachable by the fact that we don’t need a complete runtime environment to conduct our smoke test. We can often get an environment that is “good enough” with an in-memory database and mocked components.
You may think that this smoke test is a waste of time. After all, there are probably hundreds of other tests in your pipeline that go much deeper than just checking that the application can start. But don’t be fooled into thinking that smoke tests are useless due to their simplicity. Your code might pass unit and integration tests with flying colors and still fail to boot up. Smoke tests are your code’s first instance of a reality check.
Smoke tests in the CI pipeline The next logical point for a smoke test is in the build stage. In this stage, we build the binary/executable artifact and run various types of tests. Often, we can reuse the same script used to run the Git hook smoke tests.
The point of this check is to confirm that the application can boot up and perform some basic functionality, but this time in the clean environment that the CI machine provides. Ideally, you build the artifact once (and only once) and test it over the course of the pipeline. The test does not need to be exhaustive, but it must be able to expose major problems.
Smoke tests as a part of the acceptance test suite Once basic testing is done, we can tackle more comprehensive items such as acceptance tests and end-to-end tests. These types of tests usually take a long time to set up and run, because they need a production-like environment and they interact with the application via the UI.
Because of this longer runtime, failed acceptance tests are more frustrating than most other types of tests. Smoke testing presents another opportunity to fail fast and potentially save time. A few well-placed smoke tests that verify the most critical application paths can catch most problems before we begin the end-to-end testing phase.
Pre- and post-deployment tests In some cases, we can run acceptance smoke tests directly in the CI pipeline, while in others — like in our example — we can use a separate staging environment.
A staging environment presents an opportunity to test the deployment procedure. This process benefits from smoke tests at several points:
Smoke tests in DevOps Production deployment introduces a few more opportunities for smoke tests.
Like we did during acceptance testing, we’ll sanity check the environment with config and infrastructure tests, then deploy and run a post-deployment smoke test.
A few examples of smoke testing In this section, we’ll move on to a more concrete example and try to apply everything we’ve learned so far.
Let’s say we’re writing an API server and that we already have a working CI pipeline. We now want to extend the process with continuous delivery.
First, we’ll need a small test script. Pinging one of the endpoints with curl should be enough for a basic smoke test:
```
$APP_URL/is_online returns HTTP status 200set -e status=$(curl --silent --output /dev/stderr --write-out "%{http_code}" "$APP_URL"/is_alive) if [ $status -ne 200 ]; then echo "Got status code $status instead of expected 200" exit 1 fi ```
Next, we have to prepare the CI machine to run the application. The steps to do this are as follows:
We can fulfill the steps above with the following commands:
```
checkout
artifact pull workflow myapiserver.bin
sem-service start postgres psql -U postgres -c "CREATE DATABASE myApiDB;" psql -U postgres -d myApiDb -c "COPY mytable FROM 'sample_data.csv' WITH (FORMAT csv);"
export APP_URL=localhost ./myapiserver.bin & bash scripts/check-app-up.sh
```
Once the smoke test job is introduced, the pipeline looks like this.
Smoke testing deployments Encouraged by the success of the first smoke test, the next step is to deploy to the staging environment and run more tests. In our example, there are two items to check before we can say that the infrastructure is ready:
Both checks can be solved with small Bash scripts. First, we check the server via SSH:
```
set -e remote_server_name=$(ssh "${SERVER_ADDRESS}" hostname) if [ ! "$remote_server_name" = "$SERVER_HOSTNAME" ]; then echo "Server name doesn't match: expected $SERVER_HOSTNAME found $remote_server_name" exit 1 fi ```
Second, to ensure the PostgreSQL database is ready, we can use pg_isready:
```
PGPASSWORD="${DB_PASS}" pg_isready \ -h "${DB_HOST}" \ -p "${DB_PORT}" \ -d "${DB_NAME}" \ -U "${DB_USER}" ```
The following screencast shows how to add a continuous deployment pipeline with two smoke tests and a deployment job.
Below you’ll find the YAML snippet for the pipeline for reference. You’ll need to create a couple of secrets to store sensitive data, such as SSH keys and database credentials, to complete the setup.
version: v1.0
name: Continuous Deployment
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
- name: Pre-deploy test
task:
jobs:
- name: Server & database
commands:
- checkout
- bash ./scripts/check-db-up.sh
- bash ./scripts/check-server-up.sh
env_vars:
- name: SERVER_HOSTNAME
value: myserver-prod
- name: SERVER_ADDRESS
value: api.myapp.com
secrets:
- name: db-params
- name: ssh-keys
- name: Deploy Application
task:
jobs:
- name: Deploy
commands:
- checkout
- bash ./scripts/deploy.sh
env_vars:
- name: SERVER_ADDRESS
value: api.myapp.com
secrets:
- name: db-params
- name: ssh-keys
- name: Post-deploy test
task:
jobs:
- name: Smoke test
commands:
- checkout
- bash ./scripts/check-app-up.sh
env_vars:
- name: APP_URL
value: api.myapp.com
- name: APP_PORT
value: '80'
Try the continuous deployment pipeline to see the smoke tests in action.
At this point, it should be fairly obvious how to extend the workflow with a production-deployment pipeline. You should repeat the steps and direct the deployment script to your production environment.
Would you like to see smoke tests and deployments in action? Let me know on Twitter and I’ll prepare and link a ready-to-use demo project.
Conclusion Smoke tests should evolve as your application grows. In the beginning, they check one or two essential functions and, over time, grow to serve the purpose of catching errors quickly and cheaply. It takes a bit of effort but can save a lot of time and headaches.
The post Smoke Testing Keeps Your Delivery Pipeline Safe and Sound appeared first on Semaphore.