Yes, I already know: when developers read the word “testing,” a mix of emotions comes to their mind: fear, anxiety, frustration…

For this reason, this post aims to explain what test automation is and how developers can benefit from it, in terms of productivity and code quality.

In particular, it discusses what unit testing and integration testing are, how they differ from each other, and how to implement them in Python using Flask, by providing a step-by-step tutorial.

So, fear no more, and let’s dive right into it!

What is Test Automation?Test automation is the process of automating the testing activities developers do to ensure that their code works as expected. It involves writing scripts or programs that simulate user interactions with an application: this helps verify its behavior under different conditions, and report any issues found.

Manual VS Automated TestingManual testing is the first step developers need to overcome when they learn to code if they want to ensure their code works as expected when users will use it.

This process needs total human intervention, which means that every time you perform a new test, you’ll need to manually execute each step involved in your test case. So, as understandable, this can be very time-consuming and prone to errors due to human mistakes.

Also, the process of manual testing is generally very simple. For example, a typical use case in Python is to write some print() here and there along the code to check whether everything works fine. But, indeed, this does not ensure the code is working correctly: it only shows that the code runs without crashing until a specific point.

On the other hand, automated testing—while automating the process of testing—also provides processes and procedures that actually ensure the code works as expected.

Benefits of Test AutomationSo, let’s point out some of the benefits of using automated testing:

• Standardization: Test automation equips developers with a standardized approach to testing by providing frameworks and processes that actually ensure the code works as expected.
• Reduced human errors: As with every automation, test automation eliminates the human errors that can occur in the processes.
• Increased code quality: By ensuring the code works as expected, test automation helps improve the overall quality of the code. In fact, test automation facilitates consistent and thorough testing: this reduces the number of bugs entering into production, thus improving the overall quality of the code.
• Faster code delivery: With automated testing, developers can quickly identify and fix issues before releasing software updates. This speeds up the development cycle and allows teams to deliver high-quality products more frequently.
• Increased test coverage: The goal of testing software is to ensure that all parts of the code are tested thoroughly (or, at least, the majority of them). While manual testing may miss certain areas or scenarios, automated tests can cover a wider range of cases and edge situations, increasing confidence in the reliability of the system when delivered to production.

Test Automation: Understanding Unit TestsIn the context of test automation, we can define a “unit” as the smallest testable piece of a software application. This means that a unit could be a function, method, class, module—or anything related—depending on how granular you want to get or depending on the software you are developing.

So, unit testing is the test automation practice that focuses on testing individual components of a program to ensure they work as intended, separately from the other parts of the same program. For this reason, this is generally the first level of testing performed during the development process.

For example, in a web application that manages online purchasing after users have logged in, a unit test can focus on the login phase, ensuring the login credentials are validated properly, and that the user is redirected to the correct page upon successful authentication.

While this testing practice does not guarantee that the entire program works as expected, it ensures that each component behaves correctly independently from the others. For this reason, this does not mean that unit tests are unuseful; instead, they provide some benefits during the software lifecycle like:

• Early bug detection: Unit testing is often practiced by developers themselves while coding. This allows them to identify and fix bugs easily during the development phase. This early detection allows addressing issues before they become more complex and, eventually, costly.
• Enabling frequent releases: When unit tests are implemented effectively, they help ensure that changes made to one part of the codebase don’t break other parts. This makes it easier for teams to make frequent releases without worrying about introducing new issues or regressions.
• Simplified refactoring and maintenance: Well-written unit tests act as documentation for the codebase, making it easier for developers to understand how different parts interact. This serves as safety nets, helping teams against unintended side effects when refactoring or modifying existing functionality during maintenance tasks.

Characteristics of Effective Unit Tests: The FIRST PrincipleDue to the nature of unit testing, effective unit tests should follow the so-called FIRST principle:

• Fast: Staring at the screen during the first of 10 unit tests running, knowing the first will take 15 minutes alone to complete, is not acceptable. Instead, unit tests should run quickly, ideally within milliseconds. This enables developers to run them frequently during the development process, allowing for rapid feedback and iteration. Fast tests, in fact, encourage developers to run them frequently, integrating them seamlessly into the development workflow with the aim to catch defects early and accelerate the development cycle.
• Independent: Each unit test should be independent of other tests, meaning they can be executed in any order without affecting the results. This promotes parallel execution, teamwork, and efficient test suites, which is a way to speed up the testing process.
• Repeatable: If you run a unit test today obtaining a different result than yesterday, something went wrong. Instead, unit tests should produce consistent results regardless of the environment or machine where they’re run. This ensures reliable outcomes across different setups.
• Self-validating: A good unit test should use assertions to validate outcomes, providing a clear pass or fail based solely on its own logic result without requiring external input, interpretation, or manual intervention.
• Timely: Ideally, tests should be written just before the production code that makes them pass. This enables the test to guide your design choices, aligning with the so-called Test-Driven Development (or TDD). To get the idea, consider that for some developers “timely” means writing tests in the same timeframe as the code itself.

Test Automation: Understanding Integration TestsIntegration testing is another important aspect of test automation that focuses on testing the interactions between multiple units of a software application. So, unlike unit tests, integration tests verify how these components work together to achieve the desired outcome as a whole.

The main idea behind integration testing is to simulate real-world usage scenarios and detect potential issues that arise when combining different parts of the application. These tests help uncover problems such as data inconsistencies, communication failures, or unexpected behaviors caused by interactions between various components.

As an example, let’s consider the previously-mentioned web application scenario. In this case, an integration test might involve simulating the entire purchasing flow, including logging in, selecting items, adding them to the cart, proceeding to checkout, and completing payment. Such a comprehensive test ensures that all aspects of the purchase process function correctly when integrated, allowing deployment to production with greater confidence.

Integration Testing MethodsThere are several methods used for integration testing, each with its own advantages and trade-offs. Here are some common approaches:

• Big-bang: This testing method involves integrating all units simultaneously and testing them as a complete system. While this is a thorough method, it can lead to longer test cycles since fixing bugs discovered later in the process requires retesting earlier stages. For this reason, this method is typically used when the components are relatively independent and can be tested individually.
• Top-down: Top-down integration starts with higher-level units and gradually integrates with lower-level ones. This approach helps isolate issues closer to the top levels but may overlook deeper-seated problems in lower layers. So, while this method is highly effective in the early detection of interface issues and major control points, it requires the creation of mock objects for lower-level units, which can be complex and time-consuming.
• Bottom-up: Bottom-up integration tests start with lower-level units and progressively integrate them into higher-level ones. This method is useful for identifying issues related to internal interfaces and dependencies among components. Its great advantage is that it allows developers to test the lower-level modules and their interactions before integrating them into the higher levels, generally with no need for mock objects. However, it delays the testing of the higher-level functionalities, which may push back the identification of defects, slowing down the development process.
• Incremental testing: Incremental testing combines the elements of the other testing approaches. It begins by testing smaller units and gradually increases complexity by integrating additional components over time when the tested ones work as expected. This method strikes a balance between early defect detection and timely completion of higher-level tests, offering flexibility and adaptability throughout the development lifecycle. However, it is time-consuming and may not be suitable when there’s the need to match tight deadlines.

Key Differences Between Unit and Integration TestingNow that we’ve covered both unit and integration testing, let’s highlight the key differences between them to provide a clearer overview:

• Scope and focus: Unit tests concentrate on testing individual units, whereas integration tests examine how those components interact with each other. So, in other words, unit tests focus on ensuring each component functions correctly, while integration tests ensure that the combined components behave as expected as a whole.
• Isolation vs interaction: Unit tests operate in isolation, focusing on a single unit’s behavior without considering its interactions with other components. On the contrary, integration tests evaluate how units collaborate and communicate with each other, taking into account their relationships and dependencies. To do so, they require mocking or stubbing techniques to simulate the behavior of dependent components, as users supposedly will use them.
• Complexity and execution time: Unit tests tend to be simpler and faster to execute compared to integration tests. This is because unit tests usually involve fewer steps and less setup, while integration tests require setting up a more realistic environment that mimics actual usage scenarios.
• Debugging and failure analysis: Debugging unit tests is generally straightforward, as they target isolated components. On the other hand, debugging integration tests can be more challenging due to the increased complexity and interdependencies between components. This is why failure analysis in integration tests may involve tracing through multiple interconnected units to pinpoint the root cause of issues.
• Actors involved: While this topic depends on the project structure and organization, we can say that unit tests are primarily conducted by developers who create and execute them to ensure the proper functioning of individual units. Instead, as integration tests are more complex, they are generally performed by QA teams; yet, it may involve collaboration between developers and testers, as well as stakeholders representing end-users’ perspectives.

Test Automation: A Step-by-Step Python Tutorial Using FlaskAfter all this theory, it’s now time to get’s hands on code!

In this section, you will learn how to create a simple Flask application and how to test it. For the sake of simplicity, the application can be tought of as an online calculator that performs basic arithmetical operations. Specifically, it adds and multiply numbers.

But before diving into the code, let’s start by listing all you need to correctly set up your environment.

Prerequisites, Requirements, and Repository StructureBefore you begin, make sure you have Python 3.8+ installed on your computer.

Then, create a repository – I named it flask_app – with the following structure:

├── app.py├── templates/│ ├── index.html│ └── result.html└── tests/ ├── __init__.py ├── test_unit.py └── test_integration.py The venv/ folder contains a virtual environment. You can create it by typing:

python3 -m venv venv To acvitate it on Windows, type:

venv\Scripts\activate And on Linux/MacOS, type:

source ./venv/bin/activate After the virtual environment has been activated, install the required packages by typing:

pip install Flask pytest requests Now you are ready to write your code!

Building The ApplicationNow you can create your online calculator by writing the following code inside app.py:

from flask import Flask, render_template, request, redirect, url_for, jsonifyapp = Flask(__name__)def add_numbers(a, b): """Adds two numbers and returns the result.""" return a + bdef multiply_numbers(a, b): """Multiplies two numbers and returns the result.""" return a * b@app.route('/')def index(): """Render the main page with forms.""" return render_template('index.html')@app.route('/calculate', methods=['POST'])def calculate(): """Handle form submission and display the result.""" operation = request.form.get('operation') a = request.form.get('a') b = request.form.get('b') c = request.form.get('c') try: a = float(a) b = float(b) # Perform the selected operation if operation == 'add': result = add_numbers(a, b) return render_template('result.html', result=result) elif operation == 'multiply': result = multiply_numbers(a, b) return render_template('result.html', result=result) elif operation == 'add_multiply': c = float(c) sum_result = add_numbers(a, b) result = multiply_numbers(sum_result, c) return render_template('result.html', result=result) else: return render_template('result.html', error="Invalid operation selected.") except (TypeError, ValueError): return render_template('result.html', error="Invalid input provided.")@app.route('/add', methods=['GET'])def add(): """API endpoint to add two numbers.""" try: a = float(request.args.get('a')) b = float(request.args.get('b')) result = add_numbers(a, b) return jsonify({'result': result}) except (TypeError, ValueError): return jsonify({'error': 'Invalid input'}), 400@app.route('/multiply', methods=['GET'])def multiply(): """API endpoint to multiply two numbers.""" try: a = float(request.args.get('a')) b = float(request.args.get('b')) result = multiply_numbers(a, b) return jsonify({'result': result}) except (TypeError, ValueError): return jsonify({'error': 'Invalid input'}), 400@app.route('/add_multiply', methods=['GET'])def add_and_multiply(): """API endpoint to add two numbers and then multiply the result by a third number.""" try: a = float(request.args.get('a')) b = float(request.args.get('b')) c = float(request.args.get('c')) sum_result = add_numbers(a, b) final_result = multiply_numbers(sum_result, c) return jsonify({'result': final_result}) except (TypeError, ValueError): return jsonify({'error': 'Invalid input'}), 400if __name__ == '__main__': app.run(debug=True) So, this code created an app that does the following:

• Adds two numbers (with the add function).
• Multiply two numbers (with the multiply function).
• Creates three endpoints that can be usable as a web app. One endpoint sums numbers (the add endpoint), one multiply numbers (the multiply endpoint), and the last one adds two numbers and multiplies the result by a third number (with the add_multiplyendpoint). Each endpoint manages the expected data types and errors, in case of bad inputs.

To make it locally working, you also need to write the following code into the index.html file:

Calculator App Calculator App Add Two Numbers First Number (a): Second Number (b): Add Multiply Two Numbers First Number (a): Second Number (b): Multiply Add Two Numbers and Multiply the Result First Number (a): Second Number (b): Multiplier (c): Calculate This will manage the UI of the app.

Also, you need to add the following code into the result.html file:

``` Calculation Result Calculation Result {% if result is defined %} The result is: {{ result }}

{% elif error is defined %}        Error: {{ error }}

{% endif %}    Perform another calculation

``` This will manage the UI of the result page.

Now you can run the app by typing:

python app.py and open your browser at http://localhost:5000/ to see it live.

You are now ready to test it!

Manual TestingFor the sake of completeness, let’s briefly discuss how to manually test this application.

For each of the three endpoints, you should manually:

• Verify all the possible acceptable cases. This means testing numbers, which means: integers and floating (as well as negative numbers, null values, etc…).
• Verify all the possible unacceptable cases. This means testing strings, which means: letters, symbols, etc. This also means that the application should return an error message to say that the input is invalid, not break the code.

For brevity, let’s just manually verify one endpoint. For example, let’s test the add one.

In the positive case, you should insert two numbers:

When clickinc on add, you should be redirected to another page showing the result of the addiction:

If you now insert a letter instead of a number, you should receive an error message:

As understandable, even in the case of a basic app, the effort when manually testing code takes a lot of time and resources.

So, let’s now use automated testing to save our time!

Unit TestsTo create automated tests based on unit testing, you can consider the add_numbers() and multiply_numbers() functions to be units. In this scenario, you can write the following code into the test_unit.py in the tests/ folder:

import unittestimport sysimport os# Add the parent directory to sys.path to import app.pysys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))from app import add_numbers, multiply_numbersclass TestMathFunctions(unittest.TestCase): """Unit tests for math functions with broad test cases.""" def test_add_numbers_valid(self): """Test the add_numbers function with valid numeric inputs.""" # Test with positive integers self.assertEqual(add_numbers(1, 2), 3) # Test with negative integers self.assertEqual(add_numbers(-1, -1), -2) # Test with zero self.assertEqual(add_numbers(0, 0), 0) # Test with positive floats self.assertEqual(add_numbers(1.5, 2.5), 4.0) # Test with negative floats self.assertEqual(add_numbers(-1.5, -2.5), -4.0) # Test with mixed integer and float self.assertEqual(add_numbers(1, 2.5), 3.5) # Test with very large numbers self.assertEqual(add_numbers(1e20, 1e20), 2e20) # Test with very small numbers self.assertAlmostEqual(add_numbers(1e-20, 1e-20), 2e-20) def test_add_numbers_invalid(self): """Test the add_numbers function with invalid (non-numeric) inputs.""" # Test with strings (letters) with self.assertRaises(TypeError): add_numbers('a', 'b') # Test with strings that look like numbers with self.assertRaises(TypeError): add_numbers('1', '2') # Test with None with self.assertRaises(TypeError): add_numbers(None, 2) # Test with lists with self.assertRaises(TypeError): add_numbers([1, 2], 3) # Test with dictionaries with self.assertRaises(TypeError): add_numbers({'a': 1}, {'b': 2}) # Test with boolean values with self.assertRaises(TypeError): add_numbers(True, False) def test_multiply_numbers_valid(self): """Test the multiply_numbers function with valid numeric inputs.""" # Test with positive integers self.assertEqual(multiply_numbers(2, 3), 6) # Test with negative integers self.assertEqual(multiply_numbers(-2, -3), 6) # Test with positive and negative integers self.assertEqual(multiply_numbers(-2, 3), -6) # Test with zero self.assertEqual(multiply_numbers(0, 100), 0) # Test with positive floats self.assertEqual(multiply_numbers(2.5, 4), 10.0) # Test with negative floats self.assertEqual(multiply_numbers(-2.5, -4), 10.0) # Test with mixed integer and float self.assertEqual(multiply_numbers(3, 0.5), 1.5) # Test with very large numbers self.assertEqual(multiply_numbers(1e10, 1e10), 1e20) # Test with very small numbers self.assertAlmostEqual(multiply_numbers(1e-10, 1e-10), 1e-20) def test_multiply_numbers_invalid(self): """Test the multiply_numbers function with invalid (non-numeric) inputs.""" # Test with strings (letters) with self.assertRaises(TypeError): multiply_numbers('a', 'b') # Test with strings that look like numbers with self.assertRaises(TypeError): multiply_numbers('1', '2') # Test with None with self.assertRaises(TypeError): multiply_numbers(None, 2) # Test with lists with self.assertRaises(TypeError): multiply_numbers([1, 2], 3) # Test with dictionaries with self.assertRaises(TypeError): multiply_numbers({'a': 1}, {'b': 2}) # Test with boolean values with self.assertRaises(TypeError): multiply_numbers(True, False)if __name__ == '__main__': unittest.main()

NOTE: The use of the sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) is to allow importing from the parent directory (..) where the app.py file resides. This is necessary because the current script is located within the tests/ subdirectory, and thus needs to navigate upwards to access files outside its immediate scope.

As you can see, with only a few lines of code, this test file takes care of different possibilities like inserting negative numbers, very big numbers, letters, and more.

Now, if you go into the tests/ folder and launch the test_unit.py file by tiping python3 test_unit.py you should receive an output like this one:

Ran 4 tests in 0.001sFAILED (failures=2) So, in this case, you have two tests failed. Why has this happended? Return up to the previous code; as you can see, it reports various functions that represent different cases; let’s consider two of them:

• The test_add_numbers_valid() function creates and tests positive cases for the add_numbers() function. This means that the values tested are acceptable ones.
• The test_add_numbers_invalid() function, on the other hand, creates and tests negative cases for the add_numbers() function. This means that the values tested are not acceptable ones.

So, a question may arise now: have you managed unacceptable values in the functions in the app.py file? Well, the answer is no! In fact, if you scroll the terminal, you should see an output like this one:

AssertionError: TypeError not raised This happens because the add_numbers() and multiply_numbers() functions in the app.py are not raising a TypeError when provided with invalid (non-numeric) inputs, as your tests expect.

Hooray! Here’s another big result achieved! Not only you can test multiple scenarios automatically, but you can also find ways to improve your code. This is why the Test-Driven Development approach is so powerful!

So, if you want to improve your code in the app.py by also considering to raise a TypeError with invalid inputs, you can modify the functions like so:

def add_numbers(a, b): """Adds two numbers and returns the result.""" if isinstance(a, bool) or isinstance(b, bool): raise TypeError("Boolean values are not allowed") if not isinstance(a, (int, float)) or not isinstance(b, (int, float)): raise TypeError("Both inputs must be int or float") return a + bdef multiply_numbers(a, b): """Multiplies two numbers and returns the result.""" if isinstance(a, bool) or isinstance(b, bool): raise TypeError("Boolean values are not allowed") if not isinstance(a, (int, float)) or not isinstance(b, (int, float)): raise TypeError("Both inputs must be int or float") return a * b At this point, you should get no more errors and the result should be something like this one:

Ran 4 tests in 0.001sOK Which means that 4 test have been performed in 0.001 seconds and everything went fine!

Integration TestingLet’s now use a botton-up approach to perform integration testing. In this scenario, you can write the following code into the integration_unit.py in the tests/ folder:

import unittestimport sysimport os# Add the parent directory to sys.path to import app.pysys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))from app import appclass TestAppIntegration(unittest.TestCase): """Integration tests for the Flask application.""" def setUp(self): """Set up the test client.""" # Configure the app for testing app.config['TESTING'] = True self.client = app.test_client() def test_add_endpoint(self): """Test the /add endpoint with valid and invalid inputs.""" # Valid input test response = self.client.get('/add?a=10&b=20') self.assertEqual(response.status_code, 200) self.assertEqual(response.get_json(), {'result': 30.0}) # Invalid input test (non-numeric values) response = self.client.get('/add?a=foo&b=bar') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) # Missing parameter test (missing 'b') response = self.client.get('/add?a=10') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) def test_multiply_endpoint(self): """Test the /multiply endpoint with valid and invalid inputs.""" # Valid input test response = self.client.get('/multiply?a=5&b=4') self.assertEqual(response.status_code, 200) self.assertEqual(response.get_json(), {'result': 20.0}) # Invalid input test (non-numeric value for 'b') response = self.client.get('/multiply?a=5&b=bar') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) # Missing parameter test (missing 'b') response = self.client.get('/multiply?a=5') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) def test_add_multiply_endpoint(self): """Test the /add_multiply endpoint with valid and invalid inputs.""" # Valid input test response = self.client.get('/add_multiply?a=2&b=3&c=4') self.assertEqual(response.status_code, 200) self.assertEqual(response.get_json(), {'result': 20.0}) # (2 + 3) * 4 = 20 # Invalid input test (non-numeric value for 'a') response = self.client.get('/add_multiply?a=foo&b=3&c=4') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) # Missing parameter test (missing 'c') response = self.client.get('/add_multiply?a=2&b=3') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) def test_calculate_route(self): """Test the /calculate route for form submissions.""" # Test addition via form submission response = self.client.post('/calculate', data={ 'operation': 'add', 'a': '5', 'b': '7' }) self.assertIn(b'The result is: 12.0', response.data) # Test multiplication via form submission response = self.client.post('/calculate', data={ 'operation': 'multiply', 'a': '4', 'b': '6' }) self.assertIn(b'The result is: 24.0', response.data) # Test add and multiply via form submission response = self.client.post('/calculate', data={ 'operation': 'add_multiply', 'a': '2', 'b': '3', 'c': '5' }) self.assertIn(b'The result is: 25.0', response.data) # Test invalid input via form submission response = self.client.post('/calculate', data={ 'operation': 'add', 'a': 'foo', 'b': 'bar' }) self.assertIn(b'Error: Invalid input provided.', response.data)if __name__ == '__main__': unittest.main() And you chould obtain the following result:

Ran 4 tests in 0.023sOK So, here’s what this code does

• Validates end-to-end functionality: It ensures that the application handles requests properly from the client’s perspective, including processing inputs, performing computations, and returning appropriate responses for each endpoint.
• Tests interactions between components: Verifies that the individual functions (add_numbers, multiply_numbers) integrate correctly with the Flask routes and handle data as expected.
• Checks input validation and handles errors: Confirms that the application responds correctly to invalid inputs or missing data, returning appropriate error messages and HTTP status codes.
• Simulates real user behavior: Uses the Flask test client to mimic real-world interactions without running the actual server, facilitating efficient and automated testing.

Note that, with this code, you tested all the endpoints and their expected functionalities from a user’s perspective. The add_multiplyendpoint is particularly to be mentioned because it does not perform its logic independently from the others two, so it couldn’t be tested with only unit tests (if you’d write a unit test for it, you’d end up testing individual components – add_numbers() or multiply_numbers() – in isolation.)

ConclusionsIn this article, we presented the basics of test automation, discussing both unit testing and integration testing approaches.

While the theory is important, understanding these concepts requires practical experience and time. For this reason, the step-by-step guide provided offers a hands-on approach to learning about test automation using Python and Flask. But this is only the first step towards mastering this topic, so keep practicing and exploring the various techniques to become proficient in test automation!

Also, for improving even more your experience, you find all the code in this public repository. When you clone it, consider that you can do even more. In fact, to improve your automated testing skills and experience even more, I have created a CI using Semaphore CI: this firse up your automation journey!

Happy coding!

The post Unit Testing vs. Integration Testing: Test Automation Basics appeared first on Semaphore.