One of the main targets we try to achieve with Loadero is to make load and performance testing as accessible and simple as possible. As a major step toward making the test creation easier, a whole new test builder was designed and developed, with major benefits:
- Intuitive test configuration flow
- Fewer actions to get to the final result
- Higher test customisability
Whether you have never used Loadero before, or are an existing user that has gotten used to the old test builder, this blog post will guide you through setting up a test in Loadero via our new test builder.
Prerequisites
In order to create a test in Loadero, you will need an account with an active project. You can create an account here, it’s free. Once that is done, log in, begin a free trial and you are good to go. Upon creating a project, you should see something like this:
Click the “+ New test” button and let’s begin with creating your first test.
Test parameters
We begin test creation in the Details tab, which handles all the general test parameters. You can find more detailed descriptions of those in our documentation. We’ll just briefly explain them in this post.
Title
We suggest having a title that best describes the purpose of the test:
Good title – “Register 500 users and join a 10 minute call”
Bad title – “Test 1”
It allows for easier navigation through test results later on. We will simply name our test “Search Bing for Loadero”, as that is what our example test will be doing.
Project language
The script of a test can be written in one of 3 languages:
- Javascript + Nightwatch.js
- Java + TestUI
- Python + Py-TestUI
The project language you see in the test builder shows you what the currently set project language is. If you are making a test and notice that the language specified in this view does not match the language you would like to work with, you can change the language in project settings. Note that this is the language the test automation script will be written in, it is not the language of the website you are writing the script for.
Test mode
The next parameter in this tab is test mode, Loadero has three of them:
- “Load test” mode is suitable for launching large-scale tests with a lot of test participants in a single test run, however, this comes with the drawback of reduced Selenium log verbosity.
- “Performance test” mode is limited to 50 participants in a test run max, but additionally provides machine statistics.
- “Session recording test” mode is limited to 10 participants in a test run max, but provides a screen recording video for each participant.
Tip: If you plan to run a test with less than 50 participants, we suggest always using the “Performance test” mode instead of “Load test”. It provides you with more metrics and data in the results report.
Note: You can always change the test configuration afterwards, if you would like to run the same test in a different mode. If you intend to run a test in “Load test” mode with 50+ participants, we suggest running this test at a smaller scale in a different mode first, to make sure the test is working properly.
We’ll be running a small-scale test with two participants, so “Performance test” is the test mode we’ll use.
Increment strategy
The increment strategy determines the pacing of participant initialization:
- The “Linear participant” strategy ensures that the participants begin the test one by one, with the same time interval between each participant.
- The “Random participant” strategy makes the pacing of the participants random – the intervals between participants will vary and participants may even begin the test at the same time.
- The group strategies are similar to the previous two with the modification that participants belonging to the same group will always start the test simultaneously.
Start interval
The start interval is the time within which all participants will start executing the test.
So if you are configuring a large-scale test with thousands of participants, a longer start interval would be reasonable. That way you will not DDoS your service. If all participants should begin at the same time, then just set it to 1 second.
Note: Having a too-long start interval may cause some participants to finish test execution before other participants have started.
We’ll set the start interval to be 5 seconds and the increment strategy to be “Linear participant”. Since our test will consist of only two participants, then this means that 5 seconds will pass between the first and the second participant starting the test.
Participant timeout
Participant timeout determines the maximum amount of time a participant can take to execute the test before it is forced to cease execution. This is not a test run duration setting, but rather a condition that force-stops a participant if its execution time has exceeded the participant timeout value.
If the timeout is reached, the participant will be marked as timed out in the results report.
Important: Participant timeout has to be longer than it normally takes to execute your whole test script, otherwise your participants will always be forced to cease execution before they actually get to finish the script.
We’ll set the participant timeout to 10 minutes, which should allow the execution of our test script to finish. If you followed the instructions so far, you should have the following parameters:
- Title: Title of your choice
- Test mode: Performance test
- Increment strategy: Linear participant
- Start interval: 5 seconds
- Participant timeout: 10 minutes
Script
Test script is a UI automation script that simulates participant actions using Selenium. To make the test participants interact with UI elements you’ll have to use element locators.
Tip: If you’re new to writing test automation scripts, you can learn the basics from our guides to test automation for Javascript, Java, or Python.
For our example, we’ll have a script that makes test participants visit Bing, do a search and take a couple of screenshots along the way. Here is the test script we’ll use:
client => {
// Example of locating elements using CSS selectors
client
// Navigate to website bing.com
.url("https://www.bing.com")
// Wait up to 10 seconds until page is visible and take a screenshot
.waitForElementVisible('.dimmer', 10 * 1000)
.takeScreenshot("bing.png")
// Type 'Loadero' in the search bar and press 'Enter'
.waitForElementVisible("[type=search]", 10 * 1000)
.setValue("[type=search]", ["QA Processes", client.Keys.ENTER])
// Wait up to 10 seconds until search results are visible and take a screenshot
.waitForElementVisible("[aria-label='Search Results']", 10 * 1000)
.takeScreenshot("search_results.png");
}
Tip: Loadero uses just one test script that will be executed by all the test participants, but if you need to make different participants perform different actions, you can still accomplish that within a single script.
Now that your test parameters are set and the test script is ready, the last mandatory step before you can launch a test run is adding test participants to your test. Click “Next step” to proceed.
Test participants
What we call a test participant in Loadero is essentially a virtual user – a cloud-hosted browser with Selenium web UI automation running on it, which executes your test script to simulate what actual users might do on the app. To simulate real users as close to real life as possible we also allow configuring participant parameters.
Tip: You can have test participants in groups or ungrouped. In more complex tests with multiple participant configurations, it may be a good idea to group them by some parameter: location, actions to take during a test run, browser version, etc., to have an easier time managing them.
Title
The title is just a name for the participant that helps us differentiate between participants later. You can input anything there, the title does not have an impact on the test runs. We’ll call our participant “John Doe”.
Count
The count is the number of identical copies of this specific participant to be added to the test. Each copy will have exactly the same participant configuration. You can use count to easily add more participants to the test and simulate a higher load.
Tip: Participant groups also have the “Count” parameter, which works the same. If you want more copies of that group with exactly the same participant configurations to be added to your test, you can just increase the count for the group.
Our test will have two participants with identical configurations, so we are setting “Count” to 2.
Compute units
Compute units are how we at Loadero measure compute power assigned to a test participant. A single compute unit or G1 is 50% of a CPU core and 1 GB RAM, G2 is double that, etc. Compute unit selection should be made depending on how demanding the application under test is. . To learn how to choose the correct compute power for your test participants you can read this blog post about Loadero’s compute units.
Tip: We recommend always starting your testing with G2 in “Performance test” mode and exploring the machine statistics in the results report to see if going lower might be fine to successfully run your test.
We will set G2 for our test participants in this example.
Important: Loadero bills you for the number of compute units you use in your test runs. As such, it is a good idea to use a smaller number of participants when debugging a test. However, do still choose a compute unit value that ensures that participants will have the necessary resources. While using G0.5 is cheaper, it comes with a higher risk that the participants won’t have enough resources to run the app, which may lead to inaccurate performance metrics or the test crashing. As such we don’t suggest going lower than G1 unless your application is incredibly lightweight.
Browser
As mentioned earlier, Loadero test participants are cloud-hosted browsers. And to allow you to properly simulate real users who may have different browser versions, you can configure your test participants to use different browser versions too. Loadero supports Google Chrome and Mozilla Firefox, and we always have the five latest versions of the browsers available to use. We’ll set our test participants to use the latest Google Chrome version.
Location
The next participant parameter is the location, which is the geographical location where your test participant will be hosted, which allows you to properly simulate users accessing the application from different parts of the world Loadero has 12 locations available, you can see the full list here.
We’ll set our participants’ location as “US West – Oregon”.
Network conditions
Just like the real users of your application, test participants can be configured to have imperfect network connections. Loadero allows users to choose between various packet loss, high jitter, high latency, and bandwidth limitations. This allows our customers to check if their applications are usable and perform well for users with a poor network connection and how an application handles changing network conditions.
For this example we’ll use “Default network settings”, which have no limitations added by Loadero and is the best connection we can get from the provider, basically, it’s a carrier-grade network connection.
Tip: If you would like to customize your own network limitations or change them dynamically during the test, you can use Loadero’s custom “Update Network” command.
Audio feed and video feed
The last two parameters are the audio and video feed. Loadero allows you to use built-in media files to simulate users with microphones and webcams. Mainly this is used for WebRTC application testing. You can pick a feed from the range of options we provide, but if your test does not use the webcam and mic, like this example, you can pick any feed, as it will have no impact on the test run process.
After you’ve set the audio and video feeds the test participant configuration is done, and your participant should have the following parameters:
- Title: John Doe
- Count: 2
- Compute units: G2
- Browser: Latest Google Chrome
- Location: US West – Oregon
- Network conditions: Default network conditions
- Audio feed: Default audio feed
- Video feed: Default video feed
Click the “Add participant” button. Now we’re almost ready to run our test. If you followed the steps described so far, you should have this on your screen:
Asserts
The last step of creating a test in Loadero is adding asserts, however, this step is entirely optional. Asserts automatically check if performance metric values were within the expected range. This means that you don’t have to check whether metrics are within an acceptable range manually, which can save you quite some time, especially during larger scale tests. We use asserts in most of the tests we create for customers and you should consider setting some asserts for your test, especially if it’s a recurring test. But since having asserts isn’t mandatory to launch a test run, we’ll skip this step and proceed to finalizing our test. If you were following our instructions, you can just click “Create test”.
Running your test
If you followed the steps described in this post so far you should be ready to launch your test. You should see something similar to this:
You can just click on the “Run test” button and let Loadero run the test you just configured, so let’s proceed with launching it.
Your test run will go through multiple stages after you launch it. You can find out what stage your test run is currently in from the project’s “Tests” or “Recent runs” tab.
We will not describe all the possible test statuses here, but keep in mind that your test can spend some time in the queue which may range from seconds to a few minutes (“Pending”) until there is enough compute capacity to launch your test. After that your test run will have to go through the “Initializing Environment” stage before the actual test script execution begins. You can read about all the possible statuses and what they mean here.
The duration of your test run depends on various parameters, mainly – your test script and the start interval. After a few minutes our test run successfully finished and all the test participants executed the test script successfully.
If your test run didn’t finish with a 100% success rate, your test might need some debugging. There are many different ways to find issues in the test script and test configuration, we have shared the ones we use most in this blog post, make sure to read it if you need to debug your test.
Once a test run has finished, regardless of its success rate, it’s time to analyze its results. You can get as much (sometimes even more) valuable data and insights from tests that failed as from the successful ones.
If you followed our instructions, then you already have a fully configured test with two participants. You can now use this setup, change the test script to a custom one and try running a PoC test on your application with Loadero’s free trial. If you have any questions about using Loadero, feel free to use our support chat to ask them, our team is here to help. Good luck with your tests!
