Setting a correct increment strategy for load generation is crucial when you are load testing a website or service. It’s especially important to make sure your website will not be bombarded by a huge number of participants all at once. Choosing correct increment strategy helps you organize your test run flow. You are making sure that participants start exactly how you expect. That can be especially useful for WebRTC service testing. You can monitor how participants continuously joining a call can impact performance and user experience. In this blog post we will describe increment strategies available in Loadero and explain their differences. So you will be able to choose the best according to your testing needs.
Start interval and Participant timeout
Before we get to the different increment strategies, describing two test parameters is necessary for a full understanding. On the test configuration screen in Loadero, besides the test script, you can see the test name, testing mode, increment strategy, start interval and participant timeout. The start interval is very closely connected to how the load generation will happen. Participant timeout is necessary to control your test duration. So let’s first take a look at what exactly these two parameters are, and how to use them.
Start interval
This is the time within which all participants will start executing the test. The delay between participants depends on how long the start interval has been set. So if you are configuring a large scale test with thousands of participants, a longer start interval would be reasonable. With that you will not be DDoSing your service. If all participants should begin at the same time, then just set it to 1 second.
Important: Having a too long start interval may cause some participants to finish test execution before other participants have started.
Participant timeout
Participant timeout is the duration in which the participant has to complete the test execution. Otherwise it will be stopped and marked as “Timeout” in the results report. This is necessary to prevent too long test execution. It may happen in cases when the defined commands may freeze, require way longer execution times or go in an infinite loop. Start interval has no impact on participant timeout as it is added before that. So even if your start interval is higher than the participant timeout, participants will not timeout.
Different types of increment strategies
There are 4 available options in Loadero: Linear participant, Linear group, Random participant, Random group. They can be accessed on the right side of the test edit page when creating or updating a test.
By default, new tests are set to Random participant, which means that all participants will join at random time periods within the provided start interval. To choose the best load generation strategy for your needs you must figure out if you need a linear or a random strategy, and if it has to be based on participant groups or individual participants.
Linear strategies
These types of increments specify participants to begin the test execution after a specific time period from when the previous one has started. Linear strategy takes the start interval and divides it into equal intervals. All participants are starting the test linearly, creating a continuous and monotonic load increase.
For example, if the start interval is set at 10 seconds and there are 5 participants, then each participant will join every 2 seconds during the start interval. This can help control the flow of participants as necessary.
Important: The order in which participants start does not guarantee that they all will access the page in the same order. It is possible that for participant A it took 8 seconds to load the page but for participant B it took 6 seconds. So even with a 1-second delay between each of these participants, the participant B would be ahead in test execution.
Controlling the start interval is key when using linear increment strategies. Setting this interval too long or short can potentially impact your test results, depending on how well your service handles multiple users joining in a short period of time. This can be useful for both load and performance testing to monitor how incrementally increasing the number of test participants impacts your service and internal architecture’s performance. With linear increment, you get rather predictable results because there will be a precise calculable delay between participants. So if you require load generation to be predictable, one of the linear strategies is the correct choice for you.
Tip: The order in which participants start a test is set by the global participant ID which is unique for every participant in one test. Each participant in the test has its own IDs: there is a global ID, which is unique for every participant in the whole test and there is just participant ID, which is unique for each participant in a group. So each group would start counting participant ID from 0 again. Global ID would continue counting further from the previous group.
Check out our wiki for more information about participant IDs and other Loadero variables: JavaScript+Nightwatch variables and Java+TestUI variables.
Random strategies
These options are just as they sound like, random. You have limited control over when each participant starts the test. The start interval is less important for these settings, but it still can affect the outcome. For example, if the start interval is 10 seconds, then with a random increment setting it is possible for participants or groups to join at any second and multiple participants might join at the same time. Basically, participants or groups would pick a random second out of the given start interval and join then.
Using random increment strategies could be useful when you want to simulate user flow closer to real-world situations. Just like in real life it is not possible to predict how often and how many users will join your service at any given time. You can also use a random strategy when it is not important how and when participants start the test.
Group strategies and participant strategies
In essence, group strategies work exactly like participant ones. The difference is that instead of starting execution depending on each participant’s global ID, it is done by group ID, so all participants sharing a group will begin executing the test at the same time. Let’s assume we have a test with 5 groups with a different number of participants, 9 in total. Our start interval is 10 seconds. With the Linear participant strategy a participant will join the test run roughly every second. If with the same test configuration we would choose the Linear group strategy, a group will join the test run every 2 seconds, and all participants in the same group will join at the same time.
Group increment strategies can be useful when your test participants are organized in specific groups, and you want to control how each group of participants start the test. For example, if you want to test multiple video call rooms at once, and each room will have a new participant joining simultaneously, participants can be organized in groups, and Linear group load generation strategy can be used.
Another example is setting up a video call with an administrator or an initial participant that sets up the meeting. The script could differ for that first participant or group of participants, but be the same for the others. If it is a load test, and multiple calls will be created, separating administrator participants into a group is a good choice. You will have to adjust your script, so that only a specific group performs call setup steps. Are working on a test with participants performing different actions? Make sure to read this blog post that explains how a multiple flow test can be set up in Loadero.
Summary
In the end, choosing an increment strategy entirely depends on what your intentions for the test are. If you wish for participants to join with predictable delays, then either Linear participant or Linear group should be your choice. Control over the frequency of new participants joining is useful for load testing to not overload your service. Pick Random increment strategies when you wish to simulate more realistic scenarios. In many real-world cases the flow of users can be quite unpredictable. Choosing between group and participant strategies mostly depends on the design of your test.
Choosing a load generation strategy is a very important step in planning and configuring your test. In case you ever need a reminder, you can check out our wiki page about test parameters. If you have any questions about increment strategies or any other Loadero feature, feel free to contact our support team by email or use web chat in the Loadero interface for this, we’d be happy to help you.
