Loadero is a versatile tool that can be used for different types of web application tests. In order to provide a comfortable way of running the tests that you need, we have different test modes. In this blog post, we describe each of those in detail so you can pick the appropriate one every time you are launching a test run.
There are currently 3 available test modes to choose from when creating your own Loadero test. The user should choose between them depending on their testing needs and requirements as each test mode suits different use cases. The provided test result data also differs among the different test modes. These test modes are:
- Performance test
- Load test
- Session recording test
The test mode is set in the test configuration view on the right-hand side. In this post, we give detailed explanations of each of the available options so that you can pick the most appropriate one.
Performance test mode
Pros:
- Verbose Selenium log output.
- Machine resource usage statistics for each participant.
- Ability to test file downloads with available artifacts afterward.
- Full post-run assertion availability.
Cons:
- Maximum limit of 50 participants.
- Small extra resource usage for collecting all the additional data.
This mode provides the most performance metrics for each test run participant. There is a limitation of 50 participants maximum per test using performance test mode as this gathers the most amount of data out of all test modes and processing it requires additional time and resources.
Most notably this mode includes machine statistics that reflect CPU, RAM, and network metrics which give insight into how demanding the test flow is and help pinpoint possible issues within a web application, as well as helps to determine the amount of computing power that should be assigned to each corresponding test participant. This data will be collected during the whole test for each participant in 1-second intervals.
Performance tests will also gather WebRTC metrics if the web application establishes a video call. If these WebRTC metrics and their graphs are thoroughly analyzed, they may provide a lot of meaningful information. Loadero has already processed some of the important information in easier-to-access views, but there is still a ton of additional information that can be found by examining the raw data logs.
Tests can also use all types of post-run assertion, which will provide easier test quality assessment. This is made possible because of all the machine and WebRTC statistics collected throughout the whole test run.
After running a test in performance test mode verbose Selenium logs are provided for each test run participant, which provides more details on what happened during the test and how long each activity took.
Another feature that can be tested only using performance test mode is file downloads. In performance tests, each participant has a 100MB limit of downloaded file artifacts which will be available to download after a test run is completed. These files will be located in the report screen for the corresponding test run participant under the “Artifacts” tab.
If the test does not exceed 50 test participants, then almost in all cases this should be the desirable test mode of choice, as the data gathering is not adding too much of a strain on test participants and the test results will provide more information than running the test in other test modes.
Load test mode
Pros:
- No limit on the number of participants in the test.
- Collection of WebRTC statistics and post-run assertions on these metrics.
- Least amount of additional machine resources necessary for data collection out of all test modes.
Cons:
- Reduced Selenium log verbosity.
- No machine resource usage statistics.
This mode is perfect for simulating large amounts of test participants. There generally is no limit to the number of participants in the test apart from the limitation of 15 000 compute units used in a single location or the limits of a paid plan used for the project.
Tests run using this mode will have base Selenium logs, with reduced verbosity as this information is not that important at a higher load scale. Also, machine statistics will not be available, therefore machine post-run assertions also won’t be available. WebRTC statistics are still collected in this mode, so WebRTC post-run assertions will still be working the same as in using performance test mode.
Using load test mode generally should be done once the test script has been fully developed, base statistics have been examined and appropriate test resources are allocated for each test participant. Afterward, when running these tests they should be run with gradually increasing participant count intervals, to determine when the web applications might start hitting some bottlenecks. If you are getting ready to scale up your test, instructions for this can be found here.
Session recording test mode
Pros:
- Gives the best and easiest way to understand what the participant is doing during test execution.
- Each participant will have a session recording available at 15 FPS with audio.
- Visual representation of test participant actions eases debugging process in different scenarios.
Cons:
- Maximum of 10 participants per test.
- Additional machine resource usage overhead is necessary.
- No machine and WebRTC statistics are collected, resulting in no post-run assertion availability.
- Not available in trial projects.
This test mode is generally used for easier debugging of your test script during its development phase, and for ease of understanding what the test participant is doing during test execution. This test mode is limited to a maximum number of 10 participants, as the amount of data to process after such a test is a lot higher than even with the Performance test mode. The verbosity of Selenium logs is also reduced for this test mode, the same as it is done for load test mode.
Each participant during test script execution will be making a screen recording, with audio as well. The recording will be recorded in 15 frames per second, so these videos should generally not be used for quality assessment but for in-depth debugging.
Tip: As recording a session will be using resources from the machine on top of the regular test execution actions, additional machine resource overhead should be allocated for each participant.
After the test execution has been finished session recording will be available in the report screen for the corresponding test run participant under the “Artifacts” tab.
As additional resources are required for the session recording itself tests using this test mode will not be collecting machine statistics and WebRTC statistics to reduce the necessary already higher computing power for running these tests. As neither of those statistics is collected, post-run assertions will not be available, as the data to process that information is not available.
Now that you know how the test modes differ in Loadero, you can pick the best-fitting one for your needs. But if you think that your plan for testing is different and not sure which mode would be the right one, feel free to ask for help in our support chat, we are there to help you.
