You have just clicked the button to run your test, but how soon will the first participant start the test? And when will you be able to explore the results reports? Each test run goes through multiple stages before a results report is generated. In this blog post we are going to explain everything you need to know about test run statuses you can see in Loadero!
What are the possible statuses?
At this moment there are more than 10 possible statuses that your test run can have and each one of them indicate a different stage in test execution or a failure that was caught and was not possible to handle to continue test execution. You can see the current status in the Results view of your project.
Status: Pending
Before we pick up a test from the queue it’s always going to have a “Pending” status, which indicates that this test run is just waiting for its turn. Usually, this status can be only seen for a couple of seconds, but if there are a lot of tests currently running in the Loadero platform, then it’s going to wait until the test run queue has enough compute unit capacity freed up to fully provide resources for the next test run. No worries, if you see the status for some time, your test is ready for running and will start soon. To better understand how our test run queue works, check out our wiki page about it.
Status: Initializing environment
The test run is setting up a test environment, this can take a while – usually about 2 to 3 minutes. During this time we are deploying the servers where the test participants will be hosted and synchronizing their execution. This is an important step to ensure that our test participants are as close to real-world users as possible.
Status: Running
If you see this status for your test – great! Test execution is in progress and at least one of the Loadero participants is executing the test. This test run status will still be used even if some of the participants have finished execution and will be changed only when all participants have finished execution. Now you should wait until the test run finishes for all participants you configured, the duration of the test run depends on your test script and test settings.
Status: Waiting for results
All participants have finished their execution with four possible results – pass, fail, timeout, aborted. During this time we are processing the results – checking all post-run assertions that apply to the participant and as well as calculating metrics. Also participant logs and artifacts are being saved during this status.
Status: Done
The test has finished its execution successfully and all of the results are available. This status does not necessarily mean that all participants executed successfully – the success rate is always available. If this test status is set we have calculated all data for your test run. At this point you can click on the “View results” button to access the test run results report.
Status: Stopping
If you decide to stop a running test and will click on the “Stop” button, the status will change to “Stopping”. Stopping a running test might be a good idea if you notice that your app is being overloaded or the test is misconfigured or was just launched accidentally. During this time we are sending out requests for participants that have not finished the test yet to stop the execution, the test isn’t stopped immediately to provide as much value as possible by stopping gracefully and preserving results even for aborted participants. Soon the test status will change to “Aborted”
Status: Aborted
The test run has been aborted by the user and all results that could be gathered have been saved. Similar to “Done”, but since the test was stopped early, some participants may not have full results available. This status is also set if test execution did not even start and still was in “Pending” or “Initializing environment” status and an abort request was sent. To make sure your test runs provide all the data possible in the results reports, you shouldn’t stop your test runs, but should always wait for them to stop naturally.
Status: Server error
At some point in the test run’s life cycle Loadero encountered an error that could not be handled to continue test execution. This means that the issue was on our side and you will not get billed for the test run. Sometimes this is a result of a one-off problem, and relaunching the same test may start a successful test run. If such status is set for your test run, feel free to contact our support team via the webchat in the Loadero’s interface.
Status: Database error
At some point during the test run we encountered an error that was caused by loss of database connection or overall database query issues. Similar to “Server error”, this means that the test run will not impact your billing.
Status: AWS error
During test initialization, there was an issue that was caused by Amazon Web Services and we could not handle it to continue with test execution – for example, because AWS could not provide as many server instances as would be needed to handle all test participants. If this happens to your test run, don’t worry, compute units configured for it will not be added to your billing! You can try relaunching the same test, if the test run still doesn’t work correctly – let us know, our support team is there for you.
Status: Insufficient resources
During the “Initializing environment” phase, when trying to set up a test environment, we encountered an issue. It caused the test execution not to start because not enough resources were available to host all test participants. The available test resources change frequently, so you can try launching your test run again after some time with the same configuration. If the issue is still there and you get `”Insufficient resources” status again, contact our support team, so we can investigate why this happens. No worries, for test runs like this, used compute units are not added to billing, because the test was not fully prepared and did not run.
Status: Timeout exceeded
This status is set for a test run if at some point in the run’s life cycle one of the stages took too long and timed out. Usually, this does not happen, if you get the status, most of the time it does not indicate a critical issue in our system. It’s safe to retry launching the test, in case that you get this status again, feel free to contact our support team! For such test runs the used compute units will not be added to project billing.
When are compute units deducted?
A very popular question that we get is – how each test status impacts project billing (compute unit usage). Short answer – compute units are only deducted if the test ran its full life cycle without an error on our side and if the user aborted the test run after the “Pending” state. If participants failed because of Selenium error or assertions – they will be billed, even if they failed because of compilation error. To make sure that you don’t have to pay for the tests that were not properly set and ready for a successful run, it is a good practice to always first run a test with a small number of compute units to validate that the script works correctly and all the settings are set according to your needs. And if you already started a test run and it is in progress, you should let it run and finish naturally to get all the possible results, even if the test will fail. This might help you easier debug the test to fit your needs.
We try to provide as descriptive test run statuses as we can to let users know what is happening to their test runs at any given moment. Half of the statuses actually indicate what went wrong for the test run – but hopefully, you won’t be seeing them! Anyway, if you have a feeling that something is not right, feel free to contact our support team using Loadero’s webchat and we will look into it as soon as possible!
