The last thing a business wants is to be known as an unreliable and poorly performing service, especially if there are similar solutions a few clicks away. So being aware of the performance of a WebRTC application or any other software solution is a must to avoid issues in the future. A solution can be developed by experienced people and tested before it is released, but even so it doesn’t mean performance degradations will never appear. Noticing issues early and taking action accordingly can help a lot to make the users have a better experience.
Imagine that your video calling application has issues when packet loss occurs. Packet loss can cause choppy audio, frozen video, or missing video frames, making it hard for users to understand what is being said or see what is happening on the call. High packet loss can also make it difficult for the app to establish and maintain a connection, leading to dropped calls or video chats. If you and your team didn’t experience packet loss while working on the application’s development, there is one way you can find out there is an issue – a disappointed customer can inform you about their unpleasant experience. Of course, it would be a lot better to find out about the issue and fix it before users experience it.
This is why monitoring is such a beneficial practice – regularly observing the performance and behavior of the product over time allows you to react quickly to any issues that arise and maintain a reliable and high-performing service. This not only improves the user experience, leading to increased customer satisfaction and loyalty, but also gives your business a competitive edge in the market.
Benefits from regular application monitoring:
- Getting notifications about issues early. By setting up the monitoring and test failure notifications, you’ll be alerted to problems in real-time and be able to take action in a timely manner before users have a poor experience.
- Monitoring provides detailed reports that can help you identify the root cause of failures and pinpoint areas of your app that need improvement.
- Saves lots of time. The monitoring is an automated process, which means that you can create the solution once and just let it work with little to no maintenance.
- Confidence that your solution works fine. When you have the monitoring in place, you can have the peace of mind that your application is being checked regularly, and works fine if you didn’t get any notifications recently.
- Ability to compare performance over time. When your monitoring setup runs the same tests regularly and does measurements, you can later aggregate the data and see how the performance was changing over time.
How WebRTC app monitoring works with Loadero
Loadero is a load and performance testing tool that is capable of simulating real user behavior and interaction with websites by using web automation technologies, such as “Javascript + Nightwatch”, “Java + TestUI”, or “Python + Py-TestUI”. Along with that, it provides you with all the necessary WebRTC and machine statistics such as FPS, bitrate, jitter, round-trip time, and others, that were gathered during the test run and is capable of asserting them.
As monitoring requires constant observation, e.g., constant test runs in our case, we would also need to periodically trigger test runs. For that purpose a CI/CD pipeline is a great choice as it is flexible and is not complicated to configure for our task. Besides running the tests periodically, the pipeline can also be used to automate testing after each deployment to make sure the performance is fine.
In order to start monitoring your WebRTC solution with Loadero a setup consisting of 3 parts is required:
- A Loadero test that will assess your WebRTC app’s performance. You can find an example here.
- A script that uses Loadero API to launch the test, get the results and notify about the failure.
- A CI/CD pipeline to automatically run the script.
Setting up a test for the monitoring setup
Let’s start by setting up the Loadero test for our monitoring setup example. If you already have a test in Loadero, which can be launched to check some of the performance metrics, you can skip this part and jump to the part about launching tests via the pipeline. If it is your first time working on a test in Loadero, a complete step-by-step guide on how to create a test in Loadero can be found in this blog post. If you already have a WebRTC performance test in another service, see how you can migrate your test to Loadero here. In our case, we will have a “Javascript + Nightwatch” test. The scenario of it will be a one-minute long 1-on-1 call in Jitsi.
In the test two participants will join a call and stay in it for a minute, taking a few screenshots midway for additional connection verification.
The participants will connect from US Oregon, will use the latest Google Chrome version (109v as of writing this blog), and will use default video + audio to simulate the output signal.
We will also use Loadero’s post-run assertions. They allow you to specify the “pass” criteria for WebRTC and/or machine metrics in your test, like “if average FPS ≥ 10, then pass”. After the run is complete, assert results are calculated automatically for each participant to check if the given values have met the pass criteria. If the assertion failed for a participant, this participant’s status in the results report will be “Fail” too.
With the asserts, we will assess the incoming and outcoming bitrate and packets for both audio and video as well as FPS.
The configuration of our test would be something like this:
- Test mode: ‘Performance test’
- Increment strategy: ‘Linear participant’
- Start interval: 1s
- Participant timeout: 5min
In this example we have a Loadero test script written in Javascript + Nightwatch, but the same can be done with Java + TestUI or Python + Py-TestUI.
client => {
client
// Open the page
.url(`https://meet.jit.si/LoaderoTests`)
// Wait until the username field is visible
// And enter the username
.waitForElementVisible('[placeholder]', 30 * 1000)
.sendKeys('[placeholder]', 'User')
// Wait until the "Join" button is visible
// And join the call by pressing the button
.waitForElementVisible('[aria-label="Join meeting"]', 10 * 1000)
.click('[aria-label="Join meeting"]')
// Another thing you can do is to take screenshots during the test
// Which could help you to identify the cause of a failure
// And give visual feeback about the test run
.takeScreenshot('pre_call_screenshot.png')
// Stay in the call for half a minute
.pause(30 * 1000)
// Take a mid-call screenshot
.takeScreenshot('mid_call_screenshot.png')
// Stay in the call for another half a minute
.pause(30 * 1000)
// Take a post-call screenshot
.takeScreenshot('post_call_screenshot.png');
}
Setting WebRTC performance expectations
Each WebRTC application is different and will have slightly different performance. Here we aim to define the thresholds where we want to be immediately notified if the performance expectations are not met, even if that happens during the night. To do this we’ll use a set of Loadero’s post-run assertions for WebRTC metrics, which will make the whole test run fail if WebRTC performance metrics values are not as good as we’d like to see them. Hence the target values should not be set on narrow margins, but we should allow for the tests to be occasionally slightly worse than we would ideally like, but still within reasonable performance. The values we have set here are examples and you may want to adjust them based on the application specifics (for example, if you prioritize a high frame rate over network bandwidth, you might want to increase the minimum frame rate you want to see and decrease the bitrate limits). As a starting point, you can use the asserts list from below:
Here is the list of assertions and their values that we set for this example test:
- Assert that the video has at least 10 frames per second for more than 75% of the call duration in both incoming and outgoing video streams. Additionally, check that the fluctuations in framerate are small, no more than 2 frames per second, by setting an assertion on standard deviation:
webrtc/video/fps/out/25th >= 10webrtc/video/fps/in/25th >= 10webrtc/video/fps/out/stddev < 2webrtc/video/fps/in/stddev < 2
- Checking the values of packets sent per second is important to make sure that the performance is optimal. Sending too many packets per second will introduce more overhead, but by sending more packets the jitter can be lower as the delay between packets is smaller. A good balance must be found and it can differ from application to application. In this example we’ll check if packets sent per second number is between 40 and 100:
webrtc/audio/packets/out/avg > 40/secwebrtc/video/packets/out/avg > 100/secwebrtc/audio/packets/in/avg > 40/secwebrtc/video/packets/in/avg > 100/sec
- Bitrate indicates the amount of data sent per second. Generally the higher quality media is sent, the higher the bitrate. However some applications will attempt to minimize the bitrate to work better over bad networks and consume less data. With these asserts we check that data consumption doesn’t exceed set values of 25 kbit/sec for incoming and outgoing audio and 1000 kbit/sec for incoming and outgoing video for 95% of the duration of the test:
webrtc/audio/bitrate/out/95th <= 25 kbit/secwebrtc/video/bitrate/out/95th <= 1000 kbit/secwebrtc/audio/bitrate/in/95th <= 25 kbit/secwebrtc/video/bitrate/in/95th <= 1000 kbit/sec
Configuring test participants and running the test
Finally, we have to configure test participants. We are going to have 1 group of participants which will have 2 participants with the same configuration that will join the call. The participants’ setup is the following:
- Title – ‘Participant’
- Count – 2
- Compute units: G2
- Browser – ‘Latest Google Chrome’
- Location – ‘US West – Oregon’
- Network – ‘Default network settings’
- Audio feed – ‘Default audio feed’
- Video feed – ‘Default video feed’
Tip: if you want to have many participants with the exact same configuration, then in the configuration menu increase the Count value. In order to save your time, we suggest creating participants individually only in case when they need to have different configurations.
For the test configuration that would be it. But as we intend to have our test run regularly, before we proceed to configure our monitoring setup, run the test and verify that the script is not faulty by checking the Selenium logs and screenshots that participants took during the test run. If it is your first time launching a test in Loadero, or you just are not sure if it was configured correctly, use this blog post to check if your test is ready for launching.
As mentioned previously, participants may still fail if the WebRTC metric assertions fail and the success rate might not be 100%. This happened for our test too.
This doesn’t necessarily mean that the test is faulty, only that the app does not meet the assertion criteria you have set. But if your test failed due to another reason, not the assertions set, this blog post explains some ways to debug your test.
The asserts we set are just a general baseline not tailored to the application we test. You can use the list and start with those values, but the app you test might differ and the metric results could be very different too.
Tip: One good way to set your own assertions would be to do 5 test runs, look at the participant metrics and evaluate reasonable goals.
Setting those assertions and analyzing results to find out why the asserts failed is a complex task itself, so we’ll not go into too much detail about it in this blog post. Also, the fact that our example test failed due to an assertion fail simulates exactly the case we need, when a test failed and a notification about the failure is sent, so we will leave it as it is. If Selenium logs show that you haven’t encountered any errors and screenshots confirm that all the necessary actions were taken, then the test is still good to go.
Launching tests regularly from the pipeline
Our team has already prepared a few blog posts on how to integrate tests in your development pipeline: using JS and Loadero’s Python client. Therefore here we are going to rely on them. For our setup we will use what is suggested in the blog post that uses Python, GitHub and its CI/CD implementation – Workflows along with Loadero Python client.
Note: some information might be skipped. For in-depth instructions refer to the original Loadero Python blog post.
For our workflow we will need:
- A GitHub repository
- A YAML file that defines the workflow
- A Python script to run the Loadero test and report the results
Create a new repository on GitHub or use an existing one.
In your repository in the .github/workflows directory create a notify-on-fail.yml file. This is the file that is going to contain instructions on how to set up the environment and launch the Loadero test.
Let’s start defining the workflow by specifying the triggers in the notify-on-fail.yml.
on:
schedule:
- cron: '0 9-18 * * *'
workflow_dispatch:
schedule allows you to trigger a workflow at a scheduled time. Therefore it is the place where you define the frequency of the test runs. In our example we have set the schedule to run the test every hour from 9am till 6pm, as it is the time when the team is most likely to be able to react to failure. In case you may need to run tests throughout the whole day-night cycle, you may prefer to run them every 4 or so hours. Thus you monitor the app even when nobody is awake. Pay attention, that schedule uses a specific syntax more about which you can learn here.
Tip: When setting the frequency of tests, consider that the period between the test runs should be longer than the duration of the test, as your tests may interfere with each other.
The second trigger – workflow_dispatch – allows triggering the pipeline manually via GitHub web application if need be.
In our jobs section of the script we specify the environment and all the dependencies that we will use. For our purpose the configuration from the Loadero Python blog post fits perfectly, so don’t hesitate to copy-paste it.
jobs:
notify-on-fail:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.10"
- run: pip install loadero-python
- run: python run.py
env:
LOADERO_ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
LOADERO_PROJECT_ID: ${{ secrets.PROJECT_ID }}
LOADERO_TEST_ID: ${{ secrets.TEST_ID }}
Important: pay attention, that in line - run: python run.py, after the python we have the path to your run.py file relative to the repository’s root. In my case the file structure will look like this:
Another thing to pay attention to is the credentials. Firstly, you can find test and project ID in Loadero, but you will also need a project API access token – you can create one in Project Settings under “API Access” tab. Secondly, the credentials are used as GitHub Actions secrets. This keeps your Loadero access token private. Secrets can be configured in repository settings -> Security -> Secrets and variables -> Actions -> New repository secret.
For the detailed step-by-step instructions about the secrets refer to the previously mentioned blog post.
So the workflow configuration for now should look something like this:
name: Notify about failing tests in Loadero
on:
schedule:
- cron: '0 9-18 * * *'
workflow_dispatch:
jobs:
notify-on-fail:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.10"
- run: pip install loadero-python
- run: python run.py
env:
LOADERO_ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
LOADERO_PROJECT_ID: ${{ secrets.PROJECT_ID }}
LOADERO_TEST_ID: ${{ secrets.TEST_ID }}
Now let’s add to our setup the Python script to interact with Loadero. Once again, the script from Loadero Python blog is a great starting point, so we will use it and modify it later for our needs.
import os
from loadero_python.api_client import APIClient
from loadero_python.resources.test import Test
project_id = os.environ.get("LOADERO_PROJECT_ID", None)
access_token = os.environ.get("LOADERO_ACCESS_TOKEN", None)
test_id = os.environ.get("LOADERO_TEST_ID", None)
if project_id is None or access_token is None or test_id is None:
raise Exception(
"Please set the "
"LOADERO_PROJECT_ID and LOADERO_ACCESS_TOKEN AND LOADERO_TEST_ID "
"environment variables."
)
APIClient(
project_id=project_id,
access_token=access_token,
)
run = Test(test_id=test_id).launch().poll()
print(run)
for result in run.results()[0]:
print(result)
if run.params.success_rate != 1:
raise Exception("Test failed")
Sending notifications about failed tests
Now it is time to modify our Python script in order to implement our notifications. In this example the alert will be sent to a Discord channel. Although the way you implement the notifications is completely optional, since it depends on your personal preferences.
Let’s update our Python file by importing the requests library, ResultStatus and AssertStatus classes.
import requests import os from loadero_python.api_client import APIClient from loadero_python.resources.test import Test from loadero_python.resources.classificator import ResultStatus, AssertStatus
As well as updating our YAML file to install the requests library.
- run: pip install loadero-python - run: pip install requests - run: python run.py
In case the pipeline is misconfigured and the value of any of the credentials is None, we should notify our Discord channel about it by sending a POST request with the error message.
missing_credentials_message = (
"Please set the "
"LOADERO_PROJECT_ID and LOADERO_ACCESS_TOKEN AND LOADERO_TEST_ID "
"environment variables."
)
def send_notification(message):
requests.post(
"https://discordapp.com/api/webhooks/{id}",
data={"content": message},
)
if project_id is None or access_token is None or test_id is None:
send_notification(missing_credentials_message)
raise Exception(missing_credentials_message)
If the test fails, we would like to know the reason. Here we add the participant’s verification. If a participant fails, we send the error message with the following structure:
- Participants name
- Selenium result
- Status
- Failing assert’s path
- Run status
Additionally we should get rid of the exception the initial script had, as it is excessive here
run_failure_message = ""
for result in run.results()[0]:
result.params.run_id = run.params.run_id
result.read()
if (
result.params.selenium_result.value != ResultStatus.RS_PASS
or result.params.status.value != ResultStatus.RS_PASS
):
run_failure_message += (
f"{result.params.participant_details.participant_name}:\n"
f"-Selenium result: {result.params.selenium_result.value}\n"
f"-Participant status: {result.params.status.value}\n"
)
if result.params.asserts:
run_failure_message += "-Failing asserts:\n"
for assertion in result.params.asserts:
if assertion.status != AssertStatus.AS_PASS:
run_failure_message += f"--{assertion.path.value}\n"
run_failure_message += "\n"
run_failure_message += f"Run status: {run.params.status.value}"
if run.params.success_rate != 1:
send_notification(run_failure_message)
And also let’s send a message for the successful test run:
if run.params.success_rate != 1:
send_notification(run_failure_message)
else:
send_notification(f"The {run.params.test_name} test has been finished successfully")
As the final verification we should wrap the whole API call into try-except in case the connection with Loadero API is faulty. The final python script altogether looks like this:
import os
import requests
from loadero_python.api_client import APIClient
from loadero_python.resources.test import Test
from loadero_python.resources.classificator import ResultStatus, AssertStatus
project_id = os.environ.get("LOADERO_PROJECT_ID", None)
access_token = os.environ.get("LOADERO_ACCESS_TOKEN", None)
test_id = os.environ.get("LOADERO_TEST_ID", None)
missing_credentials_message = (
"Please set the "
"LOADERO_PROJECT_ID and LOADERO_ACCESS_TOKEN AND LOADERO_TEST_ID "
"environment variables."
)
def send_notification(message):
requests.post(
"https://discordapp.com/api/webhooks/{id}",
data={"content": message},
)
if project_id is None or access_token is None or test_id is None:
send_notification(missing_credentials_message)
raise Exception(missing_credentials_message)
try:
APIClient(
project_id=project_id,
access_token=access_token,
)
run = Test(test_id=test_id).launch().poll()
print(run)
run_failure_message = ""
for result in run.results()[0]:
result.params.run_id = run.params.run_id
result.read()
if (
result.params.selenium_result.value != ResultStatus.RS_PASS
or result.params.status.value != ResultStatus.RS_PASS
):
run_failure_message += (
f"{result.params.participant_details.participant_name}:\n"
f"-Selenium result: {result.params.selenium_result.value}\n"
f"-Participant status: {result.params.status.value}\n"
)
if result.params.asserts:
run_failure_message += "-Failing asserts:\n"
for assertion in result.params.asserts:
if assertion.status != AssertStatus.AS_PASS:
run_failure_message += f"--{assertion.path.value}\n"
run_failure_message += "\n"
run_failure_message += f"Run status: {run.params.status.value}"
if run.params.success_rate != 1:
send_notification(run_failure_message)
else:
send_notification(
f"The {run.params.test_name} test has been finished successfully"
)
except Exception as err:
send_notification(f"Error while running Loadero test: {err}")
Inspecting WebRTC monitoring results
Now our test automatically runs every hour from 9am till 6pm and assesses whether the app performs as expected.
After the test execution has finished, in case of failing the test you get notified about the failure.
In our case the Jitsi test has failed due to FPS and packets not meeting our criteria, which can be seen in the Asserts tab of the test result. The assert results are also accessible for every participant individually, so you can verify whether the problem has occurred for all of the participants or only for a portion of them.
The values provided above for the asserts can serve as a beginning reference, and it seems that in our case they don’t align with the target performance of Jitsi. So don’t hesitate to explore how your app performs and what asserts fit your app the best, to ensure that the monitoring process is optimal.
Note: just by looking at the asserts of our test we can note that there are entire sections that output zeros throughout the test run, which ultimately impacts test results.
Tip: If your test failed, you can navigate to the WebRTC statistics tab, where you can find various graphs with data and get more information about the metric that has caused the failure.
In this blog post, we have provided an example of how you may monitor your WebRTC application with the assistance of Loadero, and GitHub workflows. With the use of monitoring you are better equipped to deal with any issues that may arise in your application. It may also be wise to create other scenarios with different conditions for a more holistic view on the application’s performance. Setting such monitoring can be quite a complex task. Would you like Loadero team to create a similar monitoring setup for you? Feel free to contact us at support@loadero.com and let’s discuss how we can help you regularly and automatically monitor your WebRTC solution.
