Three years ago, we benchmarked RSAS, Icecast-KH, and SHOUTcast to see how each performed with 10,000 listeners connected. RSAS was shown to be the fastest streaming audio server, with the lowest CPU usage, which is the main bottleneck. With the release of RSAS 1.0, we decided it was time for a new round of benchmarks.
This year, we're going to try ramping up each server to 240,000 listeners and see what happens.
We'll also be trying to improve our benchmarking technique and answer two questions:
1. Which streaming audio server can handle the most listeners?
This question is important for broadcasters because more efficient streaming server software means being able to serve more listeners on the same (or even slower) hardware, leading to cost savings. It is also important to test the maximum capacity of streaming server to check scaling behaviour:
- Does the streaming server effectively use all available cores?
- Are there any internal bottlenecks in the streaming server that prevent it from scaling?
This year, we're also testing vanilla Xiph Icecast. Although the "Icecast-KH" fork of Icecast is regarded as having better performance, we wanted to see how Xiph Icecast actually compares.
2. How consistent is the performance of each at high loads?
We've heard from numerous users that Icecast has a listener limit of around 24,000 listeners, after which playback goes choppy. Today, we're going to investigate this and see if we can reproduce it. We're going to push each streaming server to its limit and see what happens.
Test Setup
We've upgraded our test systems since our last benchmarks, so we can try higher listeners counts:
- Server PC: Ryzen 2700 (8c/16t), 32 GB DDR4 (@ 3200 MHz), Ubuntu 20.04
- Client PC: Ryzen 5950X (16c/32t), 32 GB DDR4 (@ 3600 MHz), Ubuntu 20.04
Both PCs were directly connected using a pair of Mellanox ConnectX-3 Ethernet cards and a 40 Gbps direct attach copper cable.
On the client PC, multiple instances of ab were ramped up and connected via HTTP to the server PC, each simulating 1000 concurrent listeners,
until we reached 240,000 or the server's limit. Standard tuning was also applied to each PC to allow them to handle high numbers of network connections (increased socket descriptor limits, etc.).
Listener counts were recorded by polling the status page of each server.
The server PC was configured with the following software:
- RSAS 1.0.0 pre-release
- Icecast 2.4.4
- Icecast-KH 2.4.0-kh15
- SHOUTcast 2.6.0.753
These are all the latest versions as of August 30th, 2021. A single 64 kbps MP3 relay was configured on each Icecast-compatible server. For Shoutcast, a single 64 kbps MP3 stream was encoded by Rocket Broadcaster using the Shoutcast 1 protocol.
Benchmark Results
CPU Usage vs. Listeners
Xiph Icecast
Surprisingly, vanilla Xiph Icecast didn't even get off the ground and failed our test immediately. Spinning up 3000 listeners immediately overloaded Icecast, causing it to become unresponsive and freeze. We even slowed down our test to only spin up 100 listeners every 3 seconds, and this too caused it to become overloaded. We were unable to collect data because when our benchmark suite polled the Icecast status page to retrieve the listener count, Icecast was unresponsive.
Since it couldn't run the test, Xiph Icecast is omitted from our graphs and the rest of our analysis.
Icecast-KH
Icecast-KH seems to fair well for the first 50 seconds of the test, as it ramps up to about 30,000 listeners. However, beyond this point, we see the CPU usage going flat. Icecast-KH is accepting new connections, yet somehow the CPU usage is not increasing. What's going on?
When manually connecting with a player, we can hear the problem - We're able to connect, but Icecast-KH doesn't send us any audio. Despite low CPU usage, once the Icecast reaches this saturation point, listeners are able to connect, but are no longer being served streaming audio. This is problematic because it means your production Icecast server may be failing to serve listeners and hitting an internal bottleneck without you realizing.
We'll see that this theory is confirmed in the bandwidth graph below.
Shoutcast
Shoutcast fairs similarly to Icecast-KH in this test, but exhibits slightly different behaviour. After about 20,000 listeners, Shoutcast's CPU usage stops increasing indicating the server has reached saturation and the rate at which it can accept new listeners drops. Once it reaches saturation, it is no longer able to keep up with adding new listeners, so by the end of the test, it only reached 60,000 listeners. At saturation, response times increased majorly and new listeners were not able to receive audio.
RSAS
RSAS is the only streaming server that manages to run this test successfully, accepting and serving audio to 240,000 listeners simultaneously in our test setup. The graph shows what we want to see: CPU usage increasing linearly as listener counts increasing. The spikes are showing the extra work RSAS is doing to accept a batch of 1000 new listeners and are normal behaviour. Above 200,000 we see that the rate at which it accepts new listeners slows down, but it manages to keep up and successfully reaches 240,000 listeners and serves live audio to every listener.
How do we know RSAS was serving audio successfully to each listener?
To help verify this, we measured the bandwidth usage while running these tests, in addition to doing manual testing. Before running each test, we connected as a real listener to ensure the audio was not choppy as the load test ran. We also connected as a new listener at the end of each test, to verify new listeners were still receiving audio.
Bandwidth vs. Listeners
To help interpret the odd CPU usage scaling behaviour and reports of choppy audio with Icecast at saturation, we measured the bandwidth used during each test:
The bandwidth graphs above are consistent with the CPU usage behaviour we noticed earlier and explains why listeners receive choppy audio or no audio with Icecast-KH and Shoutcast after saturation - Those servers simply stop sending audio!
RSAS is the only server that actually manages to serve live streaming audio to 240,000 listeners. We can see that the bandwidth scales up linearly with the listener count, as it should, and that it reaches the expected bandwidth usage for this number of listeners (64 kbps x 240,000 = 15.4 Gbps).
RSAS is able to accomplish this by efficient use of multithreading and optimizations for modern hardware. We optimize RSAS not just for bragging rights, but so that it scales efficiently on your hardware and is easy to monitor in production. Even at 95% CPU usage, RSAS is still able to serve smooth audio to your listeners, whereas Icecast and Shoutcast start having problems at much lower load levels, which make them difficult to monitor and troubleshoot in production.
"Why is my stream choppy?" - Here we showed that the answer can be that your Icecast or Shoutcast server is saturated, despite having low CPU usage.
The Bottom Line
RSAS comes out on top again as the highest performing streaming audio server, allowing broadcasters to serve more listeners than ever before with the same hardware.
Questions about RSAS or planning high capacity streaming audio? Get in touch!