As many of you may have noticed, your streaming services went offline at or around 01:00 hours on Saturday, July 30th, 2016 MST. Our network operations department immediately notified me (kicking me out of bed) to investigate the issue.
After a quick preliminary investigation I found that the hard disks for the production farm (cabhs3x) had been emptied of all customer and application data. In essence the hard disks were erased. However our main customer and service database (which sits in an FBI protected location in the United States) remained intact and preliminary investigation shows that all customer and service data was protected from the attack. Simply stated: The attackers DID NOT get your personal or payment information.
So…When can I get back to business?
We were hacked in 2013 and the lessons learned are making it possible to get your services back online quickly so that you can at least broadcast using your source software by this evening at the latest. The most of the time will be brought in re-installing services and BoomBox on each server node.
Details of the Attack
Our information on the attack is still developing however this much we know thus far: On or about 20:00 hours on Friday, July 29th, 2016 UTC we began receiving system notifications that brute force attacks were being made on the server farm. Our network operations team immediately blocked the ip-addresses: 220.127.116.11 (lh28409.voxility.net), 18.104.22.168 (chomsky.torservers.net), 22.214.171.124 (torrelay4.tomhek.net) and performed a security audit on the servers. Once I was sure no more attacks were occurring, I went to bed, but the network operations folks remained vigilant.
Roughly 3 hours later I was contacted by network operations that services had gone down and began to investigate. I found that on all production servers and hot swaps the hard disks were deleted of all customer and application data and the operating systems could no longer function normally.
I immediately contacted the FBI protected operations center in the US to verify that customer and application information was safe. They are still checking, but thus far have found no indication that any attacks were made. I have been promised a comprehensive report by the end of next week.
So, what did the hackers get?
As far as we can tell thus far, nothing. The production server farm (where your streaming services are located) is by nature unsafe and functions as “dummy” slave to the server farm located in the US. Meaning that NO CUSTOMER DATA (not even your email address) was stored any where on those systems.
Okay. So why is it taking so long to get things working again?
Whenever a server is hacked, you have to assume that the entire system is compromised and the only way to ensure it isn’t, is to completely re-install the system. In this case the hackers deleted application data leaving the operating system in an non-functional state. So, we have no choice but to re-install the systems. That takes some time (roughly 3-5 hours per server) plus testing and security lock down.
The hacker(s) also took advantage of an exploit in the hot swap servers which allowed them to power up the nodes and delete data on them as well. Hot swap servers are mirrored versions of the production server we can bring online in an emergency. Server nodes in the hot swap platform must be cognizant of each other in order to achieve failover and we believe that by hacking one server node, the hackers we able to access other server nodes as well.
Also as you know BoomBox is under heavy development and we need to collate and test the latest changes to make sure they show up on the servers once everything has been installed. Thankfully most of these processes happen simultaneously minimizing downtime.
Why are the production servers inherently unsafe?
Customers need to have write access to certain areas in order to upload files, manage configurations and start services and even though we filter and scan what is uploaded, most customers are laymen (and therefore not as security minded as we are) and could, inadvertently, upload something that would allow hackers to get in or give their passwords to someone they thought they could trust.
For that reason we keep the entire service configuration in a database located somewhere else so hackers can’t get to it and BoomBox is designed in such a way, so that restoring a service configuration only takes seconds. That is of course unless all the servers are hacked simultaneously and we have to re-install them as was here the case.
Okay. So, what was lost?
Just your uploaded music files and before you groan, the reason we do not backup music files is that they change (in some cases) on a minute to minute basis and the cost of backing up everybody’s music files all the time would cause prices to skyrocket. Besides the time it would take to restore all those files would add hours to the time needed to get your services back online.
I’m just as angry as you are, because now I have to re-upload the entire global music library.
So, how are you going to ensure this doesn’t happen again?
I’m not sure we or anybody can. Hackers only have to be right once. We have to be right all the time and there is more of them then there are of us. I don’t know for sure yet if it was a brute force attack or an Trojan uploaded, again inadvertently, by a customer. The server nodes are locked down according to FBI standards and we perform regular security audits, but Hackers are persistent and if you punch somebody in the face long enough, eventually they’ll go down.
We do have our enemies. In 2013, the last time we were hacked, the FBI found, what they deemed, a high confidence of evidence that the German Udo Poschen and the German performance rights organization GEMA were somehow involved, but apparently there was not enough evidence to be actionable. I do know they were contacted by the German government, because I still have the nasty emails they sent me afterward. I tried to explain that we had nothing to do with the investigation and that we are the victims here, but you know how people are.
Nobody has claimed responsibility for the most recent attack. I don’t believe we are big or important enough for anybody to make any such announcement. I think some random hackers found us through our website or through shoutcast.com and, as I said before, the FBI is investigating. I wouldn’t be surprised if they found out that it is some competitor however, because it was one in 2013.
On more thing. This downtime is considered an “Act of God” in our terms and services agreement. I always have customers who scream for refunds when we’re down for even a couple of seconds, so NO there won’t be any refunds or compensation. The hackers could have just as easily hacked your personal computer and you would have an equal amount of recourse had that happened.
Not to cause anybody alarm, but it is theoretically possible that the hackers got your ip-address and may have already made attempts on your computer or those of your listeners. It might be a good idea to scan your computer to see if it’s been compromised. I did.
I hope I was able to cover the most relevant topics for you. Remember, we’re going to be busy working to get things back up, so support might be a bit slow for a few days.
One final thing. Most people consider those who have been hacked as being incompetent, lazy and unreliable. Let me remind you that governments and corporations who have much more money to invest in security than we do are hacked on a daily basis. And more often than not the hackers get customers personal and credit card information. I’m proud of the fact that no such thing happened to us and I will continue to keep it that way. I hope you keep that in mind when you render your service cancellation.
I will attach new information as I have it to this blog post.
7/31/2016: All accounts have been restored and servers setup and started. Broadcast credentials were sent to individual users via email. The broadcast credentials, hostname, port and any other settings remained as before.
The reason this took longer than expected was because routines were need to restore accounts from the database settings instead of simply creating new accounts which would have overwritten configuration settings.
8/1/2016: The BoomBox Radio Automation Control Panel is back online with limited functionality. Customers may now start or stop their servers. Other features and components (auto-dj, playlist/user management…) will be released throughout the day as they complete testing. Customers who cannot login, start or stop their servers are to contact support immediately.
Implemented multi factor authentication on the hot swap servers. As stated above, part of the reason this attack was so devastating, was that the hackers were also able to access the hot swap backup servers through the production servers. Now human intervention is required for changes in the operating system. This will ensure the integrity of the system and immediate fail over in case a production server is hacked again.
8/2/2016 (07:43 MST): Released statistics and server management components.
8/6/2016 (04:35 MST): Released basic transcoder management.
Thank you all again for your patience.
We’ve been obviously slammed with support requests and development issues while getting BoomBox back up and running. If you haven’t logged in recently, we’ve made quite a bit of progress and hope to have more done by the end of business today.
The hold-up was centered around finding a version of the shoutcast transcoder that would work with the current version of the shoutcast server. In case you didn’t know, nullsoft (and those who bought them later) stopped development of the shoutcast transcoder in 2011 and it never got out of beta which means there are a lot of undocumented “features”. We were challenged to find a version that had most “features” working and still performed basic operations properly.
Historically we were committed to using the shoutcast transcoder because of the 3rd party software we migrated off of. We were looking into working in other transcoders (like liquidsoap) just before we were hacked which threw development off course.
I understand that some customers have been frustrated, but unlike Radionomy and like providers, we are committed to communicating the issues as bandwidth permits. Service cancellations have been thus far almost non-existent and we like believe that has to do with said communication policies and each day means more progress which we demonstrate by releasing incremental updates to the BoomBox platform.
There is still a bit of work to do, but we are now on the fast track to success thanks to recent developments. We ask that you remain patient while we slog through the issues. Your patience and loyalty will be rewarded.
8/6/2016 (07:45 MST): Manual server upgrade routine released.
Each customer is required to perform the manual server upgrade in order to guarantee that their radio stream is functioning properly. This is the first step in releasing the transcoder components (user, playlists…).
The reason for the upgrade: What we discovered is that the shoutcast server and transcoder versions compiled for 64bit operating systems are fundamentally different than those compiled for 32bit (The BoomBox platform is 64bit). This caused major issues with extensive memory leaks which lead to buffering, the reading of music tracks tags, buffer overrun (speeding up of track playback) and mini server (admin.cgi) crashes. It also adversely affected the way the authhash was sent to yp.shoutcast.com.
What we’ve done is wrap the 32bit stable versions of the shoutcast server and transcoder in a type of “virtual” machine code which executes base functions in 32bit while interfacing with BoomBox in 64bit. This is a gross simplification of the process, but easiest to understand. The result is a much more stable and buffer free service.
Because the server must be restarted, customers are required to login to BoomBox and perform the upgrade manually. The process only takes a few seconds, but we recommend that customers notify their listeners before performing the upgrade.