FTP Security Alert

***** IMPORTANT SECURITY ALERT *****

***** FTP HAS BEEN GLOBALLY DISABLED *****

A handful of customers have reported that their music and log folders have suddenly disappeared and after inspecting the associated ftp log files, we were able to find a single common ip-address (95.173.136.168) associated with deletion of files.

Although we cannot prove that the aforementioned ip-address is the culprit, it is highly suspicious that a single ip-address would be associated with the deletion of files across multiple systems.

Needless to say we have temporarily banned the ip network (belonging to Russia) pending a security review and are evaluating other points of exploit.

The impact has been limited to a small portion (at current count about 8) of retail customers located primarily in the Netherlands and South America. At this time, we find that producers in other regions (North America, Asia and the rest of Europa) have not been affected.

Out of an abundance of caution, all customers are encouraged to login to the customer control area (www.sonixcast.com) and to change their passwords both on their account and individual services.

Because of the limited scope of the event, we have no reason to believe that any wide ranging exploit or brute force attack is occuring. FTP passwords are commonly shared by Producers for diverse reasons and we believe a bad actor with malicious intent might be the culprit. However, we highly recommend that all customers update their passwords just to be on the safe side.

***** UPDATE 2018/15/03 *****

We were contacted by facebook and DHS over a month ago that the SoniXCast Producers Group (https://goo.gl/xHzsbN) was being targeted by Trolls, Hackers and Propagandists. We are cooperating with both organizations in order to weed out culprits and were instructed to make no announcements.

However, now that we feel investigation has advanced far enough and innocent individuals are beginning to be affected, I feel it is my duty to inform all to tread cautiously (like elsewhere on facebook) with the information and individuals in the SoniXCast Producers Group. Especially if it concerns unofficial network or system announcements as the majority of the Trolls and Propagandists investigated so far are tied to SoniXCast competitors.

Hackers will try to make direct contact, so I recommend taking extreme caution sharing sensitive information (hostnames, ports, username, passwords and the like) with individuals in the SoniXCast Producers Group.

The only official channels for trusted corporate communications is our website (www.sonixcast.com) including the knowledgebase, announcements, emails from support and ticketing systems, our info blog (info.sonixcast.com) and the official SoniXCast Support Group (https://goo.gl/yquHLX). We are very transparent about our network and system availability which can be viewed in real-time at https://goo.gl/p3gzwf.

To demonstrate the gravity of the situation, last week a handful of customers had their streaming accounts hacked into and all data erased (no personal data was compromised as that sits in a separate highly secure area with no access to the internet). With the help of diverse government agencies, we were able locate the attackers and alert the local authorities. More info here: https://goo.gl/wrsvhA

We view the SoniXCast Producers Group as a discussion group where producers can help each other and exchange ideas. The support team will NOT monitor the SoniXCast Producers Group for support issues. For support use the aforementioned official channels of communication. However, Vincent Reilly will continue to administer the group and forward possible support issues onto the main support team.

IMPORTANT: Those who rant or make outrageous claims should be treated with extreme caution as the threat is ONGOING and there are many eager to make your life difficult. Let common sense prevail.

The Affect of Russian Hacking

As previously reported, The US Justice Department recently indicted 13 conspirators of which one Sergey Pavlovich Polozov was personally responsible for hacking into SoniXCast systems in 2014. We now know why don’t we?

Just in case you missed it

The indicted Mr. Polozov was tasked with putting together systems in North America to be used to mount the so called “Information War” against the United States 2016 elections. In the runup, in June 2014 he along with 2 other co-conspirators broke into and took control of SoniXCast Virtual Private Servers in Toronto, Montreal and New York City. Since we couldn’t wrest control back from the hackers, we had to destroy the VPS’s and rebuild customer accounts and services from backups of our customer database. That was an “all hands on deck” effort which took about a week until everything was back to normal.

However, the hackers didn’t cover their tracks very well and we were able to obtain their personal information which we passed onto the Department of Homeland Security. As far as we know, Canada does not have an active counter espionage unit. To date: no Federal Agency has ever contacted us for further information on the subject, so we exacted some self help against the hackers that was sure to have caused some inconvenience.

One of the co-conspirators was the German national Udo Poschen who also Trolled for the German Performance Rights Organization GEMA. In a separate lawsuit, we were able attain Mr. Poschens bank records that ultimately showed that he had received payments both from the Russians and GEMA which was also reported to the authorities. Since then, Mr. Poschen has been pretty much inactive on the Internet.

But that is only a drop in the bucket

On an average day, our network team responds to over 300 network attacks. Most are very amatuer attempts to brute force into our systems or some really laughable ransom attempts, but at least once a week we receive a serious threat. An overwhelming amount (over 80%) come from US-based systems including the most recent attack that took out the router (according to OVH) in our Montreal Data Center. Over 60% originate from Russia. The remaining 40% originate from Europe and China.

So, What’re ya doing to mitigate the effects?

Immediately after the 2014 instance we moved sensitive data into a private network inaccessible from the Internet and implemented aggressive backup plans to ensure we could quickly rebuild our systems in the event of catastrophe. We also developed a sort of “Hot Swap” concept that would allow us to quickly bring a mirror of each server online in the event of attack.

The biggest crux in our efforts has been the provider. We have to host in Canada because of the broadcast license and despite popular belief, the Canadians just aren’t as technically savvy as US providers are. Further, it takes an act of congress to get exasperated OVH technician to do anything. The folks at iWeb are much more responsive (and friendly), but still technically really, really weak.

Edge firewalling is not even in the average Canadian providers vocabulary and the concept of intrusion detection systems might as well be aliens from outer space. Leaving providers like us to resort to self help in securing our systems against hackers and the occasional overzealous technician (they cause more downtime than you think).

Recently we came across a US provider who built their own cutting edge data center in Canada and other worldwide locations. After testing their technology for months, we feel confident that we’ve found a data center provider that can meet our security and networking needs and have begun moving our infrastructure over to them.

Moving is a complex multi-pronged effort that will take months to complete fully. We’ve already moved most of our web infrastructure over (which is why the websites are faster now). Next will be the cluster and relay networks, commercial partners, then finally retail customer services. We expect some obvious short (seconds maybe minutes) downtimes with the final phase, but we’ll give customers plenty of time to plan for it.

More importantly we are moving from a network that is ripe for the picking by hackers and where monkeys hammer on the hardware in order to keep it running to a shiny new government grade adaptive network where the features are not just marketing hype and SECURITY is written large.

For example; one of the things we tested was the server “Hot Swap” technique we pioneered, but never got to work efficiently because of the limitations of the provider. In a recent test with over 50 server services running, we were able to swap a server within a single ping (10ms) and not one listener dropped. Cool Huh?