Migration BB23 and BB24

At 00:00 hours UTC Tuesday, February 27, 2018 we will be moving bb23.sonixcast.com and bb24.sonixcast.com into our new Data Center in Toronto Canada. This will also be a hardware upgrade giving both systems more resources to better serve client resources. Expected downtime is nominal and should only be a few minutes, but clients on these systems should expect longer downtimes of up to one hour.

THE IP-ADDRESS FOR BOTH SYSTEMS WILL CHANGE

If you are using the ip-address (instead of the hostname as you should), be advised that the ip-address will change and that you will have to contact support to get the updated ip-address after the migration is complete. We DO NOT post ip-addresses publicly on our websites or in social media for security reasons.

Additional Security Information

All systems going forward will block ICMP requests which means ping and traceroute operations from client computers will no longer function. Please refer to the customer area to check system status from this point forward.

Networks allocated to Russia and Russian aligned nations have been quarantined and will be extensively monitored going forward. Clients with computers on these networks should expect some slowing of the connection as backtraces are performed to determine the MAC address of your computer. If this test fails you will be blocked from our network.

Networks originating from China and North Korea have been permanently banned as well as any proxy networks and services in all countries.

New Web Hosting Products

We are proud to announce the release of our new cPanel Web Hosting Cluster. Now faster and more available than ever before! Existing customers can get their own mywebsite.sonixfm.com subdomain and Web Hosting FREE for one year by using the promo code FREEWEB when ordering their product.

Why Should I host with SoniXCast?

Blazing fast and highly available: Our new infrastructure sports 3 new AnyCast DNS servers and 24 geographically located store and forward caching proxies that ensure that your website is as equally fast regardless of where you are in the world.

Government grade security: Hardware intrusion detection will stop DDoS and brute force attacks in their tracks and strict quarantine of “Bad Actor” nations updated daily by the US State Department and the Department of Homeland Security will ensure your data stays secure from ransom attacks and takedown attempts.

Ease of Use: Integrated tightly with our customer area, cPanel is the World’s Number 1 Web Hosting Control Panel. Well documented and designed for folks who are new to Web Hosting, setup takes only seconds and with One-Click installs, you website will be up and running in minutes!

BoomBox and SoniXFM integration: Upcoming releases of both BoomBox and SoniXFM will give producers more flexibility in popularizing their radio station. We’ve built our cPanel servers to integrate tightly with both. Besides; producers should have their websites hosted in Canada anyways to avoid license disputes with their local Performance Rights Organization.

One Time Offer for existing Customers

Now! For a limited time; existing radio producers can order SoniXCast Web Hosting for One Year absolutely FREE! Just use the Promo code FREEWEB when ordering your product. Simply select the “use subdomain” option and enter your desired name (like the name and or call-sign of your radio) to get started quickly and for FREE. Later you will be able to link your account with BoomBox, Relays and SoniXFM! You can always add your own vanity domain (www.mydomain.com) later, but keep in mind that it may cost extra to do so.

Only the following products apply for the FREEWEB discount

  • SX-100
  • SX-150
  • SX-200

The Affect of Russian Hacking

As previously reported, The US Justice Department recently indicted 13 conspirators of which one Sergey Pavlovich Polozov was personally responsible for hacking into SoniXCast systems in 2014. We now know why don’t we?

Just in case you missed it

The indicted Mr. Polozov was tasked with putting together systems in North America to be used to mount the so called “Information War” against the United States 2016 elections. In the runup, in June 2014 he along with 2 other co-conspirators broke into and took control of SoniXCast Virtual Private Servers in Toronto, Montreal and New York City. Since we couldn’t wrest control back from the hackers, we had to destroy the VPS’s and rebuild customer accounts and services from backups of our customer database. That was an “all hands on deck” effort which took about a week until everything was back to normal.

However, the hackers didn’t cover their tracks very well and we were able to obtain their personal information which we passed onto the Department of Homeland Security. As far as we know, Canada does not have an active counter espionage unit. To date: no Federal Agency has ever contacted us for further information on the subject, so we exacted some self help against the hackers that was sure to have caused some inconvenience.

One of the co-conspirators was the German national Udo Poschen who also Trolled for the German Performance Rights Organization GEMA. In a separate lawsuit, we were able attain Mr. Poschens bank records that ultimately showed that he had received payments both from the Russians and GEMA which was also reported to the authorities. Since then, Mr. Poschen has been pretty much inactive on the Internet.

But that is only a drop in the bucket

On an average day, our network team responds to over 300 network attacks. Most are very amatuer attempts to brute force into our systems or some really laughable ransom attempts, but at least once a week we receive a serious threat. An overwhelming amount (over 80%) come from US-based systems including the most recent attack that took out the router (according to OVH) in our Montreal Data Center. Over 60% originate from Russia. The remaining 40% originate from Europe and China.

So, What’re ya doing to mitigate the effects?

Immediately after the 2014 instance we moved sensitive data into a private network inaccessible from the Internet and implemented aggressive backup plans to ensure we could quickly rebuild our systems in the event of catastrophe. We also developed a sort of “Hot Swap” concept that would allow us to quickly bring a mirror of each server online in the event of attack.

The biggest crux in our efforts has been the provider. We have to host in Canada because of the broadcast license and despite popular belief, the Canadians just aren’t as technically savvy as US providers are. Further, it takes an act of congress to get exasperated OVH technician to do anything. The folks at iWeb are much more responsive (and friendly), but still technically really, really weak.

Edge firewalling is not even in the average Canadian providers vocabulary and the concept of intrusion detection systems might as well be aliens from outer space. Leaving providers like us to resort to self help in securing our systems against hackers and the occasional overzealous technician (they cause more downtime than you think).

Recently we came across a US provider who built their own cutting edge data center in Canada and other worldwide locations. After testing their technology for months, we feel confident that we’ve found a data center provider that can meet our security and networking needs and have begun moving our infrastructure over to them.

Moving is a complex multi-pronged effort that will take months to complete fully. We’ve already moved most of our web infrastructure over (which is why the websites are faster now). Next will be the cluster and relay networks, commercial partners, then finally retail customer services. We expect some obvious short (seconds maybe minutes) downtimes with the final phase, but we’ll give customers plenty of time to plan for it.

More importantly we are moving from a network that is ripe for the picking by hackers and where monkeys hammer on the hardware in order to keep it running to a shiny new government grade adaptive network where the features are not just marketing hype and SECURITY is written large.

For example; one of the things we tested was the server “Hot Swap” technique we pioneered, but never got to work efficiently because of the limitations of the provider. In a recent test with over 50 server services running, we were able to swap a server within a single ping (10ms) and not one listener dropped. Cool Huh?

Outage in the Montreal Datacenter

The main router in the Montreal Datacenter (CABHS) which enables access to the Internet suddenly stopped working at or around 09:00 hours UTC on 2/12/2018. It took a couple of hours to install and re-configure a new router and operations resumed at 18:00 hours UTC.

Preliminary autopsy results indicates a massive network attack occurred shortly before the router went offline, however there are indications that the router itself was hacked. The network team will present a full report by the end of business on Friday 2/16/2018.

The outage did not affect any other SoniXCast Datacenters.

We deem such an issue as untenable and our infrastructure team will make adjustments to ensure it does not happen again. This can include, moving accounts and services to other Datacenters within the SoniXCast Broadcast Network.

Mandatory BoomBox Upgrade 3.0.3

IMPORTANT INFORMATION: As of immediately all servers have been updated to BoomBox 3.0.3. Users must complete the upgrade by logging into BoomBox and following the upgrade instructions. The upgrade to version 3.0.3 is MANDATORY for all users and must be performed MANUALLY upon login into the BoomBox control panel. USERS WILL NOT BE ABLE TO MANAGE THEIR SERVICES UNTIL THE UPGRADE HAS BEEN COMPLETED.

The upgrade process will normally take only a few seconds, but can take longer for users with large music collections. You must contact support via the ticket system in the customer area at www.sonixcast.com with any issues that may occur with the upgrade. SUPPORT WILL NOT BE PROVIDED VIA 3RD PARTY PROVIDERS (i.e. facebook, twitter). THE TICKET SYSTEM IS THE ONLY MEANS ACCEPTED FOR CUSTOMER SUPPORT REQUESTS.

Music files and folders are being re-organized with this upgrade and therefore services (Server, AutoDJ) WILL be restarted upon completion of the upgrade process in order to ensure proper operation. Please plan your station outage accordingly.

Migration

All files and folders will be moved from playlists (Finder ‘Playlists’) folder to the music folder (Finder: ‘My Music’) and the playlists folder deleted in order to enable full access to file and folder operations (i.e. copy, move, delete). In addition, folders and files will be sanitized (renamed) in order to ensure proper service operations. Track metadata will not be affected by the sanitization operation. The sanitization utility has been added to the Finder App for future discretionary use.

Changes in version 3.0.3:

  • Better crash protection for services (Server, AutoDJ) to ensure longer uptimes.
  • Auto-monitoring of services that will attempt restart of a service up to 10 times in succession before giving up and sending an email notification to the station operator.
  • Rewrite of quota and traffic management services and display widgets. AS OF IMMEDIATELY DISK SPACE USAGE OVERAGES WILL APPLY.
  • Rewrite of the service management pages (Server, AutoDJ) to include full control (start, stop, restart) of services by the user. Save operations will automatically restart a service in order to apply changes.
  • Less aggressive service polling in the BoomBox interface and elimination of ‘blinking’ working icons that confused some users.
  • Added the Sanitize Utility (Folder, Brush icon) to the Finder App for discretionary use users who prefer uploading via FTP. Auto file and folder sanitization and normalization has been built into the Finder uploader. USERS WHO UPLOAD VIA THE FINDER ARE NOT REQUIRED TO RUN THE SANITIZATION UTILITY AS FILES AND FOLDERS ARE NORMALIZED UPON UPLOAD.
  • Added playlist management functionality to the Finder App. Users can right click on a folder, select the ‘Edit Playlist’ option and create or edit existing playlist settings associated with the folder. Folders that are deleted in the Finder App will also delete the associated playlist settings in the Playlist manager.
  • Playlists can now be associated with a folder in the music library. A dropdown has been added to the playlist editor that allows the user to select a folder from the music library containing tracks for playback. IF NO FOLDER IS SELECTED THE PLAYLIST WILL AUTOMATICALLY BE DISABLED.
  • Auto disabling of the AutoDJ service will occur if all playlists are disabled or the associated folders are empty.
  • Diverse UI upgrades and enhancements to eliminate confusion and better user experience and usability.