Rumble2012 Nox Solutions Tech Social Media Fail Case Study

A debate both entertaining and valuable took place today between Jon Stewart and Bill O’Reilly. Unfortunately it was accompanied by a monumental tech and social media monitoring and reaction failure related to their paid live streaming internet solution.

Earlier this week Kitchen Aid fielded a social media disaster very well. Not so for The Daily Show and the O’Reilly Factor. Neither of their Twitter accounts @TheDailyShow nor @OReillyFactor monitored the event nor responded to customer complaints.

The worst display on record in responding to a public tech crisis goes to Nox Solutions. Apparently, Bill O’Reilly set up the debate and used his tech company to manage the live streaming. Their capability to handle this was not adequately vetted by either of the participants. $4.95 would be collected to watch a live stream of the debate, and to have access to three downloads. Half of the net profits would go to charity. Obviously, the expenses that were paid before net profit included Nox Solutions.

What happened is that people who attempted to register close to the event could not get access to the site which was already buckling under customers coming in early to set up to watch. On Twitter and on Google Plus there were hundreds of complaints. The Twitter hash tag for the debate was #Rumble2012. A new hash tag sprung up called #RumbleFail. Nox Solutions does not have company accounts on Twitter nor Google Plus.

Fifty minutes into the debate there was finally an acknowledgement on the Twitter stream set up for the debate @theRumble2012 that there was an issue. They apologized and told followers that a download would be available later after the debate. I don’t know who owns that Twitter account.

Nox Solutions an hour into the debate posted an apology on their Facebook saying don’t worry, the download would be available later. People immediately began commenting on the thread asking when the download would be available. Apparently frightened, Nox Solutions shut down their Facebook wall completely!

Nox-Solutions-Facebook-screenshot

What is the damage so far to Nox Solutions? Early indications can be found with some disgruntled customers who found their Yelp account and began pounding it with negative reviews.

Ironically Nox Solutions claims on their website that they deliver solutions for “online reputation management”!

Next steps:
There should be apologies and explanations on all of the major social media channels.
The debate parties should ensure that the archive is provided free to the public. They might wait a day or two to provide the value of seeing it first to those who paid for the video.
There are those demanding refunds. That is a bigger issue that I am not going to attempt to address here.

Lessons to be learned:

1. Select your service providers carefully. Just because a company you already work with claims they can provide other services, you should still do your due diligence to make sure they have that capability. Especially if it is a big important project
2. Never leave your social media accounts unmonitored. It doesn’t matter if it is Saturday. Especially during a big event.

I would love to know more about what went wrong with the Nox Solutions servers. I would assume that the type of solution a company like RackSpace would provide where cloud bandwidth is increased when necessary, or Amazon backup is what they needed in this circumstance.

Alex Kantrowitz from Forbes on the incident.

I got busy on a client call and didn’t get to registering for #Rumble2012 until 15 minutes before the event. The tech platform was already failing then. I decided to just follow the tweets since I couldn’t get on. I tweeted a bit then chronicled the debate and the tech failure on my Google Plus – where there were complaints but no live posts about what was happening in the debate.

I had been looking forward to Rumble 2012 since seeing it introduced on the Daily Show with a charming visit from Bill O’Reilly. I put the link for registration up on Facebook, Google Plus and Twitter.

Kathy Gill did a very nice Storify for the Moderate Voice using tweets from the debate. Several of the tweets she selected were people exclaiming that they were very happy to spend $4.95 to watch this intellectually pleasing, substantive yet entertaining debate.

The debate is worth watching. However you finally watch it, I’m sure you will enjoy it. There should be something for everyone.

Rumble 2012 Jon Stewart vs Bill O'Reilly Debate Poster

Updates:
We all looked for a Twitter account for Nox Solutions during the event, couldn’t find one and gave up. One popped up around 9PM est on the night of the incident with apologies and some responses to tweets. At first I thought that it was real and they were trying to be funny but Kathy Gill pointed out that it is much more likely this is a spoof account and I have to agree.

As of 10:00pm EST Tuesday Oct 9 – still no tweets from @TheDailyShow nor @OReillyFactor regarding the tech failure.
After watching these two Twitter accounts for the past few days, I have come to the conclusion that the tweets are proposed, approved and pre-scheduled well in advance of posting and there is no monitoring of or response to social mentions.

I JUST found the Facebook Page for the event. This page did provide a place for customers to vent. Unfortunately, there is no link to this on the Runble2012 event website although they did provide a link to the event Twitter stream.

The Wall Street Journal pre-announced they would live blog the event. They had trouble getting on but finally did blog some of the action. There were 1,346 comments as of Oct 8 1:30pm est and the I would say over 90% of them were about the technical failure. One comment at Oct 7, 2:27am est, said that Nox Solutions “have gotten caught in the past with a security breach of their systems, which allowed thousands of customer’s private information to be compromised because they store everything in ‘plain text’” but I could not find anything about that online.

I caught a comment from Benjamin Hubbard on the event Facebook Page saying, “Bill is using his website to give the streaming away for free if you buy his 1 month membership at the same price meaning less money would go to the charities…please share.” I have not verified this myself by going to Bill O’Reilly’s website.

Actually I was on the Facebook event page looking for a comment I had seen the day before (which I should have copied off then) from someone without a Facebook avatar, with the name Micky(?), saying something to the effect that they would come back to the Facebook fans next week regarding the refund issue. But they did communicate the same thing by adding a post directing us to their website for “More info about on-demand access and all of the charities benefiting from The Rumble” and the therumble2012.com website said the following as of 7:30pm on October 8. But they later deleted the apology and mention of a refund in the first three sentences.

We understand many viewers had difficulty streaming the debate when it began. These issues have been resolved and the show is now available both on-demand and via download. For anyone who was unable to view The Rumble live and no longer wishes to do so, refund information will be available early next week. We regret any inconvenience this may have caused. One-half of the net profits from “The Rumble 2012” will be donated by Jon and Bill to benefit charitable causes such as: New York Collaborates for Autism, Wounded Warrior Project, Fisher House Foundation, Alzheimer’s Association, Boomer Esiason Foundation, Doctors Without Borders, Tuesday’s Children, Portrait Project, NYC Coalition for the Homeless, Project A.L.S., Bob Woodruff Foundation, Haitian Health Foundation, The USO, Responsibility, Childcare, Best Friends Foundation and FealGood Foundation.

O’Reilly said that a refund will be provided for those who request it. His statement is at the beginning of a 4 minute video on the Huffington Post of O’Reilly speaking to George Stephanopoulos on “ABC This Week” on Sunday. He said that servers crashed and people can still buy the video for $4.95. NO apology. There is no indication that it will made generally available.

Someone mentioned that there were apologies at the news conference following the debate. I searched for something about that and found this article from the George Washington University (the venue) student newspaper but it doesn’t mention apologies. If anyone has found apologies from O’Reilly or Stewart, please let me know.

Monday night, October 8th on The Daily Show, Jon Stewart immediately addressed the tech issue with a sincere apology. I was very pleased. Here is the full episode video archive from The Daily Show site.

October 9th, Kathy Gill provided a useful update of the tech and social media snafus to PBS Digital Shift that includes her tweet Storify that I mentioned earlier.