Rumble2012 Nox Solutions Tech Social Media Fail Case Study

A debate both entertaining and valuable took place today between Jon Stewart and Bill O’Reilly. Unfortunately it was accompanied by a monumental tech and social media monitoring and reaction failure related to their paid live streaming internet solution.

Earlier this week Kitchen Aid fielded a social media disaster very well. Not so for The Daily Show and the O’Reilly Factor. Neither of their Twitter accounts @TheDailyShow nor @OReillyFactor monitored the event nor responded to customer complaints.

The worst display on record in responding to a public tech crisis goes to Nox Solutions. Apparently, Bill O’Reilly set up the debate and used his tech company to manage the live streaming. Their capability to handle this was not adequately vetted by either of the participants. $4.95 would be collected to watch a live stream of the debate, and to have access to three downloads. Half of the net profits would go to charity. Obviously, the expenses that were paid before net profit included Nox Solutions.

What happened is that people who attempted to register close to the event could not get access to the site which was already buckling under customers coming in early to set up to watch. On Twitter and on Google Plus there were hundreds of complaints. The Twitter hash tag for the debate was #Rumble2012. A new hash tag sprung up called #RumbleFail. Nox Solutions does not have company accounts on Twitter nor Google Plus.

Fifty minutes into the debate there was finally an acknowledgement on the Twitter stream set up for the debate @theRumble2012 that there was an issue. They apologized and told followers that a download would be available later after the debate. I don’t know who owns that Twitter account.

Nox Solutions an hour into the debate posted an apology on their Facebook saying don’t worry, the download would be available later. People immediately began commenting on the thread asking when the download would be available. Apparently frightened, Nox Solutions shut down their Facebook wall completely!

Nox-Solutions-Facebook-screenshot

What is the damage so far to Nox Solutions? Early indications can be found with some disgruntled customers who found their Yelp account and began pounding it with negative reviews.

Ironically Nox Solutions claims on their website that they deliver solutions for “online reputation management”!

Next steps:
There should be apologies and explanations on all of the major social media channels.
The debate parties should ensure that the archive is provided free to the public. They might wait a day or two to provide the value of seeing it first to those who paid for the video.
There are those demanding refunds. That is a bigger issue that I am not going to attempt to address here.

Lessons to be learned:

1. Select your service providers carefully. Just because a company you already work with claims they can provide other services, you should still do your due diligence to make sure they have that capability. Especially if it is a big important project
2. Never leave your social media accounts unmonitored. It doesn’t matter if it is Saturday. Especially during a big event.

I would love to know more about what went wrong with the Nox Solutions servers. I would assume that the type of solution a company like RackSpace would provide where cloud bandwidth is increased when necessary, or Amazon backup is what they needed in this circumstance.

Alex Kantrowitz from Forbes on the incident.

I got busy on a client call and didn’t get to registering for #Rumble2012 until 15 minutes before the event. The tech platform was already failing then. I decided to just follow the tweets since I couldn’t get on. I tweeted a bit then chronicled the debate and the tech failure on my Google Plus – where there were complaints but no live posts about what was happening in the debate.

I had been looking forward to Rumble 2012 since seeing it introduced on the Daily Show with a charming visit from Bill O’Reilly. I put the link for registration up on Facebook, Google Plus and Twitter.

Kathy Gill did a very nice Storify for the Moderate Voice using tweets from the debate. Several of the tweets she selected were people exclaiming that they were very happy to spend $4.95 to watch this intellectually pleasing, substantive yet entertaining debate.

The debate is worth watching. However you finally watch it, I’m sure you will enjoy it. There should be something for everyone.

Rumble 2012 Jon Stewart vs Bill O'Reilly Debate Poster

Updates:
We all looked for a Twitter account for Nox Solutions during the event, couldn’t find one and gave up. One popped up around 9PM est on the night of the incident with apologies and some responses to tweets. At first I thought that it was real and they were trying to be funny but Kathy Gill pointed out that it is much more likely this is a spoof account and I have to agree.

As of 10:00pm EST Tuesday Oct 9 – still no tweets from @TheDailyShow nor @OReillyFactor regarding the tech failure.
After watching these two Twitter accounts for the past few days, I have come to the conclusion that the tweets are proposed, approved and pre-scheduled well in advance of posting and there is no monitoring of or response to social mentions.

I JUST found the Facebook Page for the event. This page did provide a place for customers to vent. Unfortunately, there is no link to this on the Runble2012 event website although they did provide a link to the event Twitter stream.

The Wall Street Journal pre-announced they would live blog the event. They had trouble getting on but finally did blog some of the action. There were 1,346 comments as of Oct 8 1:30pm est and the I would say over 90% of them were about the technical failure. One comment at Oct 7, 2:27am est, said that Nox Solutions “have gotten caught in the past with a security breach of their systems, which allowed thousands of customer’s private information to be compromised because they store everything in ‘plain text’” but I could not find anything about that online.

I caught a comment from Benjamin Hubbard on the event Facebook Page saying, “Bill is using his website to give the streaming away for free if you buy his 1 month membership at the same price meaning less money would go to the charities…please share.” I have not verified this myself by going to Bill O’Reilly’s website.

Actually I was on the Facebook event page looking for a comment I had seen the day before (which I should have copied off then) from someone without a Facebook avatar, with the name Micky(?), saying something to the effect that they would come back to the Facebook fans next week regarding the refund issue. But they did communicate the same thing by adding a post directing us to their website for “More info about on-demand access and all of the charities benefiting from The Rumble” and the therumble2012.com website said the following as of 7:30pm on October 8. But they later deleted the apology and mention of a refund in the first three sentences.

We understand many viewers had difficulty streaming the debate when it began. These issues have been resolved and the show is now available both on-demand and via download. For anyone who was unable to view The Rumble live and no longer wishes to do so, refund information will be available early next week. We regret any inconvenience this may have caused. One-half of the net profits from “The Rumble 2012″ will be donated by Jon and Bill to benefit charitable causes such as: New York Collaborates for Autism, Wounded Warrior Project, Fisher House Foundation, Alzheimer’s Association, Boomer Esiason Foundation, Doctors Without Borders, Tuesday’s Children, Portrait Project, NYC Coalition for the Homeless, Project A.L.S., Bob Woodruff Foundation, Haitian Health Foundation, The USO, Responsibility, Childcare, Best Friends Foundation and FealGood Foundation.

O’Reilly said that a refund will be provided for those who request it. His statement is at the beginning of a 4 minute video on the Huffington Post of O’Reilly speaking to George Stephanopoulos on “ABC This Week” on Sunday. He said that servers crashed and people can still buy the video for $4.95. NO apology. There is no indication that it will made generally available.

Someone mentioned that there were apologies at the news conference following the debate. I searched for something about that and found this article from the George Washington University (the venue) student newspaper but it doesn’t mention apologies. If anyone has found apologies from O’Reilly or Stewart, please let me know.

Monday night, October 8th on The Daily Show, Jon Stewart immediately addressed the tech issue with a sincere apology. I was very pleased. Here is the full episode video archive from The Daily Show site.

October 9th, Kathy Gill provided a useful update of the tech and social media snafus to PBS Digital Shift that includes her tweet Storify that I mentioned earlier.

  • http://twitter.com/LSDigest LiveStreamDigest

    I wish the NY Times had read your article. They put it on Roku.

    • http://ItsDifferent4girls.com/my-links Linda Sherman

      Thanks for your kind tweet. Sorry to hear that about the New York Times. I had the advantage of being online from just before the event to hours after it, looking at everything that was going on. Sometimes big media has to respond quickly and may wing it without all the facts. I didn’t have time to do eloquent writing in between finally seeing the video and going out to dinner on Kauai but at least I could see what had happened. 

      • http://twitter.com/LSDigest LiveStreamDigest

        One of the most brilliant things you did was tweet the link to Nox’s Yelp page. I’m more interested in tech, but your social media critique is brilliant.

        • http://ItsDifferent4girls.com/my-links Linda Sherman

          Thank you! You had some very interesting tech things to say in your article (#RumbleFail trackback below)

  • Pingback: #RumbleFail()

  • http://s.hryx.net/ hryx

    The debate video is available for download now. The “very high” quality selection has an stunning frame rate of about… 14fps. The lower resolutions are slightly higher (not 30fps) but look like they were encoded with codecs from 1999. I don’t even know if I can bring myself to watch it anymore.

    • http://ItsDifferent4girls.com/my-links Linda Sherman

      Thank you hryx. I love getting the 411 on the tech. Do watch it though. The content is great.

  • ebuttscpa

    Linda – this whole sequence events really is fascinating. With regards to the point that you didn’t cover above, the answer is straightforward. When you fail to deliver on a just-in-time service like this, a refund request should not be disputed. Can you imagine HBO crumbling under demand for a boxing PPV fight and not giving refunds if they somehow failed to deliver? Focus needs to be on managing their long-term reputation and not short-term profits in this case.

    On a separate note, with the resources available to these two participants, they both should be embarrassed by what happened with this event. If they really want to show some good faith, they should cover the charity portion lost as a result of the refunds that had to be given to disgruntled customers.

    • http://ItsDifferent4girls.com/my-links Linda Sherman

      I was looking to see what the update was on the topic of refunds before answering you and added a couple of things to the bottom of my update section on the post. 

      First I noticed that someone on the Facebook Page said, ““Bill is using his website to give the streaming away for free if you buy his 1 month membership at the same price meaning less money would go to the charities.” We cannot necessarily draw the conclusion that Bill would not make up for that by funneling that money to charity but it would be nice if that became transparent.

      I agree that the two debate participants are wealthy and I expect them to do the right thing here. 

      It’s 6PM est Monday and there is still no mention of the incident on the Twitter streams for @TheDailyShow nor @OReillyFactor I also checked the Daily Show Facebook Page. Lots of comments about the issue but I didn’t see a response (they allow update comments).  I found the Facebook Page for O’Reilly by going to BillOReilly.com. It is  facebook.com/billoreillyfnc . Fewer fans than Stewart and no cover photo but his audience is apparently older than Stewart’s. I found no mentions on the Facebook Page nor his website. Much fewer complaints about the tech issues on O’Reilly’s page in the comments on the post about the debate, they don’t allow fans to post updates.
      I do hope the tech issue gets addressed soon that we can concentrate on the messages that were in the debate – which were good. It was nice to see a civilized discussion of the key topics they covered..

  • http://HonuaLani.com/ Jai Roberts

    It is unbelievable to me that Nox Solutions would send out those kind of tweets when they advertise that they specialize in online reputation management! What were they thinking?

    • http://ItsDifferent4girls.com/my-links Linda Sherman

      Apparently someone advised them to be funny. That works in some heated circumstances but I don’t think it is helping in this one. Their Facebook Page is still shut down.

  • http://WiredPen.com/ kegill

    Linda, I’m 99.9% certain that the Nox Twitter account is a spoof. 

    • http://ItsDifferent4girls.com/my-links Linda Sherman

      You are so right Kathy. Thank you.