(blogs let others gawk)

February 9, 2014

Single point of failure (or how important is your data?)

So, this is a story I don’t tell too often but in light of some recent conversations about performing backups following the news about the Iron Mountain fire, I felt it would be insightful to share.

Back in 1997/1998 I learned a very hard lesson about data loss and the publication I Co-Edited named Game Zero magazine.

First the back story to explain how this situation ended up the way it did.

We started our web presence near the end of 1994 with a user account with a local Arizona company named Primenet who offered users the traditional array of features (WWW, POP mail, etc…). This worked out great except for a couple of problems. The first was that even though we had registered the domain gamezero.com for our site, Primenet’s server name resolution would sometimes flip a visitor’s browser to the primenet.com/team-0 URL while the person was traversing the site. This caused lots of people to create bookmarks and links to the site by the wrong URL (this comes into play later).

The second and later problem, although not a technical issue, was the cost associated with bandwidth for WWW visitors to the site. Towards the end of our time with Primenet we were hitting fees of a few hundred dollars a month for bandwidth from our 700,000+ page views a month. Fortunately we had designed our site incredibly light, so that helped keep costs low, but traffic and fees were climbing. Ultimately I set my sights to moving us to new “discount” hosting services which were becoming a thing in 1997. It was obvious we could save a significant amount of money by moving the site.

For backups, we had our production computer which housed all the original and developing web content, including the active mirror of the website and remote publishing tools as well as our POP e-mail client for all business e-mail. Additionally, we kept backups of web content and e-mails on a collection of Zip disks along with some limited content on a random assortment of floppies.

In 1997 hard drives where expensive! We’re talking a few hundred dollars for a 1GB drive. Our production PC had something like a 120MB drive, as I recall, so we had lots of data off loaded on the Zip disks.

Also around this time we also received word that the provider which had been handling our FTP based video repository was getting out of the hosting business. I decided it best to roll the video content into the new web hosting arrangement as the price would still be reasonable. We quickly migrated everything over, changed DNS entries, started sending out e-mails to people who had the old primenet.com addresses to please update their links, etc… Following the migration we only published a few major updates on the new server consisting of a couple of new videos and some articles which only existed on the website, our production system and our Zip drive backups.

Then problems started…

  1. Traffic tanked on the new server.
  2. My crawling the web looking for bad links suddenly made me aware of just how bad the extent of the linking issue was and a significant amount of traffic was still going to the old Primenet URL. Fortunately right before we closed our Primenet account we setup a root page that linked to the proper URL along with a notice about the move which Primenet was kind enough to leave up at no cost, but it wasn’t a full site wide redirect though. Just the root pages.
  3. A few months into running on the new provider their servers went dark. When I contacted them to find out what happened, I reached a voicemail that informed me that they had filed bankruptcy and closed business. Done, gone… No contact and no way to recover any of the data from the web server.
  4. We now had a domain name that didn’t respond, our old provider’s server was pointing traffic to that very same dead URL and since we had long since closed the Primenet account we had no ability to log in and change the redirect notice or make other modifications to redirect traffic someplace else.
  5. While scrambling to find new hosting, the hard drive on our production computer completely and utterly failed. 100% data loss.
  6. After getting a new hard drive I went to start rebuilding from our Zip disks and to my horror none of them would read. We had now become a victim of what became to be known as the “click of death”. We lost some 20-30 Zip disks in total. Almost everything was gone except for a mirror of the website from before the migration to the new hosting and other random items scattered around. We also had a limited number of hard copies of e-mails and other documents.
  7. Lastly, while the Internet Archive now is a great way to recover website content. At this point in time it was still just getting started and their “Wayback Machine” had only just taken a partial snapshot of our sites (in both the US and Italy). Par for this story, the lost content was pages that had not been crawled yet except for the index pages for the missing videos. I could view the archive of the video pages… but the linked videos were too large at that time and were not mirrored.

Coming into this, I felt we had a pretty good data backup arrangement. But I learned the hard way that it wasn’t good enough. We lost all of the magazine’s e-mail archives including thousands of XBand correspondences as well as innumerable e-mails with publishers and developers. We lost two videos that had been produced and published. We lost a few articles and reviews. We also lost nearly all of the “in progress” content as well as a number of interviews.

At this point the staff agreed to stop spending money on the publication and formally end the magazine, especially since some of them were already making natural transitions into their careers and school. While we had stopped actively publishing at then end of 1996/start of 1997, if you were to ask me if there was a hard line for the the true end of the magazine, this was it.

Ultimately I did get the site back up as an archive which you can still read today. But, that’s another story.

The lesson of this story is to remember that there is no fool-proof backup situation. Only you can be responsible for you (or your company’s) data and you must always be aware that no matter what your best efforts are, data loss is always a possibility.

99.9% guarantees are great except for  that 0.1% chance, which is still a chance! and if someone is selling you a 100% guarantee let me know because I’ve got the title for this bridge in Brooklyn I might consider selling you for a deal.

What could I have done differently?

  1. Spread out our backups across more than one media type and one location. Simply having a duplicate set of Zip disks and a second drive off site where there was no cross-mixing would have made a huge difference here.
  2. More frequent backups of critical business data such as e-mail.
  3. Retained the master account with the old service provider until we were sure traffic migration had been completed.
  4. Upon the first sign of Click of Death observed. I should have isolated both the problematic media and drive from use and looked for a second drive as the damage propagated once manifest but nobody had enough information about the problem at the time and the manufacture kept denying the problem existed.

Granted some of these would have likely added overhead cost, but the the question is would that cost balance against the value of the data lost? I don’t know. But since this happened I have been far more diligent in my data storage strategies where I now factor in the value and importance of the data with the breadth and depth of the backup plan and go with the best possible solution I can devise.

I have had only one significant data loss in the years since this happened. It was just last fall and I was doing some data re-organization as part of a desktop upgrade. A USB drive I was using for temporary storage fell over and become damaged in such a way that it would no longer read the disk. I then discovered that the data on the drive hadn’t been synchronized with the backup repository for a couple of months for some reason. Fortunately it was non-critical, personal data (downloaded drivers and install packages that I was able to re-download from the Internet). So all in all the only loss here was in my time. But it was a reminder to me that even though I am way more careful than before, accidents can still happen.

September 24, 2010

Push and Pull

Filed under: General,Internet Rant,Perspective — Bryan @ 4:07 am

In the world of advertising and marketing you can identify communications as either push or pull. Push is where you push content out to viewers and Pull is where viewers come to you to get content.

Mediums such as TV, newsletters and banner ads are identified as push based communications in that they are designed to push information about new products and services to you. The idea is to inform you about a solution to a perceived need you never knew you had.

Internet content in general (and by extension most social media) is considered pull based communications because the viewer has to actively go out and request the information in order to receive it. There is currently no way for you to force someone to read your blog or watch your YouTube video (not that people aren’t trying to figure out a way to do it though). Granted you can design a website that forces a pop-up ad to show before you can read the page you want to visit. In this case the add is push, but the content that that provided you to the chance to see the ad is pull.

In many ways, modern TV has evolved to this state. Historically when all we had generally were three channels run by national networks to choose from in any given U.S. city an advertiser could simply run their ad on all three networks at peek viewing hours for their target demographic and be confident that just about everyone would see the ad. Now with literally thousands of TV channels between broadcast, cable, satellite, internet, etc… combined with time shifting technology (ie, DVR devices such as Tivo) advertisers are moving to running ads specifically related to a particular program. In very much the same dynamic as a web site (eg, you want to run an advertisement that I.T. workers will see? Run it on TV during a show like MythBusters, and then put your online ads on a site like Slashdot.

Really, the bottom line is consumer choice vs. advertiser choice. The less choice a consumer has the more push communications can be applied unilaterally. The unfortunate thing here is that as people have more choice, marketers must get more intrusive in order to push their messaging out. I’m left thinking of a scene in Minority Report where the Tom Cruise character is walking along and all of the wall ads are narrow focusing audio at him with personalized sales pitches.

In the decade since blogs first formally appeared as a platform, people have been trying to figure out a way to make them more Push based instead of Pull. Why? Because as a Pull based technology your readership is limited to who you can draw into your site and this is where link aggregation services such as Digg, etc… come into play. Effectively people have been trying to find a way to make a things like blogs work in the way TV communications used to work.

A site like Digg or Technorati are Pull services, but they have such high volume and high visibility that they can effectively becomes a Push agent for everyone else down stream because they have critical mass audience based on being an aggregate.

I suppose in the rare cases, there have been blogs that have attained that same space but in all of the cases I am aware of the blogs are working as an aggregate site. Examples would be Slashdot mentioned above as well as sites like BoingBoing or in the most extreme FARK.

So, I guess the lesson learned here is if you want to get the word about about your product, you either need to advertise everywhere you can (print, TV, web, etc…), go door-to-door like the old days (or site to site posting comments — don’t spam!) or get your site/product referenced on one of these aggrigator sites (you better have something unique going on to get their attention). Hmm… that’s all that comes to mind at the moment. Does anyone else have ideas on this subject?

July 14, 2010

The industy is dead! Long live the industry! (Won’t someone think of the children!?!)

Since I touched on the subject of media transition touched on briefly in my post about going mobile friendly, I think this is a good chance to highlight some historical hysteria regarding entrenched business models collapsing to be replaced by new ones.

Let’s specifically look at the history of music distribution over the last 100 years.

Going into the 1900’s piano rolls and sheet music were the predominant methods of music distribution. Granted there were also broadsides, but those were considered a medium for the working class and were typically lyric sheets with no music score, commonly notated with statements like ” sung to the tune of -fill in the blank- “.

Even in a time where the average worker earned around $600 a year, 25-60 cents for a copy of sheet music was a premium purchase for many. That said, sheet music was big business and when the phonograph came around, sheet music publishers saw the new medium as a threat and fought tooth and nail to kill the medium. “Oh! We can’t let this happen”, “This will destroy the traditional family gathering in song”, “nobody will learn to play music”, “someone think of the children”, etc…

But, the reality on the ground was that pianos are expensive both to purchase and maintain. On the other hand, phonographs are cheaper to produce, cheaper to maintain, easier to operate and you didn’t have to be able to play music to enjoy them. In the end music producers actually sold more copies of music because they now had a larger audience and the companies that adopted to the new business model profited greatly.

Most sheet music publishers failed to adopt the medium hardly failed to die. Granted some of the fears were well founded, the days of the families gathering around a piano to sing together for leisure were lost (if they truly were all that common to begin with). But sheet music is still produced and sold in most music stores. Granted these days it’s mostly piano and guitar based, but, those are the popular instruments for people learning to play music so it only makes sense.

Let’s roll ahead to the next big jump to radio. When radio hit the scene phonograph publishers went crazy. “Oh! We can’t let this happen”, “People will quit buying music when then can get it for free over the radio”, “This will make it impossible for musicians (sic, publishers) to get paid for their work”, etc…

On the contrary to most concerns, radio actually increased sales for two reasons. People were exposed to a larger variety of music and they like the convenience of listing to a song on-demand so naturally they went out to buy their favorite songs in order to have them available to listen to. Publishers that added value to their product saw even better profits (e.g., B-Sides) among core fans.

Then over then next 60 years not much changed aside from improvements in recording and distribution. Granted there were fights over the introduction of cassette tapes and fears that people would just copy music instead of buying it. The same thing happened with CD based music. But don’t forget it’s always been a steady lowering of the bar of the cost of entry into the world of listening to music contrasted against the publishers desires to maximize profits from that same music. Also, people on the whole will more often than not pay for something when they feel it is being sold at a value they perceive as fair. Don’t believe me? Ask Trent Reznor (of the band Nine Inch Nail) or for another example in a different entertainment industry, ask the video game publisher Stardock. Both have spoken out at length about the success of this business model to their sales.

Since the who knows how long the common perception (supported by much anecdotal evidence and statements from artists) is that artists get paid little if nothing for their work when they publish music through a publisher and that publishers takes all of the profit from sales. The big money for the artists more often than not is in concerts and live performances and endorsements. A popular song will generate larger ticket sales and everyone wins (hopefully). This situation set the scale for next big crisis for publishers. The Internet combined with the modern computer.

In the mid-to-late 90’s the barrier of cost related to copying and sharing music finally broke down and anyone with a computer and internet connection suddenly found a plethora of methods to acquire music to listen for free (sometimes pirated, sometimes not) that weren’t available before. And the net result? Sales increased! What’s that? That doesn’t make any sense. The music publishers told us that people stealing music was loosing them money. Wrong. People downloading music was gaining them customers. The money they were supposedly loosing was based on estimates of “if every single downloaded song on the Internet had instead been purchased we would have profited this much”.

… publishers used the same logic with radio by the way.

The reality was that people who used to be pigeonholed to a particular music style suddenly had a inexpensive way to explore new music that they might not have been willing to pay for (on the risk that they might not like it). When they discovered a new artists or new genre they enjoyed, they then frequently went down to the record store to find more of that music to purchase. You had punk rockers buy classical music and country lovers buying speed metal.

But the industry could only focus on the “lost sales” not factoring in that these weren’t really lost sales. Anecdotal evidence from the time indicates that were more like samples. To compare, yes, we know there is always going to be the guy who lives off samples at the grocery store for dinner, but most people actually buy their food and the samples are good because they primarily encourage regular customers to try things they never tried before. The guy living off of them is a cost of business.

I always said at the time that music companies should have jumped on this immediately and put their entire catalogs online at 56kbps bit-rate (radio quality), with an easy click to purchase the higher quality version priced as a convenience item. They would have made a killing, but instead they decided to fight their customers (and still do). Effectively deciding to sue anyone who eats a square of cheese at the deli counter without buying the whole wedge.

When asked years later why they pursued this course of action, one executive answered that they were so scared of the changes happening and knew that they didn’t have a clue about what was going on that they feared everyone was out to rob them and that even the consultants couldn’t be trusted. So they fell back on the only tool they could trust, their lawyers.

Sadly this fight is still playing out even to this day but in the last year some significant changes have happened that probably mark the end for some of the large publishers in this space. The barrier of entry for recording equipment has vanished and a lot of bands, frustrated with publishers and finding greater profitability by simply going solo on the Internet is increasing. When you strip away all of the fluff YouTube is now the largest publisher of music on the Internet at this time. So large that other publishers now effectively turn their content over to them just to get exposure for their artists.

The more things change the more they stay the same. Business is always evolving and those that learn and adopt quickly are well positioned to profit from their observation skills. Others are destined to dig in their heels and ultimately become a footnote to history.

….some references that helped in the creation of this article are listed below.

  • Media-Morphosis: How the Internet Will Devour, Transform, or Destroy Your Favorite Medium: http://www.internetevolution.com/document.asp?doc_id=171555&
  • http://www.econlib.org/library/Enc1/WagesandWorkingConditions.html
  • http://www.bls.gov/opub/cwc/cm20030124ar03p1.htm
  • Perspective: Radio/photograph was going to destroy print: http://web.mit.edu/comm-forum/papers/murphy.html
  • Sheetmusic and broadsides…: http://popmusic.mtsu.edu/dbtw-wpd/textbase/broad/broadside_ex.htm and http://www.phonobooks.com/BirthRec.htm
  • http://cultureandcommunication.org/deadmedia/index.php/The_Victrola
  • Radio was going to destroy the records: http://en.wikipedia.org/wiki/History_of_radio#Legal_issues_with_radio (although the Internet distributed music has revolutionized the way records are sold, it still hasn’t destroyed them)

April 23, 2010

Computer vs Google

Filed under: General,Perspective — Tags: , , , , — Bryan @ 6:23 am

The other evening I was inspired to see what the top three search results were in Google for the word Computer since I expected it to be generic enough to not be branded yet specific in such a way that all languages referring to it would be identifying the exact same concept in their word. To make it interesting I first stopped by the Babel Fish translation service and from the twelve non-English languages I translated the word computer into that language in the native character set.

Then where possible I went to the Google landing page for that country and executed the search by pasting in the translated word. Eg, for Japan I went to www.google.co.jp and entered “コンピュータ” which is what Babel Fish told me was the Japanese translation for the word.

What follows are my results followed by some observations.

In Dutch: “computer”

  1. http://www.mycom.nl/
  2. http://www.computertotaal.nl/
  3. http://nl.wikipedia.org/wiki/Computer

In French: “ordinateur”

  1. http://fr.wikipedia.org/wiki/Ordinateur
  2. http://www.ordinateur.com/
  3. http://www.dicofr.com/cgi-bin/n.pl/dicofr/definition/20010101003926

In German: “computer”

  1. http://www.computerbild.de/
  2. http://de.wikipedia.org/wiki/Computer
  3. http://www.atelco.de/

In Greek: “υπολογιστής”

  1. http://el.wikipedia.org/wiki/%CE%97%CE%BB%CE%B5%CE%BA%CF%84%CF%81%CE%BF%CE%BD%CE%B9%CE%BA%CF%8C%CF%82_%CF%85%CF%80%CE%BF%CE%BB%CE%BF%CE%B3%CE%B9%CF%83%CF%84%CE%AE%CF%82
  2. http://en.wiktionary.org/wiki/%CF%85%CF%80%CE%BF%CE%BB%CE%BF%CE%B3%CE%B9%CF%83%CF%84%CE%AE%CF%82
  3. http://www.komvos.edu.gr/periodiko/default.htm

In Italian: “calcolatore”

  1. http://en.wiktionary.org/wiki/calcolatore
  2. http://www.freeonline.org/calcolatrice_dtml
  3. http://it.wikipedia.org/wiki/Computer

In Japanese: “コンピュータ”

  1. http://ja.wikipedia.org/wiki/%E3%82%B3%E3%83%B3%E3%83%94%E3%83%A5%E3%83%BC%E3%82%BF
  2. http://www.dell.co.jp/
  3. http://www.apple.com/jp/

In Korean: “컴퓨터”

  1. http://ko.wikipedia.org/wiki/%EC%BB%B4%ED%93%A8%ED%84%B0
  2. http://www.compuzone.co.kr/
  3. http://www.trigem.co.kr/

In Portuguese: “computador”

  1. http://pt.wikipedia.org/wiki/Computador
  2. http://en.wiktionary.org/wiki/computador
  3. http://www.dell.com.br/

In Russian: “компьютер”

  1. http://ru.wikipedia.org/wiki/%D0%9A%D0%BE%D0%BC%D0%BF%D1%8C%D1%8E%D1%82%D0%B5%D1%80
  2. http://www.sigmacomputers.ru/
  3. http://www.depo.ru/

In Spanish: “computadora”

  1. http://es.wikipedia.org/wiki/Computadora
  2. http://www.monografias.com/trabajos15/computadoras/computadoras.shtml
  3. http://www.alegsa.com.ar/Dic/computadora.php

In Chinese-Simplified: “计算机”

  1. http://baike.baidu.com/view/3314.htm
  2. http://product.enet.com.cn/price/plist3.shtml
  3. http://www.enet.com.cn/computer/

In Chinese-Traditional: “計算機”

  1. http://home.educities.edu.tw/tky999/top/top-right/tool/FinanceTools/calculator.htm
  2. http://www.acm.org
  3. http://zh.wikipedia.org/wiki/%E9%87%8F%E5%AD%90%E8%AE%A1%E7%AE%97%E6%9C%BA

And last but not least, in English: “computer”

  1. http://en.wikipedia.org/wiki/Computer
  2. http://www.dell.com/
  3. http://www.apple.com/

On a lark I dug into the English results to see where IBM showed up since I would think that they would be synonymous with the word computer to some degree… After lots of clicking I found it. At the bottom of page #57 in my results: http://www.research.ibm.com/compsci/

This kind of blew me away so I quickly did a search for the word “server” in English as well and found IBM at the bottom of page #9 with a link to: http://www.ibm.com/software/webservers/appserv/was/

While I was not surprised to see Wikipedia frequently showing up as the #1 result in most countries/languages I did expect to see more academic material represented. I was surprised to see how well Dell and Apple were represented. I was also taken aback by the Chinese Traditional #1 which is a bland page with a giant basic math calculator square in the middle of the page.

Also, looking at this from a global SEO frame of reference, I think this nicely highlights that when you are doing global SEO analysis, it can pay to develop localized strategies. For instance it would seem to me that Russia, Korea and Spain might be low hanging fruit for an aggressive campaign to capture visibility on the word computer.

Tell me what you think about these results.


April 8, 2010

The case for a mobile friendly website

In 2007, it was estimated that 36%-40% of the world’s population carried a mobile device giving us an estimate of 2.4-2.7 billion people carrying at least one phone. At that same time several writers projected that based on current growth estimates, sometime around 2010 to 2012 (depending on who you asked) we might hit 3.3-3.6 billion mobile devices.

Well, here we are in 2010 and according to a UN report published in March 2009 it was estimated that there were already 4.1 billion mobile phone subscribers  at the end of 2008 (60% of the world population), with the fastest growing country being… Pakistan.

Additionally, it was noted that there had been a clear shift from fixed to mobile cellular phone use and that in the same reporting period there were over three times more mobile cellular phone subscriptions than fixed telephone lines globally. Two thirds of those mobile phones are found in the developing world compared to less than half in 2002.

Why does this matter?

Because among Gen Y (and younger) and throughout many parts of the developing world the cell phone or other mobile devices are becoming the first device of choice (or necessity) for interacting with the Internet, for making online purchases, for banking, etc… South Korea, a country considered to be at the leading edge of digital communications is a place now where nearly everything is done through your cell phone and the simple idea of getting a plastic card to use for purchasing is archaic and offered as a courtesy option to banking customers who think they might be traveling out of the country.

This leads us to another thing to be aware of when you look at that graph above. The racing climb of mobile devices also represents a growing class of web users that may be visiting your website, buying your products, trying to get customer support.

It was estimated that in 2008, the number of mobile Internet users had reached 1.05 billion, surpassing the number of PC web users (1 billion) for the first time ever.

The natural questions become, why isn’t your website mobile friendly? And, if you’re doing any kind of e-commerce, why isn’t your store front not only mobile friendly but able to accept payments in the common methods of payment frequented by your customer base?

If you don’t have the answer, your business could be in trouble. Even the best established business relationship or brand loyalty can dissolve in the blink of an eye when there is a major change in the way society communicates.

You can see this taking place right now across all aspects of the publishing world. Non-Internet based media companies have spent the greater part of decade trying to figure out how to apply their business models to Internet communications rather than the other way around. For example let’s take the contemporary case of the newspaper classifieds. Classifieds were the mainstay of newspaper revenue for over a hundred years in the U.S. but they were never a perfect process for users. Limited words, fees, trying to figure out what days you wanted your ad to show up… these things all presented challenges to customers using the service. Then a very basic website called Craigslist showed up which was free to read, free to post (short of fees for job postings and some services) and had no real limits on word counts or listing durations. Within a few short years the classifieds industry was decimated and many newspapers soon found themselves going out of business because they could not or would not adapt.

So let’s bring this all around… in short everything I’m talking about here relates to location and convenience. These fundamental elements have been key to business success since the dawn of time. Real-estate agents get it “Location, Location, Location!”. Traditional marketing people get it “Go to where the customers are”.

People are on the web, people are using their cell phones to use the web. People are using their cell phones for the majority of their day to day communications when you take in voice, SMS, web, chat, gaming, etc… Do your children have their own cell phone? Does each child have their own cell phone? When they grow up to be a consumer, will your company be positioned to communicate to them in a way that they expect to be talked to or are you simply expecting them to learn an archaic way to talk to you based on how you talk today? When you get that Fax with their answer let me know.

Get it? Good. Next you need to actually have something relevant on your website for visitors to see when they get there on their cell phone, but that’s another conversation all together.

My thanks to the following resources for my data:

  • http://communities-dominate.blogs.com/brands/2007/01/putting_27_bill.html
  • http://www.itu.int/newsroom/press_releases/2009/07.html
  • http://www.nytimes.com/2009/05/25/technology/25iht-mobile.html
  • http://wapreview.com/blog/?p=3019
  • http://www.tomiahonen.com/ebook/almanac.html
  • http://www.shirky.com/weblog/2009/03/newspapers-and-thinking-the-unthinkable/
« Newer PostsOlder Posts »