This article was originally published on June 5, 2014, and has since been updated.
Big data has been generating a lot of buzzes recently because of its numerous capabilities and the relative ease with which the road to Hadoop can now be accessed and used. It’s an impressive combination that is allowing businesses to innovate and implement objectives that only a few years ago would have seemed impossible. Thanks to technological advances in Hadoop and cloud computing, there are greater possibilities today than ever before.
Tweet This: 55% of big data projects fail. Learn the key to big data success.
In his Information Week article, Dough Henschen overviewed two examples of big data success from British Airways and Metlife as covered in a recent big data conference at Interop New York.
British Airways struggled to gain an in-depth knowledge of its customers because the data was siloed across programs, but with the help of analytics software, was able to combine the data into a single program.
“Previously it took six to nine months to build a single analytical application at [British Airways], but now they can do it in two to three weeks,” Arnab Gupta, CEO of Opera Solutions, was quoted as saying in the article. “That has let loose their innovative energy, and if they want to come up with a seating upgrade offer, they can quickly figure out who to target.”
Metlife was able to create the Metlife Wall, a NoSQL project that uses MongoDB to capture information from 70 different internal systems and gives the company a total customer view. All this was done in just 90 days.
Big data technology can make a difference.
The problem, however, is that too many companies jump into projects without really understanding why they’re using big data and what they want to achieve. According to an article from Gigaom, 55% of big data projects fail. That’s an extremely high number. It goes to illustrate the enigma that big data can be for many companies who don’t take the time to understand its capabilities.
To better examine big data success, let’s first look at some of the mistakes uninformed companies make that can lead to big data failure:
The “bigger is better” assumption
According to experts, many companies see big data as the solution to all their problems. And so they rush into projects without knowing what they are looking for or hope to learn from the mountains of information they have accumulated. Projects with unclear objectives are destined to fail.
Underestimating the talent gap
According to research by the McKinsey Global Institute (MGI),
“…by 2018, the United States alone may face a 50 to 60 percent gap between supply and the requisite demand of deep analytic talent…”
Companies that fail to address this growing shortage of skilled data analysts—seeking out new talent and offering existing employees new big data training—will find themselves ill-equipped to mine big data for insights and competitive advantage going forward.
Failure to keep data organized
Companies have gotten really good at collecting massive pools of data, but keeping data organized is something many companies are not doing well. The challenge is that so much unstructured data is flowing in from various sources; it’s hard to keep track of it all. As a result, companies need to conduct “data audits” before starting a project to ensure that all of the information data to be analyzed is first organized into a single database and format. Otherwise, the project is likely to produce lackluster results.
Failure to avoid infighting
One of the big questions for a big data project is which department owns it. This can lead to organizational friction and disharmony which if not addressed and dealt with by a duly appointed C-level executive—“chief analytics officer” for example—could jeopardize or completely undermine the project’s intended purposes and benefits.
Going too big too soon
Lulled by big promises and big expectations, many eager and uninformed companies go right out of the gate with expensive, high-risk big data initiatives. To avoid becoming a part of the 55% failure rate statistic, a better tactic is to start with smaller projects and build on their successes toward larger projects.
Fortunately, the essentials to a successful big data project are simple and can be done by any size company. Here are four tips to make your next big data project successful.
1. Define objectives
Before you begin a project you need to define the business objective. What’s the purpose of the project? How will it benefit your company? Why do you need big data? What are potential problems that might arise? Some examples of successful business goals include:
- Personalization and micro-segmentation in digital advertising. Learn more
- Understand consumer behavior in order to optimize a product. Learn more
- Identify which products to stock and at which price. Learn more
2. What is big data technology?
Big data technology is new and evolving. Before you begin a project you need to understand what it is. Until you know at least a little about it, there’s no way you can successfully use it.
3. What are its capabilities?
If you don’t understand what big data technology is and isn’t capable of, then there’s no way you’re going to have realistic expectations for your project. Either you’ll expect too little or too much — either way it hinders the project and your company.
4. Project management
Potentially the most important aspect of a successful big data project is having the right people oversee it. You need to have people who are familiar with it and know what they’re doing. You also need to have people who understand its potential and are willing to go through tough times in order to find the jewels that big data can offer.
The tools for success are available to companies big and small through big data technology. It can make a tremendous difference in your business today and in the future, but only if it’s done correctly. If not it will hinder more than help. By taking the proper steps in the beginning you can set yourself up for project success.
