Enterprising Developments

Breaching Big Data's Biggest Challenge: The Skills Gap

Joe McKendrick
Insurance Experts' Forum, August 23, 2012

Last week, I read that Gartner designated “big data” as one of the most over-hyped terms of the year, right behind cloud computing.

Nevertheless, the analyst group acknowledges that big data will have a profound, transformational impact on businesses. This is advice to which the insurance industry needs to pay close attention.

Where and how does big data make a difference? The bottom line is that big data helps organizations make better business decisions. A majority of respondents to a new survey, sponsored by Rainstor, found 75 percent agree that big data is extremely important for improving overall business value.

However, there are challenges. Thirty-seven percent of survey respondents indicate that the biggest challenges around managing big data are the speed of data creation (velocity), increase in types of data (variety), and the ability to provide analytics against data.

Perhaps the most vexing challenge is simply finding people with the skills to translate big data into big analytics and big decisions. Switching from data warehouse to Hadoop-style environments is one of the moves being considered—whereas standard SQL still appears to be the enterprise standard when running queries and analysis against existing data warehouses, new skills are needed to make new approaches like this possible.

When asked what the limiting factors are for those looking to adopt and deploy Hadoop, more than half of the survey's respondents agreed that they either lack the skilled resources or they simply have limited bandwidth to take on new technologies and projects. Despite the fact that managing and analyzing big data is high on the business and IT priority list, it is still challenging to achieve inside a large enterprise. Certainly, existing IT personnel are seasoned users, administrators and operators of relational and columnar-type databases, yet they are not seasoned java programmers or engineers that can easily deploy and support open-source Hadoop and then provide the analytics that the business demands.

The ability to find people to make the transformation to an analytics-driven culture will be a top priority for insurance and financial services companies in the years to come. Remember, data management is no longer some role in the database or IT department. Insurance companies, perhaps more than other types of companies, are now data companies. It is the core of the business.

Joe McKendrick is an author, consultant, blogger and frequent INN contributor specializing in information technology.

Readers are encouraged to respond to Joe using the “Add Your Comments” box below. He can also be reached at joe@mckendrickresearch.com.

This blog was exclusively written for Insurance Networking News. It may not be reposted or reused without permission from Insurance Networking News.

The opinions of bloggers on www.insurancenetworking.com do not necessarily reflect those of Insurance Networking News.

Comments (3)

Joe, very insightful article. We are seeing an increase in businesses seeking specialized skills to help address challenges that arose with the era of big data. The HPCC Systems platform from LexisNexis helps to fill this gap by allowing data analysts themselves to own the complete data lifecycle. Designed by data scientists, ECL is a declarative programming language used to express data algorithms across the entire HPCC platform. Their built-in analytics libraries for Machine Learning and BI integration provide a complete integrated solution from data ingestion and data processing to data delivery. More at http://hpccsystems.com

Posted by: HAANA M | August 31, 2012 3:31 PM

Report this Comment


Pretty good article. You missed the reflexive nature of the definition of 'Big data' though - when you make a statement like:

"limited bandwidth to take on new technologies and projects"

That is inherent in the definition of 'big data' - it is a volume/complexity of data which exceeds your organizational abilities to absorb it; if it was easy, it wouldn't be 'big'. Organizations which used to have Terabytes of data & gradually worked their way up to Petabytes while keeping their toolset built out don't consider their data to be 'big'

Posted by: Aborth | August 23, 2012 1:07 PM

Report this Comment


Joe -- great blog and so very true. Knowing data and then being able to understand it or so critical. Many times we don't have the building blocks/foundation from even to start. Proper skills and the building blocks are key.

Posted by: Cindy M | August 23, 2012 12:29 PM

Report this Comment

Add Your Comments...

Already Registered?

If you have already registered to Insurance Networking News, please use the form below to login. When completed you will immeditely be directed to post a comment.

Forgot your password?

Not Registered?

You must be registered to post a comment. Click here to register.

Blog Archive

Digital Vision vs. Harsh Reality

Much work remains to reconcile insurers' digital vision with the digital reality that seems to be arriving for other industries.

Vendors Embrace Mobile Technology

IT leaders at software firms clearly recognize the importance of mobility to drive their businesses forward. Almost 70 percent see mobility as mission critical or important to their organization today.

Big Data Is Paying Off

Insurers are getting business benefit out of their big data projects, but these projects alone won't grow their business.

What Can Insurers Learn from Home Depot?

The latest cyber-attack highlights the importance of helping policy holders defend themselves.

Not Your Father’s Insurance Company

Carriers need to look at new and impactful ways to be there for their customers.

How to Attract Top Tech Talent

When it comes to rankings of the best places to work, insurers are few and far between. Here’s what those who make the lists do to appeal to IT professionals.