Enterprising Developments

Breaching Big Data's Biggest Challenge: The Skills Gap

Joe McKendrick
Insurance Experts' Forum, August 23, 2012

Last week, I read that Gartner designated “big data” as one of the most over-hyped terms of the year, right behind cloud computing.

Nevertheless, the analyst group acknowledges that big data will have a profound, transformational impact on businesses. This is advice to which the insurance industry needs to pay close attention.

Where and how does big data make a difference? The bottom line is that big data helps organizations make better business decisions. A majority of respondents to a new survey, sponsored by Rainstor, found 75 percent agree that big data is extremely important for improving overall business value.

However, there are challenges. Thirty-seven percent of survey respondents indicate that the biggest challenges around managing big data are the speed of data creation (velocity), increase in types of data (variety), and the ability to provide analytics against data.

Perhaps the most vexing challenge is simply finding people with the skills to translate big data into big analytics and big decisions. Switching from data warehouse to Hadoop-style environments is one of the moves being considered—whereas standard SQL still appears to be the enterprise standard when running queries and analysis against existing data warehouses, new skills are needed to make new approaches like this possible.

When asked what the limiting factors are for those looking to adopt and deploy Hadoop, more than half of the survey's respondents agreed that they either lack the skilled resources or they simply have limited bandwidth to take on new technologies and projects. Despite the fact that managing and analyzing big data is high on the business and IT priority list, it is still challenging to achieve inside a large enterprise. Certainly, existing IT personnel are seasoned users, administrators and operators of relational and columnar-type databases, yet they are not seasoned java programmers or engineers that can easily deploy and support open-source Hadoop and then provide the analytics that the business demands.

The ability to find people to make the transformation to an analytics-driven culture will be a top priority for insurance and financial services companies in the years to come. Remember, data management is no longer some role in the database or IT department. Insurance companies, perhaps more than other types of companies, are now data companies. It is the core of the business.

Joe McKendrick is an author, consultant, blogger and frequent INN contributor specializing in information technology.

Readers are encouraged to respond to Joe using the “Add Your Comments” box below. He can also be reached at joe@mckendrickresearch.com.

This blog was exclusively written for Insurance Networking News. It may not be reposted or reused without permission from Insurance Networking News.

The opinions of bloggers on www.insurancenetworking.com do not necessarily reflect those of Insurance Networking News.

Comments (3)

Joe, very insightful article. We are seeing an increase in businesses seeking specialized skills to help address challenges that arose with the era of big data. The HPCC Systems platform from LexisNexis helps to fill this gap by allowing data analysts themselves to own the complete data lifecycle. Designed by data scientists, ECL is a declarative programming language used to express data algorithms across the entire HPCC platform. Their built-in analytics libraries for Machine Learning and BI integration provide a complete integrated solution from data ingestion and data processing to data delivery. More at http://hpccsystems.com

Posted by: HAANA M | August 31, 2012 3:31 PM

Report this Comment


Pretty good article. You missed the reflexive nature of the definition of 'Big data' though - when you make a statement like:

"limited bandwidth to take on new technologies and projects"

That is inherent in the definition of 'big data' - it is a volume/complexity of data which exceeds your organizational abilities to absorb it; if it was easy, it wouldn't be 'big'. Organizations which used to have Terabytes of data & gradually worked their way up to Petabytes while keeping their toolset built out don't consider their data to be 'big'

Posted by: Aborth | August 23, 2012 1:07 PM

Report this Comment


Joe -- great blog and so very true. Knowing data and then being able to understand it or so critical. Many times we don't have the building blocks/foundation from even to start. Proper skills and the building blocks are key.

Posted by: Cindy M | August 23, 2012 12:29 PM

Report this Comment

Add Your Comments...

Already Registered?

If you have already registered to Insurance Networking News, please use the form below to login. When completed you will immeditely be directed to post a comment.

Forgot your password?

Not Registered?

You must be registered to post a comment. Click here to register.

Blog Archive

It’s Okay to Take a Breather from the Technology Maelstrom

Even in technology, good things may take time.

Customers for Life

Insurers once had a monopoly on lifetime customers, but technology has changed the game.

Smarter Tablet Use Could Transform Insurance

By reducing administrative tasks and automating paperwork, tablets can increase agentsí selling time and help them respond to customers in seconds, not hours.

Insurance Wake-Up Call: Embrace the Shared Economy Opportunities

SMA believes that insurers must embrace a "shared economy," crowdsourcing and open innovation to get ahead in the new marketplace.

The Lion and the Mouse: Start-ups Pitch to Top Insurer

Insurers should be on the lookout for innovative partnership arrangements that produce unique and valuable solutions.

Silicon Valley Ventures

A trip to area hotbed of technological innovation calls into question the potential viability of insurers' legacy systems, operations and processes.