BigInsights on Cloud: Major benefits and future enhancements

IBM BigInsights is an enterprise-ready Hadoop distribution designed to enable analytics on larger volumes of data more cost-effectively. We sat down with Rohan Vaidyanathan, the offering manager for BigInsights on Cloud, to talk about the major benefits of the cloud service and get the scoop on the enhancements coming down the pike this year. In part two, we’ll talk about how companies are actively using BigInsights on their big data projects.

Hi Rohan! We wanted to sit down with you today to dig into BigInsights on Cloud. Let’s start with a quick overview. What is IBM BigInsights?

BigInsights is IBM’s enterprise-grade Hadoop distribution. It is based on pure open source Apache Hadoop. It includes IBM Open Platform, which is a distribution of open source Hadoop that includes the latest and most stable components one would need to run the Hadoop ecosystem. It’s not just the Hadoop components, the main Hadoop runtime, but also the other open source projects that you need to be successful with your Hadoop or big data project. It also provides high value analytics tools to enable users to get insights from the data.

What’s the value of having BigInsights on the cloud?

Hadoop is one of the most flexible and at present, best technologies to analyze data in a cost-efficient manner. It relies on commodity hardware, which is cheap hardware, and basically you keep adding more hardware and infrastructure as your business requirements grow. While it’s a more flexible and relatively cheaper option, it also brings a lot of questions with it:

• How do you grow your infrastructure?
• How do you manage such a large amount of machines?
• Do you have the talent in your organization for someone to keep a tab on all of your machines, and grow the infrastructure?
• Do you even have that amount of space in your data center?
• From an economics point of view, do you want to invest in more infrastructure than you need at any given point in time?

On the cloud, you can scale when you need to. You don’t have to worry about the infrastructure or administration. You just bring your data to Hadoop, you write your applications, and you run your analytics. So it takes away a lot of those questions and most importantly, it lets you get started quickly. If you think about a typical Hadoop project, you would have to size out the environment, go look at the architecture, build out the environment, etc. All of that just disappears on the cloud.

You have a cluster ready to go at the click of a few buttons and you have expertise available at IBM to take you through the process of getting data in. When you’re analyzing that data, you have access to Hadoop experts who can help you optimize the analysis.

There are several tools available in BigInsights that focus on data analysis through slightly different lenses. Why did you make that design choice?

When we designed BigInsights, we had an important idea in mind: data needs to be democratized – it needs to go to each and every individual in an organization and each of them should be able to get insights from the same piece of data irrespective of their analytics skill level. BigInsights brings that level of analytics to the table. With other Hadoop distributions, you would have to go find and integrate tools to enable any analytics across the organization. That’s the basis of design.

BigInsights includes several analytics components. One of the most important is Big SQL, which is a fully ANSI SQL compliant MPP spread across Hadoop. It uses the data from HDFS or HBase or Hive and presents a familiar SQL interface that anyone familiar with SQL can use to query data residing in Hadoop. You could also use this to connect to existing business intelligence tools. This is really powerful. To give you an example, if you were using Hive, you might end up spending a lot in time, money and resources to rewrite queries. With BigSQL, you have standard JDBC or ODBC drivers that let you connect directly into the Hadoop data, so you can reuse your existing business intelligence tools or your existing reports without any change.

Another analytics component is Text Analytics which is a very easy to use drag and drop tool to do analysis of free from text. And then you have Big R which is an implementation of R algorithms with the ability to parallelize them using machine learning language.

These components are built to support users in an organization who are using data analysis in different ways, to perform different task with different skills. With this level of support, BigInsights has the ability to bring analytics to all of the users who need to support the business with insight.

What changes are coming this year to our BigInsights on Cloud offering?

There are some exciting changes in the pipeline. With our release in June, we took more responsibility on ourselves for the monitoring and maintenance of the infrastructure of the Hadoop platform, as well as some of the most important components. With this change, we took away a lot of customer pain in administering the platform. We are going to be even more aggressive with our advances on that front moving forward, bringing more administration under the IBM cloud operations’ sphere of responsibility, and freeing customers up to focus on their data, analytics and insight: the things they really care about.

We are also making a lot of security enhancements to comply with industry standards like ISO27K and SOC2. So more administration, more managed operations capability, more security and standards compliance. That is our focus this year. We are also actively working to make the environment more flexible, providing a model that allows our users to get started cheaper and scale faster. There will be more on that in the next few months.

This all sounds really exciting! Thank you for taking the time to talk to me today. I’m going to catch up with you next week to get more details about those changes.

Thank you. Looking forward to it.