Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in ecommerce
    Analytics Technology Drives Conversions for Your eCommerce Site
    5 Min Read
    CRM Analytics
    CRM Analytics Helps Content Creators Develop an Edge in a Saturated Market
    5 Min Read
    data analytics and commerce media
    Leveraging Commerce Media & Data Analytics in Ecommerce
    8 Min Read
    big data in healthcare
    Leveraging Big Data and Analytics to Enhance Patient-Centered Care
    5 Min Read
    instagram visibility
    Data Analytics Plays a Key Role in Improving Instagram Visibility
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Keep an Eye on the emerging Open-Source Analytics Stack
Share
Notification Show More
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Software > Open Source > Keep an Eye on the emerging Open-Source Analytics Stack
AnalyticsOpen Source

Keep an Eye on the emerging Open-Source Analytics Stack

DavidMSmith
Last updated: March 3, 2011 5:47 pm
DavidMSmith
8 Min Read
SHARE

This post is contributed by Revolution Analytics CEO Norman Nie, and cross-posted from the Future of Open Source Forum.

This post is contributed by Revolution Analytics CEO Norman Nie, and cross-posted from the Future of Open Source Forum.

A lot of attention has been focused recently on Big Data, and rightly so: Big Data is a Big Deal. (See this LinuxInsider article, Big Data, Big Open Source Tools, for a compehensive overview of Big Data issues.) But what, exactly, is Big Data? Ed Dumbill of O’Reilly defines Big Data as “data that becomes large enough that it cannot be processed using conventional methods”. That’s a good definition: Big Data is primarily about your capability of making use of the data, not just its size in terabytes or petabytes.

More Read

3 Ways Big Data Is Changing Financial Institutions Forever

Predictive Analytics is a Proven Salvation for Nonprofits
Integrating Predictive Analytics and BRM to Improve Health Plan Member Experience
Use AI to Get the Most Out of Your Social Media Marketing Strategy
Minding the KPI Gap – A Critical Part of the EPM Process

In that same article, Dumbill coins a term to represent the emerging software infrastructure — typically open source, distributed, and running on commodity hardware — for handling Big Data: “A stack for big data systems has emerged, comprising layers of Storage, MapReduce and Query (SMAQ).” You might notice that a letter remains undefined in the acronym, SMAQ. I think it’s obvious what the ‘A’ should stand for: A stands for Analytics.

Because while much attention has been paid to storing, organizing, and accessing Big Data, the real value of data is in what you can do with it. The goal is simple: achieve return on what’s likely to have been a significant investment in collecting and storing information, by using data analysis to better understand your business and anticipate future outcomes. To achieve this goal, on top of a platform for storing and accessing data we also need a layer for analyzing the data: a platform where the talents of skilled data scientists can be brought to bear to combine the latest statistical and machine learning techniques to create predictive models of unprecedented power. And on top of that we need another layer, to be used by the ultimate consumer of the intelligence: a platform for the business user looking at reports, or interacting with a BI dashboard, for example. As with the SMAQ stack, these layers will typically be open-source, distributed, and designed to exploit the power of multi-processor commodity hardware.

Together, these data, analysis and reporting layers form an emerging software stack for deriving value from data: an open-source Analytics Stack, if you will. Where SMAQ provides a platform for Big Data storage and access, the open-source analytics stack provides a platform for extracting intelligence and providing insights from those data.

In other words, the end-result of the open-source analytics stack is high-value, easy-to-use business intelligence. From a purely business perspective, the true value of the stack is its ability to generate predictive analytics at a comparatively low cost. Unlike monolithic BI platforms, which excel primarily at providing tabular summaries of historical data, the open-source analytics stack is forward-looking: predictive models provide estimates of future outcomes and (equally important) the uncertainty around those estimates. That makes it ideal for business scenarios in which competition is ultra-intense and market demands are constantly changing.

But hold on a second: what about those traditional software vendors with their monolithic BI solutions? Don’t some of those heavyweight solutions aim to provide the same kind of usable business intelligence that we’re talking about?

To be fair, the answer is yes — sort of. But when you compare the cost and flexibility of open-source resources to the cost and flexibility of proprietary systems, there’s no contest.

Open source wins, hands down.

Let’s take a moment to look within the multiple layers of the open-source analytics stack. The bottom layer is your operating system, typically a Linux variant, running on inexpensive yet powerful multiprocessor hardware in a local cluster or perhaps in the cloud. Then you have a data layer, which would be Hadoop, MySQL or some similar tool that enables you to reliably and efficiently store your data in volume. Next is an ETL layer, for processing the data and extracting segments for analysis. On top of that is the analytics layer for which R, the open-source programming language that has become the lingua franca of statistical analysis, is the ideal candidate. And on top of the analytics layer is the presentation or BI layer, through which the results of these predictive analytics models will be conveyed.

With all of these layers, interoperability is really critical. As the value of the open source analytics stack becomes more widely acknowledged, more players will emerge to provide tools that work and play well with the other tools in the stack. Companies such as Cloudera, Talend, Revolution Analytics, and Jaspersoft already have products available that work together to make the open source analytics stack a fully-functioning reality. It’s entirely reasonable to assume that at some point in the near future, dozens of companies will offer products and services designed expressly to work within this stack or to support value-added operations around it.

These are indeed exciting times, and it will be fun to watch this open-source analytics stack evolve. Given the size, passion and commitment of the open-source community, I predict that its evolution will be quick and dramatic.

I also can’t help wondering what will happen to the traditional vendors and their monolithic BI solutions. Frankly, I think they’re on the wrong evolutionary track.

For the business user, the key takeaway is that this data analytics stack, built on commodity hardware and leading-edge open-source software, and a is a lower-cost, higher-value alternative to the existing status quo solutions offered by traditional vendors. Just a couple of years ago, these types of robust analytic capabilities were only available through major vendors. Today, the open-source community provides everything that the traditional vendors provide — and more. With open-source, you have choice, support, lower costs and faster cycles of innovation. The open-source analytics stack is more than a handy collection of interoperable tools — it’s an intelligence platform.

In that sense, the open-source analytics stack is genuinely revolutionary.

Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

trusted data management
The Future of Trusted Data Management: Striking a Balance between AI and Human Collaboration
Artificial Intelligence Big Data Data Management
data analytics in ecommerce
Analytics Technology Drives Conversions for Your eCommerce Site
Analytics Exclusive
data grids in big data apps
Best Practices for Integrating Data Grids into Data-Intensive Apps
Big Data Exclusive
AI helps create discord server bots
AI-Driven Discord Bots Can Track Server Stats
Artificial Intelligence Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

What Is Your Dashboard Telling You?

7 Min Read

Peer into the Analytics Crystal Ball for 2015

7 Min Read

Last night at the offices of blogging software company Six…

1 Min Read

Value of Decommissioning Legacy Systems

5 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-24 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?