BI feeds

Runar Lie, founder of Office123, posted his list of the top 10 open source applications for businesses. Nice to see Pentaho on the list.

https://www.office123.net/uhelp/index.php/blog/2010/01/31/8-the-top-10-open-source-business-applications?lang=en

Red Hat CEO Jim Whitehurst, wrote recently about why car manufacturers should look into open source – Why Toyota Should Go Open Source

He has some valid points. We have a Bose system at home that about once a month stops responding to the remote and needs rebooting. If I add up all the time I’ve spent doing that (and talking family members through it over the phone), I’d gladly take a look at the source code and try to fix it.

When it comes to cars, however, I really like the approach taken by Local Motors. Their design and manufacturing process is very open:

• They have open competitions to select the car design. At this stage it is mainly concept art done by graphic artists and 3-D modelers. The range and imagination of the designs is very cool. The amount of work put in by the contributors is amazing. See the design wall.
• They have a voting process to select the designs that will be made into cars
• They have an open build process and CAD files are available – Build Process
• They have an open manufacturing process where you can participate in the building of your car – Build Experience

How cool is that?

Now they just need an embedded Linux-based OS for all the electronics that we can contribute to. Maybe some kind of serial bus that can be used universally by all the sensors and switches.

One of the participants in the latest round of open core bashing was Eric Barroca, CEO of Nuxeo. He chimed in with over 2000 words on why the open core model is ‘fundamentally flawed’ (his words). You can read his post if you click here.

In this post I am not trying to attack him or his company. But I think his post highlights how confusing and subjective this issue is.

Here are a few quotes from his post:

I’m deeply convinced that “open core” is fundamentally flawed

I would add that a subscription-based model (where subscription is for maintenance and support services) is superior (to an open core model)

I believe in clear business models. Successful companies use clear business models because that’s what enable trust from customers. Open core is not one of them: there is no clear line between open source / proprietary, neither serious justification for the customer.

That’s why we – at Nuxeo - won’t use the open core model even if it could increase short term revenue.

So Eric is fairly clear and outspoken about his view of the open core model. The following are snippets from the comments on the blog post

• James Dixon – Maybe I have this wrong, but doesn’t Nuxeo Connect include ‘Premium Tools’?
• Eric Barroca – Nuxeo Connect include SaaS. Not tools / code / additional features to apply on the open source software. Pretty much like RedHat for its OS or for JBoss. So, no, we’re not an open core company.
• Rahul Sundaram – So let’s keep this simple: Is any part of Nuxeo that you sell proprietary code is it all 100% free and open source? Otherwise, the comparison to Red Hat model is invalid.
• Eric Barroca – 100% of the code we sell / you install and run is LGPL. Period. So I think the comparison is very valid.

When Eric says ’100% of the code we sell / you install and run is LGPL’, I guess that’s true. But it seems there is code they sell (via SaaS), that is not installed, and is not open.

• Customers of Nuxeo have access to value-added design, configuration, monitoring and management tools. These features are not open source, they are proprietary, and only available to customers. These features are available via a SaaS offering.
• Customers of Pentaho have access to value-added design, configuration, monitoring and management tools. These features are not open source, they are proprietary, and only available to customers. These features are available on-premise.

As I understand it, having proprietary extensions to open source software – no matter how many extensions, or how many customers use them, or how they are delivered – makes you an open core company. As far as I see it the only difference between Nuxeo’s model and Pentaho’s model is that Nuxeo’s proprietary extensions are only available as SaaS, whereas Pentaho’s are available on premise.

If Pentaho stopped offering on-premise installations, and only offered SaaS, would Pentaho suddenly no longer be open core?

Do Nuxeo and Pentaho have different models, one open core, one not? If so, how come?

If Nuxeo and Pentaho have the same model, what model is it? Open core or what?

Who’s more confused, me or Eric?

Your thoughts?

A good review of Pentaho Data Integration V4.0 from Seth Grimes Doug Henschen (my bad, sorry Doug) at Intelligent Enterprise -  Pentaho Release Blends Data Integration and BI

Analysts of the software industry have some challenges when it comes to talking about open source. I know several analysts from a few different analyst companies, and they are good people with good intentions. This post is not a criticism of them – just my view of some challenges their industry faces.

Many analysts base their work on the experiences they had prior to becoming analysts. A lot of them worked for decades in large and small software companies. This experience has served them well to date. Unfortunately they now have to talk about open source companies – and none of them have ever worked in one. They now have to deal with new models, new terminology, and new concepts – all without getting direct experience.

This lack of experience leads them to make mistakes and say some odd things. Here is a recent examples (not naming names – you know who you are). This is from an analyst at a very prominent analyst firm, talking about open source companies using the open core model:

Even the very definition of “community” is being adapted to suit the open core narrative. What has largely interested the corporate IT world is the concept of a community as a collection of code contributors working outside a normal project/company structure. But now open core providers are extending the term community to include users and even resellers. That, of course, is what we’ve all been calling a software ecosystem for the last twenty years. Same old, same old – just co-opted terminology used to describe it.

What this analyst is saying is that open core companies have modified the term ‘community’ to suit their needs, by including the user community and not just the developer community. Anyone familiar with open source sees how ridiculous this statement is. Within the free software and open source worlds the community includes all constituents, not just developers – wikipedia definition.

Secondly they state that an open source community is the same as the ecosystem of a proprietary vendor. This is not the case in my experience – the participants and the kinds of interactions are very different. This analyst, due to inexperience, ends up making inaccurate statements that reduce their credibility.

To help counter this issue analyst firms are adding open source specialists. This is a decent initial step but it doesn’t really solve the problem. When it comes to our domain (Business Intelligence) most analyst firms have one analyst that understands open source, and a different analyst that understands the BI market. Unfortunately, neither of them is able to understand how open source will impact the BI market.

The first thing analysts need to do is to review the questions they ask. In many cases the questions asked are about the amount of money spent on software licenses. This is fine as long as 100k spent on package X is roughly equivalent in value to 100 copies of package Y which costs 1k. In the past this has been roughly true. But now open source companies are providing equivalent products for a fraction of the price charged by the old vendors. Additionally, there are companies that sell subscriptions, not licenses.

1. Stop asking about licenses
2. Stop asking about money
3. Start asking about CPU/cores
4. Start asking about usage

This is an issue that has to be tackled domain by domain. The analysts in the operating system and database domains get this (took them years). Analysts in the BI space don’t get it yet.

One of the metrics that some analysts use when assessing the impact of a vendor is ‘how often do people call me asking about the vendor’. The reason people call asking this question is that they are not sure if the software will suit their purpose or solve their problem. They are calling the analyst because they have no other way to get an unbiased opinion. When consumers have the ability to download the software and prove to their own satisfaction it does what they need, they don’t need to call an analyst.

An analyst saying ‘no-one is using open source software X’ is like a travel agent saying that no-one goes to Disney anymore. They do, in droves, they just don’t have to call you first.

Another challenge facing analysts is that they don’t know how to factor community participation and usage into their models. One tactic has been ‘if you don’t pay for support, we don’t consider you’. Obviously this doesn’t work in the long term. Lets say Apache HTTP server ends up with 90% of the existing web server market, and is used in an equivalent number of installs at companies that could not have afforded WebSphere, WebLogic or IIS. From the perspective of the analyst the web server market has shrunk considerably, whereas it has actually grown considerably.

One way analysts could tackle this is to introduce a new visualization of the market – one that shows usage or impact or reach. Maybe the other axis could be ’sophistication’ or ‘completeness’. In the BI market this would make sense because now Excel has a spot of its own. Excel is commonly used to solve BI needs but is typically not shown as its own point.

My response to Garner’s Brian Prentice’s reply to my post responding to his post got too big for a comment field… So here it is

What I care about is whether your business model creates incremental value for your intended customers. Those are the people I spend most of my time speaking to. And those are the people who clearly understand the value that Gartner provides in an increasingly open-world. What they want from us is to put vendor claims to the test – and that’s what I did in my blog…

If I have this correct, you’re saying that providing an equivalent product at 0% of the license cost is of no value? I’m skeptical.

…Believe it or not James I actually do understand the dynamics around S&M costs. I spent a good part of my career in product management, marketing and sales roles at software vendors. Those managing software companies should be concerned about these costs. But the nature of those cost structures are far more complex and entrenched than advocates of open core models make out…

I agree its complicated. I don’t doubt your claim that you understand the factors that influence proprietary software vendors. But you have no experience working for an open source vendor, and so you are starting to make (invalid) assumptions. What I don’t see in your posts is an indication that you understand the differences that exist between these business models. If you really understood these things I’m guessing you would point them out (to add value).

…And the jury is definitely out whether the open core approach will create a lasting impact. I personally think it won’t…

Let’s look at another domain that is in transition – cars. We can attempt to state that hybrid cars will create no lasting impact. This is almost certainly true given a long-enough timescale. Eventually hybrid cars will die, because, eventually, there will be no more oil. But probably, in order to get to an oil-free scenario, hybrids are vital during the transition. So can you really state ‘no lasting impact’?
Open core might be of value only during the transition from a proprieatry world to an open source one. If we ever get to an end game (20-30 years) my opinion is that open core will vital during the (long) transition

…But again James – so what? If you can lower your S&M costs you’ve increased your profit margin. Where’s the value to your clients?…

Thank you for proving my point. You don’t understand the model. We don’t lower our S&M costs to increase our profit margin. We lower our S&M costs to completely eliminate the up-front license fee. The cost of our subscription is less than the maintenance fee of the proprietary vendors, with 0% license fee. The software you previously paid $2million for, with a $400k maintenance fee, you can now get for $0 up-front and a $100k subscription. You see no value in that? Really?

…I think you should be spending more of your time blogging on that topic…

I do. You either don’t read, or don’t comprehend

…What is about open core that makes companies like Pentaho uniquely valuable as a potential supplier…

We provide an equivalent product at 0% of the license cost. Simple.

… And there’s the rub – there isn’t. …

0%. Still not compelling?

…That doesn’t doesn’t detract from your company or product. It just means you’re like every other software company out there vying for people’s business. Therefore you should be treated by those potential customers no differently…

I absolutely agree with you. From the perspective of the mainstream customer, when they come looking for a solution, they should treat every vendor the same – whether proprietary, open core, or pure-play. In fact we do best under those conditions – better than both proprietary and pure-play open source vendors.

Over to you, Brian.

We just wrapped up our most popular webinar ever.

Click here to watch the playback

Brian Prentice at Gartner has a new post titled Open Core: The Emporer’s New Clothes – click here to read it

I met Brian at OSBC recently and chatted with him for a while about several topics including patents and Microsoft’s open source strategy. While I like Brian, and we agree on many things, I (somewhat predictably) don’t agree with this latest post.

There are some important things that are missing from his piece.

For most proprietary software companies the biggest expenses are sales and marketing.

Let’s look at a proprietary competitor in Pentaho’s market – MicroStrategy (stock symbol MSTR). In their last quarterly report they state that

• Revenue from Product Licenses $34.4m
• Revenue from Support and Services $69.6m
• Sales and Marketing (S&M) Expenses $31.5m
• Research and Development (R&D) Expenses $11.4m
• G&A Expenses $14.2m

MicroStrategy’s figures are fairly typical for an independent proprietary software vendor. You can see from these figures that R&D is the smallest expense, and S&M is the biggest. In fact the S&M expenses are more than all other expenses combined. You can see that revenue from licenses is about the same as the S&M budget. It takes almost as much money to convince people to buy the software as they get from the sales. The word ‘convince’ is important – the S&M teams have to convince companies that the software both does what they need, and is worth the cost. Each quarter the money made from selling licenses is poured into next quarter’s sales and marketing budget. It is therefore support and services where they make most of their money and all of their profit.

To re-iterate, because the S&M expenses are so high, MicroStrategy make no money selling software licenses, they make most of their money (and all of their profit) on support and services. From this perspective they sound very similar to the pure-play open source companies – who make no money selling software licenses, they make it all on support and services.

If you contrast the S&M strategy and budget of a proprietary software company with a commercial open source one you will see fundamental differences. The commercial open source company provides open source software. This software has to be be fully-functional and useful, otherwise their will be no adoption. The R&D costs of this software will be lower and, more importantly, development will be quicker than that of a proprietary company. But, as proved above, the R&D costs are the lowest costs anyway – lowering your lowest cost does not provide you with a significant benefit. The major difference is that the open source community edition of the software enables an efficient, cheap, in-bound marketing strategy instead of an expense, out-bound, enterprise marketing strategy. By switching to a primarily in-bound sales model a commercial open source company can drop its marketing budget dramatically. The reduction of the S&M budget allows the license fee to be eradicated. So the tasks and strategies of S&M in an open-core company is significantly different from those of a proprietary company.

These facts about the S&M activities of proprietary and open source companies seem to be largely ignored by many analysts and commentators:

• Tarus Balog (CEO of The OpenNMS Group), says that open core vendors use open source as a marketing gimmick. As far as I know Tarus has no prior experience running a software company so I’ll forgive him his confusion. This is no gimmick, this is a fundamental part of the business model.
• Matt Aslett (451) and Brian Prentice (Gartner) talk about code contributions and R&D practices but rarely mention S&M

I have worked my entire professional career in either proprietary software development or commercial open source software development. I have experience of the differences between these models first hand. I have only covered differences in the S&M domain. There are also numerous, significant differences in the development, QA, partnering, business development, support and service departments etc. Some of these I cover in part II of the Bees and the Trees

As I have said before, the software industry is in a transition from proprietary software, to some mix of proprietary and open source software. Is that eventual mix 0% proprietary and 100% open source? I don’t know. I do know this transition has been in progress for over a decade and has a long time (decades) to go. Maybe the open core model will only be relevant during this transitional period. But this transitional period is long, and many start-ups and existing proprietary companies have no viable way through this transition without the open core model.

Brian Prentice seems to be trying to find something to say about open source. Using invalid comparisons with ‘freemium’ and SMB strategies isn’t a good idea. Heavy-handed, one-sided statements, about an entire business model  – based on seemingly incomplete knowledge of that model, doesn’t seem good either. Companies like Gartner are struggling to show value in an increasingly-open world. Brian’s latest post, in my mind, doesn’t exactly help them do that.

Pentaho Defines a Better Way to Build BI Solutions. Join Pentaho, Ventana Research and Mozilla, to see how Pentaho is Using Agile BI to redefine the way that BI applications are built.

Featured Speakers:
Mark Smith, CEO & Executive Vice President of Research, Ventana Research

Richard Daley, CEO, Pentaho Corporation

Daniel Einspanjer, Metrics Software Engineer, Mozilla

Joe Nicholson, Product Management, Pentaho Corporation

http://www.pentaho.com/events/20100331_big_news_webcast/index.php

Here is a new recorded demo of the modeling and visualization features in Pentaho Data Integration V4.0

http://www.pentaho.com/products/demos/showNtell.php?tab=demos&article=pdi4-intro

After the Open Source Business Conference last week a few analysts have been chiming in on the ‘open core’ business model for commercial open source companies:

Matt Aslett : http://blogs.the451group.com/opensource/2010/03/25/winning-and-losing-with-open-core/

Brian Prentice : http://blogs.gartner.com/brian_prentice/2010/03/23/open-sources-reality-distortion-field/

Matt’s first point is that:

The proprietary vendors are now using open source collaborative development and code to lower the costs and improve the quality of their own product development, and are better quipped to compete with open core vendors thanks to their installed base, larger resources, and product maturity.

What I don’t think Matt appreciates is that than open source business model is not just a different development strategy. Executing well using this model affects every department within a company – particularly sales and marketing. You use mass-marketing techniques instead of enterprise-sales techniques. I don’t see the large proprietary vendors changing that much. Also you have to factor in ratios of open source code to proprietary code: from 100% open in a pure-play, 80-99% in an open-core play, and 0-2% for the big proprietary vendors. If a large vendor manages to reduce their development costs by 10%, you are not going to see them drop their license fee by 80%, particularly because the license fees cover the sales and marketing costs.

Another mistake is to view the ‘product maturity’ of the incumbent proprietary vendors as an advantage for them. Some of these product are bloated, and very hard to install and configure. This is not an advantage against open source software which needs to be easily to consume – otherwise the model does not work. In many cases the large vendors want to limit (or carefully manage) access to the software before customers purchase it, because they know the experience of using it is not great.

Matt’s second point is:

The shift towards software services (SaaS, managed hosting, cloud-based delivery), where the value being delivered is via services, rather than products.

Matt’s point here is that an open-core approach is irrelevant in this environment. That’s true, but you can say the same of pure-play open source and proprietary software. Since this applies to all software development strategies I don’t see the relevance of it.

Brian’s first point is about the VC-centric nature of open source start-up in the USA:

There’s a yawning gap between the value open source provides a venture capitalist (VC) and what it provides an end user.

I think this is the wrong comparison. Brian seems to think that VC-based companies and self-funded companies have different attitudes towards making money. Having done two self-funded start-ups, and now a VC-backed one (Pentaho), I can say that we would still be using an open-core model if we were using our own money.

His second point:

Open-core is a largely a re-tread of tired, old SMB packaging strategies which have almost universally failed in the market.

This is ridiculous. An open source distribution model, and the adoption of the software by an active community, is nothing like the SMB packaging strategies of the proprietary vendors. The community and enterprise editions are marketed consistently to all companies of every size – there is no different positioning or pricing for the SMB space. Making software available for $0 and converting a small (<1%) set of the community into customers, is very different from taking a huge expensive piece of enterprise software and trying to sell a cut-down version of it for less down-market.

Second point:

I’m not sure most open-core business models have been successful in building large external code contributions

Brian is fixating on code contributions – a common failure. Ignoring the other (in my opinion) 95-99% of all contributions. Take Apache Tomcat – the majority of the code has been written by a small group of people (lets assume 100), but the user base is in the millions. Certainly the software would not exist without those 100 developers, but without the large community of millions, Tomcat would be a hobby for 100 developers, nothing more.

His third point is about the companies that acquire these start-ups:

Who are those likely buyers? Increasingly it appears to be the very same established vendor community that are saying “ya, I do open source too!” So much for a compelling new business model!

When it comes to the M&A decisions and processes of large proprietary software vendors (and I’ve been through a few of these), I doubt whether the issue of ‘open-core’ or ‘pure-open’ would ever come up. It will come down to sales, brand, traction, and community. If ‘pure-play’ open source companies are as successful (or more successful), than the open-core companies, they are just as likely (or more likely) to be acquired by proprietary vendors as open-core ones. I don’t see the relevance of this point.

Recently a couple of analysts have shown confusion about dual-licensing of open source software.

A Matt Aslett (451 Group)  survey shows that he was asking which business model open source companies are using ‘open-core licensing strategy’, ’single open source license’, or ‘dual-licensing’. This make it looks like dual-licensing is an alternative to the other options.

During Brian Prentice’s (Gartner)  session at OSBC last week he inter-changed the terms ‘open-core’ and ‘dual-licensing’ were the same thing.

In reality dual-licensing is not open-core or an alternative to it. Dual-licensing is the provision of the same software under two licenses, an open source one (usually GPL), and a commercial one. Dual-licensing can be applied to open source software regardless whether the company also offers proprietary software or not. So you can have an open-core model with or without dual-licensing. You can also have a pure-play open source model with or without dual-licensing.

Enterprise Data World is the business world’s most comprehensive vendor-neutral educational event about data and information management.I am honored to be invited to speak on Trends in Business Intelligence. Please join me wednesday, March 17th at 11:00 am in San Francisco, USA. Information is crucial to fight challenges but also in trying to identify chances and possibilities. What are the major trends in Business Intelligence and how is it impacting your organization? Capgemini explains how you can make use of data in order to make the necessary decisions under these fast changing circumstances.

My video Interview with Hans Lamboo from DB Magazine on www.biplatform.nl is no longer online.However you can still find the material on youtube.http://www.youtube.com/watch?v=78Wa52Oxa2I

Whatever happened to CPM or Corporate Perfomance Management?  Do you remember when it was on top of the hype cycle? Gartner defined it as “… an umbrella term. It comprises all the relevant methods and all the processes, metrics and systems that enterprises put in place to measure and manage business performance”. Performance Management main objective is to improve the organizations performance and finds its starting point in the corporate strategy. Based on this strategy plans are made, results are measured and evaluated and actions are defined. CPM can even lead to new plans or strategies.

If we take a look at classic Business Intelligence we find that it is focused on using company data to report and analyze the current (or past) performance of the organization (lag). However, the Return-On-Intelligence (ROI) increases as organizations use this data to take a proactive approach in their decision making (lead). A good performance management framework therefore often consists of both lead and lag measures. 

Let’s break down the definition of CPM. Firstly, we take a look at “relevant methods”. This refers to such methods as the Business Balanced Scorecard (Kaplan & Norton), Lean or Six Sigma. Secondly, it’s about “processes”, or in other words a performance improvement process. The third element is “Metrics and systems”. Another, widely accepted, name for metrics can be Key Performance Indicator or KPI. The system to support this would be Datawarehousing & Business Intelligence. But all these afore mentioned elements are used to “measure and manage business performance”. And this means that CPM is focused on delivering a system with set of Key Performance Indicators to measure (by definition) past performance and use this to manage (or plan) future business performance.

So how is this different form the new number one on the hype cycle chart: Analytics? The main difference would be that analytics focuses not so much on measuring and managing but on analyzing data to gain insights and real understanding of (hidden) business performance. It is the next step in the evolution of Information Management. With CPM we can answer such questions as what happened and where (which we really need to know) but with Analytics we begin to understand the why. It will open up the road to not only making better decisions but maybe even the best decision.

http://www.haarlemmermeer.nl/badhoevedorp/Please vote and make photo week 43 your choice so I can win!Thanks.

A new book on BI will be published on January 26th during the Heliview Business Intelligence conference in FIGI ZEIST for which I wrote one chapter (On BI in Retail).

See also  http://bi.heliview.nl/

The title of the book is: Over business intelligence. Data is zilver, informatie is goud (About Business Intelligence. Data is silver, information is gold). 

The book can also be ordered for € 26,50 (excl. € 2,60 postage and packing) on www.uitgeverijtiem.nl.

You can also view the index and some summaries on this website.

The long expected 2nd BI Dutch Event will take place on thursday afternoon (13:00) the 4th of march.The theme for this event will be Self Service BI.Location and agenda will be announced soon. See also: http://nlbi.blogspot.com/2010/01/aankondiging-bi-dutch-jaarcongres.html Register online using LinkedIn:  http://events.linkedin.com/BI-Dutch-jaarcongres-Self-Service/pub/197525

How to mix Business Intelligence and Social Media to enable decision making

Children are constantly learning. First they crawl than they learn how to walk. And the same happens with playing.  At first they start out doing this alone and they find it difficult to share toys with others. They want to keep those fun things themselves.  But in time the children learn that playing together is more fun. And that playing together also means that you have to share your toys with others. This increases the fun and before you know it you have a new best friend. We have an expression in the Netherlands which goes like this: “Playing together is sharing together”. This Dutch rime helps children share their toys with other children while playing. As an adult the world looks pretty much the same. We also find it difficult to share our toys, let alone enterprise data or corporate analyses. However sharing can create enormous advantages and will increase your Return-On-Information. In this article we look at the role of corporate information (or business intelligence) and how social networks (or internet social media like twitter) can help decision making. Perhaps in the near future a manager will publish his or hers corporate data on the net with a $50 reward for the person that comes up with the best solution.

From bearskin suits to intelligent use of information

We used to learn this in school during history lessons. The first inhabitants of the Netherlands were fishers, hunters and collectors. They wore bearskin suits and chased after mammoths. But after a while they started working the land and established small towns and villages. This urbanization allowed them to split their roles and tasks. The cow gave milk to the farmer which he used to buy bread from the baker. The industrial revolution ended these small scale economic (trading) activities. The modern time demanded mass production and oil and steel were the new, o so crucial, cows.

The next big break was the information revolution. The introduction of the computer was just the beginning. Everything bit of information was changed into bits and bytes. Information became the new oil and is the lubricant for today’s economy. Data or information has become an essential production factor just like labor or capital. But the information revolution came to us in waves. First, information was broadcasted by one source for the benefit of many. Think about mass media like radio and television. But the internet has changed this model for ever. The net maybe started out somewhat one dimensional with website after website where all possible interest known to mankind was broadcasted. But soon after that it became possible to react on the sender’s opinion thus creating some kind of dialogue. The next step was sharing information or creating a dialogue between larger groups of congenial persons. Communities were being formed.

The importance of information has been recognized by many.  Information has become an economic factor of production and therefore it is one of the things that enable enterprises to achieve competitive advantages. For example, by knowing trends and other developments before others. It has given rise to a field of expertise where collecting information to improve performance has become important: Business Intelligence (BI). Intelligence, not by coincidence, reminds is of the CIA. An organization that also values information. Only the BI field has been dominated by IT. As a result there is a focus on the technology with regard to information supply instead of using information to increase business opportunities. BI is mainly engaged with opening up enterprise data to get insights in the corporate performance. Social media are also engaged with information collection. Only the platform is not the corporate IT systems but the internet.

Oil is scarce, information is abundant

In time commodities like oil and steel are going to run out. With information it’s exactly the other way round. There is an abundance of information available. From every corner comes data or information. From our systems like cash register, book keeping, HR or the ERP system. But also from external sources like internet. This information can take on different shapes and forms. Sometimes it’s a comment placed in an online forum, or a picture, but it can also be a pdf file containing market analyses or a press release. The challenge is to find the famous needle in the haystack, or rather in football fields filled with haystacks.

A first step is finding or locating data that can be useful, in other words: where is it? The next step will be controlling the data. To make heads and tails of it. For example by capturing it in a model to create insight. The central question here would be: What is it? After that we have to look at the value of the captured information. Is it indeed relevant? Does it have some sort of economic value? What can I do with it? The last step is an interpretation of the data. By giving the data some kind of rating or importance. In other words: what can I do with it?

The World is continuously in motion and freezing an existing situation is therefore not smart as the value of information decreases directly. Above mentioned steps are therefore not suggested as a well managed linearly process. Especially for social media controlling information can only be done up and until the process of interpretation. For classical BI the process might be a bit more linear and controlled but still some phases will melt together, as pictured in the information cloud example below.

 Locate: Where is it?

Business Intelligence (BI) is still very traditional in retrieving or finding information. Most information comes from the organizations own systems. In every organization IT systems are being used for capturing data, such as ERP systems, Cash Registers or HR. This data capture is needed to support the operational process. For example, in HR we need to know what the employees name is, how long he has been working for the company and so on. Also we need to know how much we have sold and to who or where. All this data is being stored in the operational systems. But the data can also change in time or even be deleted. Historically this is why we have built datawarehouses. It allows us to capture all the data in time and create a historical view of facts and dimensions. For BI the datawarehouse is often the information or data source of choice. Its limitation is that it often offers only internal data. The type of data is also often static, meaning that is consists of number, tables. Retrieving information from a datawarehouse can be done with BI tools which build reports or list. Often these reports are developed by the IT department based on user requirements. All and all it takes a lot of steps before information can be retrieved.

Locating data or information on the internet is not necessary anymore. Any which way you use the net, you are constantly surrounded with information, coming from WebPages or our own social network. The internet is like an array of millions of datawarehouses that all can possibly contribute to your information supply. For example, this is the first year that computer generate more data than people do. Every minute, ten hours of video is added to Youtube. Every week over ten billion pieces of content are added to facebook. Service like lazyfeed and DailyPerfect can,  based on some criteria (or topics), retrieve all the information that can be of relevance. They can do this much faster and more elaborate than any human could have done. Data is everywhere when you are on the web. It is on websites, in RSS feeds, in your social network or hidden within a simple Google query. Data is more than available, rich in content, from many sources and better still, easy to find.

Where BI is focused on retrieving structured data from corporate systems or datawarehouses, social media do not need to focus on data retrieval. The information is just there. This makes BI very internally focused and Social Media externally focused. The internet is just an array of datawarehouses filled with an inexhaustible amount of information. But where to find the relevant information? Search engines can help here, just as web services or having a network of trusted people and websites. Both worlds (BI & Social Media) are complementary. By combining BI and Social Media both structured as well as unstructured data is captured. But it is also a combination of internal and external data. The real value is in taking either an inside out approach of better still an outside in approach to enrich already existing information. Finding the right combinations is the final challenge.

Control: What is it?

Controlling the information from datawarehouses is done with a BI tool in the form or reports, list. Sometimes even OLAP cubes are introduced. These are larger datasets where the facts or measures can be analyzed from different dimensions like date, customer or product. It looks a bit like a pivot table in Excel. Modeling information in a datawarehouse is difficult and often a challenge. Reproducing this data in reports or cubes is even more difficult. The biggest challenge here is getting the definitions right and the same across the whole organization. If this is not done everybody can interpret the data the way they want it. That way there will always be multiple versions of the truth. Too many meeting have been spent on discussing the validity of the figures instead of using them. Controlling internal data might remain difficult, but when done correctly offers a multi dimensional view across the entire organization. With this one version of the truth comes a great return on intelligence.

Controlling data is also a big issue for Social Media. There is so much data, that if you want to read it all, this is impossible. So you need to filter and make choices in what you need. This is often done by working with trusted feeds, people and topics. By selecting a number of websites, data feeds or subscriptions to trusted source you can easily control your dataset. You can also use your social network to get information. Some people are part of your network because you trust them or because they can add value. You can also choose to subscribe to certain topics. Services like Google Alert of Lazyfeed scramble the internet for you. This way you can create a controllable dataset, without drowning in the sea of information that is available online. Also by working with trusted feeds, people and topics you know that the information you receive probably hold value for you. It is not just static or a waste of time.

In order to control the information BI uses data models. Data is captured, clarified and distributed using reports or cubes. The information is there, relevant or not. But it does provide one stable version of the truth. BI often models this information in line with the organization operational processes (finance, sales, and logistics). This makes these processes explicit and the supporting data known. The added value of this that you can create insight into your own performance: it is looking inside.

Controlling information in Social Media is impossible. This means that you have to filter. By using trusted feeds, people, sources or topics. It is about limiting the dataset. If it is not relevant it does not exist. Therefore a dynamic version of the truth exists. The social media often model the information based on trust. This is done based on your gut feeling, so often implicit. The added value of this is that you can create insight into the performance of others: it is looking outside.

BI and Social Media are complementary as they allow for looking inside and outside. BI on its own is not enough. The same applies for Social Media. But together they offer a 360 view. Just think about the sales director that explained a decline in sales for Region X based on a newspaper clipping he found on the internet that a megastore was opened by the competition.

 Value: Can I use it?

The gap between business and IT seems to grow every day. This makes sense as each party has its own tasks, responsibilities and interests. But at the same time this causes problems, because as long as business and IT do not work together, the BI competence will face many challenges. A BI Competence Center where business and IT work together and align their activities is still considered to be the best practice. So why does this gap still exist?

IT is focusing on operational excellence. Their aim is to get a minimum of changes, defects or incidents. In their struggle with a declining budget they are constantly striving to reduce the total cost of ownership. But how can we reduce the costs of BI? The simplest solution is reduce the amount of changes. That means less resources and releases. The result of this is a decline in the information supply. On the other hand business is focusing on a concept we call business excellence. They are confronted on a daily basis with many questions and they need answers fast. They need to reduce the amount of uncertainty by increasing the amount of information. If IT cannot supply this to them they will take control themselves. The rise of all kind of (managed) self service reporting is exemplary. Information does not only reduce uncertainty it can also create competitive advantage. By knowing things first the competition is put at a disadvantage. This means that the demand for information will increase.  In conclusion, data supply by IT is down and data demand by business is up. This will lead to higher value of the information. The problem here is how to value this information? Because it is not only the information but also the experiences of people using this information, their skills and attitude. BI can help structure data which leads to explicit information. However its real value is made possible through implicit interactions. Humans add value to the information, for example by making decisions or taking actions. But why limit something a powerful as that to only one person?

 And what is the value of information that you have retrieved from trusted feeds, people and topics. How do you determine which information is important to you? To determine the value is something very personal. What some people consider valuable others might throw away. All information you received is by definition subjective. The information differs in value based on the subject or source you retrieved your data from. Some sources are more valuable than others. This is based on past experiences, known expert knowledge of the sources or because the source was recommended by a trusted person.

 But there are also some semi-objective measures to rate the value of information. Semi-objective because every opinion is a group result and groups needs some conditions to come to a sound conclusion (diversity, independency, specialization and the ability to aggregate various opinions according to James Sorowiecki in his wonderful book Wisdom of the Crowds). Possible way to quantify the value of information is to look how many social bookmarking sites have acknowledged a certain website, how some article are rated on social news sites (like DIGG) or what the Google Pagerank is, or how many people subscribe to a certain RSS feed.

Netflix

Netflix is a company that rents out movies on DVD, Blueray as well as online streaming. Its customers can use a recommendation engine (relevant suggestions based on content) which was improved by using social media. Netflix published a set of anonymous data and ask ‘strangers’ if they could improve their recommendation engine by 1%. The person with the best improvement could earn one million dollar. The final result was an improvement of their engine of 10%. This leads to much higher revenue and the costs of the competition were easily compensated. Also these kinds of improvement if done by their own IT could not have been done or would have cost them more than the 1 million dollar. It was much cheaper to just give the data to the social network.

In conclusion, it can be said that the value of information is increasing and it is the human being that decides the real value. Classical BI has a somewhat carefully, safe way of using data. It source is often trusted (own corporate system or datawarehouse). Social media are completely different. They use a multitude of sources. Also the safety or reliability cannot always be guaranteed. But in order to come to an optimal performance it is not always black or white. Business Intelligence can create a foundation for decision making, it can formulate a thesis, and social media can serve as some kind of standard deviation. They can make or break the thesis, confirm or reject it, but always enrich it. 

This article was written together with Rick Mans - Social Media evangelist at Capgemini

There is a new hype on Twitter: bashing BI trends. This is so much fun. Since everybody is making their BI trends list for next year, let’s all do something else. Let’s find out which trends are real and which are BS or let me rephrase that editorials from the marketing department. The winner of the best bashing of the most repugnant, far away, non inspired BI trend of 2010 will receive many thanks and perhaps a small price.