The next frontier for software

Forrester Research is predicting Smart Process Applications market to grow to $34 billion by 2015.

These are software applications designed to support business activities that are people-intensive, highly variable, loosely structured, and subject to frequent change. This next generation of packaged apps will encapsulate current best practices in these collaborative business processes yet make it possible for the app to reflect continuous improvements in how collaboration can occur.

Smart Process Application  is just one sub-set of larger smart application market. The emergence of smart applications is going to revolutionize software industry and define the next set of frontiers in that market.


The connected word is creating more and more opportunities for analytics and knowledge modeling applications.

While most of the attention is given to mobile devices, there are many other areas that are making quite some nice progress.

According to a new study by research firm Analysis Mason Machine-to-machine (M2M) connections will grow to 2.1 billion by 2021, up from roughly 100 million last year.

Some of the newer M2M examples in wired and Wireless sensors and devices connectivity include vending machines, parking meters, home security systems, electric, gas and water meters, and healthcare devices.

Obviously we have already passed the first stage; the connectivity. From now on it is all about making sense and use of available data.

The report provides an example where applied knowledge models have helped cities with 40% boost in their revenue stream by dynamically adjusting parking meter rates based on events and situations!

Of course these days everything is getting labeled under Big Data umbrella!
But in many M2M cases the amount of transferred data is really not that high. So regardless of buzzwords the key focus should be how to create reusable knowledge models for making sense and benefiting from available data!


According to CB Insight VC have invested $1.1B in BigData startups

Following chart shows that while the investments have been slow in Q1 2012, but the number of deals has gone up.

More specifically in Q2 2012 financing and deals have been up 150% and 304%, respectively.

As powerful as predictive analytics can be, it can also sometimes misfire.

I recently read an article on how one of Target’s marketing campaigns caused some funny family discussions by producing false positives on engagement!

And similar stories of how Target knew a teenager was pregnant before her parents did.
(and sending out coupons for baby products!)


There is a fine art of finding the right balance between overfitting and generalization of models.

Nevertheless there is rarely a guaranteed prediction, which highlights the importance of conscious attention to cost-of-error.


BigData Trend


Obviously the BigData trend is growing rapidly. Just a look at job growth in that area will validate that.


Not having even recognized its potentials, people now started talking about a bubble!



To me BigData is an enabler of KnowledgeModeling, which will drive the technological innovations for the years to come. I don’t expect to see any slowdown in that trend for a decade or two.


During MIT's Brains, Minds, and Machines symposium, which started on May 3rd, some of the founders of artificial intelligence and cognitive science tried to explain the lack of progress in AI over the last few decades.

AI pioneer Marvin Minsky started by asking: "Why aren't there any robots that you can send in to fix the Japanese reactors? The answer is that there was a lot of progress in robotics in the 1960s and 70s and then something went wrong.  Today you'll find students excited over robots that play basketball or soccer or dance or make funny faces at you. But they're not making them smarter."

Ultimately the general consensus is that the stagnation is due to decline in funding after Cold War and on early attempts to commercialize AI and because the research is focusing on ever-narrower specialties rather than seeking the bigger questions underneath.


Read more at:


Decision Paralysis

Until not long time ago our challenge was to find the relevant information for decision making. These days our challenge is to cope with information overload in decision making!

Not only the access to information has opened up, but also our universe of available "options" and "selection parameters" have expanded in all possible dimensions.

To provide context, lets first look at the volume of data we are talking about:

  • In entire human history, until 2003, 5 exabytes of information was created. (one exabyte equals 10^18 bytes).
  • During 2006, approximately 161 exabytes of information were created.
  • In 2010 over 900 exabytes of info got created or more than 2 exabytes every Day!! 
    (That includes over 200 billion emails and more than 28,000 hours of YouTube video uploads per day)
  • And in 2020 we expect to create 35,000 exabytes of data, or 35 Zettabyte.
This exponential growth of information will have a profound effect in our life. 
On one hand it will result to game changing advancements in AI and machine learning.   On the other hand it will have negative impact on our decision making abilities.

According to a recent research at Center for Neural Decision Making at Temple University our decision making ability and quality decreases with information overload. People start making stupid mistakes and bad choices.

Along the way subject matter experts are adjusting and learning how to filter noises. But for everybody else decision making is getting harder (a.k.a decision paralysis syndrome!).

This to me is another reason why knowledge modeling is going to become more and more important. Knowledge Models can focus on what matters and learn to filter noise, as experts do.



Watson winning Jeopardy ..


These days we hear a lot about Watson, the IBM AI supercomputer that defeated two greatest Jeopardy players on February 16th 2011.

Watson is in size of 8 refrigerators, a cluster of ninety IBM Power 750 servers with a total of about 3000 processor cores and about 16-terabyte of memory. (For comparison the US Library of Congress contains about 10 terabyte of data)

The large memory is used hold ontological object graph of concepts, entities, properties and their relationships. 


Putting it in Knowledge Modeling context Watson works using a chain of Knowledge Models as followed:

  Diagnostic Model -> Explorative Model -> Analytic Model -> Selective Model


- The Diagnostic Model utilizes Natural Language Processing (NLP) to decipher the Jeopardy question and identify the "problem" (what is being asked for)


- An Explorative Model utilizes the internal memory to present all possible answers, each represented as a Hypothesis.


- The Analytic Model is in charge of evaluating and calculating the confidence level of each Hypothesis on multiple dimensions.


- And finally, the Selective Model ranks the options and chooses the best possible answer based on predefined thresholds and dimensional weightings.


According to the IBM research team Watson is using a dynamic weight setting  that enables it to learn during the game. (For example adjusting its confidence level according to Jeopardy category)


To me, one of the most impressive part of Watson is the massive parallelism that makes a timely response possible.


Apparently the overwhelming victory in Jeopardy resulted in a partnership between IBM and Nuance to apply Watson technology in Healthcare industry, with the first commercial offering planned within 24 months!!



Would you allow a computer to pick movies for you to watch?


Going through Netflix movie list I was thinking about the idea of delegating my selection process to a Knowledge Model.

This is actually similar to Netflix initiative in 2006 for creating a new recommendation engine, however in a more personalized way.
(Read more about Netflix competition with $1 million prize and the winner here: http://www.netflixprize.com//community/viewtopic.php?id=1537)


So I started by monitoring my own behavior during selection process.

In the first iteration I captured all the key attributes that I used when selecting a movie, such as: genres, main Actors/Actresses, year the movie was made, director and user ratings.

Then I started with capturing situation based parameters such as day of the week, my mood and so on.

For example I would normally pick a comedy when I am tired (e.g. during the week) and an action movie during holidays. 

So far so good … I just need to pass those additional parameters to the model.

But then I noticed my selection changes dramatically, when I have company. Not only my own preferences, but also my perception of their personality would impact the selection. For example I wouldn't pick a Mystery movie with someone who can't sit quiet!


Gradually few more variables popped up and the model was getting more complicated, however still manageable.

But ultimately I decided to consider the selection process as part of the entertainment and dropped automating it.

Nevertheless that was a fun exercise …

What is Enterprise Knowledge Bus?

First lets see what is Knowledge good for.

Well, except when you want to show off, there is really no benefit to knowledge unless it is put into action!

Looking back that is where software has been used; to take actions.
But so far we have been focused on optimizing the execution part of action.

As the software industry advanced, we started a paradigm shift by introducing ESB (Enterprise Service Bus) and SOA (Service Oriented architecture) to further optimize execution and facilitate interoperability. 

But by definition action implies also need for knowledge.

Entering 21st century, armed with cheaper hardware, increasing processing power and larger storage capacities now we need to turn our focus to knowledge part of actions.


- What if I could apply the same strategic planning knowledge that I use in Chess gameto product selection or go to market strategy in my business?

- What if there was a way to store all pieces of knowledge available in my organization into a centralized repository where people and systems could pick, pull and chain to make actions more effective and efficient.


This might remind you of Knowledge Management movement of 90s.

However to me the current KM systems are mostly about an "information management system" plus "collection of best practices for processing information". While that is an excellent way of managing artifacts, it is not solving the "direct" needs for knowledge. 

Think of the following scenario:

You have a dispute with your neighbor.

- I provide you with a searchable Federal and State law repository and consider your problem as resolved!

- Or I can ask you few questions and provide you with 3 options ranked based on your preferred time and financial commitment to resolving the issue along with likelihood of outcomes. Upon picking your option, I will provide you with a formal letter to be mail to your local authorities.

In this context I would refer to as "enablement" vs. "empowerment".


With the vast array of information produced and published these days, accessibility will not go a long way.


The Enterprise Knowledge Bus (EKB) is a concept that I have been playing with in my mind for years. The idea was born when I realized that 1) the amount of generated knowledge is growing much faster than one can absorb and 2) we are increasingly outsource out knowledge needs to "trusted" providers.

In a nutshell EKB is a system that not only facilitates access to knowledge related artifacts, but it is also able to apply those to certain situation and contexts to produce actionable outcomes.
In practice it facilitates sharing, reusing and combining knowledge models to be used by systems and rational agents. 


For more details on Enterprise Knowledge Bus check my related paper.