Fluctuating Tails II

November 22, 2015

In my last post, “Fluctuating Tails I,” we explored the effects of a black swan on a single normal distribution. In this post, we will look at a the effects of a black swan on a multinormal distribution from the perspective of a linear regression.

Lets start off with the results of a linear regression of multidimensional data. These regressions give rise to ellipse containing the multidimensional data. This data also gives rise to many normal distributions summed into a multinormal distribution.

 Black Swan Imacts on Multivarite Distribution 1

I modified the underlying figure. The thick purple line on the distribution on the p(y) plane represents the first black swan. The thin purple line projects the black swan across the ellipse resulting from the regression. The data to the right of the black swan is lost. The perpendicular brown lines help us project the impacts on to the distribution on the p(x) plane. The black swan would change the shape of the light green ellipse, and it would change the shape of the distribution, shown in orange, on the p(x) plane.

In the next figure, we draw another black swan on the p(y) plane distribution further down the tail. We use a thin black line to represent the second black swan. This black swan has fewer impacts.

Black Swan Imacts on Multivarite Distribution 4

In neither of these figures did I project the black swan onto the p(x) plane distribution, or draw the new x’ and y’ axes as we did in the last post. I’ll do that now.

Black Swan Imacts on Multivarite Distribution 3

Here we have projected the black swan and moved the x and y axes.

Notice that the black swan is asymmetrical, so the means of the new distributions would shift. This means that any hypothesis testing done with the distributions before the black swan would have to be done again. Correlation and strength tests depend on the distance between the means of the hypotheses (distributions).

Parameter Distributions

After drawing these figures, I went looking for Levy flight parameters. I wanted to show how a black swan would affect pumps in a Levy random walk. I settled instead on a Rice distribution.

 Rice Distribution

The shades of blue in the figure are the standard deviations of sigmas from the mean. Sigma is one parameter of the Rice distribution. V is another.

Rice Distribution Parameters

Here are the PDFs and CDFs of a Rice distribution given the relevant parameter values. The blue vertical line through both of the graphs is an arbitrary black swan. Some of the distributions are hardly impacted by the black swan. A particular distribution would be selected by the value for the parameter v. The distributions would have to be redrawn after the black swan to account or the change in the ranges of the distributions. Once redrawn, the means would move if the black swan was asymmetrical. This is the case for the Rice distribution and any normal distributions involved.

If the parameters themselves were distributions, a black swan would eliminate parameter values and the distributions for those parameter values.

When we base decisions on statistical hypothesis testing, we need to deal with the impacts of black swans on those decisions.


Fluctuating Tails

November 13, 2015

On twitter tonight, @tdhopper posted a citation to a journal article on the fluctuating tails of power law distributions. In my last post, I mentioned how black swans moved the tail of a normal distribution. So I took a look at those power law distributions. We’ll talk about that first. Then, I’ll go back and look at tail fluctuations and more for normal distributions.

Power Law Distributions

I drew a two-tailed distribution. This distribution has an axis of symmetry. In the past, I talked about this axis of symmetry as being a managerial control. In the age of content marketing, a question we might ask is what is the ratio of developers to writers. The developers would have a their tail, a frequency of use per UI element histogram, and the writers would have their tail, the SEO page views histogram. Add a feature, a few pages–not just one. So that axis of symmetry becomes a means of expressing a ratio. That ration serves as a goal, or as a funding reality. Adding features or pages would constitute fluctuations in the tails of a power law distribution.

The commoditization of some underlying technology, say the day relational databases died, would result in loss of functionality, content. That would be a black swan. In it’s original sense, financial, against a normal distribution, the losses would be in stock price. In a more AI sense that I’ve written about before, world size, the losses would be in bits.

So I’ve illustrated three cases of fluctuating tails for a power law distribution.

Fluxuations in Power Law Distributions as Change in Angle of Axis of Symmetry

The first power law distribution is A shown in orange. It’s tails have a ration of 1:1. Each tail has the same length. On the figure, the arrowheads represents the point of convergence and provides us with a side of a rectangle representing the size of our world. The point of convergence is represented by a black dot for emphasis.

The second power law distribution is B shown in green. It’s tails have a ration of 2:1, as in x:y. The green arrow give us the right side of our rectangular world. Changing the angle of the axis of symmetry is one way of expressing fluctuation or volatility. The axis of symmetry runs from the origin to the opposing corner of that rectangle.

The third example is C shown in red. This power law distribution has undergone a black swan. The black swan is represented by a black vertical line intersecting the power law distribution B. That point of intersection becomes the new point of convergence for power law distribution C. Notice that this means the black swan effectively moves the x-axis. This makes the world smaller in width and height. The new x-axis is indicated by the red x’ axis on the figure. If this figure were data driven the ratio for the axis of symmetry could be determined. Black swans are another means of expressing fluctuation. Realize that stock price changes act similarly to black swans, so there is daily volatility as to the location of the x-axis.

Normal Distributions

I’ve talked about the normal distributions and black swans in the past. But, this time I found some tools for making accurate normal distributions where I freehanded them in the past. The technology adoption lifecycle is represented by a normal distribution. The truth is that it is at least four different normal distributions and a bowling ally’s worth of Poisson distributions. And, if you take the process back to the origination of a concept in the invisible college you’ll find a Dirac function.

Let’s look at a standard normal, a normal with a mean of zero and a standard deviation of 1, and a normal with a mean of zero and a larger standard deviation. The value of that larger standard deviation was limited by the tool I was using, but the point I wanted to make is still obvious.

Lets just say that the standard normal is the technology adoption lifecycle (TALC). Since I focus on discontinuous innovation, I start with the sale to the B2B early adopter. That sale is a single lane in the bowling ally. That sale can likewise be represented by a Poisson distribution within the  bowling ally. The bowling ally as a whole can be represented as a Poisson game.

The distribution with a larger standard deviation is wider and shorter than the standard normal. That larger standard deviation happens as our organizations grow and we serve an economics of scale. Our margins fall as well. That larger standard deviation is where our startups go once they M&A. Taking a Bayesian view of the two normal, the systems under those distributions are by necessity very different. The larger normal is where F2000 corporations live, and what MBAs are taught to manage. Since VCs get their exits by selling the startup to the acquirer, the VCs look for a management that looks good to the acquirer. They are not looking for good managers of startups.

After drawing the previous figure, I started on the normal undergoing a black swan. With a better tool, I was surprised.

Now a warning, I started this figure thinking about Ito processes beyond Markov processes, and how iteration and recursion played there. Programmers see iteration and recursion as interchangeable. Reading the definitions makes it hard to imagine the difference between the two. The critical difference is where the memory or memories live. Ultimately, there is a convergent sequence, aka there is a tail. The figure is annotated with some of that thinking.

So the figure.

Black Swan 01

I started with a standard normal, shown in dark blue. The gray horizontal line at y=4 is the top of the rectangle representing the size of the world, world n, associated with the standard normal. This is the world before the black swan.

The black swan is shown in red. The new x-axis, x’, runs through the point where the normal intersects with the horizontal line representing the black swan. Notice that the normal is a two-tailed distribution, so the new x-axis cuts the normal at two points. Those points define the points of convergence for a new thinner normal distribution. I drew that normal in by hand, so it’s not at all accurate. The new normal is shown in light blue. The red rectangle represents the new distribution’s world, world n+1.

The new distribution is taller. This is one of the surprises. I know the areas under the two normal equal one, but how many times have you heard that without grasping all of that property’s consequences. Where you can see the new normal in the diagram, what you are looking at is the new learning that’s needed. Again, taking a Bayesian/TALC point of view.

Between the new x-axis and the old x-axis, we have lost corporate memory and financial value.  The width of the new distribution is also thinner than the original distribution. This thinning results from corporate memory loss.

I also annotated some time considerations. This would be TALC related. The black swan happens at some very recent past, which we can consider as happening now. Using the black swan as an anchor for a timeline lets us see how a black swan affects our pasts, and our futures. Those memory losses happen in our tails.

The original x-axis represents, in the AI sense, the boundary between the implicit and explicit knowledge. I know that’s pushing it, but think about it as the line between what we assert and on what we run experiments.

I drew an Dirac distribution on the diagram, but it doesn’t happen at mean or where a normal would be. It is a weak signal. It happens prior to any TALC related Poisson games. Oddly enough, I’ve crossed paths with a Dirac when I asserted a conditional probability.

So here is a Dirac distribution, not normalized, just for the fun of seeing one. This from Wikipedia.


Please leave comments. Thanks.

Flashbulb Pop!

November 2, 2015

I’ve went through more of that math for life sciences book. It’s taking forever and it’s at that point where you want it to end, but there is one more chapter taunting you. The discussion is about how samples tend to the normal, and how the sum of normals is another normal. It sounds straightforward enough. But, there was other reading, other thinking, and a surprise.

I’ve talked about how a distribution defines a world, a rectangular world, and how a black swan chops off the smooth convergence and creates a thick tail. It moves the convergences of that world defining distribution, so you end up with a smaller world. Wall street puts a valuation on those worlds, so smaller means a massively lower price for your stock.

I’ve been reading about random walks and Levy flights. The length of the jump and the direction of the jumps in these things is controlled by several parameters each under their own distribution. So instead of having one distribution we have several. And, that black swan cuts through them as well. If we are making a jump in a Levy flight, and we’re not there yet, that black swan would force us to backtrack and make a different jump. We’d stop drilling in the Arctic. That black swan is operating on our networks.

I’ve also come across the notion of causation. Correlation is not causation. We hear that all the time. But, what is correlation? Correlation is a pair of nodes in a network. Causation is an arc between nodes in a network. The network is a collection of distributions connected by other distributions. This was the lesson of “Wetware.” [I tried to find the link, but I’m not certain at this time.] In biochemistry, we had the Krebs cycle, a nice circular pathway describing metabolism. Alas, the cycle isn’t real. It’s a name we put on a collection of random walks constrained by physical shapes.

Our networks include value chains, and they get cut by our black swan as well. That smaller world that the black swan brings us involves all of our networks, not just the one describing our progress across the technology adoption lifecycle, or across our market populations. What we really have is a multinormal collection of distributions all being sliced at the same time. We can’t make the strategic jumps we intended to make. We can’t continue our daily routine in the tactical space either.

The multinormal distribution is also the best way to think about populations for discontinuous innovation. Innovating for the same and adjacent populations, the populations of our economies of scale is continuous–one of our normals. Discontinuous innovation has us addressing and tuning ourselves to a population beyond our economies of scale, a yet to be discovered population–another normal. Keeping those normals separate is essential, but Christensen couldn’t sell that idea to cost accountants, despite it being the way to creating economic wealth, rather than just capturing more cash. Keeping those normals separate would be essential to our organizations, because our current position in our category’s technology adoption lifecycle is tuned into built organizational structure.

You can’t sell to late market pragmatists and early adopters with the same sales force, or same marketing, or same business model. Is it any wonder that existing companies don’t do discontinuous innovation? Is it any wonder that the typical analysis of an F2000 company doesn’t work for discontinuous innovation? The first assumption would be our experience, our templates, our economies of scale matter. Well, no, and that’s long before you get to the differences in the geometries of the pre-six sigma company and the 42 sigma company. It fundamentally boils down to separate populations and their normals. And, that huge paper slicer we call the black swan. Chop. Opportunities gone in a pop of a flashbulb, in a moment unsuspected, but delusionally well known.


HDR for Product Managers

August 4, 2015

I discovered HDR a while back. It’s been a while. I stopped paying attention to cameras a long time ago. Then, just browsing through the photography section at B&N, I came across my first mention of HDR. I came to a rudimentary understanding of it. I thought it seemed like a data warehouse and I left that thought there to bounce around in my head. So it’s been bouncing ever since.

I bought a book on some photo manipulating software last year. It’s not like I use that kind of software. But, it was something to read on an airplane, but not what you’d call an airplane read. Tonight, I was wondering if I could sell it at Half-Price Books, but no, there was a chapter on HDR, so I have to read that. I have a better understanding of what it is.

What is HDR?

HDR Imaging, high-dynamic range imaging, captures a larger dynamic range than camera w/o it. It captures 32-bits where a normal camera captures 16-bits. In a photograph that translates into a larger tonal differences between white and black. It lets you do the Zone System in your camera rather than in a darkroom. It attempts to see they way an eye sees. A camera takes one image at one point in time. The human eye sees a series of images and adjusts the contrast on the fly as it attends to various locations in the scene. In the mathematical sense, HDR is local to the normal camera’s global. In the ordinary photograph the global swamps the local and details get lost for better or worse.

In the old days you bracketed the exposure and took three shots -1,0,and +1 hoping that when you got back to the darkroom you’d have a shot you could use. HDR automatically takes a much wider sequence of exposure settings, and constructs a photo where all the details were captured and then puts all the pieces back together from the different shots. Back in the old days, one of the shots was going to be the picture, the best one. In HDR, each shot might have something to contribute to the final picture.

So enough already. How does that get me to something a product manager could use?

Frequency of Use Histograms

As product managers (PMs) we might be envious of the product marketing managers (PMMs) histograms that they get from their SEO analytics and log files. They get a long tail. They understand which conversions worked and which ones didn’t work. They know how their content network is pushing prospects, buying roles, customers, and users to their next sale or use. They can optimize their content to their audiences. Product managers could have the same thing.

Each content conversion has a frequency that you get from the analytics that sum up the clicks recorded in the server logs. Notice the clicks. They don’t look like UI elements, but at an abstract level, that exactly what they are. So in the parallel universe of product managers, we get the use frequencies for every control in our UI. Since that isn’t an off-the-shelf thing, it would have to be built in. When a user click on button A, the button makes a request to a server, the server logs the request, serves nothing finishing the request. Then, using the same SEO analytics tools, sum up the requests in various ways over various periods, and that gives you the frequency of use histogram for the use of the controls in a collection of controls inside your application or across a collection of applications. Product marketers and product marketing managers would have analytic equality. They both have their frequency of use histograms.

I’ve written about these frequency of use histograms in other posts.

I wrote a post where I put the PM histogram on the x-axis and the PMM histogram on the y-axis and coordinated them across the axis of symmetry of an exponential curve. But, it must be lost on a prior blog. That axis of symmetry is one point of control. It would determine the length of the long tails of the product document set/touchpoint collection and product controls.

So we have our frequency of use histograms. In the product managers histogram, each bar would represent a single feature or a rollup of feature frequencies in a give use case. The aggregation would depend on the analyst, the product manager.

Data Warehouses

A data warehouse aggregates data in different ways. The summing of a single data item could be represented by a histogram; aggregates another. Aggregates can also be represented by pie charts. In the end, data warehouses contains histograms.

Back to the HDR

In photo editing software a photo is, likewise, represented via histograms. A data warehouse is like an ordinary photo. It represents a firm at one moment in time, one interval. Cameras use exposure setting to define the time interval that becomes a photos one moment in time. HDR captures a sequence of various intervals of a given scene, and aggregates the various components of the scene through a wide range of aggregations or data fusions. A data warehouse has captured all of its data, all of its light. A wide range of aggregations, exposures within a data warehouse would be delivered as a result of different SQL queries later aggregated to show the local objects in a global picture. Integration with the firm’s or customer’s factor analysis might drive the contrasts within the system.

Prospects talk to marketers every time they click. Users talk to product marketers every time they click. Make sure every click in your lean experiments get logged. Listen to what users are saying to you with every click.

A Night at the Bookstore, yes, a Bricks and Mortar Space

July 23, 2015

When I go to the bookstore or a university library, I pick out a stack of books in my areas of interest, and try to scan through them enough to justify taking them off the shelf. I was supposed to finish a particular book, but that didn’t happen. Instead, I spent some time looking through the following at a high level:

  1. EMC^2 (the author), Data Science and Big Data Analytics,
  2. Lea Verou, CSS Secrets, and
  3. Adam Morgan et.al, A Beautiful Constraint.

In Data Science …, I came across a very clear diagram of how power (or significance) gets narrow and taller as sample size increases. Consider each sample to be a unit of time. That leads us to the idea that power arrives over time. These statistics don’t depend on the data. They are about the framing of the underlying studies. The data might change the means and the standard deviations. If the means are narrowly separated, you’re going to need a larger sample size to get the distributions to be narrow enough to be clearly separated, which is the point of the power statistic. Their arrival and departures will change the logic of the various hypotheses. You could under this paradigm see the disruptions of Richard Foster’s Innovation, a book Christensen referenced in his Inventor’s Dilemma before Christensen took an inside-out view of disruption, a view of the scientist/engineer-free innovation, as the arrival of the steeper slopes of the price-performance curve intersections and the departures of same.

As an aside, This week in a twitter linked blog post by a never to be named product manager, I came across the weakest definition of our “all the rage” disruptive innovation, as being akin to a classroom disruption, so far has our vocabulary fallen. No. No. But, it is a buzzword after all. Louder with the buzz please. “I can’t hear you.”

There was also a graph of Centroids (Clusters) that turn out to look like a factor analysis in the sense of steep and long to ever flatter and shorter spans.

There was also a discussion of trees. A branching node in the middle of the tree was called an internal node. I typically divide a tree into its branch nodes and it’s leaf leave nodes. I didn’t read it closely, so the distinction is lost on me.

This book is not an easy elementary statistics book.  I will buy it and take a slow read through it.

In CSS Secrets, there were a lot of things new to me. I did some CSS back in the day, so sprinting through this was interesting. Yes, you can do that now. What? Align text on any path, use embedded SVG. The real shocker was tied to Bezier curves and animation. Various curves in a cubic-Bezier curve showed how to “Ease In;” “Ease In and Out,” which looks like the S-curve of price-performance fame; “Ease Out”; and the familiar “Linear.” The names of the curves could be framed as talking about business results. There were more curves, but there are only a limited number of cubic-Bezier curves. Higher-order curves were not discussed. A cubic-Bezier curve has two end points, and two control points. In the animation sense, the curve feeds values to the animated object. The cubic-Bezier curve is not capable of driving, by itself, full-fledged character animation, but it’s a beginning. We, the computer industry, are easing out of Moore’s law as we speak.

In A Beautiful Constraint, we are looking at a biz book, in the self-help sense. It describes the mindset, method, and motivation for overcoming constraints on one’s performance. We start out as victims. We have to overcome path dependence. We do that with propelling questions and what the author calls Can-If questions. With a Can-If question we are asking about the “How,” sort of the developer’s how, rather than the requirements elicitor’s what. Breaking the path dependency has us asking ourselves or our team about believing it’s possible, knowing where to start, and knowing how much do we want to do it.

An interesting statement was that Moore’s law is actually a path dependence. Intel’s people didn’t let the law break. They always found a way to preserve the “law.” But, Moore’s law was really a sigmoid curve. It flattens at the top. The investment to break the constraint requires much more investment and delivers almost no return, so Intel’s people easing out of it. They like Microsoft will have to find another discontinuous innovation to ride.  The cloud is not such a thing. In fact, the cloud is old and there won’t be a near monopolist in that category. It’s not the next discontinuous innovation. It is really the disappearance, the phobic and non-adopter phases–the phases at the convergence at the end of the category. The device space is that the laggard, yes laggard, but it is still 10x bigger than pre-merger late mainstreet. The normal of Moore’s technology adoption lifecycle is really a sum of a bunch of normals, which leave us unable to see the reality of the category that the discontinuous innovation gave rise to. The end is near.

Anyway, that was tonight’s reading/browsing/carousing. Enjoy.

Geometric Progressions vs Constraints

July 6, 2015

You have a glass of water sitting on the table. You smash the glass. The once organized water now approaches maximum entropy. You throw down a paper towel that absorbs some of that water. That paper towel has organized some of that water. Entropy is lower after that paper towel does its job. That glass acts as a constraint on the behavior of the water the glass contains. That paper towel acts as a constraint on the behavior of the water, as well.

In electronics, this sort of thing is called a ground. If no ground is applied to a light bulb it won’t light up.  Electrons flow. They flow from source to ground, or from ground to source depending on who you are. Yes, there are different perspectives on this. Asking won’t reveal this. But, more to the point, the flow of electrons are organized by constraints.

Graph theory mathematicians talk about completely connected graphs. Add an edge to a large completely connected graph gets you a large number of connections. Notice that this graph is not organized by constraints. In effect, it is ground. The mathematics under the hood is a geometric progression.

This morning I woke up being challenged about some tweet about team size and communications. I didn’t really think about it long enough. A tweet or two touched on the core issues. But, I’ll go into my view here.

Organizations are organized. Organizations are constrained. So when your team size, as a product manager increases by a new staffer, your communications/leadership network does not explode. Every team member doesn’t talk to every other team member. It might be easier if they did, but it doesn’t happen. Yes, you’ll need to talk to that staffer, but you already talk to his boss, and others in the same role.

In one company, all the developers, testers, documentation people had a weekly meeting. The only thing that was really communicated was the ship date. This meeting wasn’t Scrum. While we all attended, and we all reported our status, it was a culture and team thing. The dev team had their own meeting with their lead. The real communications was lead to lead. It wasn’t team to team. If I needed something from dev, I asked the PM to be in the middle. The communications between teams was not nice. But, there was no communications/leadership network explosion. We were organized. We were constrained. We had our subnets.

Companies are organized. This subnet. That subnet. The communications arcs are stable. Being a flatter organization gets rid of some of these constraints, so we explode a bit.

Companies are temporal. Communications isn’t constant. I know I used to work all night, because there were too many interruptions during the day. Talk to me during the day at anytime. But, don’t talk to me at all at night. Those arcs are not persistent. We are not always talking to every edge, every node. We are not always conversing. Sometimes we broadcast. Like when we say what the ship date is.

Companies are cognitive. There is plenty of discussion about the 7+/-2 rule. But, consider that the mean. In different contexts the communicated content will be more or less. PowerPoint insists that we present only three things before we shift it all to long-term memory. DITA and modularized text has similar problems. It takes a lot of planning to divide up the content into nicely related chunks. UIs, task, user stories, face the cognitive limits of the audiences similarly. So when the company communicates to its staff and to its markets, it has to organize the content to the cognitive limits of its recipients.

Channels exist within and without. Those channels structure communications.

Consider that geometric progression to be ground, or maximum entropy. Organizing constraints abound. With the geometric progression we can pretend like we do with Frequentist probabilities that everything is random, but we should realize that the geometric progression doesn’t speak in the face of organization, likewise Frequentist probabilities. Seek the model that accounts for the organization, rather than ground.

Anyway, that’s my response after a few hours. That should be clearer than my immediate response to a tweet. Enjoy. Now back to my world, where the staff is self managed. I could drop dead and the product would still ship on time.

Factor Analysis–What’s Important to your Product

June 29, 2015

Earlier in the week John Cook tweeted something about Coxeter circles, so I clicked the link and was surprised by the following figure. The relationships between the diameters or radii of the circles is the same as what one would expect from a factor analysis. The first factor is the steepest and longest. The next less steep and shorter than the first. Subsequently, each factor is less steep and shorter than the previous factor. The particular angles and lengths will differ, but the subsequent factor will always be less steep and shorter.

Coxeter_circlesThe circle labeled zero is your firm. The circle labeled one would be your category. If you are focused on managing your revenues, your monetization generating your revenues would determine your category. If you are focused on something other than revenues, then place yourself in a category relative to that. The circles labeled two or three, any number above one, would be macroeconomic considerations.

A factor analysis typically covers 80% of your variance with three factors. They would be labelled with negative numbers. The area of a given circle hint at how much variance that factor covers. The factors would, as circles get smaller, or in a line graph get flatter and shorter. The statistical studies of your variance beyond those three factors gets more expensive, so your budget would constrain the number of factors your effort could be managed with. The budget is both monetary and managerial focus driven. The independence of the variables and the complexity of the data fusions giving rise to each factor would impact managerial focus.

The Coxeter circles here represent two levels of macro economic factors, your category, your firm, and your product. For wider product portfolios there would be more circles with negative numbers. Imagining this in three dimensions, as collections of spheres would demonstrate some interesting relationships.

In a firm that stretches across the technology adoption lifecycle (TALC), the factors would migrate in an animation, live and die as Ito memories and oscillate between carrier and carried considerations. In such a firm, the population considerations could be a parallel factor analysis anchored around each population’s relevant product. Economies of scale do not allow expression of the TALC.

Factor analyses need not be firm centric. The economic return on a given set of factors, places a given firm in a given value chain. In a value chain, the larger, aka steeper and longer factors may be outside of your managerial focus. A small factor for your customer would be a very large factor for your company. The key reason to outsource is to preserve managerial focus. When you tell your supplier how to do business, you are not preserving managerial focus. I realize a product manager wouldn’t do this, but when it happens it enters into your matrixed product organization.

Factor Analysis of Value ChainAd serving might be your only monetization, so you need to get and keep eyeballs, and deal with the standardized ad serving infrastructure. Your factor analysis would have holes in it. Your factor analysis would have discontinuities in it. Fast followers would have similar factors, whole product factors, and supplier factors.

In the figure, two whole products are shown: one for web, and another for mobile. One fast follower is shown. A fast follower may compete with you on a single factor. All ad serving monetized businesses might use this supplier.

The arrowheads indicate convergences defining the world size of a given value chain. That is similar to convergences in probability distributions. A factor analysis looks like a power law distribution or a long tail.

Where you have discontinuities in your value chain, you will have to establish well defined interfaces, as well as deciding how soon you would want to follow changes to the definitions of those interfaces.

Ito Processes in the Technolgy Adoption Lifecycle.

June 20, 2015

A Markov process has no (zero) memory. An Ito process has a finite memory. A Markov process is an Ito process with a memory size of n=o. All of that is, for our purposes, talking about history, or more specifically, relevant memory.

In our ordinary conversations about memory or leaning in a firm, the memory is infinite. It is not an Ito process, so it can’t be a Markov process. We talk about brand and design as if they will always be relevant, and have always been so. We talk about a whole host of things this way. But, it is the technology adoption lifecycle that makes everything finite. We try very hard to make the late mainstreet market infinite. Sloan’s invention of management leads us to the infinite firm and the management practices that make the infinite firm. Blue oceans lead us to find another structure for a category after we can’t get anymore infinity from our management practices. These notions of infinity invite us to cut costs until there are no more costs to cut. These notions of infinity kill our companies, and kill our companies fast and faster.

Innovation and management are entirely different. Sloan didn’t innovate, except in his creation of the product he called management. He did not innovate cars. He grew his company through M&As. He consolidated his category. Such consolidations are an indicator that the market leaders have been chosen. Those market leaders get a monopoly or near-monopoly position. Everyone else is stuck in promo spend territory fighting over the scraps. Everyone else is stuck with competing on brand and design, because they have no market power, and no differentiation. This is late mainstreet phase of the technology adoption lifecycle (TALC) out to laggards (devices) phase. The later you are in the TALC, the more you have to spend on brand and design, the more you have to manage your costs and processes.

When we talk about the early mainstreet IT horizontal, geek facing internet of the 90’s as if it didn’t have design, the more we ignore the lessons of the TALC, fit the population you serve. Design is not a characteristic of geek facing products. Design in a characteristic of consumer facing products. The geeks that tried to sell dog food, or any consumer product back in the 90’s, in the early mainstreet market failed. Those same geeks giving something away for free, something technical, something infrastructural, something non-consumer succeeded. We came into the late mainstreet market knowing that free worked, that customer would not pay for anything, that paywalls were wrong, …. We came into the late mainstreet market having learned the wrong lessons. We are finally forgetting those lessons. We are finally learning that consumers pay for stuff.

Alas, we learned the wrong lesson still, when we try to sell something to geeks in the late mainstreet market. No, they will not pay… We are learning the wrong lessons from our success with consumers.

The main problem in crossing the TALC is that the TALC structures our memories. We have finite memories and infinite memories. But, we only have one memory. In my prior discussion of software as media, and in my TALC slideshare, I

So back to this Ito process.

The birth of a category begins by finding the B2B early adopters. Yes, lean does not start there. Lean is late mainstreet. Lean is built on other people’s whole product. It starts well within a category’s life. The birth of a category is 90’s era internet. That’s where today’s whole product came from. Twitter is probably the only such play we’ve had in Web 2.0. Even Google is a subsequent generation in an existing category and a promo spender to boot. And, no, we hear about how B2B needs design these days, sorry, but that is late mainstreet as well. It’s consumer and laggard/phobic facing.

The category is born with a Poisson game, aka a Markov process. These vendors have nothing to leverage and face the process of building tech, product, market, and company all facing the client. Unlike lean, they are stuck with the technology whose adoption they are fosters. Unlike lean, the best practice is to automate the client’s product visualization, not your own. Well, lean lets the users provide the product visualization instead of their own. The point is that n=o, aka we have a Poisson process with no memory. But, we do have a nascent finite memory on our hands. That we intend to repeat this process, we separate our phase processes from our customer facing people. Usually, companies do not do this. For them, the end of their category leaves them without a memory of the discontinuous innovation processes, so they start over again with the disadvantage of the cost issues with trying to use their current late mainstreet process to do what they cannot do, and economies of scale that are devoid of the needed customer base. Memory problems have costs, but accountants can’t tell you how much those problems cost. Memory problems kill innovation. Separation, Christensen’s real original concept, failed to gain traction against the cost accountants.

Christensen build his consulting firm with late mainstreet people who did not provide the early mainstreet effort needed to foster adoption of the separation concept.

So we start with a Markov process. With every capability we build in our consultative product implementation processes, we add to that memory. Call it n=20. Then, we start to build our  vertical market chasm crossing processes, n=21 to n=60. But we partition these two capability collections. We keep our consultative processes going with a brand new discontinuous innovation when the time comes, when the bowling alley ends. Then, we focus on carrier, and build our IT horizontal facing processes, n=61 to n=90. Within the IT horizontal facing organization, we build our tornado capabilities, n=91 to n=100. The tornado capabilities will be harder to retain, but they only work in the tornado and in the post M&A tornado. It is hard to keep them loaded from an HR perspective. Likewise any IPO and further investor relations capabilities, again memory in terms of processes and people. Through it all our Markov process becomes Ito.

At some point we get to our six sigma normal and all things Markov/Ito become Gaussian. Memory becomes infinite. We move from discovery to enforcement, different types of machine learning. Our geometry changes from hyperbolic to Euclidean and subsequently beyond six sigma, to spherical, Euclidean and spherical being safe for management.

Still, there are events that drive us back to earlier memories. Commodification of core technologies make us go back to discontinuous innovation in the midst of our continuous innovation efforts. Mass customization forces us to focus deeply on carried like we did the B2B EA. There will also be processes that we use once and throw away. Before throwing them away, however, you need to think long and hard about reuse and load issues. If you need those people and processes don’t throw them away, and find a way to keep them loaded, rather than letting them dissipate in lateral moves.

Outsourcing is another of those late mainstreet methods for managing managerial focus that lead us to dispose of capabilities and learning, memory, that we may need again. Again, think hard. You can’t get these back after they are gone.

Devices phase leads us to gain a hardware capability beyond the software capabilities we already have. Hardware also drives new software capabilities. More memories, more people, more processes will all be required. Cloud, the phobic phase, similarly.

Like in my post on incommensurate, the water balloons, or balloon poodles model will help here. Where does the memory begin? How large does the girth of this  memory get? How long does it last? Does it produce cash or wealth or loss? What balloons are inside other balloons? What balloons are outside the others? What are the interfaces? The coupling? The cohesion?

Know that you are managing your company’s memory. Learning is good, but it takes us away from our pasts even as it takes us to our future. Learning can prevent us from going back to the parts of our past that we will need again unless we were built to flip, built to exit. Manage memory.



June 15, 2015

Next, I went back and color coded the labeled gaps. In this figure, I’ve put lines at the bridged gaps indicating the use of a new Back in 2009 or so a reader of this blog asked me to define the term incommensurate. I’ll do that again here.

I’ll start with a graph from S. Arbesman’s The Half-Life of Facts. That graph was a surprise to me. It displayed the results of fifty or so experiments about temperature. Some of the experiments intersected with other experiments. Other experiments were parallel to the existing experiments. I’ve drawn a graph here showing the same kinds of things.

BaseThe darker lines are the results of a regression of data contained by the light gray rectangle. Each rectangle represents a single experiment and its replications.

Where the lines intersect, we can call those results commensurate. They result from what Kuhn called normal science. The experiments were designed differently, but reflect a single theory. The measurements within a single experiment reflect a particular apparatus. Changing the apparatus would give you another experiment with potentially different results.

Where the lines don’t intersect, we can call those results incommensurate. I’ll point out the gaps in the next figure. These gaps reveal an inadequacy in the current theory.

This graph can show us all of the experiments at once. But, that covers up things that would be revealed better in an animation. We don’t know, from this graph, when a particular result showed up. If we attended to the temporal aspects of the underlying data, we’d be able to see other gaps. The experiments characterized the gaps across the ranges and domains of several experiments.

Continuities 00ln this figure I’ve highlighted the continuities, the intersections, with red squares. I’ve assumed that all of these intersections exist. The results of one experiment, in the top left, is shown in blue.  I’ve assumed that this experiment was incommensurate and that the experiments that intersect with it did not exist at the time. The experiment that connected it to the chain of experiments to its right happened later.

The experiments shown with red lines are still incommensurate. They exhibit gaps with those experiments to their right. At the bottom right, three experiments exhibit continuity with each other, but exhibit a gap with both the other experiments above and to their right, and the other experiments to their left.

Normal science looks like a well connected network. Extending the range and domain of the existing theory is the job of normal science. A single regression would result in a decreasing function. Where the details differ from that single regression, we have an opportunity for clear functional differentiation.

Each of those commensurate experiments enables continuous innovation that extends the life of a category after the discontinuous innovation gives birth to the category. The technology adoption lifecycle is driven by improvements in a technology’s price-performance curve or S-curve. It is the price-performance curve that delivers on the promises made when the technology was sold and purchased. The demanded performance gets easier and easier to deliver and the range and domains of the underlying experiments expand.

Discontinuities 00In the next figure,  I’ve circled the discontinuities, the gaps, the incommensurate experiments. We won’t pursue experiments to bridge the gaps labeled G and H. We won’t try moving to G, because we can already read that temperature. We might want another way to take that measurement. We could develop a pass-fail thermometer where we are just interested in knowing if we have to make a bigger effort to get a numeric reading. Then, jumping that gap would make sense. The gap H just hasn’t been worked on yet.

Discontinuities Next, I went back and color coded the labeled gaps. The black rectangles show the range and domains involved in bridging a given gap. Bridging a gap requires new theory. The gap at A is from the experiment represented by the  blue line to the experiment on the right. The gap at E can bridge to any of three branches on the right. Any one branch will do. Continuous paths can get you to the other branches. Think step functions. The gap at F actually gaps with a collection of experiments to its right. The gap at B bridges two large subnets. Bridging this gap is critical. The gap at D can bridge to the left or the right. Either will do. Again, paths exist to get to and from the left and right side.

Other parametersIn this figure, I’ve put lines at the bridged gaps indicating the use of a new parameter that enables us to bridge the gaps. These parameters are labeled p and q. Their use was described in a new theory. The dark purple lines demonstrate how a continuous path through the network resolves a branch in resolving the gap.

The gaps E and A were resolved via parameter p and the network flow. The three gaps at F were resolved by parameter p as well. The gap at B was resolved by the solution to the gap at F. The gap at G continues to be ignored. The gap at D and C was resolved via the parameter q and network flows. The gap at H, again, ignored.

In these experiments basic research has showed us where our opportunities lay. It has delivered to us the incommensurate seeds of categories, and the commensurate life blood of new growth (dollars, not populations) to lift us slightly from the swamps of the margins from our nominal operations.

Another Explanation

The simplest explanation of what incommensurate means is that every theory is a water balloon. A theory can only hold so much of what it manages to contain. When you want more than a theory can deliver, when continuous improvements run out, you need a new trick to combine two water balloons. Have fun with that.



Where to Invest?

June 4, 2015

Where to invest was the question. My answer has always been in a company doing discontinuous innovation. But, like most things finding them is the hard part. Most of what we hear about these days is continuous innovations. What we don’t hear about is discontinuous innovation.

Since I’ve worked in startups well before the web came along, my problem has always been finding startups. Truth is that I didn’t find them. They found me. But, living in a startup desert, I’m looking for ways to find them. For a job search, watch your sales tax permit applications. That’s not much help for an investor, and it’s probably way too early. I know from cold calling SIC coded companies that the SIC classification system is very wide. You’ll end up calling a lot of companies that don’t do anything remotely like selling software.

The investor alternative is to find VC funds and put your money in one of them. If you’re going with discontinuous innovation, finding that VC fund will be the issue. I don’t know if VC funds mix discontinuous and continuous innovators together in the same portfolio. I do know that the continuous investments are smaller and get less attention from the VCs. Discontinuous innovations take more time, more money, and more VC attention.

You’ll hear about the continuous innovators and more than likely you won’t hear about the discontinuous innovators. Read the journals in the fields where you expect to invest. Read the SBIRs. Take note of the investigator’s names. Check their bibliographic information. When will one of their students bring the investigator’s technology to the market?

Anyway just a few hints on where to find the discontinuous innovators. Investing in a company that creates a category, and gets the near-monopolistic position is a good place to grow your money. The quick flip of the continuous innovators or the fast followers not so much.

Remember that the technology adoption lifecycle is more than some ordered Markov process transitioning populations. The populations organize the companies serving them. Early phases grow. Late phases decline. We hide that decline in things like cost management and large numbers. Early phases create wealth. Late phases capture cash. Discontinuous innovations begin in the early phases and transition into the late phases. Continuous innovation begin in the late phase and live short lives.


Get every new post delivered to your Inbox.

Join 1,841 other followers