Archive for November, 2015

Fluctuating Tails II

November 22, 2015

In my last post, “Fluctuating Tails I,” we explored the effects of a black swan on a single normal distribution. In this post, we will look at a the effects of a black swan on a multinormal distribution from the perspective of a linear regression.

Lets start off with the results of a linear regression of multidimensional data. These regressions give rise to ellipse containing the multidimensional data. This data also gives rise to many normal distributions summed into a multinormal distribution.

 Black Swan Imacts on Multivarite Distribution 1

I modified the underlying figure. The thick purple line on the distribution on the p(y) plane represents the first black swan. The thin purple line projects the black swan across the ellipse resulting from the regression. The data to the right of the black swan is lost. The perpendicular brown lines help us project the impacts on to the distribution on the p(x) plane. The black swan would change the shape of the light green ellipse, and it would change the shape of the distribution, shown in orange, on the p(x) plane.

In the next figure, we draw another black swan on the p(y) plane distribution further down the tail. We use a thin black line to represent the second black swan. This black swan has fewer impacts.

Black Swan Imacts on Multivarite Distribution 4

In neither of these figures did I project the black swan onto the p(x) plane distribution, or draw the new x’ and y’ axes as we did in the last post. I’ll do that now.

Black Swan Imacts on Multivarite Distribution 3

Here we have projected the black swan and moved the x and y axes.

Notice that the black swan is asymmetrical, so the means of the new distributions would shift. This means that any hypothesis testing done with the distributions before the black swan would have to be done again. Correlation and strength tests depend on the distance between the means of the hypotheses (distributions).

Parameter Distributions

After drawing these figures, I went looking for Levy flight parameters. I wanted to show how a black swan would affect pumps in a Levy random walk. I settled instead on a Rice distribution.

 Rice Distribution

The shades of blue in the figure are the standard deviations of sigmas from the mean. Sigma is one parameter of the Rice distribution. V is another.

Rice Distribution Parameters

Here are the PDFs and CDFs of a Rice distribution given the relevant parameter values. The blue vertical line through both of the graphs is an arbitrary black swan. Some of the distributions are hardly impacted by the black swan. A particular distribution would be selected by the value for the parameter v. The distributions would have to be redrawn after the black swan to account or the change in the ranges of the distributions. Once redrawn, the means would move if the black swan was asymmetrical. This is the case for the Rice distribution and any normal distributions involved.

If the parameters themselves were distributions, a black swan would eliminate parameter values and the distributions for those parameter values.

When we base decisions on statistical hypothesis testing, we need to deal with the impacts of black swans on those decisions.


Fluctuating Tails

November 13, 2015

On twitter tonight, @tdhopper posted a citation to a journal article on the fluctuating tails of power law distributions. In my last post, I mentioned how black swans moved the tail of a normal distribution. So I took a look at those power law distributions. We’ll talk about that first. Then, I’ll go back and look at tail fluctuations and more for normal distributions.

Power Law Distributions

I drew a two-tailed distribution. This distribution has an axis of symmetry. In the past, I talked about this axis of symmetry as being a managerial control. In the age of content marketing, a question we might ask is what is the ratio of developers to writers. The developers would have a their tail, a frequency of use per UI element histogram, and the writers would have their tail, the SEO page views histogram. Add a feature, a few pages–not just one. So that axis of symmetry becomes a means of expressing a ratio. That ration serves as a goal, or as a funding reality. Adding features or pages would constitute fluctuations in the tails of a power law distribution.

The commoditization of some underlying technology, say the day relational databases died, would result in loss of functionality, content. That would be a black swan. In it’s original sense, financial, against a normal distribution, the losses would be in stock price. In a more AI sense that I’ve written about before, world size, the losses would be in bits.

So I’ve illustrated three cases of fluctuating tails for a power law distribution.

Fluxuations in Power Law Distributions as Change in Angle of Axis of Symmetry

The first power law distribution is A shown in orange. It’s tails have a ration of 1:1. Each tail has the same length. On the figure, the arrowheads represents the point of convergence and provides us with a side of a rectangle representing the size of our world. The point of convergence is represented by a black dot for emphasis.

The second power law distribution is B shown in green. It’s tails have a ration of 2:1, as in x:y. The green arrow give us the right side of our rectangular world. Changing the angle of the axis of symmetry is one way of expressing fluctuation or volatility. The axis of symmetry runs from the origin to the opposing corner of that rectangle.

The third example is C shown in red. This power law distribution has undergone a black swan. The black swan is represented by a black vertical line intersecting the power law distribution B. That point of intersection becomes the new point of convergence for power law distribution C. Notice that this means the black swan effectively moves the x-axis. This makes the world smaller in width and height. The new x-axis is indicated by the red x’ axis on the figure. If this figure were data driven the ratio for the axis of symmetry could be determined. Black swans are another means of expressing fluctuation. Realize that stock price changes act similarly to black swans, so there is daily volatility as to the location of the x-axis.

Normal Distributions

I’ve talked about the normal distributions and black swans in the past. But, this time I found some tools for making accurate normal distributions where I freehanded them in the past. The technology adoption lifecycle is represented by a normal distribution. The truth is that it is at least four different normal distributions and a bowling ally’s worth of Poisson distributions. And, if you take the process back to the origination of a concept in the invisible college you’ll find a Dirac function.

Let’s look at a standard normal, a normal with a mean of zero and a standard deviation of 1, and a normal with a mean of zero and a larger standard deviation. The value of that larger standard deviation was limited by the tool I was using, but the point I wanted to make is still obvious.

Lets just say that the standard normal is the technology adoption lifecycle (TALC). Since I focus on discontinuous innovation, I start with the sale to the B2B early adopter. That sale is a single lane in the bowling ally. That sale can likewise be represented by a Poisson distribution within the  bowling ally. The bowling ally as a whole can be represented as a Poisson game.

The distribution with a larger standard deviation is wider and shorter than the standard normal. That larger standard deviation happens as our organizations grow and we serve an economics of scale. Our margins fall as well. That larger standard deviation is where our startups go once they M&A. Taking a Bayesian view of the two normal, the systems under those distributions are by necessity very different. The larger normal is where F2000 corporations live, and what MBAs are taught to manage. Since VCs get their exits by selling the startup to the acquirer, the VCs look for a management that looks good to the acquirer. They are not looking for good managers of startups.

After drawing the previous figure, I started on the normal undergoing a black swan. With a better tool, I was surprised.

Now a warning, I started this figure thinking about Ito processes beyond Markov processes, and how iteration and recursion played there. Programmers see iteration and recursion as interchangeable. Reading the definitions makes it hard to imagine the difference between the two. The critical difference is where the memory or memories live. Ultimately, there is a convergent sequence, aka there is a tail. The figure is annotated with some of that thinking.

So the figure.

Black Swan 01

I started with a standard normal, shown in dark blue. The gray horizontal line at y=4 is the top of the rectangle representing the size of the world, world n, associated with the standard normal. This is the world before the black swan.

The black swan is shown in red. The new x-axis, x’, runs through the point where the normal intersects with the horizontal line representing the black swan. Notice that the normal is a two-tailed distribution, so the new x-axis cuts the normal at two points. Those points define the points of convergence for a new thinner normal distribution. I drew that normal in by hand, so it’s not at all accurate. The new normal is shown in light blue. The red rectangle represents the new distribution’s world, world n+1.

The new distribution is taller. This is one of the surprises. I know the areas under the two normal equal one, but how many times have you heard that without grasping all of that property’s consequences. Where you can see the new normal in the diagram, what you are looking at is the new learning that’s needed. Again, taking a Bayesian/TALC point of view.

Between the new x-axis and the old x-axis, we have lost corporate memory and financial value.  The width of the new distribution is also thinner than the original distribution. This thinning results from corporate memory loss.

I also annotated some time considerations. This would be TALC related. The black swan happens at some very recent past, which we can consider as happening now. Using the black swan as an anchor for a timeline lets us see how a black swan affects our pasts, and our futures. Those memory losses happen in our tails.

The original x-axis represents, in the AI sense, the boundary between the implicit and explicit knowledge. I know that’s pushing it, but think about it as the line between what we assert and on what we run experiments.

I drew an Dirac distribution on the diagram, but it doesn’t happen at mean or where a normal would be. It is a weak signal. It happens prior to any TALC related Poisson games. Oddly enough, I’ve crossed paths with a Dirac when I asserted a conditional probability.

So here is a Dirac distribution, not normalized, just for the fun of seeing one. This from Wikipedia.


Please leave comments. Thanks.

Flashbulb Pop!

November 2, 2015

I’ve went through more of that math for life sciences book. It’s taking forever and it’s at that point where you want it to end, but there is one more chapter taunting you. The discussion is about how samples tend to the normal, and how the sum of normals is another normal. It sounds straightforward enough. But, there was other reading, other thinking, and a surprise.

I’ve talked about how a distribution defines a world, a rectangular world, and how a black swan chops off the smooth convergence and creates a thick tail. It moves the convergences of that world defining distribution, so you end up with a smaller world. Wall street puts a valuation on those worlds, so smaller means a massively lower price for your stock.

I’ve been reading about random walks and Levy flights. The length of the jump and the direction of the jumps in these things is controlled by several parameters each under their own distribution. So instead of having one distribution we have several. And, that black swan cuts through them as well. If we are making a jump in a Levy flight, and we’re not there yet, that black swan would force us to backtrack and make a different jump. We’d stop drilling in the Arctic. That black swan is operating on our networks.

I’ve also come across the notion of causation. Correlation is not causation. We hear that all the time. But, what is correlation? Correlation is a pair of nodes in a network. Causation is an arc between nodes in a network. The network is a collection of distributions connected by other distributions. This was the lesson of “Wetware.” [I tried to find the link, but I’m not certain at this time.] In biochemistry, we had the Krebs cycle, a nice circular pathway describing metabolism. Alas, the cycle isn’t real. It’s a name we put on a collection of random walks constrained by physical shapes.

Our networks include value chains, and they get cut by our black swan as well. That smaller world that the black swan brings us involves all of our networks, not just the one describing our progress across the technology adoption lifecycle, or across our market populations. What we really have is a multinormal collection of distributions all being sliced at the same time. We can’t make the strategic jumps we intended to make. We can’t continue our daily routine in the tactical space either.

The multinormal distribution is also the best way to think about populations for discontinuous innovation. Innovating for the same and adjacent populations, the populations of our economies of scale is continuous–one of our normals. Discontinuous innovation has us addressing and tuning ourselves to a population beyond our economies of scale, a yet to be discovered population–another normal. Keeping those normals separate is essential, but Christensen couldn’t sell that idea to cost accountants, despite it being the way to creating economic wealth, rather than just capturing more cash. Keeping those normals separate would be essential to our organizations, because our current position in our category’s technology adoption lifecycle is tuned into built organizational structure.

You can’t sell to late market pragmatists and early adopters with the same sales force, or same marketing, or same business model. Is it any wonder that existing companies don’t do discontinuous innovation? Is it any wonder that the typical analysis of an F2000 company doesn’t work for discontinuous innovation? The first assumption would be our experience, our templates, our economies of scale matter. Well, no, and that’s long before you get to the differences in the geometries of the pre-six sigma company and the 42 sigma company. It fundamentally boils down to separate populations and their normals. And, that huge paper slicer we call the black swan. Chop. Opportunities gone in a pop of a flashbulb, in a moment unsuspected, but delusionally well known.