The Waterfall Infrastructure Behind Agile

During last Thursday’s Twitter #aopm, Anthropology of Product Management, chat, we discussed how functional cultures would provide better requirements than statistical aggregation. Later that night, it came to me that requirements elicitation and basing it on statistically aggregated populations was so waterfall!

So I wrote down the following list  of waterfall vs. Agile enabling notions:

Requirements/User Stories
Delayed Value Delivery/Frequent, Incremental Value Delivery
Statistical Aggregation/Functional Cultures
Mass Population/Niche Populations
Nash Games/Poisson Games
Complete Information/Adoption
Fixed Population/Population Uncertainty
Normal Distribution/Poisson Distribution
Bayesian Processes/Markov Processes
Gaussian Processes/Hidden Markov Nodes

The items to the left represent the waterfall way. Those on the right a potential agile way.

We talk about user stories, but at the same time we talk about requirements. Well, which is it going to be: waterfall or agile? Clearly, requirements came from software engineering and exist to support the waterfall.

Releases, sure, don’t get me wrong. Agile still delivers a release, but iterations matter. In Feature Driven Design (FDD), an Agile method, you ask how many iterations it will take to release a single unit of minimal marketable functionality. This means that Agile delivers more often. When we move from the waterfall to Agile, we release more often.

A release is where we ship value to the customer. This regardless of whether we are doing the waterfall or agile, but with agile we ship value to the customer more often. Agile reduces our client’s problems with paying for the incremental value delivered. They pay in smaller chunks of money for small chunks of code. They also learn less after the install, so they get more use out of the code. They become productive sooner. We get feedback sooner. The waterfall delays value delivery.

We code for a client or more likely a customer, assumed to represent, the entire customer base. Coding for clients is a necessity for radical/discontinuous innovation. Most of us code sustaining/continuous applicaitons. The customer for the waterfall is someone we’ve never seen. The customer for agile is physical in the sense that it may be the product manager, but its still imaginary, because the product manager has a constructed an aggregate user in their minds. In both cases, the waterfall and agile, the customer is an aggregate of many users going so far as to create the mythical integration app user who negotiated away their meaning. These aggregates are meaningless. Software applications augment thought, or that’s that theory. As such, software applications should be meaningful, which implies that the application respects the meanings of its various users. Meaning is orgranized and defined by a culture. Users are practitioners in a particular functional domain, hence a functional culture. These cultures segment populations into smaller chunks. Functionality for such chunks would be more meaningful, more stable, less risky, unaggregated, and more addressible in the marketing sense. Functional culture is the natural target for Agile.

The statistical aggregation underlying the waterfall addresses a massed population. We use statistics to tell us about that population, but in some self fulfilling way we defined and created the population first, and only afterwards analyzed that population. Sure the regression fits. The error is small. But, the population is still mytical. Agile and functional culture would naturally address a niche population well defined by others, rather than by us. Statistical analysis of a niche population will provide better insight, than the same analysis on a massed waterfall population. Niche populations also define Moore’s custom application engagements and subsequent verticals in the bowling ally with Moore being focused on radical/discontinuous innovation.

For the waterfall to work, you need stable requirements, and a stable population from which to derive those requirements. The division between economic buyer, represented by the executive sponsor, and the users from various functional units further enhanced the myth of a mass population and the existence of a single integration vocabulary. The mythic nature of the mass population and integration vocabulary was clearly demonstrated by requirements volatility. Requirements volatility does not arise from the users changing what they do. It originates in the politics of the executive sponsor’s decision in the face of the political maneuvering of the functional units, who are motivated to protect their meanings and practices.

Agile doesn’t work better in regards to requirements volatility until it addresses a real user from a culturally determined niche population. Agile belongs to the real user, the real population, and their meanings.

In game theory, we learn the Nash game and its equalibriums. Somewhere in there the notion of complete information is mentioned, and quickly forgotten. We do x, they do y, our outcome is s, their outcome is t. But, is it true that we know what their strategic response is going to be? No. So the Nash game is mythic, instructional, but unreal and unreliable. A Poisson game, a game of population uncertainty is a better fit if you consider a population of strategies, or markets subject to adoption of their underlying category. A Nash game has a fixed population of strategies. A Nash game is built on a normal/Bayesian/Gaussian distribution. A Poisson game is built on a collection of successive Poisson distributions, which is a natural representation of Moore’s technology adoption lifecycle. Markets are Poisson.

In parametric statistics, a Normal distribution can be transformed into a Poisson distribution, and vice versa. A warning however is needed here, statistical inferences made on the transformed variates are error prone.

Processes can also be categorized by the same classification scheme I used on the distributions on games. Bayesian/Gaussian processes use all prior probabilities to determine the next state in the process. Markov process use only the information in the current state to determine the next state. Markov processes are bassed on Poisson distributions. So the waterfall is Bayesian/Gaussian, and Agile is Poisson. Again, Moore’s technology adoption lifecycle is a Markov/Poisson process, which fits Agile.

Taking a machine learning perspective, early research into artificial neurons led to hidden Markov nodes as a means of creating a fusion network of switches and weights that could learn. Hidden Markov nodes also discover things that they found on their own without regards of the intentions of the researchers. These days Gaussian processes are used rather than Hidden Markov nodes because the Gaussian approach is easeir. But, for this ease, you have to trade off discovery. Gaussian learning does not discover new aspects. Hidden Markov nodes reflect Agile; Gaussian, the Waterfall.

The Waterfall is closed to the new. Agile is open to the new. Yes, we can do change control on the waterfall and get to the new, but is it the natural outcomes of quicker releases, an interation focus, real users, less aggregation, and meaning rather than math?

We are still being dragged down by the infrastructure that supported Agile. Think about it.

Leave a comment. Thanks.


7 Responses to “The Waterfall Infrastructure Behind Agile”

  1. GregY Says:

    It would be interesting to overlay this comparative analysis with the relationship to the nature of processes being supported by software in question. I suspect that Waterfall, originally rooted in limitations of software development technologies of the time, is also more effective for automation of large enterprise wide, backbone processes, where flexibility of execution and interpretation is not welcome. Certainly these processes are now encased in “ancient” legacy systems, and most “peripheral” applications, often automating departmental/local business processes, are quite well suited for Agile methodology. However even in these cases the agreement to clearly defined process by large group of users could be a haunting and very political, Waterfall-like exercise.

    • davidwlocke Says:

      If you can see software as a media, then we can compare software to a radio. There is carrier, and a carried; means and message. I accept your premise that asserts an ontological/taxonomic difference between the two.

      Your premise screams loudly when one looks at software engineering originating with engineers, so we build from a specification that doesn’t need cultural segmentation, because it is used by the culture that built it. Excellent point.

      Software engineering practice inserted itself into IT systems easily enough, because IT shares in the same culture. IT culture has diverged now and become a subculture of its own. IT business analysts are expected to become functional culture practitioners, but how many of them are ethnographers, how many of them can detect ratcheting up, how many of them understand paradigmatic change.

      It is in the IT applications, rather than infrastructure, where culture as a segmentor becomes critical. Thanks for pointing this out.

      Integration applications exist because of economic indifference and the practice of management as a generic practice. When I posted on SoftwareCEO, it was difficult to get posters to recognize that different businesses require different sets of management practices. This and corporate culture leds to a dismissal of functional cultures giving rise to the executive sponsor and application requirements as politics.

      The engineering community is so sold on requirements volatility as the norm that it has been difficult to point out how the underlying functional cultures in integration applications do not in fact change that much or that quickly. Change is overstated, becuase requirements volatility is a weekly event. Users do not change their minds. Executiver sponsors do.

    • davidwlocke Says:

      Thanks for your comment. I will be pointing out the difference between infrastructural and application software going forward. Important distinction.

      My hope is that developers discover Aspect-Oriented Programing (AOP), and how it can be applied to deliver functional culture mindful applications. Doing so will drive down negative use costs, support costs, and development costs. Executive sponsors and requirements volatility can be a thing of the past.

      If you are interested in product management, you can join our tweet streams via #aopm, #prodmgmt, #pvm. I tweet as DavidWLocke.

  2. davidwlocke Says:

    Several commenters expressed interest, but were somehow classified as spam. When I moderated that spam, their posts inadvertantly disappeared.

    Someone asked if this blog was a hobby. No. But, I’m not happy with the post that appears in the leftmost column. If you know how to fix that, let me know. I moved here from another blog platform. I moved to another, because of the right post problem. I’m back, because I cannot access my last blog platform, nevermind why.

    I post here as a continuation of my product management blogs, and as a place to provide long responses to the product managment tweet streams I participate in. I tweet to promote my blog.

    Thanks to all who have commented on my blog.

  3. davidwlocke Says:

    Now that I tweeted my response, it strikes me that maybe the title of this post wasn’t on point. I was trying to say that these waterfall practices still impede Agile practices. We need to ditch these waterfall practices to realize the full value of Agile.

  4. Cultures in Our Markets « Product Strategist Says:

    […] Product Strategist Just another weblog « The Waterfall Infrastructure Behind Agile […]

  5. How I Lost Thirty Pounds in Thirty Days Says:

    Hi, nice post. I have been pondering this topic,so thanks for sharing. I will certainly be coming back to your posts.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: