view as multiple pages
by Max Guernsey, III - Managing Member, Hexagon Software LLC
If you haven’t read the
introduction for this channel of articles,
I highly recommend you do so now. For those who just need a brief reminder, the point is this:
In each article, we pick a practice used in traditional, program-centric Agility. We deconstruct that practice until we have
reduced it to its essence: principles and values. Then, we put Humpty-Dumpty together again, with data in mind. The result
should be a new practice which fills the same need as the one with which we started but is applicable to persistent data
In this installment we discuss what I think is one of the most central but least portable practices: evolutionary
software development. Why is this so important? It's the practice that let's designs emerge. Without the ability to change
our design for the better, we have to plan for every kind of variation; if not every variant.
Why does it not work with data? Simple: databases can't evolve.
Why Evolution Works in Software
It's more than just a buzzword we use or some loosely-fitted analogy. The word "evolution" actually very keenly
describes how code changes in the software world. This section covers why. We cover several traits of evolution,
as we understand it in biology, and how those relate to an Agile software product.
Evolution acts on species, not individuals. While we often like to think of our code as "the program," it is not.
Source code is the definition of a species of programs or components. They are like DNA: a plan for how to
build an individual program.
Evolution also functions by way of new generations replacing older ones. When we change the source code, we evolve
the species. Over time the new generation of that species – the new version of our software – supplants its
predecessors. The analogy is very strong so far.
Evolution in biological species is, primarily, driven by environmental factors: new meteorological conditions,
new competitors, changes in available resources... these are all the kinds of things that can accelerate the
evolutionary process. All of these things are true of source code, almost verbatim. Changing environmental
factors lead to new tests which, in turn, drive changes in source code.
Now that we've established that the term "evolution" does apply to software in an Agile environment, let's
consider why it is beneficial to think of changing code as evolving. If code is evolving, it is always getting
better. If code is always getting better, then we don’t have to do a big design up front; instead, we can
design just to the needs of today in a way that allows for change tomorrow. This is a key element of Lean and
Agile software development: emergence.
Why Evolution does not Work with Data Structures
Thinking in terms of evolution is appropriate in an iterative software development environment and adds value.
However, it does not apply to data storage systems and, as a result, such a mindset can actually cost us something.
Let’s go through why:
Whereas evolution drives change within a species by adding new attributes and subtracting useless ones, an individual
database is likely to change the information it contains more in a single week, than all of the feature-related changes
throughout its entire life. In other words, there's something else that drives database change.
We've already seen that the vehicle of evolution is the death of one generation and the birth of another. That applies
to code but not to databases. Individual databases in an agile environment need to be able to survive multiple iterations.
This is especially true of the information they contain.
While evolution’s focus is on the species, we find that individual databases often take center stage. Rarely do we care
if we lose a binary - we can always just rebuild it - but I'm sure that, at some point, you've heard someone say (or been
the one who said) "the database is down!"
While there are lots of reasons why databases and evolution do not match, the real deal-breaker is the fact that databases
are persistent things that need to live for a long time but evolution is a process of constant destruction and renewal.
Databases still have to Change
We started applying the evolutionary model to code as a way of responding to change. Up until that point, change
was considered bad. Now we realize that change is not bad, the inability to respond to change is the problem.
Change is inevitable.
This is the principle of emergence. For a detailed exploration of this principle, I recommend Scott Bain’s upcoming book:
Emergent Design: The Evolutionary Nature of Professional Software Development
Just because we cannot apply the evolutionary analogy to our database development doesn’t mean we can go back to
thinking "change is bad." What we must do, instead, is find another model we can use to react to change.
Data Stores and Metamorphosis
In order to attain any degree of Agility, we need to shift how we think about databases. We can no longer afford to
think of them as things we build and then leave standing like some Roman bridge. Nor can they evolve like our code.
I propose that even though databases do not evolve, they do metamorphose.
Let’s look at how metamorphosis matches up with database development...
Metamorphosis allows individuals to change form and databases are individual objects which need to be able to change
forms. Metamorphosis allows a single entity to gain new attributes and shed old ones and database instances need
the ability to change their design while retaining their identity. Metamorphosis uses the individual as a focal
point and database instances (which are individuals) tend to play a large role in development efforts.
Hopefully, you agree that the word "metamorphosis" is very apt as a description for how an Agile database would need to change.
The value of evolutionary thinking in Lean/Agile software development is the facilitation of emergence. However,
evolution does not allow for database designs to emerge because data cannot evolve in the way we want them to. To permit
emergent design in the database world, we need to adopt metamorphism as the model for change in persistent data structures.
Once we shift our paradigm in that way, we may then be able to produce tools, practices, and processes to support truly agile
Introduction to this Series,
Emergent Design (Bain)