Rethinking Agility in Databases - Part IV: Change

Skip Navigation Links

Rethinking Agility in Databases 3/8/2008 5:46 PM

view as multiple pages

by Max Guernsey, III - Managing Member, Hexagon Software LLC

Introduction

This installment breaks from the trend set by its predecessors a little bit. Instead of digging into a specific practice, we are going to look at a paradigm. The reason for this is that the thinking we are going to examine is one of the underpinnings for most of the Agile practices that we have developed.

We are going to talk about change; the kinds of change we have to address and how we embrace change with Agility and Lean. The concepts herein are linked to Part I (Evolution), but you can probably read either article in any order.

Embracing Change

Change is inevitable. You cannot control it or even deflect it. You can manipulate which kinds of change influence you, to some extent. You also have considerable control over how you prepare for and react to those events. Still, you do not have the ability to outright prevent change. If you do, there is really no reason for you to be reading this, or any other article… at least none of which I can conceive.

Waterfall treats change as a liability. What do we do with liabilities? We try to minimize them… to control them and to prevent them from happening or coming into existence. This does not work. It cannot work because it is based on a precept that stands in direct contradiction to the inevitability of change: the notion that we can prevent or minimize something that we understand to be unstoppable, and that we can plan something that we know to be unpredictable out of existence.

Agility and Lean, have a much healthier outlook on change: “if change is inevitable,” they say, “then maybe we should focus on being change-friendly.” They seek to build organizations that can respond to changes in their environment, rather than ones that try to predict or prevent it.

This concept is called “embracing change.” Does that mean we willingly step in front of the bus? No; of course not… it means that we are wary of busses and, when we sense one approaching, we are ready to jump out of the way. Embracing change is figuring out how to work with change, rather than against it.

How Software Changes

Programs are, primarily, containers for rules. In an ideal environment, these rules originate from customers, business owners, and other stakeholders and pass through a development team into a product.

As a result, there is really only one kind of change that has to be managed: changes to feature-sets. When new requirements emerge, the system has to evolve to meet those needs. When old requirements fade away, developers need to go in and purge the system of all relics.

If you think of the feature-set of a program or component as one extremely complex number, then we can envision these kinds of changes as motion along a single axis. More appropriately, it might be considered motion within a single space for which all of the axes are closely related to each other. I prefer the “axis” abstraction. The figure below shows motion along the feature axis over time.

Features Changing over Time

How Databases Change

Data storage systems are not so lucky. Databases are containers for information and rules. They are, ideally, continuously capturing new data and, as a consequence, being informed. They also store rules pertaining to how that information is organized: schemas, constraints, stored procedures, indexes, etc.

As a result, databases are subject to two kinds of change that we must consider: informative changes and transformative changes.

Informative changes are ones that inform the database and, thus, the system it serves. In essence, all such changes – even deletes – are adding information to a database so long as they are intentional. These should always have been initiated by a customer and there is no reason why a developer should have any part in them… unless they are also users.

Transformative changes are ones that transform a database – changes to the shape of the data that do not add or remove information. These kinds of changes should always be performed by a member of a development team but should have been commissioned by a stakeholder.

If changes to feature-sets can be thought of as motion along one axis, then changes in information might be imagined as movement along a perpendicular axis. The figure below shows motion along the feature and data axes over time.

Features & Data Changing over Time

Embracing Informative Change

Most database programmers already do a good job of embracing informative changes. We just naturally design our databases to capture information correctly. We do this so intuitively that we usually do not consider it as a kind of “change” to be “managed.” More likely, you think of informative change as what the database does. In many respects, this is true.

Because of how well we handle these kinds of changes, we probably don’t need to specifically change how we embrace them. Instead, we need to recognize that, while they are in fact what a database does, they also are part of what a database is. The same way that having read a book becomes inextricably linked with your being: information added to a system becomes a part of that system.

Embracing Transformative Change

Ironically, we don’t generally handle transformation as smoothly as information. Why is this irony? It’s the form of change that we most readily recognize as needing to be embraced by Agility.

Transformation is generally treated as a manual process; a human has to apply the changes, or at least, babysit them. Any process that is unreliable to the point of requiring a human to “make sure it goes smoothly,” is manual regardless of the degree to which it pretends to be automated.

Transformative changes are the sort that we most easily understand as needing to be embraced by Agility because they are the most similar to those we already manage well in the world of software: they are changes to rules. When we change the schema for a database, we are changing a rule for how information is stored. When we change a relationship, we are changing the rule for which data must relate to which other data.

Reconciling Informative and Transformative Change

There are two kinds of changes: informative, which we handle well but do not think about, and transformative, which we think about but do not tend to handle very well. The key to doing a better job managing transformative change may well be to start thinking more about the influence of the informative kind of change.

Where programs change on one axis (features); databases change on two (features & information). This is a much more complicated problem because we still want to keep moving forward and never backward but “forward” takes on a new meaning: we can choose which features to add but not which data are added. Moreover, the data that comes in to our system will almost assuredly end up influencing which features we start adding – especially when it comes to things like performance tuning.

At least early on, when quality is high, we tightly regiment motion on the information axis. We keep our entities well divided, add good relationships, and so on. As quality wanes, so does the control over informative change and, hence, the quality of the information itself.

To keep quality high, we need to find ways to control motion on the second axis (transformation) without impeding movement along the first. We’ve covered some of the practices that might serve this need in other articles but there is still probably a lot to be discovered.

Conclusion

We have to recognize that databases change differently than programs. The reason we haven’t is that the form of change we tend not to think as much about (in Agile terms) is informative change – which we control pretty well already – and the remaining form of change is really not that different from the way that programs change. We need to understand the two major axes of change and make them work together.

No specific practical recommendations are made, here, because this article is about raising awareness of the problem. For specifics on how to start reconciling these two kinds of change, you might look at the rest of this series. If you have ideas of your own, feel free to post them on our discussion group.