Why Software Development Requires an Evolution of Consciousness

I was recently sent a link to a set of 3 essays by Jack Reeves dating from 1992. He was a proponent of the idea that software was ALL design, even down to the area of testing, comparing it with traditional engineering processes which had a significant focus on the phase of building something. In software development building it is a miniscule part of the process, now frequently done under automatic control, especially if you are using CI (Continuous Integration) systems. This means that the traditional build phase in software is almost a non-operation (called a no-op, or nop, by techies). Click an icon and – bang – its built.

However there was a little nugget in one of Jack’s essays which caused a rather big “Aha!” moment for me. I recently gave a talk at ACCU2013 in April (slides here) which was based on my research into phenomenology and software, words you don’t usually see together, as well as my interest in the struggle between seeing the whole and the parts, as wonderfully researched by Henri Bortoft. Jack Reeves made a point which connected with these ideas of the whole and the parts, monism and dualism, etc. His point is fairly obvious to anyone who has done any significant amount of software development as he identified that the high-level software design influences the detailed low-level design (to be expected), but ALSO that the detailed design will influence (or should be allowed to influence) the high-level design.

To quote Jack:

“The high level structural design is not a complete software design; it is just a structural framework for the detailed design. We have very limited capabilities for rigorously validating a high level design. The detailed design will ultimately influence (or should be allowed to influence) the high level design at least as much as other factors. Refining all the aspects of a design is a process that should be happening throughout the design cycle. If any aspect of the design is frozen out of the refinement process, it is hardly surprising that the final design will be poor or even unworkable.”

I can hear the programmers among my readers saying: “So what?”, but seen from the perspective of how difficult it is to grasp the relationship between the whole and the parts, I now realise that it is this issue that causes such difficulty when writing software systems and why traditional hierarchical methods of management will not work. This should ring significant alarm bells for anyone wishing to manage a software project because due to the inherent complexity of systems it means that the smallest details down in the low-level design can trip up the whole project effort.

This is why I am so sceptical about large system initiatives managed in the traditional way. One of the latest casualties being the BBC DMI initiative which was abandoned after a project spend of £98 million. Ouch.

This does NOT mean that we cannot make large systems, it just means that we must factor in this tight link between the whole and the parts, the high and low level design. This is hard, even requiring a significant evolution of consciousness if we take on the ideas of Henri Bortoft and the Goethean scientific approach.

So much of our culture sits on assumptions that hierarchical structures work. Indeed a top-down approach is the usual way that I have done projects in the past. But I have to jump to the risky items FIRST, many of which will be low-level design problems that, if not solved, will mean the whole effort is a waste of time. This is why I much prefer attacking problems incrementally, which is a much more organic process and also explains the rise of Agile software development techniques.

In the end it means that as a programmer, if I am to be successful, I must be aware of the way I think and how I perceive the world. Maybe this is why I have recently become interested in watercolour painting – but that is another story…

6 thoughts on “Why Software Development Requires an Evolution of Consciousness

    • Charles, I followed Simon’s link to your slides with his annotations and stopped when I got to the Software: Imagination one. You may or may not know I’m primarily a visual thinker and one aspect of that which helps a great deal in my work is that I visualise the software. Your question, “Or are we thinking in terms of FLOW”, reminded me of something I wrote a couple of years ago:

      “With any new concept I come across, if I can form a mental image I can understand it. On the flip side, if I can’t see it in my mind’s eye I generally can’t get my head around it at all. When programming I have become very adept at seeing the structure of software. It’s not just a static picture but a dynamic visual model. It’s something like not just watching an animation but scripting, drawing and directing it all at the same time. I find this gives me an intuitive grasp of software systems and I tend to have a gut feeling for those aspects that aren’t well built.”

    • Hi Charles – great. I have a lot of articles relating to Henri, Goethe and phenomenology in the Key Articles section. I do have occasional guest writers on Transition Consciousness, so if you do write an article I would be happy to publish it. Also, you may be interested in the Facebook group I run which brings together people interested in Henri’s work: http://www.facebook.com/HenriBortoft

  1. Hi Charles, I wonder if Flow Based Programming (FBP) would be of interest? I have tinkered with this and found that Paul Morrison (who is the discoverer, as he puts it, of FBP) has the energy and attitude that you mention in “Phenomenal Software Development : Key Observations”. FBP offers (to me) the option of working at different levels within a discipline of flow. I find that I can design the overall flow of the logic somewhat separately from the low level components that manipulate detailed data. If only there were more time…
    PS this may somewhat duplicate an attempted reply to your Phenomenal post, which I think got lost.

  2. A system model that could be called hierarchical, but quite different from the systems to which “hierarchical” is usually applied is that of Stafford Beer. His books Brain of the Firm and The Heart of Enterprise go at his Viable System Model from different directions. He describes five necessary levels for a viable system. At the lowest level are multiple viable systems that together form the “body” of the larger system. He had begun work with Salvador Allende to implement these ideas in Chile when Allende was overthrown. Stafford Beer worked in Cybernetics. He introduced me to H. Ross Ashby – in particular, the “law of requisite variety” (roughly, a system needs to be able to handle a requisite variety of circumstances in order to maintain its existence).

Comments welcome. Let me know what you think...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s