Phenomenal Software Development

Over the next few posts I am going to cover the ground of Software and Phenomenology that I dealt with in my recent talks at ACCU2013, ACCU Bristol and ACCU Oxford.

Why Explore Phenomenology?

FlyingViewAs we have progressed through the industrial revolution into our current wide ranging use of information technology, there has been a big change in the form of the tools that we use. The massive impact of this transition from external physical tools to internal virtual tools has largely been unconsciously experienced.

Edgar Dykstra back in 1972 was a notable exception when he gave a talk saying:

“Automatic computers have now been with us for a quarter of a century. They have had a great impact on our society in their capacity of tools, but in that capacity their influence will be but a ripple on the surface of our culture, compared with the much more profound influence they will have in their capacity of intellectual challenge without precedent in the cultural history of mankind.”

Currently our society is heavily based upon the underlying Cartesian dualistic worldview. Along with this orientation we tend to focus primarily on results and though this has been necessary, it has some significant negative consequences. I believe that with the move to virtual tools, the cracks are beginning to show in the Cartesian worldview and its appropriateness for modern times. As computing has progressed along with this has been a questioning of just what it is to be human.

I consider that phenomenology – regardless of whether you can pronounce it or not! – can lead us to a more integrated worldview and I believe the industry needs this more balanced, more human, view if it is to constructively progress.

Overview

I will be starting by providing an overview of my own background. This is important so that you can get a sense of the experiences and thinking that have shaped my conclusions. Only then can you be free to decide what you want to take and what you want to leave.

Then I give some key observations that I’ve made through my career particularly the one about what I call ‘Boundary Crossing’, followed by a short overview of some philosophical ideas. But please note I am not an academic philosopher. Two particular philosophers I highlight are Descartes and Goethe as they represent two realms of thought that I consider relevant in their impact on software development. Notable issues here are: Knowledge Generation, Imagination and the Patterns movement.

I then have some conclusions about how we might progress into the future – both with technology development and technology use.

A Programmer’s Background : Novice – The Early Years

PrimaryI started out being interested in electronics at 17 back in 1974. Originally I was a shy young adolescent nerd who found comfort in the inner world of thought. Also I was not good at dealing with members of the opposite sex, which I believe could be quite a common phenomenon among younger software developers.

Thereafter I gained entry to Southampton University in order to study electronic engineering gaining my degree in 1979. Even at this stage I realized that I wanted to move from hardware development to software development, although I only had an unconscious sense of this physical to virtual transition.

Early programming tasks were a hobby at the time and were based on programming games in BASIC on computers I had built from a kit. There was an initial foray into trying to do an IT records management application which I messed up completely.

Then came the job in the field of media TV and film editing systems where I was definitely feeling that I was working with “cool” tech. Definitely a time of being enticed by the faery glamour of the technological toys.

A Programmer’s Background : Journeyman – The Dangerous Years

DiscusIt was the next phase of the career that I call the dangerous time. A time characterized by the following traits:

  • Wanting to play with more complex and generic structures. (Many of which did not actually get used!)
  • A focus on the tools rather than the problem.
  • The creation of unnecessarily complex systems, letting the internal idea overshadow the external problem context.
  • An arrogance about what could be achieved – soon followed by absolute sheer panic as the system got away from me.
  • No realization that the complexity of thought required to debug a system is higher than that required to originally design and code the system.

This phase of a career can last for a long time and highlights the fact that the programmer needs to become more self-aware in order to progress from this stage. In fact some people never do.

This can be a real problem when recruiting experienced programmers. When interviewing I separate the time into two sections. Initially I ensure that the interviewee has the required level of technical competence, and once I feel they are more settled I move on to see just how self-aware they are.

One question I use here is ‘So tell me about some mistakes?‘ There are two primary indicators that I am looking for in any response. The first one is the pained facial expression as they recall some past mistakes that they have made in their career and how they have improved in the light of those experiences. The second is the use of the word ‘I’.

‘I’ is an important word for me to hear as it indicates an ownership and awareness of the fact that they make mistakes without externalising or projecting it onto other people or the company. This is important because it will show the degree of openness that the interviewee has to seeing their own mistakes, learning from them, and taking feedback. A programmer who cannot take feedback is not someone I would recruit.

A Programmer’s Background : Grumpy Old Programmer

SteinadlerThis ‘Old Grump’ phase is possibly a new one that developers go through before reaching Master level. I hesitate to describe myself as Master but am currently definitely at the Old Grump stage! Traits here I have experienced are:

  • Awareness of the limitations of one’s own thinking, after realizing again and again just how many times one has been wrong in the past. Particularly easy to notice when debugging
  • Realization that maintenance is a priority, leading to a drive to make any solutions as simple and clear and minimalist as possible. Naturally the complexity of the solution will need to match if not exceed the complexity of the problem. Once one has experienced the ease with which it is possible to make mistakes it is always worth spending more time making solutions that are as simple as possible, yet do the job. An Appropriate Minimalism.
  • Code ends up looking like novice code, using complexity and ‘big guns’ when required.
  • A wish to find the true essence of a problem, but when implementing using balanced judgement to choose between perfection and pragmatism.
  • Most people think that because you are more experienced you are able to do more complex work. The paradox is that the reason you do better is that you drop back to a much more simple way of seeing the problem without layering complexity upon complexity. (This strongly correlates with the phenomenological approach)

Next I will be talking about some of the observations I have made throughout my career.
Until next time…

Why Software Development Requires an Evolution of Consciousness

I was recently sent a link to a set of 3 essays by Jack Reeves dating from 1992. He was a proponent of the idea that software was ALL design, even down to the area of testing, comparing it with traditional engineering processes which had a significant focus on the phase of building something. In software development building it is a miniscule part of the process, now frequently done under automatic control, especially if you are using CI (Continuous Integration) systems. This means that the traditional build phase in software is almost a non-operation (called a no-op, or nop, by techies). Click an icon and – bang – its built.

However there was a little nugget in one of Jack’s essays which caused a rather big “Aha!” moment for me. I recently gave a talk at ACCU2013 in April (slides here) which was based on my research into phenomenology and software, words you don’t usually see together, as well as my interest in the struggle between seeing the whole and the parts, as wonderfully researched by Henri Bortoft. Jack Reeves made a point which connected with these ideas of the whole and the parts, monism and dualism, etc. His point is fairly obvious to anyone who has done any significant amount of software development as he identified that the high-level software design influences the detailed low-level design (to be expected), but ALSO that the detailed design will influence (or should be allowed to influence) the high-level design.

To quote Jack:

“The high level structural design is not a complete software design; it is just a structural framework for the detailed design. We have very limited capabilities for rigorously validating a high level design. The detailed design will ultimately influence (or should be allowed to influence) the high level design at least as much as other factors. Refining all the aspects of a design is a process that should be happening throughout the design cycle. If any aspect of the design is frozen out of the refinement process, it is hardly surprising that the final design will be poor or even unworkable.”
http://www.developerdotstar.com/mag/articles/reeves_design.html

I can hear the programmers among my readers saying: “So what?”, but seen from the perspective of how difficult it is to grasp the relationship between the whole and the parts, I now realise that it is this issue that causes such difficulty when writing software systems and why traditional hierarchical methods of management will not work. This should ring significant alarm bells for anyone wishing to manage a software project because due to the inherent complexity of systems it means that the smallest details down in the low-level design can trip up the whole project effort.

This is why I am so sceptical about large system initiatives managed in the traditional way. One of the latest casualties being the BBC DMI initiative which was abandoned after a project spend of £98 million. Ouch.

This does NOT mean that we cannot make large systems, it just means that we must factor in this tight link between the whole and the parts, the high and low level design. This is hard, even requiring a significant evolution of consciousness if we take on the ideas of Henri Bortoft and the Goethean scientific approach.

So much of our culture sits on assumptions that hierarchical structures work. Indeed a top-down approach is the usual way that I have done projects in the past. But I have to jump to the risky items FIRST, many of which will be low-level design problems that, if not solved, will mean the whole effort is a waste of time. This is why I much prefer attacking problems incrementally, which is a much more organic process and also explains the rise of Agile software development techniques.

In the end it means that as a programmer, if I am to be successful, I must be aware of the way I think and how I perceive the world. Maybe this is why I have recently become interested in watercolour painting – but that is another story…

The Importance of being an Amateur

Just recently I have been preparing a talk that I shall be giving at ACCU 2013 in Bristol. Luckily the Bath & Bristol chapter of the ACCU asked me to come and give a dry run of the talk recently and thanks to their many constructive comments I have just finished tweaking and finishing the talk for the main conference.

In preparation for the talk my main texts have been a combination of Henri Bortoft with his “Taking Appearance Seriously”, Iain McGilchrist and his magnum opus “The Master and His Emissary”, and finally- of more import for the techies among us – another magnum opus from Christopher Alexander, his “The Nature of Order” series (which I shall refer to as NoO!).

During the preparation I have been reading these works primarily in “reference” mode, making notes and actually trying to “study” them more. However, now that the main prep is over, I decided to jump forward to the last of the four books from Alexander. So far I had only got to half way through the second one.

Given the slides I had prepared for the talk, some of which included the titles “The Importance of Energy” and “The Foundation in Play”, I was surprised to see just how well they meshed with Alexander’s approach in his Book 4 of the NoO series.

I was particularly struck by his comments about Chartres cathedral and was desperately trying to relate it to software development when a particular thought struck me between the eyes. Although Alexander never mentioned the word, one of the main drivers that the artisans making cathedrals would have used would have been the LOVE of the job, particularly given the religious context so prevalent at that time.

I then reflected upon the background history of software development and realized that it has usually been the polar opposite of this approach, since its main roots are in the military and past war efforts, particularly WW2 and the work at Los Alamos on the atomic bomb. So I then realized that a major reason why I am interested in this more human approach is in order to counteract the lack of humanity that is prevalent in software development, an easy trap to fall into given the focus on technology, and its associated roots in the military.

I then remembered the root within the word amateur – i.e. doing it for the love of it – and realized that this is an important driver for taking the time to make software development truly become a craft and an art. Thus, in order to ‘heal the Cartesian split’, as I mention in the talk, we need to bring more of this feeling of doing it for the love of it, and this is exactly what Christopher Alexander is driving at. He has a great story about getting his students to paint Easter eggs in order for them to learn how to create buildings with good centres or beings as he is also calling them.

I have included the section entitled ‘Innocence’ here as I feel it says a lot about what is needed to truly be an ‘architect’, whether of a building or of software. But unfortunately I am not usually given the time for such exercises and have had to develop this perspective in the background throughout my career. I suspect this is a common experience. But maybe I am just too much of a dreamer…

Extracted from Book 4 of “The Nature of Order : The Luminous Ground” by Christopher Alexander. pp99-100.

12 / INNOCENCE

It may help for me to describe a class I once conducted, in an effort to improve the students’ ability to form buildings from beings. I first asked each student to give an example of an innocent process of drawing or making an ornament which they had most enjoyed. I was looking for something which had been truly joyful for them, not part of their student training. They gave various answers. As I listened, I noticed that the smaller the examples were the more true – that is, the more innocent they were, the less contaminated. Then one student said, in a very soft voice, that he had enjoyed painting Easter eggs in his childhood. That was something that was pure joy, unaffected by guilt, or by a feeling that he must “do well.” At first I could not hear him. He was shy about it, didn’t want to repeat what he said. I persuaded him to speak a little more loudly, and finally we all heard him say, embarrassed, that he had loved painting Easter eggs.

I felt at once that this love, of all those which had been mentioned, was one of the most pure. It was simple. In that work, there is nothing except the egg and the pattern on its surface, no mental constraints of what one “ought” to do – only the thing itself. No one really judges or censors the outcome – so it is easy and alright, not festering with complicated concepts about architecture when you do it.

So I asked each student to make holes in the ends of a raw egg, blow out the yolk and white, and then paint the egg, decorate it like an Easter egg. I made it clear that they did not have to use the fifteen properties. All I wanted them to do was to make the egg beautiful, to enjoy what they were doing. Here are some of the eggs they painted. The shapes and spaces in the ornaments took their shape, and became what they are, just to be beautiful and to have the maker’s depth of feeling visible and shining in them. That was the only principle which governed them. And this, I believe is what one has to do to make a serious work. Naive as it sounds, it is this, too – I believe – that the great traditional builders did.

The students’ other architectural work improved greatly once they understood that making a good building is more like the joyous work of painting an Easter egg than like the practical task of being an “architect.”

People & Technology: The Boundary Problem : Bringing Work Home

At last the mainstream computing world is beginning to catch up with my warnings about unbridled technological use. The latest “Communications of the ACM” has an article entitled “Living in the Digital World” about the effect of gadget use on people’s social behaviour.

I have been a member of the ACM since 1986, having managed to get one of the early super short sexy email addresses ct(at)acm(dot)org so beloved of Unix types. Usually I have been severely unimpressed by most articles from the ACM folk about the existence of any problems with technology use, let alone a balanced view on what those problems might be. Most press has been heavily for technological use, even down into Kindergarten, with what they call the K-12 curriculum.

Oh dear me.

As someone who has programmed computers since the mid 70s I can tell you that coding for this stuff definitely does affect your social skills. I am not your usual uncommunicative nerd type – I like to think that I have quite good social skills – well – as long as you don’t get too close! What I have noticed is that the necessary criticality required to do the “day job” can spill over into your close relationships. This was a primary influence that led to the break-up of my first marriage, although of course as ever there were faults on both sides. But bringing my critical nature home definitely adversely affected my first wife, resulting in her developing allergies galore. My favourite anecdote is that apparently most of these allergies disappeared within a month of me leaving.

Hmmm…

When I found myself doing the same thing again in my second marriage, even I was not stupid enough to think that it was all the other person’s fault. I have now toned down my critical nature when at home and my kids, now of University age, are not backward in coming forward to tell me to “Chillax”. After having realised the problem existed I was extra watchful of myself during their early years and my wife and I have definitely been “good-enough” parents – or so the kids seem to think – honest!

[A problem with parenting is that it is too easy to try so hard to definitely NOT make the same mistakes our own parents did that we “slot-rattle” to the other end of the spectrum and guess what… the effect on our children can be similar to what we wanted to avoid. It seems to be a psychological law.]

So why do I call this issue of technological use The Boundary Problem?

Lets look at a number of places in my own behaviour where I did not have appropriate boundaries:

Without realising it I brought the thinking techniques from work back to the home.
I now know that the highly critical thinking required for software work must be heavily constrained within a close relationship. Of course we need some critical thought, especially if we are parents, but – as a teacher once said to me – there is value in developing a “Nelson’s Eye” and not chasing every little thing. This is easier said than done, especially when it means trying to respond rather than react to a situation that is pressing your buttons!

I was not aware of the effect of programming on my psyche.
This is a biggie and applies to almost every computer professional. In my early years it never crossed my mind that there could be a problem. Soon after the realisation hit me, I went to a computer conference and ran a session to discuss the personal aspects of being a software developer. You should have heard some of the comments! “Navel gazing” was the least abusive one. It is understandable since most technical types like to play with the toys and gadgets. Nowadays things have changed a small amount and with more “techies” you can see the penny starting to drop. I think this is mainly due to what we call “Agile” software development techniques, where you really need to focus on your programming process as well as your technical knowledge. When recruiting programmers the question “Are they aware of how they learn?” is as important as “How good is their technical knowledge?”. If someone cannot take critical feedback it can be very difficult to have them on a software team.

Of course the drive to earn more money just reinforces the “boundaryless” behaviour. You cannot expect companies to control their call on their employees’ time.

Another interesting observation is that when I was younger the gadgetry was much more enticing to me than it is now. I have spent far too many late nights programming computers into the early hours of the next morning to only see the glamorous side. You may see a nice phone. I see just how many hours coding are required to make it work well.

Well that is a small view from the inside of the industry.

In the next post on this topic I will talk about how we as parents dealt with The Boundary Problem at home: A house without computers or TV!