[Following on from my introductory poem, this is the first of a series of posts providing a transcript of my talk at ACCU2016 entitled: “Software Architecture: Living Structure, Art, or Just Hopeful Arrangements of Bytes“. I have modified it to make it read better, cutting out the usual Ums and Errs!]
The impetus for this talk came out of a chat I had with a friend, where I was ranting – as I can do – about code, and then I realized that of course it is easy to rant about other people’s code, so this prompted me to look back at my own experience. I started coding for a living back in 1980 – which doesn’t bear thinking about! – and have spent most of my career implementing high data rate video editing systems. Until recently I worked in a company that does TV and film effects and editing systems, working on a large C++ system of more than 10MLOC. I have now moved into the CAE sector.
This is quite a ‘soft’ talk and I will be following on from some points in the keynote (Balancing Bias in Software Development) by Dr. Marian Petre, although I will drop into some more grounded issues around video player pipeline design and some of the design issues that I have come across.
As I said, the point here was a sense of frustration with the quality of what was getting produced in a commercial context, and frustration in terms of finding people who could make that switch from doing the actual coding and implementation to taking a more structural view. But then as I said I started coding in 1980 and it was not until 1995 where I can say I was actually happy with what I was producing. That is quite a sobering thought. OK, maybe I have the excuse that I did not really get into Object Orientation until 1985/6, and the Dreyfus brothers say it takes 10 years to become an expert in a domain, but even so…
So I want to delve into my own experience and try to understand why this takes so long. This is an issue, not so much about teamwork, but about what we could possibly do individually drawn from my own experiences with being a practitioner with large codebases.
In terms of my inspirations, Christopher Alexander of course is one, and there is one from left field. I got involved in starting a Steiner school for my children back in the 1990s and Steiner’s epistemology, drawn from a foundation coming from Goethe, is actually quite relevant.
I will recap a bit from my talk at ACCU2013 about “Software and Phenomenology”, and my workshop in ACCU2014 about “Imagination in Software Development”, but not too much since I will be taking a slightly different slant on that content.
The Path of the Programmer
I want to start with some reflections on the path of the programmer as I have come to see it, borrowing an idea from Zen about the three phases on the path to enlightenment.
There is a NOVICE phase initially and one of the points about this phase is that you are still learning about the tools you have at your disposal.
A lot of your thinking is going to be Rule Based since you are learning the steps you need to take to do the job. The complexity of your thought is generally going to be less than the problem complexity you are dealing with when you get into ‘live’ industrial work, and hence you are producing brittle code and/or it is not doing all that it needs to do. Here you are aware of your own limits because you know you do not know things, but you are unaware of your own process. I am not here talking about team development process, I am talking about your own personal learning process.
It is thus an undisciplined self-awareness at this level. You have a little self-awareness about your own limits but the lack of knowledge about your learning process means this awareness is undisciplined.
The next phase is what I call the dangerous phase, the JOURNEYMAN phase. It was about 1984 when I was in this phase.
Here you have a better knowledge of tools, having learnt about quite a few of the programming libraries available to you, but the trap here is that the Journeyman is so very enamoured of those tools. This conforms to the upward spike in the confidence curve that Marian talked about this morning.
Here the problem is that you can get into Abstract thinking and this can lead you to having an overly complex view of the solution. Your thinking here is more complex than the problem warrants. It is quite possible that up to 80% of the code will never be used. Therefore you are unaware of your own thinking limits and this can lead to an experience of total panic, especially if you are working on larger systems. [About a quarter of the listeners raised their hand when I asked if anyone had ever experienced this] This conforms to the downward spike that occurs after the upward spike on the confidence curve.
One anecdote I have is the story of one rather over-confident colleague who was given responsibility for a project. The evening before the client was due to turn up for a demo he was still coding away. When I came into work the next morning there was a note on his desk saying ‘I RESIGN’. He had been working through the night and didn’t manage to get to any solution. Of course the contract was lost.
This highlighted the total lack of awareness about one’s own limits. In this phase I also remember having an arrogant positivity – its just software, anything is possible – as I had an undisciplined lack of self-awareness. Some people can stay in this phase for a long time, indeed their whole career. It can be characterized by an insistence on designing and coding to the limit of the complexity of their thinking, which means by definition they will have massive problems debugging it, because you need more complex thinking to debug a system than was used in creating it.
So we have gone here from one undisciplined state of partial self-awareness to another undisciplined state of no self-awareness. Of course this could be seen to be a bit of a caricature but you know if you hit that panic feeling – you are in this phase.
The next phase is the MASTER phase. In the past I have hesitated to call it the Master phase, referring to it instead as the Grumpy Old Programmer phase!
Here we have a good knowledge of tools, but the issue that is different is that you will be using a Context Based thinking. You are looking at the problem you have got in front of you and fitting the tools to that problem. There is a strong link here with a practice when flying aircraft. You need to read from the ground to map, not the other way around. You must do it correctly because there have been a number of accidents where the pilots have read from the map to ground thus misidentifying their location.
It is the same with this. Focus to the problem, bring the tools as you need them. It is interesting what Marian said about how experts can look as though they are novices – which is exactly what I feel like – sometimes I look at my code and think “That doesn’t really look that complicated”. You bring the ‘big guns’ out when you need them, hopefully abstracted down under a good interface. You know you need to keep the complexity down because there will be a lot of maintenance in the future where you or others will have to reason about the code.
So here the software complexity is of the order of the problem complexity, perhaps a bit more because you will need a bit of slack within the solution. At a personal level the big thing here is that you are aware of your own limits because in the previous phase you have been panicked.
One of the big things I have learnt through my career is the ability to handle this. For example there will be a bug. The client may panic. This is to be expected. The salesman may panic. Still possibly to be expected. As a developer if your manager panics too, you have a problem, because the buck will stop with you. Can you discipline your own thinking and your own practice so that you can calmly deal with the issue, regardless of how others are handling the situation? This is the struggle you can get in a commercial coding environment.
Implicit in this description is that you have developed a disciplined personal practice.
So in summary:
- Rule-based thinking
- Some self-awareness.
- Abstract thinking
- No (or very little) self-awareness.
- Contextual thinking
- Deep self-awareness.