ACCU2016: Talk on Software Architecture Design 7: Perceiving Organising Principles

Perceiving Organising Principles requires us to develop a living and mobile thinking perception.

Unfortunately, as programmers, we are at a disadvantage.

We work in a domain where a lot of our thinking needs to be fixed into a rule-based structure in order to imagine how a computer program will function. This can have unwanted side effects of making it difficult to think in a mobile, living way. Multi-threaded programming demands this mobile thinking and is why it is so difficult.

At a personal level if we want to develop this other way of seeing we need to engage in some activities that foster that mobile mode of cognition. Perceiving livingness, almost by definition, requires that we need to handle ambiguity. This is what is required when we are working in the ‘gap’, or whenever we are dealing with human situations.

Logical thinking can cope with known and static issues, but as programmers we need to be very aware of the boundaries of our knowledge, more so than the lay person due to the inherent fixity of the domain of computer programming.

Alexander Revisited
At this point it is useful to look at some of Christopher Alexander’s ideas about the perception of beauty and links to what I have been saying about the idea of Cognitive Feeling.

Alexander started with defining a Pattern Language to help foster good architectural design – what he called Living Structure. This metamorphosed into his masterwork, The Nature of Order where he tried to get a better understanding of why we find certain structures beautiful.

In the Nature of Order, Volume 1 Chapter 5, he identified the following 15 properties of Living Structure:

  • Levels Of Scale
  • Strong Centres
  • Boundaries
  • Alternating Repetitions
  • Positive Space
  • Good Shape
  • Local Symmetries
  • Deep Interlock And Ambiguity
  • Contrast
  • Gradients
  • Roughness
  • Echoes
  • The Void
  • Simplicity And Inner Calm
  • Not-Separateness

If you just look at this as a list of items, it can be difficult to understand how these may be useful in design, apart from as heuristic guidelines. Although useful, if we look at them in the light of the dynamic concept of the Organising Principle, they make a lot more sense.

A major point is Alexander’s use of the word: Living. As I point out, this implies ambiguity. Therefore these 15 Properties can be seen instead as Organising Principles and when we try and ‘bring them down’ into a more fixed understanding we will only be seeing one way of looking at each one.

Perceiving the Organising Principle as a Disciplined Artistic Process.
In order to develop a mobile dynamic cognition that can better perceive Organising Principles, my thesis is that we need to take up some artistic pursuit in a disciplined and self-aware way. Do whatever appeals to you. For me I find painting and dance work well.

Lets look at how the practice of these pursuits parallels software development, or indeed any technical effort.

The following image is a watercolour painting of my daughter.

Princess

Freehand painting based on photo of my daughter.

This was one of my first forays into the world of painting and like the good novice artist I was, I decided to draw the picture first, using a photograph as a reference.

It took me 3 hours!

The first effort took 2 hours. The next took 1 hour and the last two each took half an hour, with the final result intended as the basis for the final painting. Being the worried novice that I was I decided to perform a ‘colour check’ painting freehand before doing the final version. In the end this became the final painting I have included here as I found that when I tried to paint into the final drawing it did not have the same life as the freehand painting.

This is an example of the difference between the ‘master’ freehand approach as compared to the ‘journeyman’ drawn approach. Of course I do not consider myself to be a master painter, but this example illustrates the self-developmental dynamic inherent in the artistic process.

We can also see here the need to do the foundational, ‘analytic’ work, in this case the drawing; followed by the ‘gap’ of putting the drawing away and using the freehand skill to come up with the ‘solution idea’.

The following is a painting by Jim Spencer and is for me an example of how less is more and illustrates how such minimalism is an essential aspect of any mastery. In this case Jim began learning art just after the second world war. (Also see my post Minimalist Mastery)

RedSkyAtNightSmall

The third example of and artistic pursuit is that of dance, in this case Argentine Tango. This particular dance is a form strongly founded on being far more conscious about what is a primary human activity: walking. (See my post on Dance as True Movement)

Here there is a need for structure, and a mobile process of interpretation and improvisation, both founded on a disciplined form of the dance. It can take years to learn how to ‘walk’ again but if followed in a disciplined manner can lead to sublime experiences of ‘Living Structure’ as the ‘team’ of two people dance from a common centre of balance.

In conclusion I hope you have been able to see the implicit link between Art and Technology and the value of balancing ourselves up as human beings.

Thank you for your attention.

In response to my statement about dancing John Lakos (author of Large Scale C++ Software Design) asked for some tango teaching at the end of the talk! The picture was taken by Mogens Hansen.

CharlesAndJohnTangoSmall

ACCU2016: Talk on Software Architecture Design 6: Organising Principles

ACCU2016: Talk on Software Architecture Design 5: Active Design Ideas

In the last post I highlighted some specific design problems and associated solutions. Now I want to look at these solutions a little more deeply.

To refresh our memory the solutions were as follows:

  1. Separating Mutex Concerns.
  2. Sequential Resource Allocation.
  3. Global Command Identification.

I want to characterise these differently because these names sound a little like pattern titles. Although we as a software community have had success using the idea of patterns I think we have fixed the concept rather more than Christopher Alexander may have intended.

I want to rename the solutions as shown below in order to expressly highlight their dynamic behavioural aspect:

  1. Access Separation.
  2. Sequential Allocation.
  3. Operation Filtering.

You might have noticed in the third example the original concept of “Global Command Identification” represents just one possible way to implement the dynamic issue of filtering operations. Something it has in common with much of the published design pattern work where specific example solutions are mentioned. To me design patterns represent a more fixed idea that is closer to the actual implementation.

Others may come up with a better renaming, but I am just trying to get to a more mobile and dynamic definition of the solutions. Looking at the issues in this light starts to get to the core of the issue of why it is so hard to develop an architectural awareness.

If you can truly understand, or ‘grok‘, the core concept of this characterisation, regardless of the actual words, you will see that they do not really represent design patterns – not in the way we have them at the moment.

This is where there is a difference between the architecture of buildings – where design patterns originated – and the architecture of software. Although both deal with the design of fixed constructs, whether it be the building or the code, the programmer has to worry far more about the dynamic behaviour of the fixed construct (their code). Yes – a building architect does have to worry about the dynamic behaviour of people inhabiting their design, but software is an innately active artefact.

Let me recap the debugging and design fixing process in terms of the following actions that are carried out in order:

1: Delicately Empirically Collect the Data.
Here we have to be very aware of the boundaries of our knowledge and collect information in a way that does not disturb the phenomenon we are looking at. Awareness of our own thinking process is vital here.

2: Imagine into the Problem Behaviour.
We have to imagine ourselves into the current behaviour that the system is exhibiting. (This is the hard bit when you are under pressure and is what requires a strong focus in order to understand what the existing design is doing)

3: Imagine into the Required Behaviour.
We need to imagine into what the required behaviour of the system NEEDS to be and it is here that we start to meet the ‘gap’ between problem and solution. It may indeed only need a one line fix, but quite likely there is a deeper design problem. Again here is a point where our self-awareness is important. Do we have the discipline to make ourselves stop and think more carefully and widely about the presenting problem?

4: THE GAP. Cognitively Feeling for the best Solution Concept.
In this stage there is a very fine “Cognitive Feeling” in action to decide what is a good fit to the problem. For the experienced programmer this is more than just a question of “Does this solution fit the requirement?”

There is the consideration of whether the proposed solution idea is going to be a sustainable fix during the future lifetime of the project.

This question is much like asking myself if I will still find
this painting beautiful in 10 years time.

YachtClubSmall

There is a current widely held belief that the best procedure for coming up with a design solution is to produce many possible alternatives and evaluate them in order to choose the best one. In practice I have found that this very rarely – if ever – happens.

I usually arrive at a single design solution by trying out the multiplicity of possible solutions while in the ‘gap’ where I am considering various alternatives – imagining each of them in operation, possibly ‘drawing’ the thoughts out on a whiteboard as I think.

In this part of the process the more experienced programmer will slow things down to the extent of even putting in a provisional simple solution that gives them some breathing, or thinking, space. This is the idea of provisionality mentioned by Marian Petre, because this mode of design thinking requires time and reduced pressure.

It is amazing how often this happens in the shower!

Of course this is predicated on the fact that I have done the required detailed groundwork, but as I mentioned in the poem, our logical thinking can only take us to the boundary of what we know. Trying to push to go faster results in inadequate and buggy designs that are based on immature thinking.

This is the central conundrum of software development. The more we dive down into detailed analysis, the more we encounter these ‘softer’, heuristic elements.

5: Implementation.
Finally we get to the implementation. As you will have seen it is far too easy to jump into “premature implementation”. It is hard, if not impossible, to teach people just how small a part the coding is of the whole process. It needs to be experienced. Until you have seen how a good design triggers an amazing collapse in code complexity, the importance of taking the time to search for that great design is not an obvious conclusion. This is a fundamental eye of the needle that all programmers need to go through.

This is the main reason I like programming:

I get less code.
I get something I can reason about.
I get something that does the job!

Beautiful!

In the next post I am going to show how the dynamic design solution ideas and the human analysis process link to what I will call the “Organising Principle”, a term I have borrowed from Rudolf Steiner’s lexicon.

ACCU2016: Talk on Software Architecture Design 6: Organising Principles
ACCU2016: Talk on Software Architecture Design 4: A Design Example

ACCU2016: Talk on Software Architecture Design 4: A Design Example

[The following transcript is more for the techies of my readership. For those of a less technical inclination, feel free to wait for the next post on “Active Design Ideas” which I have separated out due to the length of this post.]

I am now going to ground this discussion into an example software architecture by considering some design problems that I have experienced in designing a multi-threaded video player pipeline. The issues I highlight would generally apply to many video player designs.

The following image is a highly simplified top-level schematic, the original was just an A4 pdf printed from a whiteboard which I find much better than trying to work out designs using a computer-based UML drawing tool. The gross motor movement of hand drawing “in the large” seems to help the thinking process.

Player

There are 3 basic usual commands for controlling any video player that has random access along a video timeline:

  • Show a frame
  • Play
  • Stop

In this example there is a main controller thread that handles the commands and controlling the whole pipeline. I am going to conveniently ignore the hard problem of actually reading anything off a disk fast enough to keep a high resolution high frame-rate player fed with data!

The first operation for the pipeline to do is to render the display frames in a parallel manner. The results of these parallel operations, since they will likely be produced out of order, need to be made into an ordered image stream that can then be buffered ahead to cope with any operating system latencies. The buffered images are then transferred into an output video card, which has only a relatively small amount of video frame storage. This of course needs to be modeled in the software so that (a) you know when the card is full; and (b) you know when to switch the right frame to the output without producing nasty image tearing artefacts.

These are all standard elements you will get with many video player designs, but I want to highlight three design issues that I experienced in order to get an understanding of what I will later term an “Organising Principle”.

First there was slow operation resulting in non real-time playout. Second, occasionally you would get hanging playout or stuttering frames. Third, you could very occasionally get frame jitter on stopping.

Slow operation
Given what I said about Goethe and his concept of Delicate Empiricism, the very first thing to do was to reproduce the problem and collect data, i.e. measure the phenomenon WITHOUT jumping to conclusions. In this case it required the development of logging instrumentation software within the system – implemented in a way that did not disturb the real-time operation.

With this problem I initially found that the image processing threads were taking too long, though the processes were doing their job in time once they had their data. So it was slowing down BEFORE they could get to start their processing.

The processing relied on some fairly large processing control structures that were built from some controlling metadata. Since this build process could take some time these structures were cached with their access keyed by that metadata, which was a much smaller structure. Accessing this cache could occasionally take a long time and would give slow operation, seemingly of the image processing threads. This cache only had only one mutex in its original design and this mutex was taken both for accessing the cache key and for building the data structure item. Thus when thread A was reading the cache to get at an already built data item, it would occasionally block behind thread B which was building a new data item. The single mutex was getting locked for too long while thread B built the new item and put it into the cache.

So now I knew exactly where the problem was. Notice the difference between the original assumption of the problem being with the image processing, rather than with the cache access.

It would have been all too easy to jump to an erroneous conclusion, especially prevalent in the Journeyman phase, and go in and change what was thought to be the problem. Although such a change would not actually fix the real problem, it could have changed the behaviour and timing so that the problem may not present itself, thus looking like it was fixed. But then it might resurface 3 to 6 months later, a costly and damaging process for any business.

In this case the solution here was to have finer grained mutexes: one for the key access into the cache and a separate one for accessing the data item, which was then lazily built on first access.

Hanging Playout or Stuttering Frames
The second case was either hanging or stuttering playout. This is a great example because it illustrates a principle that we need to learn when dealing with any streamed playout system.

The measurement in this case was extremely ‘old school’, simply by printing data to a log output file, although of course only a few chars per frame, because at 60fps (a typical modern frame-rate) you only have 16ms per frame.

In this case what was happening was that the streaming at the output end of the pipeline was getting out of order. Depending upon how the implementation was done, it would either lock the whole player or get a stuttered playout. Finding the cause of this took a lot of analysis of the output logs and many changes to what was being logged.

What I found was that there was an extra ‘hidden’ thread added within the output card handling in order to thread off some of the other pre-processing that needed to happen, BUT there was no enforcement of frame streaming order. This would mean that the (relatively) small amount of memory in the output card could get fully allocated and there would be a gap in the ordering. Thus it was not possible to fill that gap in the frame order with the correct frame when it eventually came along, because there was no room in the output card to put that frame. Thus usually resulting in a playout hang.

MindTheGapCropped

This is why, with a streaming pipeline where you always have limited resources at some level, allocation of those resources MUST be done in streaming order. This is a dynamic principle that can take a lot of experience to learn.

The usual Journeyman approach to such a problem is just to add more memory, i.e. more resource! This will just hide the problem because processing will still be done out of order, but because you have increased the spare capacity it will not go wrong until you next modify the system to use more resource. At this point the following statement is usually made:

“But this has been working ok for years!”

One of the things I am always saying to less experienced programmers when trying to debug such problems is:

“Do not change any of the existing functionality.
Disturb the system as little as possible.
Keep the bug reproducible so you
can measure what is happening.
Then you will truly know when you have fixed the fault.”

Frame Jitter on Stop
The third case was one of frame jitter when stopping playout. The problem was that although the various buffers would get cleared, there could still be some frames “in flight” in the handover threads. This is a classic multi-threading problem and one that needs careful thought.

In this case when it came time to show the frame at the current position, an existing playout had to be stopped and the correct frame would need to be processed for output. This correct frame for the current position would make its way through to the end of the pipeline, but could get queued behind a remnant frame from the playout. This remnant frame would most likely have been ahead of the stop position because of the pre-buffering that needed to take place. Then when it came time to re-enable the output frame viewing in order to show the correct frame, both frames would get displayed, the playout remnant one first. This would then manifest the frame jitter.

One likely fix of an inexperienced programmer would be to make the system sit around waiting for seconds while the buffers were cleared and possibly cleared again, just in case! (The truly awful “sleep” fix.) This is one of those cases where, again due to lack of deep analysis, a defensive programming strategy is used to try and force a fix of what initially seems to be the problem. Again, it is quite likely that this may SEEM to fix the problem, and is likely to happen if the person is under pressure.

The final solution to this particular problem was to use the concept of unique “command ids”. Thus each command from the controlling thread, whether it was a play request or a show frame request, would get a unique id. This id was then tagged on to each frame as it was passed through the pipeline. Then by using a globally accessible “valid command id set” the various parts of the pipeline could decide if they had a valid frame that could be allowed through, or could be quietly ignored.

When stopping the play all that had to be done was to clear the buffers, remove the relevant id from the “valid id set” and this would allow any pesky remaining “in flight” frames to be ignored since they had an invalid id. This changed the stop behaviour from being an occasional, yet persistent bug, into a completely reliable operation.

In the next post I will recap the above human process of finding and fixing the problems.

ACCU2016: Talk on Software Architecture Design 5: Active Design Ideas
ACCU2016: Talk on Software Architecture Design 3: The Issue of Doubt

ACCU2016: Talk on Software Architecture Design 1: The Path of the Programmer

[Following on from my introductory poem, this is the first of a series of posts providing a transcript of my talk at ACCU2016 entitled: “Software Architecture: Living Structure, Art, or Just Hopeful Arrangements of Bytes“. I have modified it to make it read better, cutting out the usual Ums and Errs!]

Introduction
The impetus for this talk came out of a chat I had with a friend, where I was ranting – as I can do – about code, and then I realized that of course it is easy to rant about other people’s code, so this prompted me to look back at my own experience. I started coding for a living back in 1980 – which doesn’t bear thinking about! – and have spent most of my career implementing high data rate video editing systems. Until recently I worked in a company that does TV and film effects and editing systems, working on a large C++ system of more than 10MLOC. I have now moved into the CAE sector.

This is quite a ‘soft’ talk and I will be following on from some points in the keynote (Balancing Bias in Software Development) by Dr. Marian Petre, although I will drop into some more grounded issues around video player pipeline design and some of the design issues that I have come across.

As I said, the point here was a sense of frustration with the quality of what was getting produced in a commercial context, and frustration in terms of finding people who could make that switch from doing the actual coding and implementation to taking a more structural view. But then as I said I started coding in 1980 and it was not until 1995 where I can say I was actually happy with what I was producing. That is quite a sobering thought. OK, maybe I have the excuse that I did not really get into Object Orientation until 1985/6, and the Dreyfus brothers say it takes 10 years to become an expert in a domain, but even so…

So I want to delve into my own experience and try to understand why this takes so long. This is an issue, not so much about teamwork, but about what we could possibly do individually drawn from my own experiences with being a practitioner with large codebases.

In terms of my inspirations, Christopher Alexander of course is one, and there is one from left field. I got involved in starting a Steiner school for my children back in the 1990s and Steiner’s epistemology, drawn from a foundation coming from Goethe, is actually quite relevant.

ProgrammersPath

I will recap a bit from my talk at ACCU2013 about “Software and Phenomenology”, and my workshop in ACCU2014 about “Imagination in Software Development”, but not too much since I will be taking a slightly different slant on that content.

The Path of the Programmer

I want to start with some reflections on the path of the programmer as I have come to see it, borrowing an idea from Zen about the three phases on the path to enlightenment.

There is a NOVICE phase initially and one of the points about this phase is that you are still learning about the tools you have at your disposal.

Primary

A lot of your thinking is going to be Rule Based since you are learning the steps you need to take to do the job. The complexity of your thought is generally going to be less than the problem complexity you are dealing with when you get into ‘live’ industrial work, and hence you are producing brittle code and/or it is not doing all that it needs to do. Here you are aware of your own limits because you know you do not know things, but you are unaware of your own process. I am not here talking about team development process, I am talking about your own personal learning process.

It is thus an undisciplined self-awareness at this level. You have a little self-awareness about your own limits but the lack of knowledge about your learning process means this awareness is undisciplined.

The next phase is what I call the dangerous phase, the JOURNEYMAN phase. It was about 1984 when I was in this phase.

Discus

Here you have a better knowledge of tools, having learnt about quite a few of the programming libraries available to you, but the trap here is that the Journeyman is so very enamoured of those tools. This conforms to the upward spike in the confidence curve that Marian talked about this morning.

Here the problem is that you can get into Abstract thinking and this can lead you to having an overly complex view of the solution. Your thinking here is more complex than the problem warrants. It is quite possible that up to 80% of the code will never be used. Therefore you are unaware of your own thinking limits and this can lead to an experience of total panic, especially if you are working on larger systems. [About a quarter of the listeners raised their hand when I asked if anyone had ever experienced this] This conforms to the downward spike that occurs after the upward spike on the confidence curve.

One anecdote I have is the story of one rather over-confident colleague who was given responsibility for a project. The evening before the client was due to turn up for a demo he was still coding away. When I came into work the next morning there was a note on his desk saying ‘I RESIGN’. He had been working through the night and didn’t manage to get to any solution. Of course the contract was lost.

This highlighted the total lack of awareness about one’s own limits. In this phase I also remember having an arrogant positivity – its just software, anything is possible – as I had an undisciplined lack of self-awareness. Some people can stay in this phase for a long time, indeed their whole career. It can be characterized by an insistence on designing and coding to the limit of the complexity of their thinking, which means by definition they will have massive problems debugging it, because you need more complex thinking to debug a system than was used in creating it.

So we have gone here from one undisciplined state of partial self-awareness to another undisciplined state of no self-awareness. Of course this could be seen to be a bit of a caricature but you know if you hit that panic feeling – you are in this phase.

The next phase is the MASTER phase. In the past I have hesitated to call it the Master phase, referring to it instead as the Grumpy Old Programmer phase!

PetrelLand_4x3_2kcropped

Here we have a good knowledge of tools, but the issue that is different is that you will be using a Context Based thinking. You are looking at the problem you have got in front of you and fitting the tools to that problem. There is a strong link here with a practice when flying aircraft. You need to read from the ground to map, not the other way around. You must do it correctly because there have been a number of accidents where the pilots have read from the map to ground thus misidentifying their location.

It is the same with this. Focus to the problem, bring the tools as you need them. It is interesting what Marian said about how experts can look as though they are novices – which is exactly what I feel like – sometimes I look at my code and think “That doesn’t really look that complicated”. You bring the ‘big guns’ out when you need them, hopefully abstracted down under a good interface. You know you need to keep the complexity down because there will be a lot of maintenance in the future where you or others will have to reason about the code.

So here the software complexity is of the order of the problem complexity, perhaps a bit more because you will need a bit of slack within the solution. At a personal level the big thing here is that you are aware of your own limits because in the previous phase you have been panicked.

One of the big things I have learnt through my career is the ability to handle this. For example there will be a bug. The client may panic. This is to be expected. The salesman may panic. Still possibly to be expected. As a developer if your manager panics too, you have a problem, because the buck will stop with you. Can you discipline your own thinking and your own practice so that you can calmly deal with the issue, regardless of how others are handling the situation? This is the struggle you can get in a commercial coding environment.

Implicit in this description is that you have developed a disciplined personal practice.

So in summary:

Novice

  • Rule-based thinking
  • Undisciplined
  • Some self-awareness.

Journeyman

  • Abstract thinking
  • Undisciplined
  • No (or very little) self-awareness.

Master

  • Contextual thinking
  • Disciplined
  • Deep self-awareness.

ACCU2016: Talk on Software Architecture Design 2: The Historical Context
ACCU2016: The Organising Principle

ACCU2015: Day1: Bits & Pieces

So it is rather late at the moment. Just had a great evening at a local Tango class with some great live music. Absolutely & totally sublime and completely different to the Nerdfest!

However I did retire to the bar on getting back from the dancing – it is important to network – to find that it was still buzzing. Had a great chat with a guy who works in CFD (Computational Fluid Dynamics), even though it was midnight! Amazing how you can talk about multi-processing late at night after a beer. Or have I been doing this job too long?

As for the day’s conference proceedings: Because Jim Coplien is recovering – thankfully – from a seriously illness, today the first keynote was by Pete Goodliffe about “Becoming a Better Programmer”. It was OK but not too many new points for me. Just a light presentation. The main points I took away were:

  • Money incentives work against producing better software. They are different targets. (A subject close to my heart)
  • The 4 levels of Maslow Competence Hierarchy (not the needs one): Unconscious Incompetence (Dangerous!), Conscious Incompetence, Unconscious Competence & Conscious Competence.
  • Dunning Kruger Effect and the cognitive bias of experts under-estimating their ability while novices over-estimate theirs.
  • Of course there was also the obligatory mention of the Dreyfus model of Skill Acquisition which has a good breakdown of the learning levels as well as mentioning that it takes 10000 hours (about 10 years) to become an expert in something.

Next up I went to a talk by Seb Rose, a TDD advocate, on “Less is More” making the case for adjusting the progression of test changes to make them have less ‘fidelity’ to your final test intentions during development to allow you to converge faster to the final implementation. This one left me wondering about this whole stepwise approach to knowledge generation implicit in TDD. Sometimes it does not happen in this left-brain way of small steps. Sometimes there are massive jumps as we tear down old structures and remake them. Also whatever happened to the differentiation between growth & development? Subsequent conversation with other participants showed that I was not alone in thinking about this although, of course, TDD does have its place.

One great point that Seb raised was the importance of saying “I don’t know”, which can be a difficult thing for anyone who is supposed to be considered competent. We need to assume we are initially ignorant and be happy in accepting it.

He cast aspersions on Planning Poker saying that in his experience it has always been wildly inaccurate. For light relief he showed an image of these cards from LunarLogic.
LunarLogic No Bullshit Cards
The main point I took away from this talk was just how much software development is a knowledge generation process (i.e. epistemic) and how we need to be clear about the next smallest question we need answering.

After lunch I split my attendance between a talk called “Talking to the Suits” and a more techie C++ one about the issues in converting the Total War game from Windows to OS/X which was mainly about the differences between the Microsoft and Clang C++ compilers (the Microsoft compiler is much more accepting – which might not help since it wont detect errors in template code if the template is not instantiated).

I will only mention the “Suits” talk because there should be a video of the Windows to OS/X one.

I came away from the “Suits” talk with some great cliches, e.g. Quality is not job number 1; Business loves legacy. The basic point here is that we have to be more explicit about the quantitative costs/gains when making a case for any technical work. We cannot assume that business leaders will be able to understand the ins and outs of things like IDEs, Technical Debt, etc. A good call to make sure we communicate more effectively about such things. A good idea here was the “Problem : Solution : Results” guideline when presenting information. For example: Problem: “Even simple changes take a lot (quantify) of time”; Solution: “If we improve the design of this part of the system”; Results: “we will save weeks of effort (quantify) when we add new workflows”.

That is probably enough for now since I really need my beauty(!) sleep. I have also put my name forward to give a Lightning Talk on “My Thinking is NOT for Sale”. Oh dear. I need to sort out the slides for that now!

Till later…

My Thinking is NOT for Sale

Its 2 o’clock in the morning and I am finding I cannot sleep. A thought that is so off the wall has been gripping my mind for a while now and I am finding it more and more relevant to what I have seen happen during my career as a programmer.

The title is worth restating:

My Thinking is NOT for Sale

This is not so much a shouted response to all those times that good technical effort has been driven carelessly under the steamroller of prevailing economic needs – usually those of the money swallowing monsters that are most companies – than it is a statement of an underlying truth, if only I can express it well enough and in shorter sentences. So here goes…

If you pay for software you will not get what you need. In fact you CANNOT buy software because it is not a finished product. The current economic model we have just does not fit and I believe this is why there is so much trouble in this area.

What is important about good software development?

Over my 30 odd years of work the primary creative and energizing point has been the interaction between the developer and the actual user as a system has come into being. The best of it has been the conversation between the two as they navigate the area of the user’s needs. If the developer is skilled, both technically and personally, they help facilitate both parties in mapping an unknown area, probably only vaguely expressed in the “wants” that the user can currently identify.

This is a conversation of human discovery in thinking.

It is priceless.

It is a gift.

It is a Free process. Capital F.

It cannot be bought.
It cannot be sold.
It is NOT a product.

It only makes sense if the effort is freely given by the developer. The inner costs of doing this are so high that it requires a high level of motivation that can ONLY be internal. To try and shoehorn it into our current ways of thinking about money devalues the process and I think this is what is underlying the problems I have seen happen many times.

The kicker here is that it is likely that it can only be funded by gift money. That means that there can be NO LINK between the funding and the final “product”. I use quotes because that word is a misnomer of what is actually going on.

Unrealistic?

Just go and read a book called Turing’s Cathedral by George Dyson and you will see how the Princeton Institute for Advanced Study was funded by donation. This was where John von Neumann worked and developed the architecture that underlies modern computers.

The picture of how the whole current edifice of modern computing was birthed from gift money just blows me away. I find my thinking so bound up in the capitalist model that to separate the resource – i.e. the money to give time for people to think – from the product of that thinking in such a way shows up the illusion of the current funding models for such work.

Is that enough to allow you to see it? Truly?
If you can then maybe you might understand why I am having trouble sleeping because in my tossing and turning my feelings tell me it could change everything…

Or maybe this is all just a dream and I shall be sensible when I wake up.
Hmmmm.

Post-ACCU2014 Thoughts

My thinking has been working overtime since I attended and presented at the ACCU2014 conference in Bristol.

[The delay in producing another post has been due to a lot of rather extensive personal development that has been occurring for me. Add to this some rather surreal experiences with dance – clubbing in Liverpool being one particular – and you might understand the delay. But that will be the subject of a separate post on dancing – I promise!]

But back to thoughts subsequent to my attendance at ACCU2014…

The Myth of Certification

The Bronze Badge. Small but beautiful.
One experience that really got me thinking was a pre-conference talk by Bob Martin reflecting on the path the Agile software development movement has taken since its beginnings. He mentioned an early quote from Kent Beck that Agile was meant to “heal the split between programmers and management”, and that one of the important guiding principles was transparency about the technical process.

But then there was a move to introduce a certification for what are called ‘SCRUM Masters’, key personnel – though not project managers – in an Agile software development approach. The problem is that it is just too simplistic to think that getting a ‘certified’ person involved to ‘manage’ things will sort everything out. This is never how things happen in practice and despite early successes Bob observed that subsequently Agile has not lived up its original expectations.

The transparency that the Agile founders were after has once again been lost. I consider that this happened because the crutch of certification has fostered inappropriately simplistic thinking for a domain that is inherently complex.

My inner response to this was: Well what do you expect?

I very much appreciate and value the principles of Agile, but there is a personal dimension here that we cannot get away from. If the individuals concerned do not change their ideas, and hence their behaviour, then how can we expect collective practices to improve? As I experienced when giving my recent workshop, it is so easy to fall prey to the fascination of the technological details and the seeming certainty of defined processes and certified qualifications.

I remember a conversation with my friend and co-researcher Paul in the early days of embarking upon this research into the personal area of software development. We wanted to identify the essential vision of what we were doing. The idea of maybe producing a training course with certification came up. I immediately balked at the thought of certification because I felt that an anonymising label or certificate would not help. But I could not at the time express why. However it seems that Bob’s experience bears this out and this leaves us with the difficult question:
How do we move any technical discipline forward and encourage personal development in sync with technical competence?

The Need for Dynamic Balance

K13 being winch launched, shown here having just left the ground.
This was another insight as to why I enjoy ACCU conferences so much. There is always the possibility of attending workshops about the technical details of software development and new language features on the one hand, along with other workshops that focus on the more ‘fluffy’ human side of the domain.

I live in two worlds:

  1. When programming I need to be thoroughly grounded and critically attend to detail.
  2. I am also drawn to the philosophy (can’t you tell?) and the processes of our inner life.

Perhaps the latter is to be expected after 30 years of seeing gadgets come and go and the same old messes happen. This perspective gives me a more timeless way of looking at the domain. Today’s gadget becomes tomorrow’s dinosaur – I have some of them in my garage – and you can start to see the ephemeral nature of our technology.

This is what is behind the ancient observation that the external world is Maya. For me the true reality is the path we tread as humans developing ourselves.

Also we need to embrace BOTH worlds, the inner and the outer, in order to keep balance. Indeed Balance is a watchword of mine, but I see it as being a dynamic thing. Life means movement. We cannot fall into the stasis of staying at one point between the worlds, we need to move between them and then they will cross-fertilise in a way that takes you from the parts to the whole.

In our current culture technical work is primarily seen in terms of managing details and staying grounded. But as any of my writings will testify, there is devilry lurking in those details that cannot be handled by a purely technical approach.

Teacher As Master

So John - Do I have to wear the silly hat? Well Bill, only if you want to be a REAL glider pilot.
Another epiphany that I experienced at the conference was a deeper insight into the popular misconception that teachers are not competent practitioners. There is the saying that “Those that can – Do. Those that can’t – Teach”. So there I was in a workshop wondering if that meant that because I was teaching programming, was I automatically not as good at the programming? But then a participant highlighted the fact that this was not so in traditional martial arts disciplines.

Indeed – teaching was seen as a step on the path to becoming a master.

We – hopefully – develop competence which over time tends to become implicit knowledge, but to develop further we need to start teaching. This will force us to make our knowledge explicit and give us many more connections of insight, indeed helping us to see the essential aspects of what we already know. There may be a transitional time where our competence might suffer – a well known phase in learning to teach gliding – as well as being a normal learning process whenever we take our learning to a higher level.

So I think the saying needs changing:
Those that can Do. Those that are masters – Teach.