Living with the gray areas

Agility is about living with the gray

In our travels as technologists we are constantly engaging in conversations with people about how technology works, should work, or has worked in the past. We spend time considering what we should be doing next and helping organizations to navigate through their individual challenges while attempting to deliver something great to consumers or simply to make the organization run smoother by utilizing technology to automate those things that are hard. We technologists write code, discover new things as we go along - our job as technologists is to live in the area outside of the known paths - we live inherently in the gray spaces where things haven't yet completely solidified. The idea isn't new but is just starting to come into power, this idea of living in the gray spaces was brought to light when the Agile Manifesto was penned - the manifesto opens with the following:

"We are uncovering better ways of developing software..."

Indicating that we don't yet know all the ways we might achieve what we set out to accomplish, but we'll discover at least one way. Along the way we may find many ways in which things don't work as well - but in doing so we'll learn what OUR path should be, or what path works best for us in a given situation at a given time.

Software development is all about living with the gray

When we as technologists (developers or otherwise) set out to do something new or to make changes to a system that already exists we are asked to 'create' something that meets a described need. When we start out, we may not be entirely sure how we are going to do the action of creation. At this moment we are confronted with hundreds of choices and hundreds of potential directions. We may feel a little overwhelmed but we are relatively well practiced at living with the gray space that choices create. We quickly navigate through the list of things we know, the list of things we don't know and start to devise ways in which we could experiment so as to understand what we don't know and "clear the fog" from the gray areas in our understanding.  We may write example code as a way to experiment, perhaps a scratch refactor of existing code to fit in a new feature or trying something completely new by picking up a new technology or a new software stack. We use these techniques to help us get through the gray areas, solidifying things as we go.

One of the big problems is that while technologists are aware of and deal with gray areas in their understanding fairly well, most people that they deal with do not.

Humans deal with gray areas poorly

Inherently we humans seem to like control or at least to think we are in control. Uncertainty makes us uneasy. Equally, when there are a great number of choices that we could possibly choose from, 
we feel trapped and our fight or flight reactions start to take hold. Outside of technology this is actually quite un-nerving to most people.  As I stated above just about everything in technology is about living solidly in the gray areas and slowly working to make them less gray and more certain. The businesses that we work with and for however don't see things quite that way. Every company I have ever worked for would like to think that software development is an inherently known quantity, that software development easily equates to something that they know is straight forward like building things or assembling items. Software however is a creative process that defies being estimated up front.  At the very start of a new endeavor we might be able to provide some context and information which can inform an estimate for how long it will take to accomplish but we should allow that initial estimate to more accurately relate to the gray that surrounds it, all the things we don't yet know. That first estimate should have a confidence interval - say 25% confidence with a list of all the gray areas that surround this beginning estimate and a set of steps for how those gray areas will be solidified and more well understood. This changes the discussion from - "Are we on plan for finishing X" to "What did we learn today and how did it impact our confidence we'll be able to do what was asked".

In short...

Software development, and by extension Agility, and other software development frameworks/methodologies have to help us deal with the gray areas. The systems we choose to use when developing software should allow us to embrace the gray areas. Frameworks we know about currently, Scrum, SAFe, Kanban all describe ways in which we can get context around what we are doing and allowing us to wipe away the gray. We can use small experiments to guide us to better places. We can learn to fail fast, as the faster we fail at doing something the more we learn. We can embrace the gray areas because that is where the learning is and it is where we expand our horizons.

Generativity, an open door policy

So, I have had some time to sit and consider what it is about software projects in my work a day world that make for successes and what things make for interesting types of failure. Here is where my thinking landed me:

1) Being open, and inviting is a benefit not to be undersold

It sounds so tantalizingly simple when you type it out like that, so much so that you might read that line above as "...If you build it, they will come..." but that isn't quite what that line means.

When teams develop new software, the software being developed is sometimes about a new feature or a new thing that BigCo. wants to have as a product and sometimes is to scratch an itch that the team has. In either case, this new software is likely to be able to assist a whole bunch of people which same or similar needs (it is unusual for software to actually be built 100% single purpose). Because most software can actually serve many 'close knit' needs, once you or your team has taken the time to build, you should also take the time to advertise and allow other teams to make use of what you have built. Being open, means no matter what you develop that you essentially open the door to others to build, manage or deploy with you.  Let other groups make use of the things you built, with as few a number of restrictions, provisos, caveats as humanly possible.

2) Protect the system, but keep the number of rules as low as you can

This is somewhat self explanatory - try to only put in place those rules that help you to make sure that the system is up, available, and working correctly.

3) Trust people to do whats right

When you open the door - you are opening your system up to whatever may come. To that point, you have to be able to trust that peoples self interest will also align with you and yours to some degree. Try to keep a light touch and trust that people will not go out of their way to screw you - or the system you built - that they are now using. If those who have been invited in through the open door do take advantage, know in advance how you would like to react to those coming to seek your new hot thing. Describe your "open door" so that people know what they are signing up for in advance.

4) Don't fall into the trap of N.I.H.

Part of doing work this way, with this mind set, is preventing yourself from going off into a corner and building it yourself because:

   * You believe that building it yourself is the only way to control your own destiny
   * Your team can't possibly be successful because your success is dependent on another team

It will always be simpler to modify something that is already built than it will be to go from scratch no matter how good your developers are. Work already done can trump work to be done in a great number of occasions.

5) Command and control will squash peoples desire to use what you build

Don't try to control the software and how its used too must - exercise just enough control to manage and maintain the quality of the system. I have seen plenty of people that believe that becoming the manager, exec, vp, director or what have you is all about being able to do it JUST the way they want. Nothing really could be further from the truth. The farther up the chain you go, the less desire to control you should have and the more trust you have to have that things will head in the right direction. If you attempt to stringently control things, no one will want to come be a part of the thing that was built and like holding a fist full of sand, the harder you squeeze the more sand you will lose, no way to hold it all.

Be open, be inviting, be willing to negotiate and understand how people will want to use your software. Be generative - providing space for people to get the most amount of benefit out of the building blocks you have already put together. It will eventually help the bottom line because everything being built will start to move faster, eventually.

Forward compatible data consumption

Forward compatibility means ignoring new fields should they arise ... passing them through if possible, ignoring them if they can't be passed through.  i.e. FOCUS ONLY ON WHAT YOU ALREADY KNOW and ignore the rest.

This means in XML parsing, if you are using annotation based parsing - tell it to ignore fields and objects it doesn't understand and continue with the rest, or if manually parsing to only concentrate on known objects and ignore the rest.

Same thing for json, annotations (like Jackson) need to be told to ignore fields that can't be mapped into the objects in a known way and to proceed with everything it does understand.

This way other systems can include new information that could be consumed at some later date without impact to the system at hand.

Why is all the above at all relevant or important - these steps are vitally important to the compatibility of APIs that you might be developing. In order for the API to be forward and backward compatible as changes to it are made the API system must be able to accept new things or ignore things it does not currently understand. APIs have to hold themselves accountable for what they 'know' they can do - and things that can safely be ignored.  If the API you are developing can't ignore things it doesn't understand, then errors get returned to the clients that may have otherwise resulted in a useable response from the API.

Death and Craftsmanship

I have been thinking a great deal about software craftsmanship recently. I have Uncle Bob Martin to thank for making me at least think about it. There is however a life event recently that made me consider the topic in a deep way that I am not sure I really even considered before; you see I had been taking what Uncle Bob was saying about software craftsmanship at face value that is until my Grandfather passed away this weekend.

The Craftsman

When I was younger I used to go over to my grandfathers house, which was not to far from my own home, about 2 or 3 times a month. At the age I was at the time - going to his house was about being able to spend my summer in the pool that my grandparents had, it was about having breakfast, lunch and dinner on the porch, it was about the grill and the amazing food that my grandfather could concoct using it. The time that I spent at my grandparents house was rarely about the things that my grandfather did for work or what he had done as work because my interaction with him was mostly after he retired from the work a day world.

To my knowledge my grandfather worked for Boeing as a machinist making various parts and pieces for either helicopters or for the machines that were machining other parts for the same. What I didn't know until I got older is what an amazing skill my grandfather had for doing what he did - I did NOT realize what a craftsman this man was. You see, my grandfather was able to make the machines he worked with sing and dance to create very specific parts. He was able to set up lathes and other machinery to sharpen existing tools, or create something totally new. My grandfather was able to cause these machines to create parts that had tolerances in dimension of no more then a few 10,000ths of an inch, by hand, day in and day out. When he learned how to do this work he didn't have a CNC machine to program, things were not automated in any significant fashion, he was taught how to make these machines do his bidding by hand, with the lightest of light touches. My grandfather took great pride in what he created - in retrospect, I have that same pride now for what he accomplished doing that work.

Software Development

My grandfathers death and the realization of the talent he displayed when working with machining metal lead me to this, I don't think that my software design and programming should be any different. I should be able to perform gross cutting code as well as fine grained 10,000ths of an inch type code and have it all work. I should be able to call myself a craftsman of software by being able to have someone look at what I have done and say that it is complete and well done. Software craft-persons should be able to look at someone else's work and identify their own craft in it as well as to call out the simple and small foibles made by their compatriot craft-person. Performing the 'craft' of software development, turning ideas into code and doing it with quality and precision, is not easy and not everyone can do it - just like I can't do the things my grandfather did with his machines. However - practice, apprenticeship and other things that go with thinking of software as a craft to master can help you improve what you do, can improve how you think about the work that is software development.

I know from here on out I will be striving to be NEARLY the person that my grandfather was, and nearly the craftsman I know he was in his work. I will work every day to improve how I write my software because knowing who my grandfather was and how he did his work prevents me from doing any less.