Roles for Making a great (software dev) team

I wanted to take the time to try to document the roles I think make up a great development team. This is essentially that raw brain dump - would be very interested in engaging discussion and conversation around these. What do you think - how has it worked for you - would you refine these in any way?

The Constructive tinkerer

This role describes a person that spends time looking at the problems that are facing the team and looks for ways to solve them.  The solution can be already contained in the things the team does or may be something that team has not yet thought of or is currently using in the way of technology. This person spends the time to investigate, research, experiment with the different, better, worse ways of doing what needs to be done inside the team. This individual is often not directly involved in the shit triad*, meaning this role is often not doing product development work but is instead creating nice paved road surface for the folk who are part of the shit triad* to move faster and get more done in shorter periods of time. The work this person does tends to lay the ground work for a great deal of other work that the team does. Clearing the path and making things easier, more visible, more operational.

Additionally - this role spends time providing guidance and mentorship for the other members of the team. Sometimes the discussions being had are technical in nature, things like "How should I implement this new hot thing that product asked for..." and sometimes the questions/discussions are procedural and more team oriented... "I think that we need to make this type of error more visible and loud so that we can find a root cause and eliminate it." This role plays a part in providing a sounding board for a great number of activities in the team working to help the team understand the aspects of their software running in a production data center with real users, whom we want to delight - not make angry.

The key for this role - is that the tinkering with things, the new code, the clean up of old code, the discussions are all oriented in positive "We will use this ways". When the result of this roles tinkering isn't useful to anyone, the person playing this role doesn't get upset, nor defensive - they simple tuck that work away for some later time, or they drop the work all together. The constructive tinkerer is focused on what the team working with them needs. Decisions about the longevity of something are biased towards team needs and wants but balanced with what the larger organization needs from the team as well.

The curator (Those that care do)

This is an oft overlooked role for every software development team. This role may not even be a single team member but encompass how the entire team thinks about the code base that they work in every single day. The curator(s) are those that are deeply invested in the well being, well meaning, and intention of each and every part of the code base. Curating the code is important because as we all know, code that is no cared for tends to suffer from:

Bit Rot
Code that is left to sit eventually rots away. Of course it doesn't actually rot, but what does happen is developers lose touch with what the code was supposed to be doing. There is a lack of documentation, so there is no longer any institutional knowledge of the code and no way to gain it other than wading back into the code base to find out whats in there. Bugs stop getting fixed and people eventually would rather replace all the work than 'figure it out' all over again. This can be avoided when there is a curator around - because they are constantly looking at what can be made better - seeing where problems are and will get upset when something sits for too long without getting any touch/attention.

Different points of view
It is rare these days that a code base have a single developer. All the code that I work on is managed and maintained by either a team or multiple teams of people. The curator has the un-envious job of reconciling the differing ways that things in the codebase are accomplished. Ties are broken, discussions are solidified into a 'way' in which the code will represent a specific thing. The code gets cleaner as a result. Differing points of view have to be recognized and addressed by a curator or you will easily be able to tell which sections of a code base were written by whom based on taste and patterns of development.

New developers
New developers to the team are simply new and have to be shown a way. This demonstration is made easier when they can see that the code base looks a certain way and does things in certain ways. This is made better when they are also able to consult a curation individual for a team. Work can be made even cleaner when new developers know that there is a pattern that exists that they can follow and that the team is open to discussion for new ways and new ideas to change existing patterns.

New shiny thing
A curator helps to combat running off and implementing the next hottest, best, new, shiniest thing in terms of development. Sometimes new patterns and new ways of doing things are beneficial or needed. A curator helps to understand how that can be phased into the code base and how existing patterns can be touched and adjusted as a result. The curator helps to understand how much time and effort is involved in moving the WHOLE code base into a new direction or if the new thing should only be used in certain locations under certain conditions.

Those supporting the shit trinity - Knows Shit, Gets Shit Done, Gives a Shit

So every team needs some number of people that have the ability to get things done... and get those things done well. Those that support the shit trinity make up a fair majority of a team and ideally everyone on the team would fall into this category to some degree. This means that everyone on the team has the ability to play a roll in accomplishing the goals that the team has set out to accomplish. Sometimes the goals are development oriented and related to software design and coding, other times the teams goals are about the process that they use to get their work organized and ready to work on.

The helper/garbage man

You always have to be willing to take out the trash. This is another role that the entire team should be adjusted towards. The garbage in the case of a development team is the grunt work of fixing bugs, re-organizing code when it makes sense, basically anything that would be considered the bottom of the barrel in terms of development effort. The tasks involved are generally just tasks that need to get done in advance of getting to the nitty gritty of some new development that has been planned. Every team needs at least one, and hopefully the entire team would be willing to do this work because they know it is necessary. Having the team be adjusted to taking out the trash means that no one on the team feels like it is demeaning to have some amount of their work be involved with taking out the trash in the code base.

The adage that I think goes with this is that a good manager, exec, CEO doesn't believe that certain tasks are somehow beneath them, they see that it needs to get done and that currently no one is doing it so they go in pick up that slack and get it accomplished and then later look for ways to not miss getting that bit of work done in the future. For a team - this mentality in at least one person has to exist both because it is necessary to pick up what is left behind but also because it encourages others to also pick up what is left behind. In essence having an individual on the team that "takes out the trash" encourages people to not throw stones at already broken windows but instead to fix those windows and move onto something else. 

The thinker (although they all have to be this way to some degree)

Software design, architecture and over all capability of the code to do a given task takes thought. It is not something that flows from the fingertips of generally smart people perfect the first time every time. Software development is a thought process, a process sometimes of trial (doing what you think will work) and error (seeing that what you did doesn't work) in order to get to the point where something is produced that meets the requested/stated needs of the larger organization. Sometimes the thinking on the team is encapsulated in one individual but I find more often that while one individual seems to do a great deal of the thinking/design that everyone seems to rally around that "nucleus" adding their own identity and thought process to the original. Making the thinker role more of an instigator role most of the time, using the power of the group to find all the holes in a design, to talk about all the corner cases in what we are doing and to aid in preventing the entire team from backing themselves into a development corner that they can't get out of. The thinker/instigator provides that stable guidance for architecture so that it evolves and morphs rather than needing a revolutionary re-write to get the team out of a bind.

Verifiers  (although they all have to be this way to some degree)

Everyone should be interested in verifying their work.  Sometimes this is a single individual that acts as the last safety valve for a team, making the last checks, T's crossed and I's dotted before the result of a development cycle, no matter how large, is verified to be doing what it was expected to when the development got started. In our team this has been encapsulated into a final check before deployment of the out put of the API.  When we are getting ready to do the deployment for a given day, the code that has been worked on is deployed into a staging environment and then a script is run to look at the syntactic differences in the output of the endpoints in the api space. Differences between Staging and Production are noted, discussed and either acted upon because they were unexpected or dismissed as expected due to the code being deployed. Taking this action in the form of tests that each developer wrote during their development in the early part of a cycle and doing this final check at the end causes us all to 'verify' our work. Similar to the thinker above - when the team sees that this is expected they tend to start executing that way on their own with little to no nudging required.

The reporter - Someone has to tell the world

In the end the WAY in which you work is important, but just as important if you want others to work in the same way is that you go and tell them what you are doing and how you are doing it. Enter the reporter of the team. This is someone who likes to taut what the team does and succeeds in accomplishing. In a shout from the roof top sort of way this individual helps the success of the team become more common knowledge and more well understood by other teams. This helps to spread ideas and processes and allows others to start to ask questions as they investigate their own teams and how they operate to accomplish the goals they have set out on.

Please stop RESTfully abusing HTTP

So I have noticed this rather disturbing trend while building, deploying and consuming REST service(s) and it goes something like this:

"HEY all - Lets build a REST service to meet our new product need."

Every one in the room nods and says - "That is an excellent idea." They have heard the buzz word and they understand HTTP fairly well so they feel like they have an excellent bead on REST. People may even do a little light reading on the internet around the topic. They will understand that they can use HTTP to take actions on resources. There is however an odd understanding that creeps into the discussion and that is how to go about representing both good and bad states for responses to the caller.

The most annoying representation I have run across thus far is:

POST ${JSON} … 200 OK {status: error, reason: SYNTAX}

Caller posts to the endpoint over HTTP, a json object. They get back an HTTP status 200 (indicating everything happened ok) with a json body in the response. That json body has an ERROR IN IT! This relegates HTTP to nothing more than a transport protocol. So you look at the above and you say - "...well, whats the big deal with that - looks ok to me."

HTTP 2XX Status Communicates Success to Clients

First and foremost amongst the problems with this pattern is that HTTP clients of all varieties expect that any response with a 2XX class status code can be considered success. This is in the HTTP spec and is what a large number of clients in a variety of languages understand. When I am attempting to write a client against this service there is no way for me, just looking at that HTTP Status code, to understand if my request actually did what I wanted or errored out in some way. Doing this forces the receiver to need to inspect the body of the HTTP response, breaking a programming tenant I have personally had for a while, FAIL FAST. Because I can no longer rely on the nice fast HTTP response header and I am forced to actually read the HTTP body - I have to read the ENTIRE body to see if there is anything in there that would tell me what went wrong with my call to this particular service. Which brings up the next problem.

What format is the Body in: Parsing??!?

So with a client now not being able to fail fast and being forced to read the entire response body what other problems are there you ask? Well how about, the format of the body? Is the body json? is it XML? is it some binary format? HTTP has a way to tell you what was sent with Accept/Vary header combinations, but in all those cases I have to essentially parse out the response in some way. What if that parsing FAILS? What specific format does the response have to be in in order for me to correctly interpret the type of failure that has occurred? Does a failed parse attempt lead to a generic error? a specific error? What do you tell a calling client when its not something the client can actually do anything about? Suffice to say this is a mess. It can be cleaned up a bit - you and your clients can specify what format these things should be in and where you can find specific error messages and other information, however HTTP spec already provides all this information for you in a standard way if you were using it the way it was intended rather than treating it as a simple transport protocol only.

I hear you screaming

I understand that it isn't always a clean mapping between good/bad things that the APIs do and an HTTP status code that currently exists, but you should at least try to find simple ones that make sense and then look for consistent and reasonable HTTP extensions that give you specificity that you need. Take for example returning a 403 Forbidden as an access exception - and then returning a header in the response with a subcode indicating WHY that specific request was 403'ed. Example, HTTP status 403, X-SUBCODE-ERROR: 101 X-SUBCODE-REASON: User over allowed limit at this time.

Please do not RESTfully abuse HTTP as a mere transport, use what you know, fail as soon as you reasonably know you can and for god sake, please don't make me have to parse a response body in order to know what went wrong.

Date Chicken

Through out my software development career I have been trying to understand how things work, actively pursuing the thought processes that people use to make their decisions. I spend time asking questions about the hows and the whys of what I am asked to do and produce. This action, over time, has produced a decent picture in my head of how software development should work and in many ways how it should never ever work. I have been able to understand the whys of certain actions and requests but the one 'action' for which I have yet to totally understand is the idea of software development "Date Chicken".

Date Chicken as I would describe it is the propensity of an organization to want to assign a date to a project without understanding or going through an understanding of the complexities around what has been requested to be developed. Date chicken comes about due to not having enough of the discovery executed such that any date that can be provided has to be provided with a long list of provisos and comments/risks about what in the world at large has to line up "Perfectly" in order to meet the date that is being published. So software development teams are asked to produce these "Plans" that indicate that we can meet the date IFF (If and only if) everything else in the world happens perfectly. What happens next, every other team produces the same sort of statement about their ability to meet the date and the fun begins. Management from each team begins to execute on the "PLAN" that has been produced, they organize teams, and instruct people on what needs to be done and they wait patiently for the first 'date' on the plan to come up.

Here the Chicken part of date chicken takes over. Basically the teams continue to plod along towards the date until that first date arises and a team has not produced the things that they 'said' they would produce and everyone else involved in the project gets to point out that 'their' dates will have to slip day for day with every day missed in the development team at the front of the train. Hence the 'Chicken' in date chicken.

Why is this so hard for me to understand you might ask - every software company of any size seems to do this...

Its difficult for me for the following reasons:

1) The date is a lie, perpetrated by someone else and then taken as gospel, which leaves me feeling like my management doesn't want me to tell the truth, ever. Ok - the date is not entirely a lie - because there is an OFF chance that the needed software shows up on "time" with high quality, but so far I have yet to see that happen when this game of date chicken takes hold.
2) Because the date is a lie everyone 'RUSHES' to turn out the thing that they were on the hook for before their date arrives. In turn this produces software of lessor quality, with production issues, that turns out to be hard to manage and maintain.
3) #2 causes our customer experience to essentially suck, driving the customers to complain or look for service else where, making my company look like the best of the worst.

I do understand that sometimes the date is important, there might be legal reasons or contractual reasons that a specific software development item must be completed by a specific date. However - when there is no specific pressure in this way, I would much rather have my management say "I want to give you the time to make the software 'RIGHT', and I am willing to wait to make that happen."

I don't think it appropriate for the waiting to be forever, but do think that a management statement that says something like "I would like to see something demonstrable in one quarters worth of time." or even, "I am ok to wait to get this right, but I would like it to launch this year" to be fine statements that reinforce the please build this right rather than fast. As we get closer to actually delivering the software requested it is possible to provide a VERY high confidence number on 'when' we are going to be complete. The further away from done however the less confidence in the 'plan' being the right 'plan' and the date, so much more a lie.

In the end - none of this is in my control, but I sincerely hope that it will be at some point if I either end up in a position to make changes, or have my own company making software. Date chicken reinforces all sorts of really terrible habits and I would rather not participate.

Death and Craftsmanship

I have been thinking a great deal about software craftsmanship recently. I have Uncle Bob Martin to thank for making me at least think about it. There is however a life event recently that made me consider the topic in a deep way that I am not sure I really even considered before; you see I had been taking what Uncle Bob was saying about software craftsmanship at face value that is until my Grandfather passed away this weekend.

The Craftsman

When I was younger I used to go over to my grandfathers house, which was not to far from my own home, about 2 or 3 times a month. At the age I was at the time - going to his house was about being able to spend my summer in the pool that my grandparents had, it was about having breakfast, lunch and dinner on the porch, it was about the grill and the amazing food that my grandfather could concoct using it. The time that I spent at my grandparents house was rarely about the things that my grandfather did for work or what he had done as work because my interaction with him was mostly after he retired from the work a day world.

To my knowledge my grandfather worked for Boeing as a machinist making various parts and pieces for either helicopters or for the machines that were machining other parts for the same. What I didn't know until I got older is what an amazing skill my grandfather had for doing what he did - I did NOT realize what a craftsman this man was. You see, my grandfather was able to make the machines he worked with sing and dance to create very specific parts. He was able to set up lathes and other machinery to sharpen existing tools, or create something totally new. My grandfather was able to cause these machines to create parts that had tolerances in dimension of no more then a few 10,000ths of an inch, by hand, day in and day out. When he learned how to do this work he didn't have a CNC machine to program, things were not automated in any significant fashion, he was taught how to make these machines do his bidding by hand, with the lightest of light touches. My grandfather took great pride in what he created - in retrospect, I have that same pride now for what he accomplished doing that work.

Software Development

My grandfathers death and the realization of the talent he displayed when working with machining metal lead me to this, I don't think that my software design and programming should be any different. I should be able to perform gross cutting code as well as fine grained 10,000ths of an inch type code and have it all work. I should be able to call myself a craftsman of software by being able to have someone look at what I have done and say that it is complete and well done. Software craft-persons should be able to look at someone else's work and identify their own craft in it as well as to call out the simple and small foibles made by their compatriot craft-person. Performing the 'craft' of software development, turning ideas into code and doing it with quality and precision, is not easy and not everyone can do it - just like I can't do the things my grandfather did with his machines. However - practice, apprenticeship and other things that go with thinking of software as a craft to master can help you improve what you do, can improve how you think about the work that is software development.

I know from here on out I will be striving to be NEARLY the person that my grandfather was, and nearly the craftsman I know he was in his work. I will work every day to improve how I write my software because knowing who my grandfather was and how he did his work prevents me from doing any less.