Source Code reflects personality of ours & our organization

As I see more & more code within my organization & outside, more I believe in Conway's Law (source: Wikipedia).

Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.

Its really amazing! This is not just limited to organizations, but also to the individual personalities. I believe, the code we write reflects our personality. We write code according to the way we think & behave. And also we behave according to how we write code!

Research paper from Nachi Nagappan called "The influence of Orgnaization Structure on Software Quality" provoked me to look at the things from different angle. I thought I would share my experience too. 

Let me start with something simple, but which has got profound impact.
Look at the following source code (source: http://googletesting.blogspot.in/2014/02/minimizing-unreproducible-bugs.html):
-----------------------------------------------------

void ScheduleEvent(int timeDurationMilliseconds) {

  if (timeDurationMilliseconds <= 0) {

    timeDurationMilliseconds = 1;

  }

  ...

}

-----------------------------------------------------
Apart from the fact that this superficially innocent-looking code has an devastating impact on predictability of software behavior, this style of coding also shows reflection of the Processes in the organization.

Many organizations turn bureaucratic as they grow. Reason why their processes become so complex, is that they try to accommodate every possible condition. There are some conditions which are aught to be handled. While some conditions are IMPOSSIBLE - with reasonable probability, there are also some conditions, which are probable, but can occur rarely. These conditions are typically "exceptions"!

A good programmer values simplicity. He/she doesn't penalize the core logic to make it complex, just for handling some condition, which is rare. If such condition occurs, he/she puts a try-catch block & specifies a separate flow to handle such scenarios.  On the contrary, programmers from a bureaucratic organization treat all conditions at the same level & handle them with if-then-else checks. This makes not only their code harder to understand & change, but also make their processes counter-productive.

Let me take an example. When I joined my organization, it was small. It had a very open culture. There was environment of trust. No swipes, no time sheets & unrestricted internet access. This made employees motivated & productive.  As the organization grew in size rapidly, we had less control on the intake of the employees. It was observed that a few employees were taking undue advantage - or rather were abusing internet resources. In order to tackle this, management imparted the following process:

1. Certain categories of websites including social networks & even You Tube was rejected by Proxy Server.
2. A very restrictive limit on amount of data one can download / amount of bandwidth one can consume.
3. Its quite possible that a developer may need to download some code from open-source / download libraries or even watch some video on You Tube which is related to his/her work. In that case, he/she can send an email to IT-Helpdesk. The helpdesk either temporarily allows the site or downloads the resources on developer's behalf.

It was very frustrating as a developer to be in a flow of doing some work & just realizing that I cannot download or access something. Our IT folks are very efficient & they used to serve our requests quickly. But its not just a question of time. Writing code / Testing software / or doing any work has "its own rythem". Such things used to break our rhythem. Effectively, we made our processes very complex & made ourselves less productive - just for handling a few cases of abuse.

Purpose of processes is to make our life simple, not harder!

However, if anyone really cared to count no of abuses of internet resources, they were very very few. Instead of having if-else checks everywhere, we could have dealt with them as "exceptions". In the "catch" block, we could define a "separate code" to handle such exceptional - but possible cases!

Because of one incident of an employee doing a fraud, a company made their "reimbursement process" very complex. This penalized productivity of their employees & in-turn that of the company. Nevertheless, those who wanted to cheat, could find some other way to meet their objectives. No processes are guaranteed to be fool-proof, neither the code. Recently, that organization learnt from this & decided to deal with such incidents as "exception". Effectively, provided a try-catch block to have separate logic. Process became simpler & employees rejoiced.

One very important thing that I learnt from my gurus, helped me a lot as a developer.
That is:- Understand that following three scenarios as different & mutually exclusive:

1. ERROR HANDLING: If one can cook up a test case, pass it your program as black-box & reach your code with erroneous inputs, then you are aught to handle it. 
2. ASSERTIONS: These are to be used when your code assumes certain condition (e.g. pointer to be null, bade face pointer from BREP body) to be IMPOSSIBLE. That is, it is not possible for such a condition to occur from inputs to your programs. Only possibility that this condition to occur is a programming mistake in some other part of the code. (Typically, that other part of the code is written by programmers, who are humans, like you & me! And humans do make mistakes!)
3. EXCEPTIONS: These are to be used to deal with conditions / scenarios which are possible, but rare. e.g. unavailability of server in client-server application. At times, server may go down, but we don't really want to pollute the code with if-then logic all over the place. Better we should handle it separately.

All three things above are equally useful & important. Key is to learn what to use when!

More examples to follow from my subsequent posts...

TTT Curve - Economic Growth - and the Source Code [Cont]

In my previous post we skimmed through T-T-T curve especially for people who don't have relevant background.

source: http://www.materialsengineer.com/E-Steel%20Properties%20Overview.htm

Now, let's talk about its connection with the Economy and yes - the Source Code.



Year 1991 is marked as turning point in India's economic history, when the (the then) finance minister Dr. Manmohan Singh made many reforms, which created many opportunities. Thereafter India enjoyed a massive economic growth especially during the period of year 2004 to 2008. India's GDP (Gross Domestic Product) since 1991 has always stayed impressive, when developed countries had been stagnant.  However HDI (Human Development Index) of India has continuously regressed. I used to question myself "Is too much of economic growth good or bad?" I believe in Laffer Curve, which gave me the same answer that I derived the other way.

When economy grows, it changes many things. You have new factories producing products that people will consume. The same factories create jobs for people who in turn by these products. People as they create wealth, buy houses and new cars. They migrate to cities. City cannot accommodate such a large rise in population, so we build multi-story towers which again gives business to construction companies which generates more money & employment. People go to shopping malls, multiplex theatres & restaurants to spend the money and so on... A goody goody picture painted by Capitalism.

But the picture is definitely no that rosy. All the things above have created severe Environmental Debt. I completely agree that growth has to occur. There is no progress without economic growth. However, when growth happens, the system has to change its Internal Structure. Widening the roads & increasing allowable height of the buildings does not solve the problem, rather it complicates the problems or creates ones. The city needs to be re-built with new design, infrastructure, schools, gardens, hospitals and so on - towards its new needs. But that has not happened. Capitalism has always painted the illusion that system takes care of itself. The strong force within individuals to make more money & resources actually helps the whole society to improve. To some extend its true - but only to some extend.

What makes difference here is the sheer Rate of Change. Look at TTT curve - there is something called as Critical Rate of Change. Very high rate of change (e.g. rapid economic growth) is required to achieve higher GDP, but it also creates a lot of internal stresses & distortion. Some times we cannot control rate of change - we are mercy to it. But we can certainly control "structure of the economy" (i.e. grain structure) and "Government Policies" (i.e. alloying elements) that will shift the TTT curve to right side of x-axis.

What happened to India during the period of year 2004-2008, has happened to my beloved city of Pune where I live. Historically this city has been hub for numerous  research & educational institutes like "National Institute of Virology", "India Meteorological Department", "Bhandarkar Institute for Oriental Research" and so on. However growth of city in last decade has been based only on Consumption, rather than Creation. We didn't add any more premier institutes, what we added was only things like shopping malls & restaurants.

The same thing has happened to many software companies. I know a few software companies which were small in size, but were serving a very niche market. They had great people working in a very open & professional environment nourishing high value system. Needless to mention they were making good money & wee also contributing to society. Working with those few companies was considered as a symbol of pride. During year 2004-2008, the market environment grew very rapidly. Investors & stock market pressurized companies to show every quarter to supersede market expectations. Supply of people was short. To make the billing happen & tick the business, company started recruiting in tons of people without really checking do they really fit into company's values & culture. Recruitment heads of these companies were given the "Headcount Targets"! This encouraged mediocracy & companies lost their reputation. They are somehow sustaining the growth, but they lost their much preserved organization culture. It is the sheer rate of change without proper grain structure & without alloying elements that killed those companies.

Same thing happened to the source code. It was eighties, when Fred Brooks wrote the famous & beautiful book called "The Mythical Man-Month". But it seems we are still are doing the opposite. Expected turn-around time for projects to deliver has shrunk tremendously. To achieve that, managers put lot of developers & testers on the project without smooth ramp-up. There are frequent governance reviews, so frequent  -that team members spend more time preparing for reviews than actually writing code or testing it. Projects get delayed & management once again puts more people in a hope to bring it on track. Quality of code degrades rapidly & the project never gets over.

Expected turn-around time is going to shrink further. We are going to get developers & testers who may not have required competencies. In fact gap in competencies is going to be very high.  And we may not afford to be very choosy in choosing the people, as pressure from investors & stock market will be tremendous to short QoQ results. This is fact of life, which we have accept. When economy grows fast companies are always short in supply of skilled people. We are not talking about niche companies here, but general market here - especially in context of common Indian Software Companies.

Question is; can we still make investors happy without sacrificing internal quality of code & without diluting organization structure? We need to think differently. Effect of grain size & proper alloying elements help a lot. In terms of code, fine grain structure (i.e. highly cohesive, loosely coupled software components written with Single Responsibility Principle) is able to withstand high rate of churn. The best Alloying Element is "Testability and Tests". It is often difficult for companies who recruit in masses, to teach new-comers about writing good code, software design, Design Principles, Design Patterns , Programming Paradigms, usability & rapidly changing technologies & so on - in short span of time. But is we teach people, work with them & show them how to write a testable code, many other things fall in place automatically. Testability is the way to scale. Its the best alloying element.

I learnt the magic of testability when I visited  GTAC-2010 ("Google Test Automation Conference") a couple of years back. The talk by Russ & Tracy "Flexible Design, Testable Design, You don't have to choose" really provoked me a lot. I went back to my environment. I applied to multiple code bases with projects from different challenges, its amazing to see how simple it is! It really works!

TTT Curve - Economic Growth - and the Source Code

In last few posts, we were discussing about interesting parallels between working with metals & the code.We discussed about properties of metal (& the code) like Strength, Ductility & Malleability and  Toughness. We also talked about Case Hardening. In this post, we will discuss something more interesting - i.e. Time-Temperature-Transformation (TTT) Curve & its relation to software development.

Figure 1: source: http://www.materialsengineer.com/E-Steel%20Properties%20Overview.htm

The abstract in the above site mentions:

The traditional route to high strength in steels is by quenching to form Martensite that is subsequently reheated or tempered, at an intermediate temperature, increasing the toughness of the steel without too great a loss in strength. The ability of steel to form Martensite on quenching is referred to as the hardenability. Therefore, for the optimum development of strength, steel must be first fully converted to Martensite. To achieve this, the steel must be quenched at a rate sufficiently rapid to avoid the decomposition of Austenite during cooling to other stable forms like Ferrite, Pearlite and Bainite.

In this post we are going to discuss out Hardenability. Ability of the steel to form Martensite is called Hardenability. It can be learned plotting the Time-Temperature-Transformation (TTT) curve. Steel is heated above its Austenizing temperature & cooled down at different rates. Curve is plotted where structural transformation of the steel into more stable forms begins & ends. This results in C-shaped curve.

What does the C-shaped curve signify?

The position of "C" along the x-axis leaves a narrow neck. Steel can be converted into fully Martensite - if & only if - its cooling rate escapes the "C" region of the TTT curve. The cooling rate that's tangent to the "C" is called "Critical Cooling Rate". It is the minimum rate at which the steel must be quenched in order to get fully Martensite. If steel is cooled slower than this critical rate, then it results in other structural forms like Bainite, Pearlite, which will not give required hardness.

So what's the issue? Its possible to ensure that steel is cooled rapidly enough to have desired hardness. But the negative side effect is, rapid cooling leaves undesirable internal stresses into steel which makes it brittle. Also, rate of cooling is so rapid that some of the Austenite doesn't at all get a chance to get converted into Martensite, even if its possible. This leaves some Retained Austenite, which makes properties of steel unpredictable.

Addition of Alloying elements into Steel play a very important role here. All alloying elements except Cobalt shifts the C-curve of TTTdiagram towards right side of the x-axis. This lowers the Critical Cooling Rate, which means you can afford to cool the steel a bit slowly, while completely converting to Martensite to get desired Mechanical Properties for the steel component you are making.

I have added nothing new so far. Every Mechanical Engineer knows it & every Metallurgist breaths it. If anyone notices errors in the above technical content, appreciate if you can bring it to my notice. 

Now some other interesting thing starts here. What is the relation of this post to Economic Growth and Software Development and the Source Code? 
Well I'll put it in my next post, as this one is getting longer.

Working with Metals & the Source Code - III

Now, let's talk about something more interesting - Case Hardening.

Hardness is an extremely desirable property of most mechanical moving components. If gears of transmission system of your car are not hard, then they will wear out quickly. Making the steel hard; is not that hard. Increasing amount of carbon & other alloying elements increase the hardness, as well as its grain structure which can be controlled by proper heat treatment.

However Hardness comes with a gift - Brittleness. Typically hard components have low toughness. They fail in brittle manner, without giving proper symptoms before failing. We cannot afford to break a gear!

So what we do? One who invented Case Hardening must be very innovative. We combine high-carbon hard material with low-carbon ductile one into a single component! There are various ways to do it - one is carburizing. Heat the steel component in a furnace that has controlled high carbon environment. Material absorbs the carbon, but the depth of its penetration is very carefully controlled. As a consequence; component becomes harder only at its surface up to certain case depth, while its core remains soft.

What does it mean?

When a case hardened component is subjected to impact, it initiates a crack - as the case is brittle. However the soft core doesn't allow the crack to propagate. That helps component not to fail.

Why are we talking about this, which everyone knows already? Well, just want to share an interesting analogy, though not a perfect one. Mechanical components require hardness as they need Wear Resistance. Software doesn't wear after use, but the other part of crack propagation is important.

When certain component (say library or class or function etc) fail due to unanticipated impact force such as Bad Input, we don't want that crack to propagate throughout the software. We want our software to detect & stop it. Earlier it detects, easier it can handle it. Even if it fails, it fails elegantly giving useful information to the users & developers of the software.

Yes, we are talking about Assertions.
If I have to say two most powerful tools I use in writing code, then they are (1) Commit Early, Commit Often AND (2) ASSERTs.

Asserts detect the crack right at the point where it is formed. It fails fast. This tool is useful not only while writing code, but also in personal life. After all, code reflects personality of the developer, it cannot be separated. Well, this is another interesting thing that we will talk when we see in person.

Working with Metals & the Source Code - II

In my previous post I started sharing some interesting parallels between working with metals & the code. In particular we talked about Ductility & Malleability.

In this post, let's talk about:

• Few other very important properties called Strength, Toughness, hardness AND
• What is the effect of Grain Size on these properties.

Of course these matter to the source code as well, as it matters to metals.
Strength is something obvious to imagine as its very important property. Its ability to withstand force without fracture or undesirable deformation. The strength can be  against Tensile or Compressive forces. It can be against Impact (Toughness) or cyclic forces (Fatigue).

Once we ship the software we have no control on how it will be used!

We can of course provide an elaborate documentation which no one will read. When did you last time bothered to open the user's guide of the digital camera you purchased? When you can't make out how to alter shutter speed, you play with various controls & try it out. It may work or it may not. If it doesn't you take help from your friend or dial the customer support. When nothing works, you open user guide.

That's natural! Point is, we cannot predict how the interfaces of the software will be used, in what context, in what sequence they will be used, whether it will get passed correct data or what. We cannot predict network speed at users end. If its a client-server application, we don't know how many users will try to connect it simultaneously. We can anticipate & design for that; while making some assumptions. If software crashes or corrupts user's data; it's very embarrassing & shameful for us, no matter whether or not ne used it as prescribed.

Yes, we are talking about Strength - Tensile, Compressive, Impact, Fatigue & all. In order to be sure we are shipping a robust software, we test it rigorously inside out labs as well as with beta users or some innovative ways like crowd sourcing. Once tested, we are very careful about changing the source code. In fact many teams discourage modifying the source code after testing before release. They call it Code Freeze & with some teams I have seen it happening months before release. I cannot understand "Code Freeze" - how code can be freezed? It must flow :-). But well, I understand the motivation.
Here we take help from "Open Closed Principle" (OCP), where we close the code modification, but open to extend it. That enables us to implement users requests & bug fixes. I'm fan of OCP, but its overuse has negative side effects. Too many extensions creates too many Grain Boundaries. Yes, Let's talk about grain boundaries now...

Most real materials are Poly-crystalline. Roughly crystal is how atoms or molecules in the metal are arranged together for form a Unit. Each individual crystal in poly-crystalline material is known as Grain. Regions where grains meet is known as grain boundaries.

Material with arrangement of large individual grain size have fewer grain boundaries. Its called as coarse-grain structure. A fine-grain structure is the one which contains smaller grains resulting in more grain boundaries.

Fig 1: Coarse & Fine grain structure. source: www.sciencedirect.com

What does the Grain Size and Grain boundaries mean to the software and the source code?


One can relate Grain to individual library or a package or a class or a function etc - let's say individual components of a system. If you design component with clear focus & crisp responsibilities i.e. Single Responsibility Principle,  it results in fine-grained structure. Fine grain structure has better mechanical properties like Strength, Toughness, Hardness compared to that of Course-grain structure.

Fig 2: source: http://www.southampton.ac.uk/~engmats/xtal/deformation/control.htm


However there is one problem with fine-grain structure.


Grain boundaries act as barriers to slip. This is because they have random crystallographic orientations, which means that the slip systems of adjacent grains are not aligned. When a dislocation reaches the grain boundary it must either change direction or stop moving. Changing the direction of dislocation movement uses up energy so the plastic deformation of the material becomes more difficult.

That's good! But Oh, that's not good as well!! We need to have our code fine-grained so that it has required strength, toughness & hardness, but at the same time we also want to be able to shape it when we want!

Well, large number of grain boundaries does not necessarily mean that code cannot be changed. As explained in above figure boundaries arranged in haphazard way prevent relative movement of grains. But if they are aligned, they can support the movement.

In deed, grain boundaries represent interaction between individual components of software. A badly designed software typically means badly arranged grains. There is always a "desired fine-grained structure" of the code and "undesirable fine-grain structure". The desired ones are Highly Cohesive & Loosely Coupled systems. They happen when one designs with "Open Closed Principle", "Single Responsibility Principle" & design-by-contract. Such a system has high strength, toughness & hardness, at the same time they are easy to change as well!

How "undesirable fine-grain systems" are formed? This typically happens when we have a big team formed rapid ramp-up, writing code without tests & version control discipline. Every developer starts writing code with nes half-cooked understanding of the system. There is no re-usability. In fact there is no usability in first place :-) as different components badly interface with each other, they are tightly coupled & no cohesion. No wonder such systems are brittle in nature. Compare this with the molten metal that is  cooled down very rapidly. The rapid cooling rate results in many centers of nucleation of grains which grow rapidly & meet fellow grains to form boundaries. If the material is thick, then the rate of cooling is also uneven all over the places leading to unpredictable properties of the metal.

But don't worry, they can be cured. You only need to put them into Refactoring furnace, heat it above critical temperature where grain structure gets corrected & then cool it down at controlled rate. But you need to fuel to fire your refactoring furnace, how you will do that? Yes, you guessed right, that's Testability. Build testability into your code, and most other things will fall in place. Watch this video: Flexible Design? Testable Design? You don't have to choose! - by Russ & Tracy. Its amazing, it works! It has worked for me! Wish you all the best!

Working with Metals & the Source Code

I'm a Mechanical Engineer & have always loved studying Metallurgy during my studies & profession. I'm writing software for Engineering Applications for about an decade now. There are interesting parallels between working with code & the metals. Computer scientist drew many inspirations from Metallurgy in the form of algorithms like Simulated Annealing, however this post is more about some simple stuff.

Here are a few most important properties of metals:

• Strength --> Maximum stress a metal can withstand before fracture

• Ductility --> Ability of a material to deform under tensile load

• Hardness --> Ability to withstand surface indentation

• Fracture Toughness --> Impact Energy absorbed by unit area before fracture

Strength matters to the code as well. Just like metals, the code is subjected to various forces. Forces of changing user requirements, changing hardware & software environment, adopting to different third party environment & so on. Not to mention, the pressure from investors to deliver QoQ profits and managers trying to squeeze everything in tiny budget & time!

Here is how the Stress-Strain curve looks like: (taken from Wikipedia)

What happens when you want to stretch a thin wire out of a metal - say Aluminium or Steel?

As the metal gets stretched (& so the code), it starts developing internal stress. The original design (Lattice structure) though gets deformed & strained, is able to absorb the force of changing requirements & environments.

As we stretch along, it becomes harder & harder to stretch it further. This phenomenon is called Strain Hardening. Software was not designed to accommodate unanticipated changing conditions. The grain boundaries between various interacting components resist the ductility.

A point is reached when its extremely hard to work with it & it starts breaking. A ductile code shows early signals before failure, while the brittle code & the metals fail abruptly!

So what do you do? We need wire to be stretched & sheets to be hammered. 
Our investors want to continue to make money from the software & that's how we get our salary cheque! Customers want us to produce more usable features & our managers want all these in the modest budget!  

Yes, there is a way to go forward - Heat Treatment! We put the metal into a furnace, heat it & cool down at a controlled slow rate. If its a steel, we heat it something beyond called as "Austenitizing Temperature". This is the temperature at which steel transforms into "Austenite" - from  body-centred cubic (BCC)  to face-centred cubic (FCC). With this new design of internal structure, it is further able to withstand the tensile forces without getting fractured.

Can we do this with code? Can we change its internal structure so that it can be further worked upon for some more time? I wish I could put the code into some furnace & heat it!
Indeed there exists such a furnace - that's Refactoring Furnace!

Malleability of steel & its ability to get heat treated depends mainly on its constituents called "Alloying elements". Steel is primarily made of Iron & Carbon, but it also contains alloying elements like Mn, Ni.

Our Refatoring Furnace can heat treat our code & improve its internal structure so that it can be stretched & hammered further. But its not possible without adding alloying element. We need some alloying element that would make easy to heat treat the code. That alloy is called "Testability & Tests"!

Your code is working, but do you think its badly designed? Do you have fear to change it further? And are you getting dragged because of enormous amount of Technical Debt carried over? Don't loose heart! You can still refactor it. You just need to add this alloying element called "Testability". Thanks to "Michael Feathers" for his beautiful book "Working Effectively with Legacy Code".

Clean Code That Works

TDD is something that has transformed the programmer within me. Kent Beck's popular book "Test Driven Development By Example" is a must read for any programmer who is serious in learning TDD.

In his book Kent Beck beautifully explains how TDD addresses the goal of every programmer: "Clean Code That Works".

Although it’s the goal of every programmer; there is always a dilemma.
There are two ways to achieve this:

1. I'll write a clean code first; then make it work later.
2. I'll write a working code first; clean it up later.

I have tried committing both the sins; both are disastrous.
Problem with first approach is; it leads to design that is unnecessarily complex & rigid.
By the way; what's use of clean code if it doesn't work!

The second approach "clean it up later" never happens.
We go on adding code; but never refactor. More we postpone it; more & more it becomes difficult to refactor.

It becomes an ever growing Technical Debt; that at some time becomes impossible to repay. It can potentially reach a state where one has no option but to close the shop.

Look how TDD helps to solve this problem: 

TDD goes in iteration : RED - GREEN - REFACTOR

• RED bar gives us reason to write a code.
• After GREEN bar; we have a working code; but that may be dirty.
• During REFACTORing we clean up the code. 

So at the end of every iteration we have Clean Code that Works!

And these iterations are matter of minutes.
RED - GREEN - REFACTOR really gives a nice rhythm while writing code.
---------------------------------------------------------------------------------
There is one parallel:
There are two schools of thoughts:- (1) Capitalism and (2) Communism

Communism says equality should be first achieved in the society; then we should make further economic progress.
Capitalism says we should prosper first; then we can make things equal (or cleaner).
Everyone shares the dream of prosperous society, whose structure is fair. It's just the matter of what comes first.
  
However; it doesn't happen with Capitalism. Rich become richer & poor become poorer.
But there is problem with Communism as well. If we don't prosper what to do with equality?

Kent has solved our dilemma in programming. 

But it seems this problem is still unsolved.

This is quite disturbing. Isn't it?

Any views?