Choosing The “Right” Hammer To Hit the Nail (aka the “right” programming language)

Setting the Scene

Last summer I worked as a software developer in a large firm that provided financial market data and services to clients worldwide.  The delivery of financial news and market data from exchanges is highly time-sensitive, approaching real-time. Hence their system was heavily optimised for performance with the back end primarily developed in C++. I was tasked with prototyping a survey tool for this platform. The required back end libraries for this tool were available in both Python and C++. However I was advised against developing the back end in Python to ensure consistency with the rest of the applications maintained by the team. Additionally, I was told that the Python libraries were essentially a wrapper around the C++ code so should use C++ for efficiency. I was neither experienced in C++, Python or large scale system development and of course I went along with it and used C++.

This got me thinking about the using the right development language for the task, if and when the choice was available. This blog post will discuss this with respect to the case I have outlined above.

Please note that this is not a discussion about whether Python is a better programming language than C++, or vice-versa. This post aims to highlight problems when choosing the “right” programming language with respect to the case outlined above.

Development Issues: C++ vs. Python

As a beginner to both the internal framework and C++  the development of the back end was a slow, arduous process. Python is definitely an easier language to write than C++ as it is more intuitive and it doesn’t require as much boilerplate code as C++.This meant that methods which could be written in five lines in Python instead took around twenty lines in C++ (not always the case).

I felt that using C++ for this led to more errors in my code and also a slower debugging rate. Of course I am not stating that more lines of code lead to more bugs. However given that the methods implemented in C++ were far longer than their Python equivalent and this combined with the complex system infrastructure to compile and test files made finding errors and testing a time consuming process. I felt that in this particular case, the use of Python, a more concise language than C++ would have made it easier to read code and find errors.

The sheer amount of boilerplate code required by the internal  C++  libraries also making changes a time consuming process.  This in turn made iterative development a slow process. I felt that in this case Python would have been better suited to this particular task.

With regards to performance, the Python libraries were also optimised. However they were not suitable for performance critical application development for this system because they compiled down to C++. Hence any applications dealing with larger volumes of data would be slower if developed in Python on this particular system. However my application was non performance critical so using Python would not have led to the application running noticeably slower.

In the end, I was able to deliver the prototype on time and got experience in coding in a language I had previously never used. However the slower development time, by using C++, meant that I didn’t have a great deal of time to implement all the features planned out at the start or refine the application with the feedback received from the demonstration. Due to this I felt that Python was better suited for the purposes of rapid prototyping as opposed to C++. Especially due to that fact that the back end was not performing complex tasks and was just storing and retrieving information from the database. Hence Python would have been more effective in terms of ease of use, readability of code and would have allowed me to develop and present a more complete solution.

The Other Side

I can understand why my team lead advised me against the using Python in this case. Their main concern was maintenance. The back end development across all their applications was carried out in C++ and had I developed the back end of the application in Python and left after eleven weeks. However it would be up to the team to maintain it. Hence maintenance may have taken longer due to unfamiliarity with the library being used.

Additionally, the team I worked in was quite small with only four developers who had started supporting more internal applications. Given this, there is a case to be made for the team not wanting to maintain and switch between applications developed in different languages and using various libraries. This does not make them less capable or inflexible but there are standards and procedures in software development for a particular reason – to help make the process more efficient. Hence limiting the number of languages used to develop the back end definitely helps making the development process more efficient.

Conclusion

Every programming language (the metaphorical hammer) has its own set of limitations and advantages over others. Hence different languages are used for different tasks (the nail). In this case while I may have felt that Python was better suited for rapid prototyping, I can definitely agree with the decision to use C++. However this may not always be the case and while spending massive amounts of time weighing the pros and cons of using language X versus language Y is not advised; giving a little bit of consideration about which language is better suited to a particular task is definitely worthwhile. It will help ensure that a developer or development team is making the best use of their time and delivering robust, efficient tools to their clients.

Agile Methodologies in Large Scale Projects: Not a Recipe For Disaster

This article is a response to “Agile Methodologies in Large Scale Projects: A Recipe for Disaster”. The author’s claim that “agile development has no place” in real-world software development is solely based on the values outlined in the Agile manifesto. Values are concepts that drive choices. Principles define the manner in which these values are carried out. By only considering the values, ignoring the principles and providing no empirical evidence the author fails to convince me that agile development cannot be successfully applied in real-world software development. In this blog post I will counter some of the main points made by the author to prove that agile development does have a place in real-world software development.

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

The values outlined in the Agile manifesto are very broad guidelines about what gets priority in the development of software. The manifesto clearly states that the notions on the left-hand side are given more priority than those on the right-hand side. This is not ‘infeasible’ in large scale software development as suggested by the author. While there are challenges in incorporating Agile practices into large teams, empirical evidence and anecdotal evidence from professional developers shows that not only is it feasible but it produces better results than traditional approaches.

A study conducted showed that one-third of all development is run with Agile methods and two-thirds apply a Waterfall model. While the use of Agile methods is still a minority compared to the use of the Waterfall model,  23% of these Agile projects involve teams with more than 200 people. The success rates presented in the following points prove that is not ‘infeasible’ to use Agile methods in large scale software development.

The author claims that:

“Lack of formal documentation or planning could appear unprofessional to clients”.

Following Agile development does not mean that the development process lacks formal documentation or planning. The development model just gives working software more priority than producing comprehensive documentation. Agile development aims to minimise documentation and aims to ensure that correct use of the model will produce a working software for the client. In his article, “Essential, Valuable, Timely Documentation”, Ashish Sharma states that “Documentation is an important part of every system, … , but comprehensive documentation as such does not ensure project success. In fact, it increases your chance of failure”.  I would agree that this evaluation is entirely correct. Documentation is crucial for both clients and developers to aid in understanding the software and for future maintenance. However it is certainly not more important than producing a working piece of software.

Furthermore, the author’s use of a failed government project as an example where comprehensive documentation will help them with accountability is highly exaggerated (as agreed by the author). The author concludes by saying that even if the scenario is exaggerated the outcome “would almost certainly be the same” in any other case. That is that the Agile model does not produce enough documentation that could be used to justify why a project failed. I think that this is incorrect. As pointed out in the above paragraph, the use of the Agile model does not sacrifice documentation. It simply aims to keep it at a minimum. Hence the author’s claim that a client would not have any proper documentation other than those on “napkins” is unfounded and would not result in the same outcome as having no documentation at all.

Another claim made by the author is that:

“with a detailed plan from the beginning that clearly defines the scope of the project, the requirements it must capture and other details … the project will be far more likely to be completed on time and within budget”.

Empirical evidence proves that using the Agile development model can lead to a 37% reduction in time-to-market with a 26% median cost reduction. This is proof that a more traditional model need not be followed in order to deliver a working piece of software on time or within budget. In addition to this, the study cites the Waterfall model as one of the reasons why between 50 and 70 percent of software project fails. I agree with this as the Waterfall model fails to give developers a good idea of the requirements at the start. This leads to developers working on ideas rather than facts and the scope of what needs to be completed is not realised till the end of the development cycle.

This moves onto another point made by the author that:

“the majority of this communication [between clients and developers]  should take place before the implementation begins”

I do not believe that this should be the case at all. When the Agile model is used development is an an incremental and iterative process. It allows the developers to acknowledge and fully understand the evolving needs of the clients rather than setting the ‘perceived’ requirements in stone at the start of the process as is the case with the Waterfall model. Acknowledging that a customers requirements will evolve and actively inviting them to participate in the development process will allow developers to verify the solution at an early stage and fix mistakes early on.

The Agile model has a itself to produce greater customer satisfaction. 78% of those who adopt it report greater stakeholder satisfaction and 66% report better alignment between business and IT. This proves that it giving priority to customer collaboration and responding to change over contract negotiation and following a plan. So I find the author’s claim that clients requesting changes to the product that they are paying for mid-way through the project means that they are not treating it seriously rather ridiculous. Using an iterative development process, such as Agile, will allow for any changes to be easily incorporated into the software  process without massive ‘ripple effects’ as claimed by the ‘author’.

In conclusion, the increase in customer satisfaction through lower costs, improved quality of software and shorter time-to-market proves that the Agile model does indeed have a place in real-world software development. In addition to this by not fully exploring the applications of the Agile values the author misses out on the advantages of the Agile model and deems it ‘infeasible’ in large scale software development for all the wrong reasons.

Pair Programming: Will It Make You Hate Your Colleagues?

images

Introduction

I recently interviewed for a company who believed in pair programming one hundred percent of the time on all projects. It got me thinking about how pair programming, on this scale, would impact me as a developer and whether it would actually be effective and useful. This blog post will explore and analyse the benefits and costs of pair programming one hundred percent of the time as a software development technique.

The Concept

Pair programming is an agile software development technique where two developers work together at the same computer. The developer who actively implements the program is known as the driver, while the other is the observer. The observer’s role is to continuously monitor the work of the driver to identify syntactic errors, spelling mistakes etc. The observer is also responsible for steering the design of the project in the right direction. Both developers switch roles frequently.

There are three possible pairing permutation when pair programming. Novice-novice, novice-expert and expert-expert.

The novice-novice pairing variation has shown an increase in productivity in comparison to developers working on their own [3]. The novice-expert variation makes for a great training tool to introduce a new hire to a development language or framework that they are unfamiliar with [5]. The expert-expert is a powerful variation and can lead to a boost in productivity.

However with each of these variations,  studies and developers [5] have both shown that pair programming is best for solving new, previously unseen problems. This is indicative that pair programming one hundred percent of the time may not be appropriate.

Reaping the Benefits

The aim of pair programming is to improve code quality and efficiency while simultaneously providing an outlet for developers to learn from each other.

Studies have shown that even pair programming by two novices is more productive than a solo developer coding on their own [3]. There are also definitive results that clearly show that code quality improves when pair programming is used. It has been shown to reduce defects by up to 15% [1]. Having the code the driver is writing be constantly reviewed by the observer leads, not only, to more errors being caught immediately. It is also more efficient to constantly review the code than to only review upon completion.

One of the nicest benefits of pair programming is team building.  It encourages better communication and collaboration. Programming is often depicted as a solitary task and developers as shy, quiet people. Pair programming, however, forces developers to actively work together which can lead to more productive teams, not just individuals . Most of the studies referenced below, point this out [1][2][3].

However, one key factor that is evident from all these studies is that they don’t seem to have considered what pair programming is like if it is being carried out one hundred percent of the time, all day, everyday. While it is true that developers don’t code all the day, there are a number of scenarios where pair programming one hundred percent of the time may not be the best course of action.

100% Pair Programming: Yea or Nay

While empirical studies prove that pair programming is one of the best things to happen in software development, I am not convinced by them. The one factor that they do not seem to take into account a great deal is  the ‘human’ factor.

“In the name of (short term) productivity” – Mark Needham [5]

The implementation phase of a software project is not all just about overcoming challenges. All projects have mundane and often trivial code that needs to be implemented to carry out the most basic of tasks. The problem is that the observer will just have to observe this dull, trivial  process without being able to contribute anything useful. Their skills could have been put to better use in implementing a critical piece of code, or another piece of trivial code which had yet to be implemented. The approach of pairing one hundred percent of the time can lead to a waste of a developers time and talent. This could slow down the development as there may be many other pieces of code yet to be implemented which could be implemented by the observer. This argument still holds when the developers switch roles.

Pairing all the time is also a terrible way for a developer to learn to use any development tools. Watching the driver make use of various tools, or watching them figure out how to use these tools is not educational. Some tasks need to be completed by the learner themselves to be able to fully understand them. This is similar to a person claiming they can play basketball because they have watched a few games on TV.

In addition to this, there is a time for individual work unhampered by the need when mundane tasks.

“You don’t want to waste their time. You don’t want to argue (unless the other person wants to as well). You give in more often than if you were working alone.” – Mark Wilden [6]

Pair programming depends on the abilities of two developers to successfully collaborate and work togethe. The studies below all assume that people will get along with each other the entire time. Perhaps this should be the case in a work environment as professionals are expected to put aside their feelings, but realistically that this is simply not the case. This is not implying that they may resent each other (they might), but they may disagree a lot when it comes minor design decisions. This may slow down the pace of implementation and in some cases lead to one developer believing that their opinion is not correct or worth taking into account. This can be quite demoralising, leading to feelings of resentment. This could be prevented if pair programming weren’t being carried out constantly and only major design decisions were discussed, leaving the minor decision to the coder’s discretion.

“Pair programming doesn’t encourage quiet reflection and exploration.” – Mark Wilden [6]

As a developer, getting a feel for the language and codebase is quite important when it comes to implementing designs. Just as children learn by trial and error, developers learn by exploring the code. Many innovative ideas can also stem from exploring the codebase; refactoring certain classes for example. However, if a developer never gets enough time to work their way through the code or language then they can never truly get comfortable with it.

The Bottom Line

Like Wilden and Needham, I think pair programming should be used in moderation. In my own limited experience as a developer, I faced days where my productivity was low. It could be argued that on these days pair programming all day would have helped this, but I don’t think it would have. Almost everyone who has has some work experience can relate to days where your brain does not seem to want to function properly. The mind doesn’t think straight; logic, which was always like second nature is baffling and even words escape you. On these days a developer just wants to get through the day. These unproductive days have always been compensated with extremely productive, slightly longer, working days. However if forced to pair program all day on such days then I’d have to say that it most likely would not have resulted in feelings of hatred, however it may have led me to associate negative feelings with programming with my pairing partner. This is not a failing on the concept of pair programming alone, in fact pair programming in small increments may help, but on human nature.

Pair programming is incredibly beneficial and should be used in moderation especially when used to exchange knowledge. It can make a great coaching tool and when used to solve a new, unseen problem it can help teams work together to design innovative solutions .  However pairing one hundred percent of the time should be approached with caution to ensure that colleagues do not develop mutual feelings of annoyance towards one another.

References

[1] D. Winkler, M.Kitzler, C.Steindk, S.Biffl (2013) ‘Investigating the Impact of Experience and Solo/Pair Programming on Coding Efficiency: Results and Experiences from Coding Contests’, Agile Processes in Software Engineering and Extreme Programming [Online]. Available at:http://download.springer.com/static/pdf/742/chp%253A10.1007%252F978-3-642-38314-4_8.pdf?auth66=1392496570_5aef1fdad11e1c3331f4cd4351cbf951&ext=.pdf.

[2] Erik Arisholm, Member, IEEE, Hans Gallis, Tore Dyba˚, Member, IEEE Computer Society, and Dag I.K. Sjøberg, Member, IEEE (2007) ‘Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise’, IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 33(2), [Online]. Available at:http://simula.no/research/se/publications/Arisholm.2006.2/simula_pdf_file.

[3] Kim Man Lui, Keith C.C. Chan (2006) ‘Pair programming productivity: Novice–novice vs. expert–expert’, [Online]. Available at:http://www.cs.utexas.edu/users/mckinley/305j/pair-hcs-2006.pdf.

[4] Alistair Cockburn, Laurie Williams (n.d.) ‘The Costs and Benefits of Pair Programming’,[Online]. Available at:http://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia.PDF.

[5] Mark Needham (2011) Pair Programming: The disadvantages of 100% pairing, Available at: http://www.markhneedham.com/blog/2011/09/06/pair-programming-the-disadvantages-of-100-pairing/ .

[6] Mark Wilden (2009) Why I Don’t Like Pair Programming (and Why I Left Pivotal),Available at: http://mwilden.blogspot.co.uk/2009/11/why-i-dont-like-pair-programming-and.html.