How to find underlying reason from user feedback to make your software better

1. Introduction

Everyone knows the importance of collecting user feedback of your software product. The simple reason is that user feedback reflect their satisfaction and user experience of your software product, and these can have big impact of sales of your software product.  Many software products failed because their developers have not done a good job of collecting useful feedbacks from their users. This article is going to talk about different ways to collect user feedback, analysis the advantages and disadvantages of these ways, and how to improve the effect of user feedback. I will propose some ideas to help us find underlying reason from user feedback to make our software product better.

2.Existing ways to collect user feedback

2.1 Using feedback form

The most common way to collect user feedback is feedback form. Companies post some questions on their mobile phone applications, websites and so on.

Figure 1. Mobile Feedback Form [1]


Figure 2. Website Feedback Form [2]

Advantage of feedback form is that it does not cost much money and you can collect feedback directly from your existing user. Disadvantages of using feedback form are very low rate of collecting user feedback. Although some companies try to use some reward to attract their users fill in their survey, this method can lead to another problem. Many users just fill in their survey very quickly and randomly to get the reward, this can make the result of feedback forms wrong and cause the company greater losses.

2.2 Using third-party service

Some company uses third-party service to collect users feedback. For example, Google Analytics[3] is one of the best free analytics applications for getting basic traffic and visitor data. You can use Google analytics to monitor the user’s activities of your website or mobile apps every day, such as number of visit, where your users from and so on.

Advantage of using third-party service to get user feedback is that you can get every user of your software product anonymously, and the result of the feedback is more reliable.  The disadvantages are you need to pay extra money if you need premium service, and service can not be customized by yourself if third-party company does not provide this service. Of course, data security is another issue you have to consider.

2.3 Eye tracking technology

Academia and business have big interest of eye tracking technology. Eye tracking is used in research as a means for insight into human behavior [4]. For example, Tobii Mobile Device Testing [4] solution lets you study how users experience mobile websites and apps or how they consume mobile ads on mobile devices and tablets [5].


Figure 3. Mobile Device Testing by using eye tracking technology[5]

Advantages of eye tracking technology include it is a very trendy technology and the future development of this technology are very broad. Disadvantages of eye tracking technology include it just give you answer of what area users are watching, but it does not give you why user look at this area. Correctness of this technology should be tested and verified.

3.How to find underlying reason from user feedback

One challenge of collecting user feedback is that we need to make sure the representativeness and accuracy of data. When we ask our old users why they are no longer use our software product, they always use “no time” and “no money” as a reason to get rid of us, but the real reasons are always unknown.  How could we know it is the real reason that our user like or dislike our software product? We need to find underlying reason instead of superficial reason from user feedback.

3.1 five why’s

Five why’s is invented by IDEO [6], which is a global design firm that takes a human-centered, design-based approach to helping organizations in the public and private sectors innovate and grow. Idea of this five why’s method is very simple: asking your user five questions about an issue continuously to get the deep and progressive answer, and this method allow your user express their deep thoughts and reasons. Here is an example of five why’s method from IDEO as follows, and this example shows the true reason of user exercise:


Figure 4. an example of five why’s method from IDEO [6]

When we test our software product, five why’s method can become as follows:

Game developer: “Why don’t you log in our  game?”
User: “I have no money and no time.”
Game developer: “Do you play other games which are similar to our game recently?”
User: “Yes, but I am not very interested in it.”
Game developer: ”What kind of game you like now?”
User: ”I guess some sports game.”
Game developer: ”Why you like sports game now?”
User: “I feel very exciting when I can create my own role in the game, and I feel myself play in the court.”
Game developer: “What differences do you think between our game and sport game?”
User: ”I think the sport game soundtrack are much cooler than your game, their 3D effect really exciting and I can play with my friends online. Your game does not have these functions.”

After these five question, game developer finally find the real reason of user does not play their game.  User play other games because they like better game soundtrack, visual effect and multiplayer game.

3.2 Behavior Analysis

When we use user feedback survey on the internet, it is difficult to ask questions to get deep reason. It is hard for user to answer the why they behave like this instead of that, but it is easier for them to answer with their behavior. Furthermore, it is more reliable to answer questions about behaviors. There are some questions we don’t need to ask our users directly to get the deep reason, instead, we can just analysis their behaviors to get the underlying reason.

3.3 Indirect comparison of other related products

Sometimes users are unaware of the real reasons behind their actions. We may not get specific reasons if we ask questions directly. So we can use indirect comparison of other

related products to inspire users to dig out the underlying reasons.

I will show you an example as follows:

Interviewer: why don’t you use our premium service?
User: “I have no money.”
Interviewer: “Are you still use other premium service?”
User: “Yes. I use premium service from X company.”
Interviewer: ”Why you use X company’s premium service instead of our premium service?”
User: “Because I think their service are more useful, and I have more privilege.”
Interviewer: “What privilege? Could you give us an example?”
User: “Unlimited downloading”

This use case shows us it is not because user have no money that he does not use our service. It is because our premium service is unattractive to users, and user want unlimited downloading privilege.


In this article, we talked about different ways to collect user feedback which are used by today’s technology company. However, after analysis these different ways, we find they all have their disadvantages. So we proposed some ideas to improve the accuracy of user feedback to make the result more reliable. These ideas could be rules to follow in the future if technology company want to design their method to collect user feedback. These ideas can help software company find the underlying reason why or why not the user like their product, and these more reliable user feedback could be guidelines to follow if they want improve their existing product or develop new products.


[1] Mobile Feedback Form. 2014 Neemware Inc.

[2] Preparing for a Site Redesign. March 20th, 2013. Bourke Design.

[3] Google Analytics. 2013.

[4] Tobii. 2013

[5] Tobii Mobile Device Testing Solution. 2013 Tobii Technology.

[6] IDEO

Response to “Using Social Networks to Analyse the Stakeholders of Large-Scale Software Projects”

1. Introduction


This is a response article to “Using Social Networks to Analyse the Stakeholders of Large-Scale Software Projects”.

In this article, the author proposed an concept of using social networks to analyse the stakeholders of large-scale software projects[1]. He also showed an example to explain this idea.  Furthermore, he demonstrated his own opinions about this concept.

I am going to talk about advantages and disadvantages of this article, and I think more things should be considered if we want to use author’s concept in reality. These things can make the result more reliable and correct.


2. Discussion about advantages


I think the author chose a very good topic, and he explained this concept very clearly by demonstrating a example. This concept is mostly based on the paper about StateNet [2]. I think this topic is very useful because lack of user involvement is the main cause of project failure, and success is rare [4]. Reports suggest that 34% of projects succeed in 2004, 35% in 2006, and 32% in 2009[5]. The author used subtitle to make the logic of this article very easy to follow. The author clearly stated his position in the conclusion part of the article. I totally agree with the idea that overlooking stakeholders is possibly the most common mistake in development efforts [6].

3. Discussion about disadvantages


However, I thinks there are some disadvantages of this article as follows.


3.1 Structure should be more upfront


The author spent most of his article to explain his concept, and he talked about his own opinions about this concept in the conclusion. I think his introduction section should mirror his conclusion. I think the structure of article could be improved.


3.2 More personal opinions rather than explanation of the concept


Like I said before, I think the author spent too much of his article on explaining the concept. The author proposed his own idea in the last section, which are

  1. Prioritise the stakeholders with some measures due to their importance

  2. It allows every stakeholder to have a say in this, so all people are considered in deciding the importance of all others [1].


The author talked about two advantages of this concept. However, I think the author may think about more about this concept, and it could be disadvantages of concept, what other areas can be used, future work, different criteria to evaluate the result and so on.


4. More things could be considered


To make the result more reliable and objective, I think more things should be considered.


First of all, the number of stakeholders of a software project is dynamic, in other words, the number of different roles of stakeholder is always change as well. This dynamic process should be considered if we want to build a weighted graph. Perhaps we can use other theories to make our result more reflective of real situation, such as fuzzy set theory [3], which can solve the fuzziness and uncertainty of big and dynamic data.

Secondly, the author believes his concept allows every stakeholder to have opinions, which can decide the importance of all others. In my opinion, if every stakeholder can have opinions which decide others’ importance, the objectivity of the results will be questioned. Relationship between different stakeholder should be considered.

Thirdly, the consistency of role descriptors should be considered and this could be a problem if different stakeholders have different opinions about a same stakeholder. Furthermore, one people could have different roles in real life, so how to describe him/her in the social network should be considered as well.

Finally, building a new social networks takes time and effort, and the accuracy and representation of new social networks is not sure. We have a idea that should we build the new social networks based on useful information from existing completed networks? We can use meaningful data, remove dirty data and add new data to build the new social networks. We believe this method could be more efficient and accurate.

5. Conclusion


In this article, we first made a brief introduction of Awad’s article [1]. Then, we discussed the advantages and disadvantages of this article. Finally, to improve the reliability and objectivity of the result if we use author’s concept in really, we believe more things should be considered, and we talked about these things in details.




[1] Using Social Networks to Analyse the Stakeholders of Large-Scale Software Projects. Nadi AWAD. February 17, 2014.

[2] StakeNet: Using Social Networks to Analyse theStakeholders of Large-Scale Software Projects. Soo Ling Lim, Daniele Quercia, Anthony Finkelstein. In Proceedings of the 32nd International Conference on Software Engineering, ICSE (1) 2010, pages 295-304

[3] Chen, Shouyu. philosophical basis of variable fuzzy set theory. s.l. : The Journal of Dalian University of Technology , 2005. 53-57.

[4] Standish Group. The CHAOS Report, 1994.

[5] Standish Group. CHAOS Summary 2009, 2009.

[6] D. C. Gause and G. M. Weinberg. Exploring Requirements: Quality Before Design. Dorset House Publishing, 1989.

Developers and testers: friends or foes? Using fuzzy set theory to decide on the correct ratio of developers to testers in agile teams

1. Introduction


Developer and tester are two essential roles in the development of large-scale and long-term software. In the past few years, there are a lot of discussion of developers and testers. These discussion reflect that software industry develop very fast and the importance of these two roles. However, fight and cooperation never stop between developers and testers in the real world.  In this article, we will talk about developers’ and testers’ work, and how to decide on the correct ratio of developers to testers in agile teams.


2. What a developer’s and a tester’s day really like?


Everyone has their own impression of these two roles in their mind according to their experience. Even talking about the same role, it usually differ because of the size of the company which they work for, the tasks they should finish and so on.


2.1 What a developer looks like?


There is no simple answer to describe a developer’s life. It depends on what levels you are and the environment of you work (the size of the company you work for, academia vs Industry and so on). Talking about different levels of a developer, there is an interesting article talking about “eight levels of programmers” [1] . This may give you an impression of different developers. In this article, we will focus on the average developers in industrial companies.


2.2 What a tester looks like

I am going to give you a brief description of testers based on the experience of my friend who work as tester in his real life.

I have a friend who work as Software Development Engineer in Test in Google. When asking the first impression of his work, he said it is very cool and “luxury”.

Talking about cool, he has opportunity to use the most advanced technologies in Google, such as Google Glass. He thinks his job is “luxury” because of his work environment. He has two advanced computers in his office. He has two 21-inch LCD monitor, one is for debugging and the other is for displaying the interface of product. If your task is testing mobile phone, congratulations! You will have many latest mobile phones and tablets from different companies for your testing, and these can occupy all the space in your table if you want. He thinks these devices help him improving the efficiency of his work, taking into account a variety of complex configurations and simulating customer environments when using their products.


Someone always ask him that as a tester, you do not need to write codes, right? His answer is: of course I need to write code. He used to test thousands of use cases for a module, and it is impossible to finish the testing by hand before the deadline. He need to write a program for testing automatically.


2.3 Good management of developers and testers


Good management of developers and testers in the development of large-scale and long-term software is like manufacturing good pair of spear and shield. Testing is spear and developing is shield. If we know the strengths and weaknesses of spears and shields in the early stage of manufacture, we can make the weaknesses become strengths.

figure 1: figure from [6]

3 How to decide on the correct ratio of developers to testers in an agile team


There is a long-standing problem in the software development world: what is the reasonable ratio of developers to testers in an agile team? In the real world, the developer to tester ratios range from 1:1.5 to 10:1, including organizations, such as Microsoft, which have ratios that are 1:1 [3]. You can see that if you have an agile team to develop a project, you cannot just choose the ratios from other companies. What you should do is analyze your own situation and think about what ratio is best for your team. However, the question is: what criterias you need to use to analyze your situation?


3.1 Johanna Rothman’s theory


Johanna Rothman gives us an answer.[2] He believes that you need to consider:


  1. Product(the requirements, product size, and complexity of the product)

  2. Your project and its process (how your organization develops products, and your customers’ tolerance for defects and ship delays)

  3. Your people (development and test staff’s abilities and responsibilities) [2]


This is the general aspects you need to consider of your product. Furthermore, he gives us a table to evaluate  three different projects as follows:

figure 2: Contrast Process and Project Attributes[2]


We think this is a very useful table with detailed criterias to evaluate a project, and it gives us important information to decide on the Correct Ratio of Developers to Testers in agile teams. In this table, Johanna uses ü to describe a certain company has the issue or not. For example, LineChecker has to define and inspect the product architecture,  but BussyApp and DataCrunch do not have to.  He believes every company will get a score as a result according to their answers of this table, which decide their ratio of developers to testers in their agile teams.


However, we believe that although the criterias Johanna provided are very reasonable, situation in the real world are more complex. You cannot easily say that a project has or has not a certain criteria, such as low customer tolerance for ship delay. The real world situation is that our answer is not just yes or no, we have different levels of our answer, such as: very positive, positive, moderate, negative and very negative. To make the result more reliable, we will use a theory called fuzzy set theory.


3.2 What is fuzzy set theory?


Fuzzy sets [4] were introduced simultaneously by Lotfi A.Zadeh and Dieter Klaua in 1965. What Zadeh proposed is very much a paradigm shift that first gained acceptance in the Far East and its successful application has ensured its adoption around the world. At the beginning of twenty –first century, based on the fuzzy set theory, variable fuzzy set theory [5] is proposed by professor Chen. With the development of fuzzy set theory, it is meaningful to introduce fuzzy set concept, models and approach into data analysis of complex situation.Fuzzy set theory is an important tool in mathematical modelling, which allows for the explicit consideration of professional expertise and judgment. It will be necessary to analyze the mutual relationship between fuzzy measures and their applications. Based on the main concepts underneath fuzzy set theory, we can make the ratio of developers to testers in agile teams more reasonable.


3.3 How to use fuzzy set theory to decide correct ratio of developers to testers?


Since the data collected from Johanna’s criterias are not in a perfect quality, using fuzzy set theory are capable of representing uncertain, non-exact, diverse and inaccurate data or information. We do not want to use many fancy formulas of fuzzy set theory to scare away our readers, so we try to explain the use of fuzzy set theory in this situation as simply and clearly as possible. We redesign the table of criterias which presented be fore as follows:

figure 3: table for evaluate your own project


You can see from the table above that for each criteria, you can have five choices: completely agree, agree, moderate, disagree and completely disagree. Each option has its corresponding score. After using this table to evaluate your project, you will get a much reasonable result compare with the old version(just yes or no answer).


We will make it clear that this ratio is not stable. It can be changed according to the performance assessment. You can add different criteria to evaluate your developers and testers.


Maybe someday in the future, testers and developers will become one role, and there is no obvious distinction but each person’s tasks will be different.


4. Conclusion

In this article, we talked about the developers and testers in the large-scale and long-term software developing team. We provide a new idea of using fuzzy set theory to decide on the correct ratio of developers to testers in agile teams. To explain how to use fuzzy set theory, we showed a example of table including different criteria to evaluate readers’ project.


[1] The Eight Levels of Programmers, April 3, 2009 Jeff Atwood

[2] Johanna Rothman on Jan 1, 2000 in Articles

[3] Cusamano, Michael A. and Richard W. Selby, Microsoft Secrets, Simon and Schuster, Inc., New York, 1995.

[4] Zadeh. Information and Control. 1965.

[5] Chen, Shouyu. philosophical basis of variable fuzzy set theory. s.l. : The Journal of Dalian University of Technology , 2005. 53-57.

[6] Stock Photo – Old shield with sword isolated over white