Issues and challenges large-scale system development


This  article reflects on the issues and challenges large-scale system development face. Software is hard to engineer on a small scale, but at a larger scale, engineering and management tasks are even more difficult. In the context of software product line evolution, the goal of this work is to look at current managing practice, through the lens of Systems Thinking. They develop a System Dynamics model to operationalize the notions examined here and run a variety of experiments representative of real situations, from which we learn some lessons and recommend policies that engineering leaders may use to manage large-scale software development organizations. Since large-scale development is an enormous subject, there are two main problems. First, large software projects are almost universally troubled, and second, all large-scale systems-development projects of almost every kind now involve large amounts of software. Unfortunately, this implies that almost all kinds of large-scale development projects will be troubled unless we can devise a better way to develop the software parts of these large-scale systems. The increasing need for large-scale system-development projects raises many questions and presents a significant challenge to those of us in the development business.

In today’s fast-moving world, successful software companies need to grow continuously in revenue, which translate into growth in headcount, market share, product feature set and product line-up. These also form the basis on which software companies compete.Companies that remain small typically merely cater to a niche in the larger market:

This is success by one measure, as such niches might indeed be quite large, but this is not the kind of development effort we are interested. Here we are specifically looking at what happens in larger efforts, usually multi-year projects employing hundreds or thousands of engineers. This is the environment the majority of my work experience comes from, and even though managers know instinctively that there are limits to growth, limits to how many features one can cram into a single release of the product, that freighting is best avoided, that projects easily spiral out of control if the organization promises more than it can reasonably be expected to deliver, all of these things still frequently do happen.

Emergent Properties of Systems

Perhaps the greatest single problem with large-scale system development concerns what are called the emergent properties of these systems. These are those properties of the entire system that are not embodied in any of the system’s parts. Examples are system security, safety, and performance. While individual components of a system can contribute to safety, security, and performance problems, no component by itself can generally be relied on to make a system safe, secure, or high performing.

The reason that emergent properties are a problem for large-scale systems is related to the way in which we develop these systems. As projects get larger, we structure the overall job into subsystems, then structure the subsystems into products, and refine the products even further into components and possibly even into modules or parts. The objective of this refinement process is to arrive at a series of “bite-sized projects” that development teams or individual developers can design and develop.

This refinement process can be effective as long as the interfaces among the system’s parts are well defined and the parts are sufficiently independent that they can be independently developed. Unfortunately, the nature of emergent properties is that they depend on the cooperative behavior of many, if not all, of a system’s parts. This would not be a problem if the system’s overall design could completely and precisely specify the properties required of all of the components. For large-scale systems, however, this is rarely possible.

While people have always handled big jobs by breaking them into numerous smaller jobs, this can cause problems when the jobs’ parts have inter-dependencies. System performance, for example, has always been a problem, but we have generally been able to overpower it. That is, the raw power of our technology has often provided the desired performance levels even when the system structure contains many inefficiencies and delays.

As the scale of our systems increases, and the emergent properties become increasingly important, we now face two difficult problems. First, the structural complexity of our large organizations makes the development process less efficient. Since large-scale systems are generally developed by large and complex organizations, and since these large organizations generally distribute large projects across multiple organizational units and locations, these large projects tend also to have complex structures. This added complexity both complicates the work and takes added resources and time.

The second problem is that, as the new set of emergent properties becomes more important, we can no longer rely on technology to overpower the design problem. Security, for example, is not something we can solve with a brute-force design. Security problems often result from subtle combinations of conditions that interact to produce an insecure situation. What is worse, these problems are rarely detectable at the module or part levels.

The Tropical Rain Forest

The fundamental problem of scale is illustrated by analogy to the ecological energy balance in a tropical rain forest. In essence, as the forest grows, it develops an increasingly complex structure. As the root system, undergrowth, and canopy grow more complex, it takes an increasing percentage of the ecosystem’s available energy just to sustain the jungle’s complexity. Finally, when this complexity consumes all of the available energy, growth stops.

The implication for both projects and organizations is that, as they grow, their structure gets progressively more complex, and this increasingly complex structure makes it harder and harder for the developers to do productive work. Finally, at some point, the organization gets so big and so complex that the development groups can no longer get their work done in an orderly, timely, and productive way. Since this is a drastic condition, it is important to understand the mechanisms that cause it.

Organizational Growth

In principle, organizations grow because there is more work to do than the current staff can handle. However, this problem is usually more than just a question of volume. As the scale increases, responsibilities are subdivided and issues that could once be handled informally must be handled by specialized groups. So, in scaling up the organization, we subdivide responsibilities into progressively smaller and less meaningful business elements. Tasks that could once be handled informally by the projects themselves are addressed by specialized staffs. Now, each staff has the sole job of ensuring that each project does this one aspect of its job according to the rules. Furthermore, since each staff’s responsibility is far removed from business concerns, normal business-based or marketing-based arguments are rarely effective. The staffs’ seemingly arbitrary goals and procedures must either be obeyed or overruled.

This growth process generally happens almost accidentally. A problem comes up, such as a missed schedule, and management decides that future similar problems must be prevented. So they establish a special procedure and group to concentrate on that one problem. In my case, this was a cost-estimating and planning function that required a plan from every project. Each new special procedure and group is like scar tissue and each added bit of scar tissue contributes to the inflexibility of the organization and makes it harder for the developers to do their work. Example staffs are pricing, scheduling, configuration management, system testing, quality assurance, security, and many others.

One of the most critical aspects of managing large-scale projects is making sure that decisions are properly made. The executive’s responsibility must be to identify the right people to make the decision, insist that the goals used for making the decision be defined and documented, and require that the criteria for the decision be established. While there are far too many technical decisions in large-scale projects for management to require that they all be made in this way, there are a relatively few times when technical decisions are escalated to senior management. However, whenever they are, these decisions are almost certainly technical issues that have become political.

If the executive does not insist that each of these politically tinged technical decisions is properly made, he or she is likely making a very big and possibly fatal mistake. If ever, this is the one time when the executive should insist that the decision be made in the right way. While these decision situations always come up when there is no time and when everybody, including the executive, is in a rush to get on with the job, this is precisely the time when proper decision making is most important. When executives insist that rush decisions be made in the proper way, they are demonstrating their ability to be technical executives. Therefore, the first two ground rules for the proper management of large-scale projects are the following: Insist that all technical decisions be made by the proper technical people.Make sure that, in making these decisions, the technical decision makers thoughtfully evaluate the available alternatives against clearly defined criteria.


In this article I discussed some of the development issues related to large-scale projects. Since this is an enormous subject, I cannot hope to be comprehensive but i discussed few issues.


Cloud computing


“What’s the cloud?” “Where is the cloud?” “Are we in the cloud now?!”

In the simplest terms, cloud computing means storing and accessing data and programs over the Internet instead of your computer’s hard drive. The cloud is just a metaphor for the Internet. It goes back to the days of flowcharts and presentations that would represent the gigantic server-farm infrastructure of the Internet as nothing but a puffy, white cumulonimbus cloud, accepting connections and doling out information as it floats.

In science, cloud computing is a synonym for distributed computing over a network, and means the ability to run a program or application on many connected computers at the same time.

Cloud computing is the next stage in the Internet’s evolution, providing the means through which everything — from computing power to computing infrastructure, applications, business processes to personal collaboration — can be delivered to you as a service wherever and whenever you need.

Cloud computing has four essential characteristics: elasticity and the ability to scale up and down, self –service provisioning and automatic deprovisioning , application programming interfaces (APIs), billing and metering of service usage in a pay-as-you-go model. This flexibility is what is attracting individuals and businesses to move to the cloud.
The world of the cloud has lots of participants:
• The end user who doesn’t have to know anything about the underlying technology.
• Business management who needs to take responsibility for the governance of data or services living in a cloud. Cloud service providers must provide a predictable and guaranteed service level and security to all their constituents.
• The cloud service provider who is responsible for IT assets and maintenance.

The benefits (pros) & advantages of cloud computing


Having your data and business computing programs running online, rather than exclusively on your office computers, means that you and your staff have access to them anytime, anywhere there’s an Internet connection. Small businesses like the idea of being able to access their data from home, at a client’s location, on the road, or even on a smartphone. And of course, if you know you will have to work disconnected, you can load the files you need onto a hard drive, a USB flash stick/drive or better yet use another offline/online cloud computing tool . Many of the more common cloud computing offers out there, such as Google Gears and Adobe AIR, make it possible for some Web-based programs to be used on a computer even when you’re not connected. That basically took one aspect that may have been viewed as a limitation and made it a huge plus.

Some of the more basic benefits/pros of cloud computing include:
• How it frees up physical space – Think of remote data storage, servers you’d otherwise have to to keep in house, etc.
• Cloud computing eliminates the needs for maintaining a cool enough environment for computer hardware that can easily overheat and crash.
• Saves electricity – in larger organizations this alone may be worth it.
• Eliminates the cost of maintaining one’s own IT staff to update and upgrade software or hardware because the data center handles that.
• Software & Hardware maintenance – Software provided online is upgraded and maintained by the provider, so the small business owner does not have to purchase the newest version of a software program or download fixes and patches for this or that server, router or hardware. Not having to buy a program outright but entering into a monthly or annual contract is very appealing, as is the fact that many applications are offered for free. The fixed cost allows business owners to plan rather than be caught off-guard when a costly software or hardware upgrade must be purchased.

For me personally, cloud computing has provided a sense of work-life balance. My personal budget, without any account numbers or anything except debits and credits, is in the cloud shared with my parents to simplify budgeting & updating. My to-do list is on the cloud. I email strictly now via the cloud. And best of all, my calendar is in the cloud. This affords me the flexibility to coordinate things between my personal life (more like several lives) and work. But enough about me.

The risks (cons) & disadvantages of cloud computing


The chief executive of leading security company Check Point, Gil Shwed, has warned enterprises from plunging too quickly into cloud computing, stating that the technology is inherently dangerous.

“The basic fact that you’re sharing data with others is a huge security risk. If I have my server with my data at least I know that I can protect my data, but if I mix hundreds of these [servers] there might be one bug, in one application, that will allow a virus to move to the others. A bug, a configuration error, anything.”

Shwed did not wish to discourage the move towards the cloud, however.

“I am definitely a supporter of cloud computing, but I don’t think it will take over from traditional computing – it will have some portion of the market. One of the things the enterprises want is control, and once you outsource … you lose a big portion of that control, so companies will only do this if there is a very, very, good reason,” he said.

“There is a clear and interesting challenge there, the move from controlling things to letting other people control it for you.”

While I don’t want to discount Mr. Shwed, you have to take his perspective into consideration. He’s in the security business. Naturally, it’s all about control; as evident by his quotes. As a small business or even a larger one (several larger ones actually), as I will mention in my conclusion below, some things are OK to relinquish control of provided the risks are understood and perhaps even mitigated. For example, I’m OK with relinquishing control of my email for my business or for most businesses I work with to cloud computing. The convenience and cost benefits out weight the control and risks (especially when they’re mitigated).

One of the biggest benefits of storing your data in the cloud is that you don’t have to worry about backing it up anymore. Big companies with hundreds of servers are more reliable than your little external hard drive, right? Yes. But servers do go down, and when you’re dependent on a web application to get your email or access that presentation for the big meeting, there’s always the risk that your internet connection will go down, or that the webapp’s servers will. Offline technologies like the previously mentioned Google Gears, decent export functionality, and a good backup system can alleviate this particular concern, but not all systems offer those things. Again, it’s about understanding the risks and mitigating them.

In conclusion …
It’s interesting that the government is behind cloud computing and has recently, a catalog of applications in the cloud. Despite the challenges of cloud computing, many large organizations are not only evaluating it, but are jumping to adopt it. Jaguar Land Rover moved to Google Apps. And the Los Angeles City Council today (if I wasn’t ill to write this on time – 2 days ago) voted unanimously to “Go Google,” (like Jaguar Land Rover) approving a $7.25 million contract to outsource the city’s e-mail system to Google’s cloud and transition some 30,000 city employees to the cloud over the coming year, according toa report in the Los Angeles Times.

In case you haven’t noticed, I personally am right on the cloud bandwagon with all of you. My web browser is the one app I run on my desktop at all times; I’ve entrusted the likes of Google, Facebook, Twitter, Dropbox, Amazon, and Yahoo with my data just like you have. The key is to know what you’re getting into when you make that choice, to crank up your personal security (like alternate email addresses and password choices) and to lobby for better user protection by hosting providers in the cloud.
It’s clear that cloud computing is a wave of the future. Ultimately, putting your data in the cloud involves choosing convenience and productivity at the cost of some security risk. In the real world, convenience almost always wins, and there’s nothing wrong with that. What’s important is that you understand the dangers. All you have to do is exercise some good judgment and practices and you ought to be able to alleviate many of the risks and challenges of cloud computing, leaving you to enjoy its many benefits.

Cloud Computing currently shapes the thinking of business modeling, business consulting, solution creation, service delivery, and software architecture design, development and deployment.