How to Effectively Estimate the Effort for Software Development Projects.
Part 1

In this article, project estimation entails the estimation of the work effort. The estimation of other expenses, such as premise leasing, hardware/software, and various licenses will not be included in this article.

The basic problem of estimation lies in a contradiction between the principal uncertainty of the future project parameters and the necessity to make decisions and plan the work.

We live in the world of tenders, annual budget cycles, ROI, and other concepts and practices that force us to make investment decisions in the lifecycle of the project as early as possible. This is when the risks of under- or overestimation are at their highest. Additional pressure can be made towards the project executors regarding minimizing the estimates and accepting even bigger risks.

It is essential to consider this basic contradiction during the process of project effort estimation and in subsequent phases.

Here are a few situations when you need to estimate the effort of the future software development project:

• Negotiations with the client regarding a new project or an additional project’s phase
• Response to RFP (request for proposal) or participation in a tender
• Planning of the budget and other company’s resources
• Calculating project ROI (return on investment) or making a go/no-go decision about the project launch

If you are required to estimate the effort at the stage of business requirements gathering, here are several ways of minimizing the risk:

• Based on experience gained during previous projects, estimate the maximum possible work scope even if it is not requested by the client.
• Provide the estimation interval (from… to… ), thus emphasizing the level of uncertainty and risk at this project phase. This will allow project sponsors to make a considered decision about risk acceptance.
• Develop the project iteratively by reviewing forecasted numbers on the way.

Apart from calculating the effort and budget, it is important to invest in justification of forecasted numbers and risk minimization.

Here are the key objectives of project estimation:

• Determine and limit the scope of work and establish the assumptions the estimate is valid with
• Determine the project price and cost
• Justify the project budget in front of all stakeholders
• Receive input data for other processes: sales, planning, recruitment, budgeting, etc.

Before we dive into the topic, let’s see an experiment.

During my workshops dedicated to effort estimation, I ask the audience to estimate the effort for developing a simple business application for selling flowers with the functionality of order tracking, order completing, payment and storage of the customers’ basic data.

As a result, even senior developers and architects who have been working together on the same platform, using the same input data, staying in the same location, and reading the same text, deliver absolutely different estimates. Though I provide detailed enough and realistic requirements.

Covering the entire scope of work and potential risks is an important part of the art of project effort estimation. Another critical part is communication with all project stakeholders.

Any project has its estimators and executors. Let’s discuss in detail these two project roles.

A perfect estimator is a software engineer in a senior or higher position with experience of unsuccessful estimation and project failures. This person must have a deep understanding of sales, estimation, planning processes and the principles of project execution.

At the same time, the estimator has to be psychologically stable without fear of being punished for under- or overestimation of the project tasks. Otherwise, they can be inclined to provide more pessimistic estimates or even refuse to provide them at all. The team will be happy to manage time according to their needs, which can lead to the budget overrun.

This phenomenon is clearly laid down in Parkinson’s law, which states that “work expands so as to fill the time available for its completion.” An experienced developer is always happy to find a way to spend 40 hours on a task that needs only 16. This means that overestimation can also impact actual labor effort, and the client’s trust can significantly suffer.

Even the most experienced specialists can be wrong. A cognitive bias, which prevents us from thinking rationally, is one of the reasons why people make mistakes. Here, we will outline three biases directly influencing the process of estimation.

Planning fallacy and estimation optimism

We want to think that we are better than others. This is why we estimate our own tasks with more optimism than the similar tasks of other team members. A good estimator has to adequately gauge their own strength and be ready to resist psychological pressure (such as the question “Why do you need so much time?”) and clearly argue their position.

Also, even experienced engineers are prone to set optimistic estimates. Based on the empirical observation, on average, engineers underestimate projects by 30%. Unfortunately, this estimation optimism is often supported by managers and other project stakeholders.

Probabilities estimation mistakes

Actual time for task completion is a random value. Even people who have completed a probability theory course often can’t properly operate probabilities. That is especially true for conditional, very big and very small probabilities, where, let’s say, insurance companies and lottery owners can greatly benefit. For example, learn more about the Monty Hall problem. Also, it can be hard for the estimator to differentiate the expected value of effort from the most probable effort needed for task completion.

To minimize the effect, discuss with the engineer what you mean by an optimistic, realistic, and pessimistic estimate of task completion.

Anchoring effect or focalism

In the back of their mind, people strive for the values they hear from someone or see somewhere. Marketing specialists are aware of this phenomenon and make use of it. In the realm of our topic, this can mean the following:

• If you want to receive independent estimation values from all estimators, do not share the result you receive from each of them.
• Do not provide any hints, expectations, starting points, or budget options before the estimator provides their own independent estimates. Otherwise, the estimate will be greatly influenced by your input data.

Another difficulty is that while estimators are often senior specialists, the tasks have to be completed by middle and junior teammates based on the estimation results. According to research, the productivity of engineers of different qualifications can greatly differ.

The recommendation is to set estimation toward a middle specialist, especially if you do not possess information about the executors at the moment of estimation. This means that you have to put yourself in the shoes of the middle specialist for a more realistic estimation.

However, if you know that your team will be imbalanced in regards to the levels of qualification, adjust the estimates.

Effort vs. duration

It is recommended to estimate all projects and tasks based on the effort needed (man-hours) rather than duration (days and weeks) for the following reasons:

• At the moment of estimation the team may be not complete, and/or its composition can change in the future.
• Teams could be focusing on other projects or tasks thus reducing their capacity.
• Based on the effort, it is easy to get an approximate estimation of the project duration by a simple division by the available resources.

However, when you have a stable team, the estimation based on duration can be more intuitive and convenient within short time frames.

Man-hours over days, weeks, and months

Man-hours seem to be a more optimal measurement for project estimation. The reasons are the following:

• Different people, countries and companies work according to different calendars. Working days and weeks can consist of a different number of working hours, or someone from the team can work part-time. This is especially true for distributed development teams.
• Rates for software development are commonly determined based on man-hours. Thus, by simply multiplying the rate by the estimated number of man-hours, it is easy to calculate the approximate project budget.
• Usually, time tracking is recorded based on hours, which makes it easier to compare the planned task effort to actual progress.
• Estimation based on days is less accurate and allows engineers to treat tasks with a lower responsibility.

Powers of two

If you choose to use man-hours as a measurement, it makes sense to use powers of two to estimate separate tasks. They are intuitive enough, correspond well to 8-hour working day, and are easy to calculate:

• 0,5 – 1 hour – time needed for the smallest task
• 2 hours – time that can be comfortably spent in front of the computer without distraction
• 4 hours – half of the working day
• 8 hours – working day
• 16 hours – two working days
• 32 hours – “almost a week”

If needed, use these intermediary numbers: 6, 12, 40. Estimations above 40 hours look unreliable, and the tasks are difficult to control. In this situation, it is recommended to divide tasks into subtasks.

Working hours vs. effective hours

It’s no secret that software engineers are actively engaged in the working process 5-6 hours a day. The rest of the time is devoted to coffee, snacks, relaxing, social media, and small talk with colleagues. It is impossible to effectively do your job over a number of 8-hour working days in a row.

The recommendation here is to estimate the project based on working hours, not effective.

The only disadvantage is in the small tasks that need minimum time. Three two-hour tasks or one six-hour task will take a whole 8-hour working day to complete. Keep that in mind when estimating and planning such tasks.

As we have already clarified in the first part of the article, the purpose of estimation is not only to determine effort but also to describe the work scope, risks, and correctly present these artifacts to all project stakeholders. This is why the estimate document has to include the following:

1. Scope. The list of requirements and tasks along with items out of scope.
2. Assumptions and risks. Under what conditions the estimation is valid and what risks can take place.
3. Numbers. How much effort do we need to complete task 1 under condition 2?
4. Package. Last but not least. It is important to properly present the estimate document with clear explanations to the customer.

Note that the numbers are only in third place. The reason is that the scope definition is more important for project success than estimation accuracy.

In the first part of the article, we have discussed the main difficulties of project effort estimation, its purpose, requirements towards estimators (engineers), structure, and metrics of estimation. In the next part, we will describe the estimation methods in detail.