How to estimate efforts in software engineering? — Product Development

Have you ever found yourself starting a new project or hobby, only to realize that it’s taking much longer than you anticipated? You may have initially estimated that a certain task would take you a few hours, only to find yourself still working on it several days later. Time is a valuable resource, and accurately estimating how much time and effort a work will take can save you from frustration and disappointment.

Just like how your daily commute time can change based on several factors, like the route you take, traffic, and weather conditions, estimating the time and effort required for a personal project can be just as challenging. There are various elements to consider, such as the complexity of the task, the skill level required, and the resources available to you. These factors can all impact the time and effort required to complete the project.

However, when it comes to business or work-related activities, precise estimations are even more critical. In a professional setting, your ability to estimate the time and effort required for a task accurately can have significant implications. For example, if you are a project manager, it’s crucial to estimate how long each task will take to complete accurately. Otherwise, the entire project may be delayed, causing problems for everyone involved.

Importance of effort estimation in software engineering or product development

According to recent data, the global contract value for software engineering services is estimated to be around $200 billion. This makes software engineering one of the top services that businesses outsource to external contractors. However, in order to make informed decisions about hiring contractors or agencies, it’s crucial for businesses to accurately estimate the time and budget required for the project. Without this information, businesses would need to rely on trust when selecting a contractor, which can be risky.

Software engineering is a complex field that often requires extensive research and innovative solutions. Therefore, accurately estimating the time and effort required for an engineering task is essential. In addition to the project’s technical requirements, it’s also important to consider metrics such as the team’s expertise and experience level when making these estimations. By taking a comprehensive approach to time and effort estimation, businesses can ensure successful project outcomes and make the most out of their outsourcing partnerships.

Our Technology & Solution Consulting firm has vast experience in providing software engineering estimates, with an average of 300–400 estimates annually.

In this blog, we will discuss our tailored approach to the software development cost estimation process, including the techniques we use. Accurate estimations are crucial for businesses to make informed decisions regarding budget, resource allocation, and timelines. We will share our insights and best practices for providing reliable software development cost estimates. Before we proceed further, let us establish the role of estimates in the software development life cycle.

Software Development Life Cycle (SLDC) — The Conventional Approach

In recent decades, the complexity of software systems has been on the rise. Simultaneously, there is hardly any industry or field of knowledge that has not been affected by technology. This has resulted in the need for a standardized approach to software development process management and formalization of the process.

The Software Development Life Cycle (SDLC) was initially introduced as the formal project management framework. It outlines the major stages and tasks within a software development process.

The conventional Software Development Life Cycle comprises six stages, which are as follows:

1. Requirements gathering: In this stage, the development team works with the client to understand their needs and document the project’s requirements. This is an important stage as it sets the foundation for the rest of the development process.
2. Design: Once the requirements are gathered, the team moves on to the design stage. In this phase, the team develops a plan for how the software will be built. This includes creating wireframes, user interface design, and architecture design.
3. Implementation: After the design is finalized, the team begins the implementation stage. The development team writes code based on the design plan and creates the software product.
4. Testing: Once the product is developed, it undergoes extensive testing to ensure that it meets all the requirements and functions as intended. Testing is an iterative process that involves identifying and fixing bugs or issues that arise.
5. Deployment: After the software product is tested and validated, it is ready for deployment. In this stage, the product is released to the market, and users can start using it.
6. Maintenance: After deployment, the software product requires ongoing maintenance and support to ensure its smooth functioning. This includes regular updates, bug fixes, and user support.

The original SDLC serves as a guide for managing projects from start to finish and is the basis for several software development methodologies used today. The waterfall model is the methodology that adheres most closely to this step-by-step process.

It is most suitable for projects with clear requirements, small scope of work, and low complexity, such as solving a secondary business problem or automating a certain internal task. For such projects, a fixed-price collaboration model is possible, where effort is defined, and outcomes are predictable. However, the traditional SDLC has some downsides, such as high uncertainty during the early stages of software development.

The Problem with software engineering effort Estimation: Dealing with Uncertainty

Approximately 60% of project success is dependent on meeting cost and time estimates, as stated in a report that identified cost overrun and delivery delays as significant problems in seven out of 12 of the most well-known failed projects.

Fred Brooks, in his renowned book, “The Mythical Man-Month: Essays on Software Engineering,” similarly observed that a lack of time was a common cause of failure in software projects. Engineers, who are frequently optimistic, are also under a lot of pressure to provide accurate estimates. Even the most experienced engineers have an unwritten rule to double their estimates to prevent delays. However, these estimates are not always effective if they are made too early in the process or if project requirements are not well-defined.

To create a reliable estimate for a project, it’s important to take into account the following factors:

• Detailed specifications: having a clear understanding of the project scope and desired outcomes is crucial for accurate estimation.
• Graphical designs: more complex UI elements require additional engineering effort and may take longer to implement.
• Technology stack: depending on the project requirements, the team may need to use advanced tools and third-party APIs or develop custom solutions, which should be factored into the estimate.
• Experience and expertise: senior software engineers may complete a task much faster than a trainee, so it’s important to consider the team’s experience and skills when creating an estimate.

Here we are only scratching the surface, as even seemingly simple requirements in project documentation can involve many factors. To illustrate, consider the scope of a basic user story, such as “As a User, I want to upload photos to my profile,” which can give rise to various queries:

• What kind of images are allowed (file types, size, resolution)?
• Should there be any limits to the number of photos uploaded?
• How should the photos be displayed on the profile (grid, slider, album)?
• Should the photos be organized into categories or tags?
• Is there a need for an option to edit or delete the uploaded photos?

The number of questions raised for a seemingly simple feature is quite extensive. These questions don’t just relate to functional requirements but also encompass technical aspects such as testing, documentation, code review, and refactoring. It is often difficult to define the scope of work at the beginning of the process since the project requirements tend to become clearer over time. Typically, the more precise the project requirements become, the more accurate the estimate will be.

Estimation Techniques and Tools: Finding a Middle Ground

1. Creating a Work Breakdown Structure (WBS)

As mentioned previously, finalizing requirements and the project roadmap during the requirement-gathering stage is crucial to reducing uncertainty in software development estimation. Andrew Stellman and Jennifer Greene, well-known Agile coaches and bestselling O’Reilly authors, suggest in their book Applied Software Project Management (2005) that a dependable estimation requires the creation of a work breakdown structure (WBS).

The PMI Project Body of Knowledge® describes the WBS as a structure that breaks down the project work into smaller, hierarchical components that need to be completed by the project team to achieve the project goals and deliver the expected outcomes. This document is similar to a backlog in Agile methodologies, as it breaks the project down into measurable and manageable deliverables. The WBS emphasizes the deliverables and their visual representation, while the backlog is centered around user stories and features. The backlog is considered an Agile WBS and, like the WBS, is a living document that is continually updated throughout the development process.

2. Metrics Used for Effort Estimation in software engineering

Once a WBS has been established, engineers can provide a more detailed estimation of the required efforts to develop a software product. However, measuring these efforts remains a challenge. Some teams use relative terms, like story points, function points, or T-shirt sizes, while others prefer more concrete units, such as man-hours, days, weeks, or months.

Despite these options, customers who prioritize business-oriented aspects may find these numbers meaningless. When negotiating with potential contractors or discussing the project with in-house engineering teams, they typically want to know when the product will be finished and how much it will cost. Therefore, effort estimations are usually translated into hours, days, weeks, or months, and costs are calculated accordingly.

3. Software effort Estimation Techniques

The choice of software estimation techniques depends on the project management methodology adopted for the development process. Generally, these techniques are categorized as either Traditional, which is typically used in the Waterfall method, or Agile.

In his article titled “Software Estimation using a Combination of Techniques,” Klaus Nielsen classifies the most widely used software engineering assessment techniques as follows:

Traditional techniques for Effort Estimation in software engineering–

This includes Analogy, Effort method, Programming techniques, Expert judgment, Delphi/Wideband Delphi, Program Evaluation and Review, CoCoMo 1/CoCoMo 2, Albrects Function Points, Parametric Methods (e.g., PRICE), Various top-down or bottom-up techniques, etc.

Agile Software Development Estimation Techniques-

Relative Sizing, Wideband Delphi, Planning poker, Affinity estimation, Ideal time and elapsed time, Disaggregation, Bottom-up/top-down estimation, Constructive agile estimation algorithm, and Time, budget, and cost estimation, etc.

4. Approaches to Effort Estimation in software engineering at AnArSolutions

AnArSolutions is committed to providing accurate project estimations by adopting a structured and comprehensive approach. Our unique process is designed to ensure that all project requirements are captured right from the outset and that client needs and objectives are fully understood. As our CEO, Rupashri Gulwani, explains, “Our process is geared towards enabling us to provide clients with an accurate estimate of the time, resources, and costs required to deliver their project successfully. 

This approach allows us to provide transparency and clarity throughout the project lifecycle, and to ensure that clients are satisfied with the results.”

The first stage of our process is the Client Discussion phase. During this stage, our team engages in detailed meetings with clients to understand their vision for the software application they wish to develop. We believe that this phase is critical to project success, as it allows us to gain a deep understanding of the client’s needs and objectives.

The Client Discussion phase primarily involves asking questions such as:

AnArSolutions is committed to providing accurate project estimations by adopting a structured and comprehensive approach. Our unique process is designed to ensure that all project requirements are captured right from the outset and that client needs and objectives are fully understood. As our CEO, Rupashri Gulwani, explains, “Our process is geared towards enabling us to provide clients with an accurate estimate of the time, resources, and costs required to deliver their project successfully. 

This approach allows us to provide transparency and clarity throughout the project lifecycle, and to ensure that clients are satisfied with the results.”

The first stage of our process is the Client Discussion phase. During this stage, our team engages in detailed meetings with clients to understand their vision for the software application they wish to develop. We believe that this phase is critical to project success, as it allows us to gain a deep understanding of the client’s needs and objectives.

The Client Discussion phase primarily involves asking questions such as:

• What is the main reason behind the software project?
• What are the business incentives driving the project?
• How will the software application add value to the client’s current business operations?
• What are the main features or functionalities that you would like to see in the software application?
• Are there any specific design or branding requirements that the software application should adhere to?
• What are the key performance indicators (KPIs) that you will be tracking to measure the success of the software application?
• Do you have any existing software systems that the new application needs to integrate with?
• What is the timeline for the project, and are there any specific milestones or deadlines that need to be met?
• Are there any regulatory or compliance requirements that the software application needs to adhere to?

By asking these questions, we can gain a deeper understanding of our client’s needs and objectives. This enables us to identify any potential issues or roadblocks and develop a more accurate project timeline and budget.

Once we have completed the Client Discussion phase, our team moves on to the second stage, which is to develop a high-level scope and sizing estimation for the project. During this stage, we analyze the information gathered during the Client Discussion phase to identify the core features and functionalities that the software application will need to include. This information is used to develop a high-level scope that outlines the project’s goals and objectives, as well as the key deliverables that will be produced.

Based on this high-level scope, we then develop a ballpark figure for the project’s cost and timeline. This figure is designed to provide clients with an estimate of the overall investment required to deliver the software application successfully. 

“Developing a ballpark figure at this stage is crucial, as it allows us to ensure that we are aligned with our client’s expectations and goals. By providing a rough estimate of the project’s costs and timeline, we can ensure that our clients are fully informed and can make informed decisions about the project.”

If the ballpark figure aligns with the client’s expectations and goals, we move on to the third stage of our process, which is to decide on the project’s methodology. At this stage, we consider whether to use the Waterfall or Agile methodology for the project. The choice of methodology will depend on various factors, including the project’s complexity, the level of client involvement required, and the project timeline and budget.

Let’s see how we use these approaches for cost estimations- 

The waterfall Approach 

For waterfall projects, we primarily rely on three methods — 

1. Analogy-based estimation: This technique estimates the necessary effort by comparing it to the information gathered from past projects or similar features.
2. Statistical method: We use statistical data to estimate the effort required based on previous experience in the relevant field.
3. Expert judgment: If the internal team does not possess the expertise to develop comparable features or the project necessitates the utilization of advanced or industry-specific technology solutions, we depend on the assessment of experts to gauge the required effort.

Our standard process commences with the finalization of a high-level project vision, which is subsequently converted into the Work Breakdown Structure. The proposed solution is subdivided into smaller modules and features to ensure that our clients and we have a shared understanding of the software product’s scope and functionality. Each feature within a module is analyzed and estimated separately, utilizing one or a combination of the following methods: Analogy-based estimation, Statistical method, and Expert judgment.

Consequently, we obtain a comprehensive document with a detailed breakdown of the scope of work and the estimated duration of each task, defining a possible range of variability with minimum and maximum values. With each feature being estimated separately, we can provide a quote for the entire project by adding up the duration of all tasks. Additionally, QA and project management activities conducted in parallel with the main development activities do not typically extend the calendar duration of the project. The total estimated scope of work is measured in man-days and does not necessarily equate to the actual calendar duration.

Despite utilizing a detailed work breakdown and dedicated methodologies, there is still some variability between the minimum and maximum estimated scope. Hence, we have devised another approach to managing complex projects.

The Agile Approach

Barry Boehm wrote in his book “Software Engineering Economics” that no matter how strong a software cost estimation technique is, it cannot make up for the lack of definition or understanding of the software job to be done. Therefore, we have adopted a more robust approach to software development — the product development model.

Developing complex software projects from scratch requires flexibility and a long-term dedicated team effort because the requirements and business model may evolve during the development process. Investing extensively in research and detailed planning at this stage is not practical. This model is best used for a customer request that involves a significant engineering task, such as building an application that plays a crucial role in the client’s business.

The conventional software development life cycle (SDLC) involves project-based collaboration using the waterfall approach, while this method only follows agile methodologies. Hence, we prioritize high-level requirements and form a dedicated team that is specifically tailored to meet the client’s needs rather than allocating 30–40% of project time to planning and analysis. Following this, we follow the typical agile process by delivering working product builds in iterations. Sprint duration is limited to 2–4 weeks to ensure measurable progress and predictable results.

This approach is fundamentally agile and cannot be constrained by fixed deadlines or commitments at the outset. Thus, we recommend the Time and Material model as the optimal option. Instead of providing comprehensive project estimates, we offer quotes for a defined set of tasks for each iteration, utilizing agile software estimation techniques such as Planning Poker, Relative Sizing, Ideal Time & Elapsed Time.

• Planning Poker

Planning Poker is an estimation and planning technique that uses a deck of cards to gamify the process. Instead of verbally stating their estimates for each item on the list of proposed features for an iteration, team members select a card representing their estimation of how long the task will take. Each team member’s card is then discussed, and the team comes to a consensus on the estimate. This method helps eliminate cognitive biases and allows each team member to make an objective judgment based on their own perspective.

• Relative Sizing

Relative sizing estimation is a technique in which a feature or task is chosen as a measurement standard, and all subsequent items are estimated in relation to how they compare to this standard.

• Ideal Time & Elapsed Time

The Ideal Time & Elapsed Time technique is founded on the notion that engineers are unlikely to work for a full 8-hour day but will instead devote about 6 hours a day to the task at hand. Factoring in this focus factor, the team will estimate a 24-hour task as taking four days to complete rather than three days.

The Conclusion 

In conclusion, the statistics from the Standish Group 2015 Chaos Report make it clear that choosing the right approach for software development can significantly impact project success rates. While traditional waterfall approaches have their place, the product development model based on agile methodologies has shown to be more successful for large projects. By focusing on high-level requirements, building a dedicated team, and using agile software estimation techniques, we can deliver functioning product builds iteration by iteration, ultimately leading to predictable results. As a result, we highly recommend that clients consider this approach for their software development projects.