How to estimate efforts in software engineering? — Product Development

Starting a new project, whether personal or professional, often reveals that tasks can take much longer than expected. You might plan for something to take a few hours, only to find yourself still working on it days later. Accurate time and effort estimation can prevent these surprises and keep frustration at bay.

Just as your daily commute time can vary with factors like route, traffic, and weather, estimating the effort for any project involves several variables—such as task complexity, required skills, and available resources—all of which impact the timeline.

In a business setting, precise estimations become even more essential. For project managers, for example, accurately forecasting the time and resources each task requires is critical to avoid delays and keep projects on track, benefiting both the team and the organization.

Importance of effort estimation in software engineering or product development

Recent estimates indicate that the global market for software engineering services is valued at approximately $200 billion, making it one of the most outsourced services by businesses. To make informed decisions about hiring contractors or agencies, companies must accurately assess the time and budget needed for their projects. Without this crucial information, businesses may find themselves relying solely on trust when selecting a contractor, which can lead to risky outcomes.

Software engineering is inherently complex, often requiring extensive research and innovative problem-solving. Therefore, accurately estimating the time and effort for engineering tasks is vital. Beyond the technical specifications of a project, it’s important to factor in metrics such as the team’s expertise and experience level during the estimation process. By adopting a thorough approach to time and effort estimation, businesses can achieve successful project outcomes and maximize the benefits of their outsourcing partnerships.

At our Technology & Solution Consulting firm, we have extensive experience in providing software engineering estimates, averaging 300 to 400 estimates annually.

In this blog, we will discuss our tailored approach to the software development cost estimation process, including the techniques we use. Accurate estimations are crucial for businesses to make informed decisions regarding budget, resource allocation, and timelines. We will share our insights and best practices for providing reliable software development cost estimates. Before we proceed further, let us establish the role of estimates in the software development life cycle.

Software Development Life Cycle (SLDC) — The Conventional Approach

In recent decades, the complexity of software systems has been on the rise. Simultaneously, there is hardly any industry or field of knowledge that has not been affected by technology. This has resulted in the need for a standardized approach to software development process management and formalization of the process.

The Software Development Life Cycle (SDLC) was initially introduced as the formal project management framework. It outlines the major stages and tasks within a software development process.

The conventional Software Development Life Cycle comprises six stages, which are as follows:

1. Requirements gathering: In this stage, the development team works with the client to understand their needs and document the project’s requirements. This is an important stage as it sets the foundation for the rest of the development process.
2. Design: Once the requirements are gathered, the team moves on to the design stage. In this phase, the team develops a plan for how the software will be built. This includes creating wireframes, user interface design, and architecture design.
3. Implementation: After the design is finalized, the team begins the implementation stage. The development team writes code based on the design plan and creates the software product.
4. Testing: Once the product is developed, it undergoes extensive testing to ensure that it meets all the requirements and functions as intended. Testing is an iterative process that involves identifying and fixing bugs or issues that arise.
5. Deployment: After the software product is tested and validated, it is ready for deployment. In this stage, the product is released to the market, and users can start using it.
6. Maintenance: After deployment, the software product requires ongoing maintenance and support to ensure its smooth functioning. This includes regular updates, bug fixes, and user support.

The original SDLC serves as a guide for managing projects from start to finish and is the basis for several software development methodologies used today. The waterfall model is the methodology that adheres most closely to this step-by-step process.

It is most suitable for projects with clear requirements, small scope of work, and low complexity, such as solving a secondary business problem or automating a certain internal task. For such projects, a fixed-price collaboration model is possible, where effort is defined, and outcomes are predictable. However, the traditional SDLC has some downsides, such as high uncertainty during the early stages of software development.

The Problem with software engineering effort Estimation: Dealing with Uncertainty

Approximately 60% of project success is dependent on meeting cost and time estimates, as stated in a report that identified cost overrun and delivery delays as significant problems in seven out of 12 of the most well-known failed projects.

Fred Brooks, in his renowned book, “The Mythical Man-Month: Essays on Software Engineering,” similarly observed that a lack of time was a common cause of failure in software projects. Engineers, who are frequently optimistic, are also under a lot of pressure to provide accurate estimates. Even the most experienced engineers have an unwritten rule to double their estimates to prevent delays. However, these estimates are not always effective if they are made too early in the process or if project requirements are not well-defined.

To create a reliable estimate for a project, it’s important to take into account the following factors:

• Detailed specifications: having a clear understanding of the project scope and desired outcomes is crucial for accurate estimation.
• Graphical designs: more complex UI elements require additional engineering effort and may take longer to implement.
• Technology stack: depending on the project requirements, the team may need to use advanced tools and third-party APIs or develop custom solutions, which should be factored into the estimate.
• Experience and expertise: senior software engineers may complete a task much faster than a trainee, so it’s important to consider the team’s experience and skills when creating an estimate.

Here we are only scratching the surface, as even seemingly simple requirements in project documentation can involve many factors. To illustrate, consider the scope of a basic user story, such as “As a User, I want to upload photos to my profile,” which can give rise to various queries:

• What kind of images are allowed (file types, size, resolution)?
• Should there be any limits to the number of photos uploaded?
• How should the photos be displayed on the profile (grid, slider, album)?
• Should the photos be organized into categories or tags?
• Is there a need for an option to edit or delete the uploaded photos?

The number of questions raised for a seemingly simple feature is quite extensive. These questions don’t just relate to functional requirements but also encompass technical aspects such as testing, documentation, code review, and refactoring. It is often difficult to define the scope of work at the beginning of the process since the project requirements tend to become clearer over time. Typically, the more precise the project requirements become, the more accurate the estimate will be.

Estimation Techniques and Tools: Finding a Middle Ground

1. Creating a Work Breakdown Structure (WBS)

Finalizing requirements and the project roadmap during the requirement-gathering phase is essential for minimizing uncertainty in software development estimations. In their book Applied Software Project Management (2005), renowned Agile coaches Andrew Stellman and Jennifer Greene emphasize that reliable estimations hinge on creating a work breakdown structure (WBS).

According to the PMI Project Body of Knowledge®, the WBS is a hierarchical structure that decomposes project work into smaller, manageable components that the project team must complete to meet project objectives and deliver expected outcomes. This document is akin to a backlog in Agile methodologies, as it organizes the project into measurable deliverables. While the WBS focuses on visualizing deliverables, the backlog centers on user stories and features. The backlog can be viewed as an Agile WBS, serving as a living document that is regularly updated throughout the development process.

2. Metrics Used for Effort Estimation in software engineering

Once a work breakdown structure (WBS) is in place, engineers can provide more precise estimates of the effort needed to develop a software product. However, quantifying these efforts can be challenging. Some teams use relative measurements like story points, function points, or T-shirt sizes, while others opt for more concrete units such as man-hours, days, weeks, or months.

Despite these various methods, customers focused on business outcomes may find these metrics unhelpful. When negotiating with potential contractors or discussing projects with in-house engineering teams, their primary concerns often revolve around project completion timelines and costs. Consequently, effort estimations are typically converted into hours, days, weeks, or months, with costs calculated based on these timeframes.

3. Software effort Estimation Techniques

The choice of software estimation techniques depends on the project management methodology adopted for the development process. Generally, these techniques are categorized as either Traditional, which is typically used in the Waterfall method, or Agile.

In his article titled “Software Estimation using a Combination of Techniques,” Klaus Nielsen classifies the most widely used software engineering assessment techniques as follows:

Traditional techniques for Effort Estimation in software engineering–

This includes Analogy, Effort method, Programming techniques, Expert judgment, Delphi/Wideband Delphi, Program Evaluation and Review, CoCoMo 1/CoCoMo 2, Albrects Function Points, Parametric Methods (e.g., PRICE), Various top-down or bottom-up techniques, etc.

Agile Software Development Estimation Techniques-

Relative Sizing, Wideband Delphi, Planning poker, Affinity estimation, Ideal time and elapsed time, Disaggregation, Bottom-up/top-down estimation, Constructive agile estimation algorithm, and Time, budget, and cost estimation, etc.

4. Approaches to Effort Estimation in software engineering at AnArSolutions

AnAr Solutions is dedicated to delivering precise project estimations through a structured and comprehensive approach. Our distinct process ensures that all project requirements are captured from the beginning and that client needs and objectives are thoroughly understood. As our CEO, Rupashri Gulwani, states, ‘Our process is designed to enable us to provide clients with an accurate estimate of the time, resources, and costs necessary to successfully deliver their projects.’

This methodology promotes transparency and clarity throughout the project lifecycle, ensuring that clients are satisfied with the outcomes.

The first step in our process is the Client Discussion phase. During this phase, our team conducts in-depth meetings with clients to grasp their vision for the software application they want to develop. We consider this phase essential for project success, as it allows us to gain a comprehensive understanding of the client’s needs and objectives.

The Client Discussion phase primarily involves asking questions such as:

AnAr Solutions is dedicated to delivering precise project estimations through a structured and thorough approach. Our distinctive process ensures that all project requirements are identified from the very beginning and that we fully understand our clients’ needs and objectives. As our CEO, Rupashri Gulwani, states, ‘Our process is designed to equip us with the ability to provide clients with accurate estimates of the time, resources, and costs necessary for the successful delivery of their projects.’

This method fosters transparency and clarity throughout the project lifecycle, ensuring that our clients are pleased with the results.

The first stage of our process is the Client Discussion phase. During this stage, our team engages in detailed meetings with clients to understand their vision for the software application they wish to develop. We believe that this phase is critical to project success, as it allows us to gain a deep understanding of the client’s needs and objectives.

The Client Discussion phase primarily involves asking questions such as:

• What is the main reason behind the software project?
• What are the business incentives driving the project?
• How will the software application add value to the client’s current business operations?
• What are the main features or functionalities that you would like to see in the software application?
• Are there any specific design or branding requirements that the software application should adhere to?
• What are the key performance indicators (KPIs) that you will be tracking to measure the success of the software application?
• Do you have any existing software systems that the new application needs to integrate with?
• What is the timeline for the project, and are there any specific milestones or deadlines that need to be met?
• Are there any regulatory or compliance requirements that the software application needs to adhere to?

By asking targeted questions, we gain a deeper understanding of our clients’ needs and objectives. This insight allows us to pinpoint potential issues or roadblocks, enabling us to create a more accurate project timeline and budget.

After completing the Client Discussion phase, our team advances to the second stage, where we develop a high-level scope and sizing estimation for the project. In this phase, we analyze the information gathered to identify the essential features and functionalities required for the software application. This analysis helps us outline a high-level scope that defines the project’s goals, objectives, and key deliverables.

Using this high-level scope, we then generate a ballpark figure for the project’s cost and timeline. This estimate serves to inform clients of the overall investment needed for successful software delivery.

‘Creating a ballpark figure at this stage is critical, as it helps us align with our clients’ expectations and goals. By offering a preliminary estimate of costs and timelines, we ensure that our clients are well-informed and can make educated decisions about the project.’

If the ballpark figure meets the client’s expectations, we proceed to the third stage of our process, which involves determining the project’s methodology. At this point, we evaluate whether to adopt the Waterfall or Agile methodology, considering factors such as project complexity, required client involvement, and the overall timeline and budget.

Let’s see how we use these approaches for cost estimations- 

The waterfall Approach 

For waterfall projects, we primarily rely on three methods — 

1. Analogy-based estimation: This technique estimates the necessary effort by comparing it to the information gathered from past projects or similar features.
2. Statistical method: We use statistical data to estimate the effort required based on previous experience in the relevant field.
3. Expert judgment: If the internal team does not possess the expertise to develop comparable features or the project necessitates the utilization of advanced or industry-specific technology solutions, we depend on the assessment of experts to gauge the required effort.

Our standard process commences with the finalization of a high-level project vision, which is subsequently converted into the Work Breakdown Structure. The proposed solution is subdivided into smaller modules and features to ensure that our clients and we have a shared understanding of the software product’s scope and functionality. Each feature within a module is analyzed and estimated separately, utilizing one or a combination of the following methods: Analogy-based estimation, Statistical method, and Expert judgment.

Consequently, we obtain a comprehensive document with a detailed breakdown of the scope of work and the estimated duration of each task, defining a possible range of variability with minimum and maximum values. With each feature being estimated separately, we can provide a quote for the entire project by adding up the duration of all tasks. Additionally, QA and project management activities conducted in parallel with the main development activities do not typically extend the calendar duration of the project. The total estimated scope of work is measured in man-days and does not necessarily equate to the actual calendar duration.

Despite utilizing a detailed work breakdown and dedicated methodologies, there is still some variability between the minimum and maximum estimated scope. Hence, we have devised another approach to managing complex projects.

The Agile Approach

In his book Software Engineering Economics, Barry Boehm stated that no matter how effective a software cost estimation technique may be, it cannot compensate for a lack of clarity or understanding regarding the software requirements. To address this, we have adopted a more robust approach to software development: the product development model.

Creating complex software projects from the ground up necessitates flexibility and a sustained team effort, as requirements and business models may evolve throughout the development process. At this stage, investing heavily in research and detailed planning may not be practical. This model is particularly suited for customer requests that involve substantial engineering tasks, such as developing an application that is integral to the client’s business.

The conventional software development life cycle (SDLC) involves project-based collaboration using the waterfall approach, while this method only follows agile methodologies. Hence, we prioritize high-level requirements and form a dedicated team that is specifically tailored to meet the client’s needs rather than allocating 30–40% of project time to planning and analysis. Following this, we follow the typical agile process by delivering working product builds in iterations. Sprint duration is limited to 2–4 weeks to ensure measurable progress and predictable results.

This approach is fundamentally agile and cannot be constrained by fixed deadlines or commitments at the outset. Thus, we recommend the Time and Material model as the optimal option. Instead of providing comprehensive project estimates, we offer quotes for a defined set of tasks for each iteration, utilizing agile software estimation techniques such as Planning Poker, Relative Sizing, Ideal Time & Elapsed Time.

• Planning Poker

Planning Poker is an estimation and planning technique that uses a deck of cards to gamify the process. Instead of verbally stating their estimates for each item on the list of proposed features for an iteration, team members select a card representing their estimation of how long the task will take. Each team member’s card is then discussed, and the team comes to a consensus on the estimate. This method helps eliminate cognitive biases and allows each team member to make an objective judgment based on their own perspective.

• Relative Sizing

Relative sizing estimation is a technique in which a feature or task is chosen as a measurement standard, and all subsequent items are estimated in relation to how they compare to this standard.

• Ideal Time & Elapsed Time

The Ideal Time & Elapsed Time technique is founded on the notion that engineers are unlikely to work for a full 8-hour day but will instead devote about 6 hours a day to the task at hand. Factoring in this focus factor, the team will estimate a 24-hour task as taking four days to complete rather than three days.

The Conclusion 

In conclusion, the statistics from the Standish Group 2015 Chaos Report make it clear that choosing the right approach for software development can significantly impact project success rates. While traditional waterfall approaches have their place, the product development model based on agile methodologies has shown to be more successful for large projects. By focusing on high-level requirements, building a dedicated team, and using agile software estimation techniques, we can deliver functioning product builds iteration by iteration, ultimately leading to predictable results. As a result, we highly recommend that clients consider this approach for their software development projects.