Foundations of Software Engineering

The 4 Challenges of Software Development

Complexity
- Requirements concerning functionality and efficiency are constantly growing
Quality
- Software malfunction may cause injuries and economic losses
- A small change in a program can have huge impact on its behaviour (Discontinuity property of software)
Human factors
- Managing software teams effectively is challenging
- Programmer productivity varies vastly from person to person
Maintenance & Legacy Systems
- Long lifetime of software systems
- Software must be continuously adapted to new and changing requirements

Programming vs Software Engineering

	Programming	Software Engineering
Scale	Small	Huge
Updates	Sequential	Parallel
Life-span	Short-lived	Long-lived
Error/Bugs	Small consequences	Large consequences

Software Engineering mainly includes

determining what to build
organizing teams to cooperatively build systems
lifecycle and process management
analysis and testing
software architecture
system engineering

Software engineers deal with a very broad range of projects, kinds of systems, organizational settings, scale, criticality, domain expertise, etc.

Degree of Formality

Large projects with globally distributed teams require sufficient level of formality to communicate requirements, designs, and other aspects of development.

Degree of formality in software process depends on project scale and domain
Smaller projects with small teams (ten or less) at a single location don’t need the same degree of formality

Life Cycle and Process Models

Life Cycle: Life Cycle of software system.

phases in the use of software system

Process Models: How do we perform the activities.

activities (what it produces - work products; who performs - roles; when - dependencies; how - tools and methods used)

Life Cycle is the subset of Process Models

The orderings are:

Requirements elicitation
Specification
Design
Implementation
Integration
Maintenance phase
Retirement
…

Build and Fix model

Basically:

No planning/up-front requirements at all (Chaotic Development)

Advantage:

Appropriate for small programs written by one person

Disadvantage:

No requirements engineering happening, poor understanding of customer need
Lack of proper architecture and design
Lack of repeatable (automated) test
- Wrong system maybe built
- messy (no design) and buggy (no systemic V&V)
Understandability and maintainability decrease rapidly with increasing program size

Waterfall model

Basically:

Sequential steps/phases with feedback loops between 2 phases
Documentation-driven
- Assume each activity is carried to completion before moving to the next

Advantage:

Lightweight, Easier maintenance (Follow logical flow)
Documentation

Disadvantage:

Customer involvement in the first phase only
Product becomes available very late in the process (Risk of building wrong system due to no user feedback, Operational quality portion of product in the end)
Only work best when there is a complete and frozen specification document

Key takeaway:

High risk for new systems because of specification and design problems
Low risk for well-understood developments using familiar technology

Rapid prototyping model

Basically:

Waterfall model but no feedback loops
frequent change, then discard (A Throwaway prototype)

Advantage:

Customer involvement is strong
- Address problem of waterfall model: Risk of building wrong system due to no user feedback
- Deduce the risk of wrong system built

Disadvantage:

Higher development effort
Danger that due to schedule slip, the prototype becomes part of the product

Key takeaway:

Low risk for new applications because specification and program stay in step
High risk because of lack of process visibility

Incremental model

Basically:

Each release adds more functionality
Better than Waterfall model

Advantage:

Better Customer involvement (Operational quality portion of product within weeks)
- Frequent release and feedback

Disadvantage:

Not suitable for large and critical system
- You can’t deliver just a small prototype of an engine control for a rocket and then just try out if the rocket would fly and fix afterwards. But you can use the approach of incremental releases for a mobile app.
Bad complete design upfront
If up-front requirements are done, during building of the system and releasing increments to the customer you discover that some of the parts of the specified functionality were not what the user actually wants/needs, then some of the up-front requirements engineering effort would be wasted.
- This just acknowledges the fact that given working releases to a customer can help you refine requirements more effectively than doing everything just on paper up front. There are cases where doing up front is needed, like in the case of large and complex safety critical systems, like an airplane.

Evolutionary model (for Agile development)

Basically:

New versions implement new and evolving requirements
Agile development
don’t have up-front requirements and up-front design four the whole system done before you code
- do requirements and design for each increment
Middle ground between waterfall and prototyping

Advantage:

Constant customer involvement and validation
- Address the need for customer feedback and reduce the risk of building the wrong system
Allows for good risk management

Disadvantage:

Build-and-fix danger
- The potential risk is that if you keep requirements and design in evolutionary lean, some lazy developers might say, hey lets just eliminate requirements and design (including unit testing) and do just coding (and debugging) as needed, which would end up being just code and fix.
- Without requirements and design then it is just code and fix

V-model

Basically:

waterfall model, but represented as a V to emphasize verification and validation using different type of testing
widely used in embedded systems industry

Advantage:

Easier maintenance (Follow logical flow)
Emphases on verification and validation
- Makes clear which activities in the left branch feed reference information for which activities in the right branch

Disadvantage:

Same as waterfall model, not enough feedback
- Product becomes available very late in the process (Risk of building wrong system due to no user feedback, Operational quality portion of product in the end)
- Only work best when there is a complete and frozen specification document

Spiral model

Basically:

Waterfall model + risk analysis preceding each phase + evaluation following each phase
Prototyping for high-risk specifications
Middle ground between waterfall and prototyping
If all risks cannot be resolved, the project is immediately terminated
Appropriate only for big projects (high management overhead)
- Originally developed for big defense contracts

Hybrid process models

Large systems are usually made up of several subsystems.
The same process model need not be used for all subsystems, e.g.
- Prototyping for high-risk specifications
- Waterfall model for well-understood developments

Taylor the process to a problem.

Agile Methods

Uncovering better ways of developing software by doing it

Agile (Evolutionary) vs Waterfall

Indivduals and interactions > processes and tools
Working software > comprehensive documentation
Customer collaboration > contract negotiation
Responding to change > following a plan

The 3 prominent agile methods:

XP
Scrum
Lean

Extreme Programming (XP)

Planning Meeting:

Brainstorm list of features

Customers: proposes fixes, features and ranks them based on value/priority

Developers: design improvements and rank feature by technical difficulty, estimate effort and plan iterations

Write tests

Pair Programming

Reduce bug / code conflict

1 guy focus on the function while another guy focus on the architecture

Learning each other

Test

Integration:

Pull request to integrate new function

Evolutionary development
Collection of 12 “Best Practices”
Focus on working code that implements customer needs (rather than documents)
Testing is a crucial element of the process
Focus on flexibility and efficiency of the process
Designed for small teams (<10)

XP Practices

Small releases

Start with the smallest useful feature set
Release early and often (e.g., every two weeks), adding a few features each time

The planning game

Stakeholder meeting to plan the next iteration
Business people decide on business value of features
Developers on the technical risk of features and predicted effort per feature
Each system features typically captured as a user story

Simple Design

Always use the simplest possible design that gets the job done (runs the tests and states intentions of the programmer)

Testing

Unit tests test individual system modules
Programmers write unit tests and customers write acceptance tests
Test-first: write test, then implement it

Refactoring

Refactoring = Improving design without changing functionality
Refactoring is done continuously; the code is always kept clean

Pair programming

All production code written by two programmers
One programmer is thinking about implementing the current method, the other is thinking strategically about the whole system
Pairs are put together dynamically

Collective code ownership

Any programmer that sees an opportunity to add value to any portion of the code is required to do so at any time

Continuous Integration (CI)

Use of version and configuration management (e.g., CVS)
All changes are integrated into the code-base at least daily
The tests have to run 100% before and after the integration

On-site customer

Development team has continuous access to a real life customer/user

Coding standards

Everyone codes to the same standards
- Ideally, you should not be able to tell by looking at it who has written a specific piece of code

40-h week

Programmers go home on time
- Overtime is a symptom of a serious problem
- No errors by tired developers; better motivated developers

Additional Agile Practices:

Estimation

Keep track of feature set to be implemented in each iteration

Estimate the effort per feature at the beginning of the iteration

Keep track and review the actual effort at the end of the iteration

Use the experience to improve estimation in the next iteration

Stand-Up / Scrum Meetings

Short (< 15 min) daily morning meetings

Each team member asked

What have you done since last meeting?

What has impeded your work?

What will you do next?

Try to resolve impediments quickly, or schedule smaller meetings immediately following

Advantages of XP

Low management overhead
- no complicated procedures to follow, no documentation to maintain, direct communication, pair programming
Continuous risk management
- early feedback from the customer
Continuous effort estimation
Emphasis on testing; tests help in evolution and maintenance

Disadvantages of XP

Appropriate for small teams (up to 10 developers) only (does not scale)
Large development groups may require more structures and documents
If maintainers are not the people that developed the code, good documentation is necessary
Generic design maybe necessary to enable expected future development

Software Development Scrum

Scrum (https://www.scrumguides.org/scrum-guide.html) is an agile method

Key takeaway:

Achieve quality through transparency and monitoring, rather than processes
Daily meetings (‘daily scrum’)
Timeboxes with deliverables (‘sprints’; like XP iterations)
Small, self-managed, cross-functional teams of 3-9 people
- Developers
  - do the actual analysis, design, implementation, testing, etc.
- Product owner
  - represents the stakeholders and the business; keeps track of the backlog and what’s due for a sprint
- Scrum master
  - maintains the processes (typically in lieu of a project manager)

Stand-Up / Scrum Meetings

Short (< 15 min) daily morning meetings

Each team member asked

What have you done since last meeting?

What has impeded your work?

What will you do next?

Try to resolve impediments quickly, or schedule smaller meetings immediately following

Scrum Process

Scrum Sprint

Sprint means run at full speed over a short distance.

A short development interval (time-boxed)
- e.g. 2 weeks, but definitely no more than a month
Well-defined goal: what’s going to be implemented, what’s going to be demoed at the end of the spring
Keep track of Backlog
- Product Backlog: Prioritized list of work to be done (features, stories, requirements)
- Rough estimate of ideal number of days required to implement each item
- Sprint Backlog: Before each sprint, a planning session is held to decide what items from the backlog will be addressed by the sprint
- May use a simple text file, Wiki, a spreadsheet, or a specialized app (e.g., Jira for Confluence, Asana), or built-in functionality in gitlab or github

End of sprint:

Demo

Collect feedback

Discuss performance, process, accuracy of estimates

Plan next sprint

Lean Software Development

Lean software development is an adaptation of lean manufacturing principles from the Toyota Production System and practices to the software development domain.

Lean manufacturing is about making value flows explicit and optimizing it to eliminate waste

7 principles, with #1 being: Eliminate waste
- Everything that does not add value to the customer is waste
- E.g., code or documentation that doesn’t get used
Defines additional software practices, e.g., Kanban board

Kanban Board

basically a task board, visually depicts the status and flow of tasks within a project in a tabular way
Typical columns in a task board include
- Backlog
- In progress
- Done
- Blocked

To implement Kanban Board

A simple text file, Wiki, Notion cards, a spreadsheet, or a specialized app (e.g., Jira for Confluence, Asana), or built-in Issue Board functionality in gitlab or github

You can customize the specific columns to your needs and use additional advice from online resources

E.g., organizing tasks using multiple dimensions in Issue Board in gitlab

Summary

Some practices are key
- Short iterations with frequent deliveries
- Unit and integration tests
- Effective communication (e.g., through like daily scrum)
- Understanding the value flow in your project (Kanban board)
- Addressing technical debt through refactoring
In practice, a few you can experiment with practices from different methods and mix and match if you find them to work well for your team and project

Exercise Questions

Revisit the concepts of Process Models

Waterfall model
- Risk of building wrong system due to no user feedback, Operational quality portion of product in the end
- Only work best when there is a complete and frozen specification document
V-model
- Basically waterfall model, but represented as a V-shape to emphasize verification and validation using different type of testing
- Still have the same risk of Waterfall model.

What is the biggest drawback of the waterfall model?

Risk of building wrong system/delivering incorrect product due to not enough feedback

How can you address the main risk of the waterfall model?

Obtain more frequent feedback from customer/users; e.g., building a prototype up front, or using an iterative model

What is the main idea behind the V process?

Linking unit, integration, and system test stages with, respectively, unit design, system design, and system specification.

What is the main difference between the waterfall model and the V-model?

V-model is basically the waterfall model emphasise Verification and Validation using different type of testing

V-model is like waterfall, but it adds more detailed stages for the V&V part, including unit, integration, and system testing (and acceptance testing); each of these test stages is fed with specification information from, respectively, unit, design, and requirements development stages.

What problem does early prototyping address?

Reducing the risk of building wrong system because more customer involvement

What lifecycle model is used in agile development?

Evolutionary model

When is the waterfall model applicable?

When the customer requirements are fully clear and frozen (no uncertainties)