Software Quality - FAQ

Why does software have bugs?

• miscommunication or no communication
  as to specifics of what an application should or shouldn't do (the application's requirements).
• software complexity
  the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Windows-type interfaces, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity. And the use of object-oriented techniques can complicate instead of simplify a project unless it is well-engineered.
• programming errors
  programmers, like anyone else, can make mistakes.
• changing requirements
  the customer may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of coordinating changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control.
• time pressures
  scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.
• egos
  people prefer to say things like: 'no problem', 'piece of cake', 'I can whip that out in a few hours', 'it should be easy to update that old code', instead of: 'that adds a lot of complexity and we could end up making a lot of mistakes', 'we have no idea if we can do that; we'll wing it' ,'I can't estimate how long it will take, until I take a close look at it', 'we can't figure out what that old spaghetti code did in the first place'. If there are too many unrealistic 'no problem's', the result is bugs.
• poorly documented code
  it's tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable, maintainable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it ('if it was hard to write, it should be hard to read').
• software development tools
  visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.

How can new Software QA processes be introduced in an existing organization?

A lot depends on the size of the organization and the risks involved. For large organizations with high-risk (in terms of lives or property) projects, serious management buy-in is required and a formalized QA process is necessary. Where the risk is lower, management and organizational buy-in and QA implementation may be a slower, step-at-a-time process. QA processes should be balanced with productivity so as to keep bureaucracy from getting out of hand. For small groups or projects, a more ad-hoc process may be appropriate, depending on the type of customers and projects. A lot will depend on team leads or managers, feedback to developers, and ensuring adequate communications among customers, managers, developers, and testers. The most value for effort will often be in:

• a- requirements management processes, with a goal of clear, complete, testable requirement specifications embodied in requirements or design documentation, or in 'agile'-type environments extensive continuous coordination with end-users,
• b- design inspections and code inspections, and
• c- post-mortems/retrospectives.

What are the common problems in the software development process?

• poor requirements
  if requirements are unclear, incomplete, too general, and not testable, there will be problems.
• unrealistic schedule
  if too much work is crammed in too little time, problems are inevitable.
• inadequate testing
  no one will know whether or not the program is any good until the customer complains or systems crash.
• featuritis
  requests to pile on new features after development is underway; extremely common.
• miscommunication
  if developers don't know what's needed or customer's have erroneous expectations, problems are guaranteed.

What are the common solutions to software development problems?

• solid requirements
  clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all players. Use prototypes to help nail down requirements. In 'agile'-type environments, continuous coordination with customers/end-users is necessary.
• realistic schedules
  allow adequate time for planning, design, testing, bug fixing, re-testing, changes, and documentation; personnel should be able to complete the project without burning out.
• adequate testing
  start testing early on, re-test after fixes or changes, plan for adequate time for testing and bug-fixing. 'Early' testing ideally includes unit testing by developers and built-in testing and diagnostic capabilities.
• stick to initial requirements as much as possible
  be prepared to defend against excessive changes and additions once development has begun, and be prepared to explain consequences. If changes are necessary, they should be adequately reflected in related schedule changes. If possible, work closely with customers/end-users to manage expectations. This will provide them a higher comfort level with their requirements decisions and minimize excessive changes later on.
• communication
  require walkthroughs and inspections when appropriate; make extensive use of group communication tools - e-mail, groupware, networked bug-tracking tools and change management tools, intranet capabilities, etc.; insure that information/documentation is available and up-to-date - preferably electronic, not paper; promote teamwork and cooperation; use protoypes if possible to clarify customers' expectations.

What's the role of documentation in QA?

Critical. (Note that documentation can be electronic, not necessarily paper, may be embedded in code comments, etc.) QA practices should be documented such that they are repeatable. Specifications, designs, business rules, inspection reports, configurations, code changes, test plans, test cases, bug reports, user manuals, etc. should all be documented in some form. There should ideally be a system for easily finding and obtaining information and determining what documentation will have a particular piece of information. Change management for documentation should be used if possible.

What's the big deal about 'requirements'?

One of the most reliable methods of insuring problems, or failure, in a large, complex software project is to have poorly documented requirements specifications. Requirements are the details describing an application's externally-perceived functionality and properties. Requirements should be clear, complete, reasonably detailed, cohesive, attainable, and testable. A non-testable requirement would be, for example, 'user-friendly' (too subjective). A testable requirement would be something like 'the user must enter their previously-assigned password to access the application'. Determining and organizing requirements details in a useful and efficient way can be a difficult effort; different methods are available depending on the particular project. Many books are available that describe various approaches to this task. Care should be taken to involve ALL of a project's significant 'customers' in the requirements process. 'Customers' could be in-house personnel or out, and could include end-users, customer acceptance testers, customer contract officers, customer management, future software maintenance engineers, salespeople, etc. Anyone who could later derail the project if their expectations aren't met should be included if possible. Organizations vary considerably in their handling of requirements specifications. Ideally, the requirements are spelled out in a document with statements such as 'The product shall.....'. 'Design' specifications should not be confused with 'requirements'; design specifications should be traceable back to the requirements. In some organizations requirements may end up in high level project plans, functional specification documents, in design documents, or in other documents at various levels of detail. No matter what they are called, some type of documentation with detailed requirements will be needed by testers in order to properly plan and execute tests. Without such documentation, there will be no clear-cut way to determine if a software application is performing correctly. 'Agile' methods such as XP use methods requiring close interaction and cooperation between programmers and customers/end-users to iteratively develop requirements. The programmer uses 'Test first' development to first create automated unit testing code, which essentially embodies the requirements.

What can be done if requirements are changing continuously?

A common problem and a major headache.

• Work with the project's stakeholders early on to understand how requirements might change so that alternate test plans and strategies can be worked out in advance, if possible.
• It's helpful if the application's initial design allows for some adaptability so that later changes do not require redoing the application from scratch.
• If the code is well-commented and well-documented this makes changes easier for the developers.
• Use rapid prototyping whenever possible to help customers feel sure of their requirements and minimize changes.
• The project's initial schedule should allow for some extra time commensurate with the possibility of changes.
• Try to move new requirements to a 'Phase 2' version of an application, while using the original requirements for the 'Phase 1' version.
• Negotiate to allow only easily-implemented new requirements into the project, while moving more difficult new requirements into future versions of the application.
• Be sure that customers and management understand the scheduling impacts, inherent risks, and costs of significant requirements changes. Then let management or the customers (not the developers or testers) decide if the changes are warranted - after all, that's their job.
• Balance the effort put into setting up automated testing with the expected effort required to re-do them to deal with changes.
• Try to design some flexibility into automated test scripts.
• Focus initial automated testing on application aspects that are most likely to remain unchanged.
• Devote appropriate effort to risk analysis of changes to minimize regression testing needs.
• Design some flexibility into test cases (this is not easily done; the best bet might be to minimize the detail in the test cases, or set up only higher-level generic-type test plans)
• Focus less on detailed test plans and test cases and more on ad hoc testing (with an understanding of the added risk that this entails).

How can Software QA processes be implemented without stifling productivity?

By implementing QA processes slowly over time, using consensus to reach agreement on processes, and adjusting and experimenting as an organization grows and matures, productivity will be improved instead of stifled. Problem prevention will lessen the need for problem detection, panics and burn-out will decrease, and there will be improved focus and less wasted effort. At the same time, attempts should be made to keep processes simple and efficient, minimize paperwork, promote computer-based processes and automated tracking and reporting, minimize time required in meetings, and promote training as part of the QA process. However, no one - especially talented technical types - likes rules or bureacracy, and in the short run things may slow down a bit. A typical scenario would be that more days of planning and development will be needed, but less time will be required for late-night bug-fixing and calming of irate customers.

What if an organization is growing so fast that fixed QA processes are impossible?

This is a common problem in the software industry, especially in new technology areas. There is no easy solution in this situation, other than:

• Hire good people
• Management should 'ruthlessly prioritize' quality issues and maintain focus on the customer
• Everyone in the organization should be clear on what 'quality' means to the customer