Introduction
From Conqat
Contents |
Background
A continuous and timely quality control process is crucial to maintain a high level of quality in software products. Many quality defects are hardly reversible and once a certain number of defects has found its way into the system it will very likely suffer from rapid decay due to the broken-window effect. Manual quality control activities are inherently costly and must be substituted or supported with adequate tool support as far as possible. As a rule of thumb, the likelihood and frequency of a sophisticated quality assessments correlates with the availability of quality assurance tools. Although there are already various quality assessment tools on the market, none of tools known to us is capable of supporting a quality control process as we envisage it.
Requirements
A suitable tool support for continuous quality control has to fulfill the following requirements.
- Non-interactive operation. As the tool supported assessments need to be carried out regularly (e. g. daily) the tool must be able to work in a completely autonomous way. The assessment results should be accessible to all project participants; e. g. on a website. At this location it must provide an integrated view onto all facts assessed.
- Aggregation. Even simple data like lines of code (LOC), may become indistinct if presented in full details. Therefore, the tool must be capable of compacting the data. An example is the summation of the LOC of all files for every subsystem. Similarly important is the presentation of raw metric values. A typical approach is to use colors such as green values below a certain threshold and yellow or red otherwise. Of course, these assessments need to be aggregated to higher levels, too.
- Scalability. A quality control tool must be able to handle very large systems. Typical counter-examples are data warehouse based products which load every imaginable fact about a software system into a database before a single analysis can be run. As the fact extraction may take several hours for large systems it is unacceptable for real-time quality control.
- Diversity. The factors of software quality are diverse. Therefore a quality control tool may not be limited to a certain type of factors or artifacts it analyzes. It must e. g. not only be able to analyze source code but should provide measures to include other artifacts like documentation, models or build scripts. Specifically it should feature an interface to a review database where results of manual inspections are stored.
- Extensibility. As no tool can innately support the whole spectrum of all possible facts it must provide an extension mechanism that allows users to add further analysis or assessment modules as needed.
- Flexibility. The tool must allow an easy combination of different analysis. For different projects or different project phases the objectives of quality assessment are different. The tool must feature a flexible configuration mechanism.
ConQAT's Design
The design of ConQAT focuses on extensibility and flexibility. We therefore opted for a plug-in architecture which allows for the addition and removal of analysis modules at load time. These analysis modules (so-called processors) can be interconnected in a pipes-and-filters-style to form an analysis network. A processor may be regarded similar to a mathematical function with an arbitrary number of input channels and a single output channel. An example is given in figure blow.
It shows a network of eight processors whereas the processors for output are unamed. This processor configuration was designed to answer the following questions:
- Which classes have unused code and a limited amount of comments?
- How are these bad classes distributed over the project?
In the example the processor JavaScope is responsible for reading the projects source code from the harddisk. Processor UnusedCodeAnalyzer finds unused code and processor CommentRatioAnalyzer determines the comment ratio for each file and assess them according to a given threshold. The LOCAnalyzer counts the lines of code for each source file.
For the two questions to answer, two different aggregations and visualizations of the assessment data are used. For question 1 all files are listed in a table that shows warnings regarding the defined criteria. To limit the amount of information, the Filter filters all files that do not exhibit any warnings.
For question 2 a treemap is used for visualizations. This treemap shows all the files of a project as rectangles where the rectangles size reflects the size of the file in lines of code. The color of the rectangle is either green (no warnings), yellow (unsued code or low comment ratio) or red (unused code and low comment ratio). This visualization enables the quality engineer to get a quick but comprehensive overview of the distribution of bad files.
Flexibility
This design offers a high degree of reuse and provides a powerful configuration mechanism as processor may be interconnected in numerous ways. The functionality of processors is not restricted in any way and thereby enables the tool to cope with the diversity of factors influencing maintainability. This is underlined by the broad spectrum of processors we already implemented, including
- common metrics analyzers, e. g. lines of code, depth of inheritance
- unit test result analyzers
- documentation auditors
- checks for compliance to coding rules for code
- redundancy analysis
- architecture assessments
Particularly valuable is a set of processors which act as adapters to other quality assessment tools like PMD.
Performance
A great number of processors work on the same input data, e. g. raw files, syntactic tokens, the abstract syntax trees of source file or the dependency-graph of a system. To minimize redundant work and memory load ConQAT offers a number of libraries for these different levels of abstractions, e. g. a parser for Java source files. These libraries do not only cache their results to reduce processing times but furthermore use each other. So the lexer library uses pre-cached file contents if a file has already been processed by the lexer before. Caches are implemented in a memory-sensitive way and are able to uncache data if the system falls short of memory.
Meta-Model Based Approaches
ConQAT’s design differs fundamentally from another approach frequently chosen in the design of software analysis tools: the construction of a meta-model that describes the analyzed code on a certain level of abstraction. The meta-model is typically relational or object-oriented. Actual analysis are carried out on the model level only. Although this might appear as an elegant solution it has a major drawback: it rigidly defines the level of abstraction used for analyses and assessments. As stated above, the relevant factors for maintainability are diverse and are therefore scattered across different levels of abstraction. An example is the simple yet effective search for source code fragments with the string “FIXME”. This can’t be performed on a callgraph representation of the system and in addition to this, building the call-graph is a waste of time for this type of analyses. This problem is usually addressed by storing multiple abstraction levels in the model which it turn makes its creation very costly. Our design effectively limits analysis overhead by only carrying out those processing steps (e. g. parsing) which are actually needed while it ensures performance by caching the results produced.
Usage Scenarios
The utility of ConQAT as well as the reasons for some design decisions are best seen in the context of typical scenarios for which we have used ConQAT so far.
- Nightly Build. ConQAT was built to be integrated in your typical nightly build process, so every morning the current state (in terms of progress as well as quality metrics) of a software system under development (or maintenance) can be easily checked. The idea is to decide on criteria to be checked, configure ConQAT once, and then have a single source of information on the project.
Our observation is that having those automatic assessments greatly helps in steering a development or maintenance project the right way. But also the developers profit from the immediate feedback on their work, as reacting early on quality defects makes it easier to keep the overall software quality on a high level.
- Interactive Analysis of Software. While it was not explicitly designed for this task, with some experience ConQAT can be brought to good use in interactively exploring unknown systems and finding potential design and quality deficiencies. For example ConQAT can find important core classes of a system, help in identifying the underlying architecture as well as detect violations of architecture rules.
Using ConQAT we easily could point several partners in industry to problems in their software, which they did not know of but which clearly would hamper further development or maintenance. Having these problems identified simplified their removal.
