Half a year ago, we announced that a student, Said Salem, would work on creating a better reporting solution for ProR in the context of his master thesis.  The good news is: He completed his thesis and passed the exam.  The bad news: The resulting implementations did not mature beyond a prototype stage, meaning that we will not incorporate the results into the RMF code base for now.  He pledged to publish his code on gitHub (some is already there), allowing other parties (including Formal Mind) to pick it up and develop it to maturity.

Reporting for ReqIF: From Analysis to Evaluation (Master Thesis)

The master thesis is available as PDF, written in German.  The main parts include a problem analysis, three prototypical implementations, and an evaluation of those prototypes with respect to the results of the initial problem analysis.

In the following, you find the highlights from this work.

Problem Analysis

The problem analysis includes the responses to the questionnaire that Mr. Salem distributed, and which resulted in 59 responses.  Many responses came from subscribers of this newsletter, and therefore the result is probably not representative for users of requirements tools, but more for the ProR community:

Next, users had to indicate which features they would consider important in a reporting solution.  Here we observed that some of the dearer features were in fact not related to reporting alone, but were highly desirable features for the user interface.  Filtering, for instance, was the feature with the highest votes, and which would clearly add value in the user interface, in addition to a reporting solution.

In retrospect, the phrasing of the features was in many cases not as good as it could be, leading to ambiguous results:

You’ll find more statistics in the actual thesis.

After some analysis, Mr. Salem decided to handle the filter functionality separately and to implement it in the ProR Core, rather than in a specific reporting solution.  Due to the prototypical (incomplete) nature of the implementation, it has not yet been added to the ProR code base.

Reporting with Birt

Birt is a popular reporting framework for Eclipse, which has a rich set of features.  Further, Birt already supports many features “out of the box” that were desired by users, and more.  For instance, Birt supports the creation of reports in various formats, and has an extensive feature set for branding, data analysis (e.g. generation of graphs) and more.

On the flip side, Birt expects its data in a certain structure, and transforming the ReqIF data from ProR into such structures turned out to be tricky.  In addition, Birt does not support hierarchical information natively.  To work around that, in the prototype, the hierarchy is flattened (see Figure).

Reporting with JSON/HTML

This approach introduces an intermediate data format (using the JSON notation).  This accounts for the fact that ReqIF is more complex than needed for generating a report.  The idea is that the intermediate format can be used not only for generating a HTML report, but to generate reports in other formats as well.

Reporting with Excel

Reporting with a tool like Excel is compelling.  For one, many users are familiar with it and know how to customize it.  But more importantly, Excel contains powerful functions for advanced analysis.

In the implementation, the user must provide a template, which allows for reusable branding.  Existing libraries make it easy and fast to implement this functionality.  However, Excel has the same limitation as Birt: It cannot deal (easily) with hierarchies, and in the prototypical implementation, the structure is flattened as well.

Conclusion

By giving weight to the user wishes and evaluating the various implementations against them, Mr. Salem made an attempt to find the “best” solution.  The winner is Birt, closely followed by Excel, with HTML/JSON being a far third.

This result is mainly influenced by filtering, tables, and the desire for Excel as a reporting format.  Thus, this result has to be taken with a grain of salt as well: If a good filter is implemented in the ProR Core, then it would not affect the HTML rating.The maturity of Birt and Excel explains the remaining advantage of those two technologies.

We believe that the findings from this work are quite useful, both to the users of ProR in general, and the developers in particular.  The big downer, of course, is the fact that the resulting prototypes did not reach the level of maturity that we had hoped for.  Should Formal Mind pick up this work to develop it to maturity, we will inform you via our Blog and Newsletter.