jasonbutz.info

Amazon CodeCatalyst Quality Reports - What Reports Should Be

• AWS, Amazon CodeCatalyst, Code Quality

In my last post, I raised some concerns with Amazon CodeCatalyst and how it might be received in an enterprise environment, especially the single sign-on capabilities. After that, I spoke with some of the CodeCatalyst team and clarified my perspective for them. I got the sense they were aware of most of the issues I raised, and they seemed very open to my feedback. I can’t share all the details we discussed, but I look forward to what they have coming in the future.

One thing that the initial post left out was my favorite part of CodeCatalyst: the quality reports.

Quality Reports

Amazon CodeCatalyst supports multiple tests, code coverage, software composition analysis (SCA), and static analysis (SA) reports with little to no configuration. CodeCatalyst workflows support attempting to discover the report files automatically. You can also specify where the files are located and provide additional configuration to help fine-tune the generated reports and what causes the workflow to fail.

Test Reports

You can configure CodeCatalyst to display the results of your unit tests or any other automated test run during your builds as a report. The report will show you the pass rate of your tests, how long the tests took to run, and the different test cases and whether they passed or failed. CodeCatalyst supports the Cucumber JSON, JUnit XML, NUnit XML, NUnit3 XML, TestNG XML, and Visual Studio TRX test report formats(1).

AWS CodeCatalyst test report page, showing a failed report with a 50% pass rate with one succeeded test and one failed test

Code Coverage Reports

CodeCatalyst’s code coverage report shows the line and branch coverage from an overview and per-file basis. It also details the coverage within different folders in your file tree. It allows you to dig into the details of a specific file and see what portions of the file are covered by the tests. CodeCatalyst supports the JaCoCo XML, SimpleCov JSON, Clover XML v3, Cobertura XML, and LCOV code coverage report formats(2).

AWS CodeCatalyst code coverage report page, showing 15% line coverage and 0% branch coverage

Software Composition Analysis Reports & Static Analysis Reports

CodeCatalyst has separate sections in the documentation for software composition analysis (SCA) and static analysis (SA), but there is a lot of cross-over between the two. The documentation lists SARIF as the only supported SCA report file format(3). Three formats are supported for SA reports: PyLint, ESLint, and SARIF(4). In my testing, I had trouble getting CodeCatalyst to flag a report as being a software composition analysis report. It never automatically detected the reports, and when I manually defined them, they showed no findings, even though a review of the files indicated they should show something. I suspect there is a subtle difference in the SARIF formatted output between what CodeCatalyst expects and what the tools I tried generated. Ensure that the SARIF output from your tools matches the standard and that the reports show the output correctly. Otherwise, you could end up with a false result. CodeCatalyst detected ESLint output without any issues.

AWS CodeCatalyst static analysis report page, showing a failed report with one quality issue, zero bugs, zero security vulnerabilities, and zero 'other' issues. Shows the message associated with the single finding, 'Missing semicolon'

Reports Access

Accessing the quality reports is easy, and you can view them for a specific workflow run or directly from a pull request. If you have pull requests going through multiple revisions, you can even view the reports for the different revisions. One situation I commonly run into is working with an engineer to improve their unit test coverage and the quality of their unit tests. Being able to jump back to their updates from a week prior quickly will help save time when trying to look at incremental progress.

AWS CodeCatalyst page showing the reports associated with a specific run of a 'onPullRequestBuildAndTest' workflow. Shows the test and static analysis reports as 'succeeded' and the code coverage report as 'no status'.

AWS CodeCatalyst page showing the reports associated with revision 20 of a pull request titled 'Reports Samples'. Shows the test and static analysis reports as 'succeeded' and the code coverage report as 'no status'.

CodeCatalyst’s Differentiator?

I’ve never seen easy access to code quality reports in GitHub, GitLab, or BitBucket without going through a lot of configuration and integrations with outside tools. This is one area where CodeCatalyst truly shines. When I first saw these reports and how easy they were to set up and access, I immediately shared a project with my boss and some of my peers, and we spent a good amount of time discussing the difference this would have made for us over the years. Other tools provide many of the same features as CodeCatalyst, but when it comes to reports integrated into the solution, CodeCatalyst seems to be in the lead.