Meet Sputnik - static code analyser for Gerrit

Monday, May 26, 2014

Sputnik runs Checkstyle, PMD and FindBugs for your Gerrit patchsets

I am happy to announce a first release of Sputnik! It is a static code analyzer that runs Checkstyle, PMD and FindBugs for your Gerrit patchsets. Its main advantage over my previous project Sonar Gerrit plugin is that Sputnik is a small, lightweight and standalone Java application. You don't need any other software to run it. It bundles Checkstyle, PMD and FindBugs jars within distribution zip.

Workflow

Sputnik is intended to use with Gerrit and Continous Integration server, i. e. Jenkins. It works like this:

Your CI server is updated by ssh that a new patch is submitted to Gerrit. CI fetches this patch and builds a while project. After a build, CI server reports its result to Gerrit. It's time for Sputnik now.

Sputnik runs regardless of build result (you can change that in your CI configuration). Sputnik fetches patchset's file list from Gerrit over HTTP REST API. Then it runs an analysis only on these files! Even if your project is huge, analysis on several files takes only seconds. Sputnik collects comments from all three analysers: Checkstyle, PMD and FindBugs. It sends back all comments to Gerrit via HTTP REST API back. It's very simple and very fast!

Installation and configuration

First, you need to build https://github.com/TouK/sputnik master or download distribution zip from here: sputnik-1.0.zip. Go to you CI server and extract it to a directory of your choice. Remember that a user you run CI builds needs to have an access rights to this directory (in my case it's simply a jenkins user). Then you need to prepare your configuration file and write this file to the same directory as unzipped distribution. It is a simple Java properties file, which is pretty self-explanatory. Here is an example:

gerrit.host=gerrit.yourcompany.com
gerrit.port=8080
gerrit.username=sputnik
gerrit.password=Pa$$wo4d
checkstyle.enabled=true
checkstyle.configurationFile=/opt/jenkins/sputnik/checkstyle.xml
checkstyle.propertiesFile=
pmd.enabled=true
pmd.ruleSets=/opt/jenkins/sputnik/pmd.xml
findbugs.enabled=true
findbugs.includeFilter=/opt/jenkins/sputnik/findbugs.xml
findbugs.excludeFilter=

Now you need to configure you CI server to actually run Sputnik after a build. It is very simple for Jenkins, just add a Post-Build Step. You can adjust if Sputnik runs only on successful build or for every build - use radio buttons for this:

Last line with exit 0 is a workaround for a clean exit, even if Sputnik fails for some reason. Exit 0 guarantees you that result of this step doesn't affect overall build result.

Summary

This is an example screenshot of Sputnik's comments:

Sputnik always reports +1 as a result. It can be lacking in some network and authorisation configuration. But it's open source so please submit issues and patches to its github page: https://github.com/TouK/sputnik.

Your feedback and pull requests are heartly welcome!

Sonar Gerrit Plugin Release

Friday, December 20, 2013

Initial release

I am happy to announce a first release of my Sonar Gerrit plugin. This plugin reports Sonar violations on your patchsets to your Gerrit server. Sonar analyses full project, but only files included in patchset are commented on Gerrit. Please forward to project page for installation instructions.

This plugin is intended to use with Gerrit Trigger plugin for Jenkins CI server. Together they provide a great tool for automatic static code analysis.

How does it work?

At the moment you push a patchset to Gerrit, Jenkins is notified with a ssh event. It fetches a code with a patchset and it builds your changes. It quits when build or tests fail.

But if it succeeds, Sonar analase your project in a post-build action. This is a place where my Sonar Gerrit plugin shines. It asks Gerrit for changed files before analysis and after Sonar analysis is finished, plugin reports comments on these files as a Gerrit reviewer. Currently plugin always reports +1 for Code Review, as it's still in development. However, you should always treat these comments as hints to improve, not as direct errors.

Extras

I've released also a second plugin: Sonar File Alerts plugin. This plugin raises alerts on file level in Sonar. It extends default behaviour, which raises alerts only at root project level. It is useful when you create alert rules in Sonar like "Code Coverage < 60". Each file is checked against this rule!

If you use Sonar File Alerts plugin and an alert will be generated on some file, then a comment will be published on this file on Gerrit.

Feedback

Please provide a feedback on these plugins. Feel free to submit issues on github or comment. It's still an early stage so your input is very welcome!

Custom SonarQube rules for Unit Tests

Monday, December 2, 2013

I need a new rule

In our project we use (formely Sonar) to manage our code quality. It is a great tool and I recommend everyone to set it up and read its reports.

Recently, we've agreed that it's better to use assertj assertions in our unit tests than JUnit's. So I've decided to write a simple rule that checks if some of JUnit asserts assertTrue, assertFalse, assertNull and others are used. Then, I've discovered it's not so easy to do it with Sonar:

  • only 10 code quality rules are applied to unit tests - they are in special repository PMD Unit Tests (source)
  • these 10 rules are disabled by default, you have to enable them by hand
  • you cannot add new rules to this group

However, it turned out it is doable with a small tricks.

Custom PMD Unit Tests rule tutorial

Create your XPath expression by following this tutorial on how to create custom PMD rule. There is a visual editor to test your rules as you develop them - that's great. My XPath expression to avoid all JUnit assertions looks like this:

//PrimaryPrefix/Name[@Image='assertEquals' or @Image='assertNull' or @Image='assertNotNull' or @Image='assertSame' or @Image='assertNotSame' or @Image='assertArrayEquals' or @Image='assertTrue' or @Image='assertFalse']

Go to your Sonar installation, log in as an Administrator, head to Quality Profiles and select a profile that you use. Search for "xpath" and change Activation to Any. You should see two results like this:

Expand XPath rule template (dont' worry that it says it's deprecated) and then click Copy rule. Fill a form with message and XPath and save it. Then take a look at the bottom - you need an identifier of this rule:

You have created a PMD rule, now you need to move it to PMD Unit Tests group. Connect to Sonar's MySQL database. Search for your rule by key:

mysql> select id, plugin_rule_key, plugin_name, parent_id, status from rules where plugin_rule_key='XPathRule_1385721910';
+-----+----------------------+----------------+-----------+-------------+
| id  | plugin_rule_key      | plugin_name    | parent_id | status      |
+-----+----------------------+----------------+-----------+-------------+
| 903 | XPathRule_1385721910 | pmd            |      NULL | DEPRECATED  |
+-----+----------------------+----------------+-----------+-------------+
1 row in set (0.00 sec)

Update plugin_name and status (remember to use appropiate primary key for id column):

mysql> update rules set plugin_name='pmd-unit-tests', status='READY' where id=903;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

There is one step left. Sonar will change this rule's status to REMOVED on restart due to his boot checks. You need to trick him and change parent_id to other's PMD Unit Tests rule. List all these rules and choose one's identifier.

mysql> select id, plugin_name, status from rules where plugin_name='pmd-unit-tests';
+-----+----------------+---------+
| id  | plugin_name    | status  |
+-----+----------------+---------+
| 775 | pmd-unit-tests | READY   |
| 776 | pmd-unit-tests | READY   |
| 777 | pmd-unit-tests | READY   |
| 778 | pmd-unit-tests | READY   |
| 779 | pmd-unit-tests | READY   |
| 780 | pmd-unit-tests | READY   |
| 781 | pmd-unit-tests | READY   |
| 782 | pmd-unit-tests | READY   |
| 783 | pmd-unit-tests | READY   |
| 784 | pmd-unit-tests | READY   |
| 903 | pmd-unit-tests | READY   |
+-----+----------------+---------+
11 rows in set (0.00 sec)

Choose any id you like, let's say 775 and apply it as parent_id to your newly created rule:

mysql> update rules set parent_id=775 where id=903;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Go to your Quality profile and make sure your rule is active! Check it twice, it's easy to forget that step. It's all set up, enjoy your analysis!

Error generating web.xml file with IntelliJ IDEA

Thursday, August 1, 2013

If you use IntelliJ IDEA for your Grails development you might encounter this error running integration tests:

Error Error generating web.xml file (Use --stacktrace to see the full trace)

The reason for this is that IDEA adds classpath by default on creating integration test run configuration. Unfortunately, sometimes it causes strange errors like this one. Follow these steps to resolve:

  1. Open Run → Edit Configurations... (or press Alt+Shift+F10)
  2. Select your configuration that fails
  3. Uncheck Add --classpath checkbox
  4. You are done! Run.

Local Graphite installation on CentOS 5.5 howto

Tuesday, July 9, 2013

Feature request

Our client called:
- "I want something to monitor my application!"
- "Ok, let's use JMX, we have plenty of stats exposed there already."
- "Fine, but I want something fancy!"
- "Ok, let's install Graphite, take a look at screenshots!"
- "Excellent! Use our testing CentOS machine for this."
- "Ok, erm... wait, nooooo! We have no root access there, there is no internet access and we can't install anything!!!"
- "(hangup click)"

Setup

There I was, standing alone, looking for help...

    My scenario:
  • no internet access
  • only user account, no root account
  • gcc and make installed
  • some libraries missing
  • no development packages for installed libraries
    My goals:
  • install libraries to $HOME
  • install Graphite and stuff to $HOME/graphite

Solution

It took me a few days, but I've managed to install Graphite locally. Here is a gist for a sake of documentation and for others that may need it.

https://gist.github.com/SpOOnman/5957589

Simple trick to DRY your Grails controller

Wednesday, June 5, 2013

Grails controllers are not very DRY. It's easy to find duplicated code fragments in default generated controller. Take a look at code sample below. It is duplicated four times in show, edit, update and delete actions:

class BookController {
    def show() {
       def bookInstance = Book.get(params.id)
       if (!bookInstance) {
            flash.message = message(code: 'default.not.found.message', args: [message(code: 'book.label', default: 'Book'), params.id])
            redirect(action: "list")
            return
        }
        [bookInstance: bookInstance]
    }
}

Why is it duplicated?

There is a reason for that duplication, though. If you move this snippet to a method, it can redirect to "list" action, but it can't prevent controller from further execution. After you call redirect, response status changes to 302, but after method exits, controller still runs subsequent code.

Solution

At TouK we've implemented a simple trick to resolve that situation:

  1. wrap everything with a simple withStoppingOnRender method,
  2. whenever you want to render or redirect AND stop controller execution - throw EndRenderingException.

We call it Big Return - return from a method and return from a controller at once. Here is how it works:

class BookController {
    def show(Long id) {
        withStoppingOnRender {
            Book bookInstance = Book.get(id)
            validateInstanceExists(bookInstance)
            [bookInstance: bookInstance]
        }
    }

    protected Object withStoppingOnRender(Closure closure) {
        try {
            return closure.call()
        } catch (EndRenderingException e) {}
    }

    private void validateInstanceExists(Book instance) {
        if (!instance) {
            flash.message = message(code: 'default.not.found.message', args: [message(code: 'book.label', default: 'Book'), params.id])
            redirect(action: "list")
            throw new EndRenderingException()
        }
    }
}

class EndRenderingException extends RuntimeException {}

Example usage

For simple CRUD controllers, you can use this solution and create some BaseController class for your controllers. We use withStoppingOnRender in every controller so code doesn't look like a spaghetti, we follow DRY principle and code is self-documented. Win-win-win! Here is a more complex example:

class DealerController {
    @Transactional
    def update() {
        withStoppingOnRender {
            Dealer dealerInstance = Dealer.get(params.id)
            validateInstanceExists(dealerInstance)
            validateAccountInExternalService(dealerInstance)
            checkIfInstanceWasConcurrentlyModified(dealerInstance, params.version)
            dealerInstance.properties = params
            saveUpdatedInstance(dealerInstance)
            redirectToAfterUpdate(dealerInstance)
        }
    }
}

New HTTP Logger Grails plugin

Tuesday, May 7, 2013

I've wrote a new Grails plugin - httplogger. It logs:

  • request information (url, headers, cookies, method, body),
  • grails dispatch information (controller, action, parameters),
  • response information (elapsed time and body).

It is mostly useful for logging your REST traffic. Full HTTP web pages can be huge to log and generally waste your space. I suggest to map all of your REST controllers with the same path in UrlMappings, e.g. /rest/ and configure this plugin with this path.

Here is some simple output just to give you a taste of it.

17:16:00,331 INFO  filters.LogRawRequestInfoFilter  - << #1 GET http://localhost:8080/riddle/rest/index?username=admin&search=foo
17:16:00,340 INFO  filters.LogRawRequestInfoFilter  - << #1 headers Cookie: 'JSESSIONID=DF4EA5725AC4A4990281BD96963739B0; splashShown1.6=1', Accept-Language: 'en-US,en;q=0.8,pl;q=0.6', X-MyHeader: 'null'
17:16:00,342 INFO  filters.LogGrailsUrlsInfoFilter  - << #1 dispatched to rest/index with parsed params [username:[admin], search:[foo]].
17:16:00,731 INFO  filters.LogOutputResponseFilter  - >> #1 returned 200, took 405 ms.
17:16:00,745 INFO  filters.LogOutputResponseFilter  - >> #1 responded with '{count:0}'
17:18:55,799 INFO  filters.LogRawRequestInfoFilter  - << #2 POST http://localhost:8080/riddle/rest/login
17:18:55,799 INFO  filters.LogRawRequestInfoFilter  - << #2 headers Cookie: 'JSESSIONID=DF4EA5725AC4A4990281BD96963739B0; splashShown1.6=1', Accept-Language: 'en-US,en;q=0.8,pl;q=0.6', X-MyHeader: 'null'
17:18:55,800 INFO  filters.LogRawRequestInfoFilter  - << #2 body: 'username=admin&password=password'
17:18:55,801 INFO  filters.LogOutputResponseFilter  - >> #2 returned 404, took 3 ms.
17:18:55,802 INFO  filters.LogOutputResponseFilter  - >> #2 responded with ''

Official plugin information can be found on Grails plugins website here: http://grails.org/plugins/httplogger or you can browse code on github: TouK/grails-httplogger.