How to recover after Hibernate's OptimisticLockException

Tuesday, October 20, 2015

I've read many articles about optimistic locking and OptimisticLockException itself. Problem is that each one of them ended up getting their first exception and no word on recovery. What to do next? Repeat? If so, how? Or drop it? Is there any chance to continue? How? Even more, documentation says that if you get Hibernate exception - you're done, it's not recoverable:

An exception thrown by Hibernate means you have to rollback your database transaction and close the Session immediately (this is discussed in more detail later in the chapter). If your Session is bound to the application, you have to stop the application. Rolling back the database transaction does not put your business objects back into the state they were at the start of the transaction. This means that the database state and the business objects will be out of sync. Usually this is not a problem, because exceptions are not recoverable and you will have to start over after rollback anyway.

Here is my attempt on this: repeatable and recoverable.

Business case

Let's say we have distributed application with two web servers, connected to the same database. Applications use optimistic locking to avoid collisions. Customers buy lottery coupons, which are numbered from 1 to 100. In the same second Alice on web server 1 draws two coupons: 11 and 12. In the same moment Bob reserves two coupons on web server 2. It draws 11 and 13 for Bob and tries to write it back to database. But it fails, since Alice's commit was first. I want a web application server to draw coupons for Bob again and then - try to save again until it succeeds.

Solution

For every request Hibernate associates different Session that is flushed at the end of request processing. If you hit OptimisticLockException then this Request Session is polluted and will be rolled back. To avoid this we will create a separate Hibernate's Session especially for drawing coupons. If separate session fails - drop this session and try again in a new one. If it succeeds - merge it with a main request session. Request Session cannot be touched during draws. Take a look at the following picture:

On this picture yellow and green short-term sessions has failed with OptimisticLockException. Red session was successful and these objects are merged to a main session on the left.

Reservation entity

Key requirement here is to keep a domain you want to lock on as small as possible and not coupled directly to anything else. Best approach here is to create some Reservation entity with few fields, let's say: couponId and customerId. For each Coupon create one Reservation row and use reserved boolean field as a reservation status. For coupon and customer use weak identifiers (long) instead of real entities. This way no object tree will be loaded and Reservation stays decoupled.

import lombok.Getter;
import lombok.Setter;
import lombok.ToString;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;

import static org.apache.commons.lang3.Validate.isTrue;
import static org.springframework.util.Assert.isNull;

@Getter
@Setter
@ToString
@Entity
public class Reservation {
@Id
private long id;
@Column("coupon_id")
private long couponId;
@Column("customer_id")
private Long customerId;
@Column("is_reserved")
private boolean reserved;

public void reserve(long reservingCustomerId) {
isTrue(reserved == false);
isNull(customerId);
reserved = true;
customerId = reservingCustomerId;
}
}

Method reserve has some business assertions to keep it legit. You can also spot lombok's annotations that provide getters, setters and toString methods on the fly.

Reservation Service

Reservation Service implements my solution. I've commented some steps for better reading. Please read a source code:

import lombok.extern.slf4j.Slf4j;
import org.hibernate.*;
import org.hibernate.ejb.HibernateEntityManager;
import org.springframework.orm.hibernate4.HibernateOptimisticLockingFailureException;

import javax.persistence.OptimisticLockException;
import javax.persistence.PersistenceContext;
import java.util.List;

@Slf4j
public class ReservationService {

@PersistenceContext
private HibernateEntityManager hibernateEntityManager;

@SuppressWarnings("uncheked")
private Iterable<Reservation> reserveOptimistic(long customerId, final int count) throws NoFreeReservationsException {
log.info("Trying to reserve {} reservations for customer {}", count, customerId);

//This is the request session that needs to stay clean
Session currentSession = hibernateEntityManager.getSession();
Iterable<Reservation> reserved = null;

do {
//This is our temporary session to work on
Session newSession = hibernateEntityManager.getSession().getSessionFactory().openSession();
newSession.setFlushMode(FlushMode.COMMIT);
Transaction transaction = newSession.beginTransaction();

List<Reservation> availableReservations = null;

try {
Query query = newSession.createQuery("from Reservation r where r.reserved = false")
.setLockMode("optimistic", LockMode.OPTIMISTIC)
.setMaxResults(count);

availableReservations = query.list();

//There is no available reservations to reserve
if (availableReservations.isEmpty()) {
throw new NoFreeReservationsException();
}

for (Reservation available : availableReservations) {
available.reserve(customerId);
newSession.save(available);
}

//Commit can throw optimistic lock exception if it fails
transaction.commit();

//Commit succeeded - this reference is used outside try-catch-finally block
reserved = availableReservations;

} catch (OptimisticLockException | StaleObjectStateException | HibernateOptimisticLockingFailureException e) {
log.info("Optimistic lock exception occurred for customer {} and count {}: {} {}", customerId, count, e.getClass(), e.getMessage());

transaction.rollback();

for (Reservation availableMsisdn : availableReservations) {
newSession.evict(availableMsisdn);
}
} finally {
newSession.close();
}
//Repeat until we reserve something
} while (reserved == null);

log.info("Successfully reserved {} reservations for customer {}", count, customerId);

//Merge reserved entities to request session
for (Reservation reservedMsisdn : reserved) {
currentSession.merge(reservedMsisdn);
}

return reserved;
}
}

This code says it all. It tries to reserve some Reservations until it succeeds in a do-while loop. Main Request Session is not polluted and it achieves our goal.

I hope this example helps you in similar cases. It works as expected for a few months on our customer's production site and I recommend this solution.

Meet Sputnik - static code analyser for Gerrit

Monday, May 26, 2014

Sputnik runs Checkstyle, PMD and FindBugs for your Gerrit patchsets

I am happy to announce a first release of Sputnik! It is a static code analyzer that runs Checkstyle, PMD and FindBugs for your Gerrit patchsets. Its main advantage over my previous project Sonar Gerrit plugin is that Sputnik is a small, lightweight and standalone Java application. You don't need any other software to run it. It bundles Checkstyle, PMD and FindBugs jars within distribution zip.

Workflow

Sputnik is intended to use with Gerrit and Continous Integration server, i. e. Jenkins. It works like this:

Your CI server is updated by ssh that a new patch is submitted to Gerrit. CI fetches this patch and builds a while project. After a build, CI server reports its result to Gerrit. It's time for Sputnik now.

Sputnik runs regardless of build result (you can change that in your CI configuration). Sputnik fetches patchset's file list from Gerrit over HTTP REST API. Then it runs an analysis only on these files! Even if your project is huge, analysis on several files takes only seconds. Sputnik collects comments from all three analysers: Checkstyle, PMD and FindBugs. It sends back all comments to Gerrit via HTTP REST API back. It's very simple and very fast!

Installation and configuration

First, you need to build https://github.com/TouK/sputnik master or download distribution zip from here: sputnik-1.0.zip. Go to you CI server and extract it to a directory of your choice. Remember that a user you run CI builds needs to have an access rights to this directory (in my case it's simply a jenkins user). Then you need to prepare your configuration file and write this file to the same directory as unzipped distribution. It is a simple Java properties file, which is pretty self-explanatory. Here is an example:

gerrit.host=gerrit.yourcompany.com
gerrit.port=8080
gerrit.username=sputnik
gerrit.password=Pa$$wo4d
checkstyle.enabled=true
checkstyle.configurationFile=/opt/jenkins/sputnik/checkstyle.xml
checkstyle.propertiesFile=
pmd.enabled=true
pmd.ruleSets=/opt/jenkins/sputnik/pmd.xml
findbugs.enabled=true
findbugs.includeFilter=/opt/jenkins/sputnik/findbugs.xml
findbugs.excludeFilter=

Now you need to configure you CI server to actually run Sputnik after a build. It is very simple for Jenkins, just add a Post-Build Step. You can adjust if Sputnik runs only on successful build or for every build - use radio buttons for this:

Last line with exit 0 is a workaround for a clean exit, even if Sputnik fails for some reason. Exit 0 guarantees you that result of this step doesn't affect overall build result.

Summary

This is an example screenshot of Sputnik's comments:

Sputnik always reports +1 as a result. It can be lacking in some network and authorisation configuration. But it's open source so please submit issues and patches to its github page: https://github.com/TouK/sputnik.

Your feedback and pull requests are heartly welcome!

Sonar Gerrit Plugin Release

Friday, December 20, 2013

Initial release

I am happy to announce a first release of my Sonar Gerrit plugin. This plugin reports Sonar violations on your patchsets to your Gerrit server. Sonar analyses full project, but only files included in patchset are commented on Gerrit. Please forward to project page for installation instructions.

This plugin is intended to use with Gerrit Trigger plugin for Jenkins CI server. Together they provide a great tool for automatic static code analysis.

How does it work?

At the moment you push a patchset to Gerrit, Jenkins is notified with a ssh event. It fetches a code with a patchset and it builds your changes. It quits when build or tests fail.

But if it succeeds, Sonar analase your project in a post-build action. This is a place where my Sonar Gerrit plugin shines. It asks Gerrit for changed files before analysis and after Sonar analysis is finished, plugin reports comments on these files as a Gerrit reviewer. Currently plugin always reports +1 for Code Review, as it's still in development. However, you should always treat these comments as hints to improve, not as direct errors.

Extras

I've released also a second plugin: Sonar File Alerts plugin. This plugin raises alerts on file level in Sonar. It extends default behaviour, which raises alerts only at root project level. It is useful when you create alert rules in Sonar like "Code Coverage < 60". Each file is checked against this rule!

If you use Sonar File Alerts plugin and an alert will be generated on some file, then a comment will be published on this file on Gerrit.

Feedback

Please provide a feedback on these plugins. Feel free to submit issues on github or comment. It's still an early stage so your input is very welcome!

Custom SonarQube rules for Unit Tests

Monday, December 2, 2013

I need a new rule

In our project we use (formely Sonar) to manage our code quality. It is a great tool and I recommend everyone to set it up and read its reports.

Recently, we've agreed that it's better to use assertj assertions in our unit tests than JUnit's. So I've decided to write a simple rule that checks if some of JUnit asserts assertTrue, assertFalse, assertNull and others are used. Then, I've discovered it's not so easy to do it with Sonar:

  • only 10 code quality rules are applied to unit tests - they are in special repository PMD Unit Tests (source)
  • these 10 rules are disabled by default, you have to enable them by hand
  • you cannot add new rules to this group

However, it turned out it is doable with a small tricks.

Custom PMD Unit Tests rule tutorial

Create your XPath expression by following this tutorial on how to create custom PMD rule. There is a visual editor to test your rules as you develop them - that's great. My XPath expression to avoid all JUnit assertions looks like this:

//PrimaryPrefix/Name[@Image='assertEquals' or @Image='assertNull' or @Image='assertNotNull' or @Image='assertSame' or @Image='assertNotSame' or @Image='assertArrayEquals' or @Image='assertTrue' or @Image='assertFalse']

Go to your Sonar installation, log in as an Administrator, head to Quality Profiles and select a profile that you use. Search for "xpath" and change Activation to Any. You should see two results like this:

Expand XPath rule template (dont' worry that it says it's deprecated) and then click Copy rule. Fill a form with message and XPath and save it. Then take a look at the bottom - you need an identifier of this rule:

You have created a PMD rule, now you need to move it to PMD Unit Tests group. Connect to Sonar's MySQL database. Search for your rule by key:

mysql> select id, plugin_rule_key, plugin_name, parent_id, status from rules where plugin_rule_key='XPathRule_1385721910';
+-----+----------------------+----------------+-----------+-------------+
| id  | plugin_rule_key      | plugin_name    | parent_id | status      |
+-----+----------------------+----------------+-----------+-------------+
| 903 | XPathRule_1385721910 | pmd            |      NULL | DEPRECATED  |
+-----+----------------------+----------------+-----------+-------------+
1 row in set (0.00 sec)

Update plugin_name and status (remember to use appropiate primary key for id column):

mysql> update rules set plugin_name='pmd-unit-tests', status='READY' where id=903;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

There is one step left. Sonar will change this rule's status to REMOVED on restart due to his boot checks. You need to trick him and change parent_id to other's PMD Unit Tests rule. List all these rules and choose one's identifier.

mysql> select id, plugin_name, status from rules where plugin_name='pmd-unit-tests';
+-----+----------------+---------+
| id  | plugin_name    | status  |
+-----+----------------+---------+
| 775 | pmd-unit-tests | READY   |
| 776 | pmd-unit-tests | READY   |
| 777 | pmd-unit-tests | READY   |
| 778 | pmd-unit-tests | READY   |
| 779 | pmd-unit-tests | READY   |
| 780 | pmd-unit-tests | READY   |
| 781 | pmd-unit-tests | READY   |
| 782 | pmd-unit-tests | READY   |
| 783 | pmd-unit-tests | READY   |
| 784 | pmd-unit-tests | READY   |
| 903 | pmd-unit-tests | READY   |
+-----+----------------+---------+
11 rows in set (0.00 sec)

Choose any id you like, let's say 775 and apply it as parent_id to your newly created rule:

mysql> update rules set parent_id=775 where id=903;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Go to your Quality profile and make sure your rule is active! Check it twice, it's easy to forget that step. It's all set up, enjoy your analysis!

Error generating web.xml file with IntelliJ IDEA

Thursday, August 1, 2013

If you use IntelliJ IDEA for your Grails development you might encounter this error running integration tests:

Error Error generating web.xml file (Use --stacktrace to see the full trace)

The reason for this is that IDEA adds classpath by default on creating integration test run configuration. Unfortunately, sometimes it causes strange errors like this one. Follow these steps to resolve:

  1. Open Run → Edit Configurations... (or press Alt+Shift+F10)
  2. Select your configuration that fails
  3. Uncheck Add --classpath checkbox
  4. You are done! Run.

Local Graphite installation on CentOS 5.5 howto

Tuesday, July 9, 2013

Feature request

Our client called:
- "I want something to monitor my application!"
- "Ok, let's use JMX, we have plenty of stats exposed there already."
- "Fine, but I want something fancy!"
- "Ok, let's install Graphite, take a look at screenshots!"
- "Excellent! Use our testing CentOS machine for this."
- "Ok, erm... wait, nooooo! We have no root access there, there is no internet access and we can't install anything!!!"
- "(hangup click)"

Setup

There I was, standing alone, looking for help...

    My scenario:
  • no internet access
  • only user account, no root account
  • gcc and make installed
  • some libraries missing
  • no development packages for installed libraries
    My goals:
  • install libraries to $HOME
  • install Graphite and stuff to $HOME/graphite

Solution

It took me a few days, but I've managed to install Graphite locally. Here is a gist for a sake of documentation and for others that may need it.

https://gist.github.com/SpOOnman/5957589

Simple trick to DRY your Grails controller

Wednesday, June 5, 2013

Grails controllers are not very DRY. It's easy to find duplicated code fragments in default generated controller. Take a look at code sample below. It is duplicated four times in show, edit, update and delete actions:

class BookController {
    def show() {
       def bookInstance = Book.get(params.id)
       if (!bookInstance) {
            flash.message = message(code: 'default.not.found.message', args: [message(code: 'book.label', default: 'Book'), params.id])
            redirect(action: "list")
            return
        }
        [bookInstance: bookInstance]
    }
}

Why is it duplicated?

There is a reason for that duplication, though. If you move this snippet to a method, it can redirect to "list" action, but it can't prevent controller from further execution. After you call redirect, response status changes to 302, but after method exits, controller still runs subsequent code.

Solution

At TouK we've implemented a simple trick to resolve that situation:

  1. wrap everything with a simple withStoppingOnRender method,
  2. whenever you want to render or redirect AND stop controller execution - throw EndRenderingException.

We call it Big Return - return from a method and return from a controller at once. Here is how it works:

class BookController {
    def show(Long id) {
        withStoppingOnRender {
            Book bookInstance = Book.get(id)
            validateInstanceExists(bookInstance)
            [bookInstance: bookInstance]
        }
    }

    protected Object withStoppingOnRender(Closure closure) {
        try {
            return closure.call()
        } catch (EndRenderingException e) {}
    }

    private void validateInstanceExists(Book instance) {
        if (!instance) {
            flash.message = message(code: 'default.not.found.message', args: [message(code: 'book.label', default: 'Book'), params.id])
            redirect(action: "list")
            throw new EndRenderingException()
        }
    }
}

class EndRenderingException extends RuntimeException {}

Example usage

For simple CRUD controllers, you can use this solution and create some BaseController class for your controllers. We use withStoppingOnRender in every controller so code doesn't look like a spaghetti, we follow DRY principle and code is self-documented. Win-win-win! Here is a more complex example:

class DealerController {
    @Transactional
    def update() {
        withStoppingOnRender {
            Dealer dealerInstance = Dealer.get(params.id)
            validateInstanceExists(dealerInstance)
            validateAccountInExternalService(dealerInstance)
            checkIfInstanceWasConcurrentlyModified(dealerInstance, params.version)
            dealerInstance.properties = params
            saveUpdatedInstance(dealerInstance)
            redirectToAfterUpdate(dealerInstance)
        }
    }
}