Thursday, 24 May 2012
Migrating projects from one container to another is often problematic. Not as much with Infinispan. This article is about configuring Infinispan, using Transaction Manager for demarcating transaction boundaries, while keeping the data both in a memory and relational database - stored via JDBC cache store. I’ll demonstrate all the features on code snippets.
A complete application is located at https://github.com/mgencur/infinispan-examples and is called carmart-tx-jdbc. It’s a web application based on JSF 2, Seam 3 and Infinispan 5.1.4.FINAL, is fully working, tested with JBoss Application Server 7.1.1.Final and Tomcat 7.0.27. There is one prerequisite, though. It needs an installed and working MySQL database in your system. The database name should be carmartdb, accessible by a user with carmart/carmart username/password. First, look at what we need to configure for JBoss Application Server 7.
Infinispan will be configured via new fluent API using builders, hence the call to .build() method at the end. We need to configure aspects related to transactions and cache loaders. The configuration API for cache loaders is likely going to be changed in not-so-far future. It should be fluent and more intuitive, generally easier to use than current one.
I purposely do not show XML configuration. Configuration examples can be found at https://github.com/infinispan/infinispan/blob/master/core/src/main/resources/config-samples/sample.xml. In order to configure transactions and cache loaders, look for tags called <transaction> and <loaders> and modify that sample file according to below configuration. Tag names and attribute names are very similar for both XML and Java configuration. If that is not enough, there is always a schema in Infinispan distribution.
The configuration of Infinispan is as follows:
Lines marked with red are different in other containers/configurations, as you’ll see in a minute. The code above implies that we need to specify proper TransactionManagerLookup implementation which is, in this case, GenericTransactionManagerLookup. We also need to say: "Hey, I wanna use ManagedConnectionFactory as a connectionFactoryClass". OK, here we go. I should, as well, explain how to configure a datasource properly, right? In JBoss AS 7, this is configured as a subsystem in $JBOSS_HOME/standalone/configuration/standalone.xml:
The usage of transactions is very simple as we can obtain a transaction object by injection.
Quite easy, isn’t it …if you know how to do it. The only problem is that it does not work (at least not completely) :-) If you deploy the app, you find out that when storing a key-value pair in the cache, an exception is thrown. This exception indicates that the operation with DB (and JDBC cache store) failed. The exception says:
A complete stack trace looks similar to https://gist.github.com/2777348 There’s still an open issue in JIRA (ISPN-604) and it is being worked on.
But how do we cope with this inconvenience for now… By not using a managed datasource but rather a third party library called c3p0 (JDBC3 Connection and Statement Pooling, more information at http://www.mchange.com/projects/c3p0/index.html) Infinispan allows you to use this library for connecting to the database. If you really want to use it, you need to choose a different connectionFactoryClass which is, in this case, PooledConnectionFactory.
Infinispan configuration looks like this:
Transactions are accessible in the same way as in the previous use case. Now let’s look at configuration for Tomcat servlet container.
Tomcat does not have any Transaction Manager in it so we have to bundle one with the application. For the purpose of this exercise, we choose JBoss Transactions (http://www.jboss.org/jbosstm). See dependencies at the end.
Cache manager and cache configuration is in this form:
For Tomcat, we need to specify a different transactionManagerLookup implementation and datasourceJndiLocation. Tomcat simply places objects under a bit different JNDI locations. The datasource is defined in context.xml file which has to be on classpath. This file might look like this:
How do we get the transaction manager in the application then? Lets obtain it directly from a cache.
Infinispan knows how to find the manager and we need to know how to obtain it from Infinispan.
Sources: https://github.com/mgencur/infinispan-examples/blob/master/carmart-tx-jdbc/src/tomcat/java/org/infinispan/examples/carmart/session/CarManager.java The transaction manager provides standard methods for transactions, such as begin(), commit() and rollback().
So…which dependencies do we always need when using Infinispan with JDBC cache stores and transactions? These are infinspan-core, infinispan-cachestore-jdbc and javax.transaction.jta. The scope for jta dependency, as defined in Maven, is different for JBossAS and Tomcat.
Common dependencies for JBossAS and Tomcat
Of course, our application needs a few more dependencies but these are not directly related to Infinispan. Let’s ignore them in this article. JBoss AS 7 provides managed datasource that is accessible from Infinispan. The only specific dependency (related to transactions or Infinispan) is JTA.
Dependencies specific to JBossAS - using managed Datasource (managed by the server)
Dependencies specific to JBossAS - using c3p0
Yes, you need to bundle also MySQL connector. On the other hand, for Tomcat use case and JBossAS with managed datasource, this jar file needs do be deployed to the server separately. For Tomcat, do this simply by copying the jar file to $TOMCAT_HOME/lib. For JBoss AS 7, copy the jar file into $JBOSS_HOME/standalone/deployments.
Dependencies specific to Tomcat - using JBoss Transactions
That’s it. I hope you’ve found this article helpful. Any feedback is welcome, especially the positive one :-) If you find any problem with the application, feel free to comment here or participate in Infinispan forums (http://www.jboss.org/infinispan/forums).
Tags: database transactions c3p0 as7 cache store tomcat
Monday, 23 April 2012
The trip started in Coimbra where Sanne Grinovero and I discussed about Infinispan and OGM at the Portugal JUG. The interest was obvious from the amount of questions and the discussions that followed and ended up late in the night. A big thanks to Samuel Santos for organising this!
The trip then continued in Lisbon where we meet the CloudTM team for two good days of hacking and brainstorming. The results are a set of awesome features the CloudTM is about to contribute to Infinispan, especially around transactions:
Total Order Broadcast (TOB) - is a transaction protocol for replicated caches built on atomic broadcast. It relies on JGroups' SEQUENCER protocol for achieving total order. The benchmarks ran comparing it with the current 2PC based transaction protocol are rather promising and we hope to integrated it in our 5.2 release
Total Order Multicast (TOM) - is simplistically put TOB for distributed caches. This is the next in pipe after TOB to be integrated. Again, preliminary performance comparisons look rather promising
The CloudTM also extended Infinispan’s transactions to support one-copy serializability (that’s a fancy name for SERIALIZABLE transactions in replicated systems). The implementation is based on keeping multiple data version of data for each entry together with some housekeeping code. More on this to come!
Big thanks to CloudTM team for their effort and all the great ideas (and patches!!) they submitted to Infinispan! It was a great meeting with very good results and burning whiteboards (see below)!
Tags: JUGs transactions event
Tuesday, 17 January 2012
Over the past week we’ve been busy profiling Infinispan, in particular we’ve been trying to maximise the performance of Infinispan transactional caches which received a major overhaul int the 5.1 'Brahma' series, and the result is that instead of going final, we’ve decided to cut another candidate release, Infinispan 5.1.0.CR4 which is now available for download from the usual place. We’re confident that these improvements will be greatly appreciated by Infinispan consumers such as Hibernate second-level-cache and JBoss Application Server 7 HTTP session replication.
As part of the configuration and parser work, new configuration documentation has generated which you can now find in http://docs.jboss.org/infinispan/5.1/configdocs/.
Wednesday, 23 November 2011
Don’t be scared the name, use1PcForAutoCommitTransactions is a new feature (5.1.CR1) that does quite a cool thing: increases your transactions’s performance. Let me explain. Before Infinispan 5.1 you could access the cache both transactional and non-transactional. Naturally the non-transactional access is faster and offers less consistency guarantees.But we don’t support mixed access in Infinispan 5.1, so what what’s to be done when you need the speed of non-transactional access and you are ready to trade some consistency guarantees for it? Well here is where use1PcForAutoCommitTransactions helps you. What is does is forces an induced (autoCommit=true) transaction to commit in a single phase. So only 1 RPC instead of 2RPCs as in the case of a full 2 Phase Commit (2PC).
You might end up with inconsistent data if multiple transactions modify the same key concurrently. But if you know that’s not the case, or you can live with it then use1PcForAutoCommitTransactions will help your performance considerably.
Let’s say you do a simple put operation outside the scope of a transaction:
Now let’s see how this would behave if the cache has several different transaction configurations:
The put will happen in two RPCs/steps: a prepare message is sent around and then a commit.
The put happens in one RPC as the prepare and the commit are merged. Better performance.
Tags: transactions performance
Friday, 11 November 2011
Pessimistic transactions were added in 5.1 and are the "rebranding" of eager transactions from previous Infinispan releases. But besides the re-branding, the code also brought some worth mentioning performance optimisation:
a single RPC happens for acquiring lock on a key, disregarding the number of invocations. So if you call cache.put(k,v) in a loop, within the scope of the same transaction, there is only one remote call to the owner of k.
if the key you want to lock/write maps to the local node then no remote locks are acquired. In other words there won’t be any RPCs for writing to a key that maps locally. This can be very powerful used in conjunction with the KeyAffinityService, as it allows you to control the locality of you keys.
during the two phase commit (2PC), the prepare phase doesn’t perform any RPCs: this optimisation is based on the fact locks are already acquired on each write. This means that then number of RPCs during transactions lifespan is reduced with 1.
for some writes to the cache (e..g cache.put(k,v)) two RPCs were performed: one to acquire the remote lock and one to fetch the previous value. The obvious optimisation in this case was to make a single RPC for both operations - which we do starting with 5.1.
Tags: transactions locking
Thursday, 10 November 2011
Here’s the problem: first transaction (T1) writes to key a and b in this order. Second transaction (T2) writes to key b and a - again order is relevant. Now with some "right timing" T1 manages to acquire lock on a and T2 acquires lock on b. And then they wait one for the other to release locks so that they can progress. This is what is called a deadlock and is really bad for your system throughput - but I won’t insist on this aspect, as I’ve mentioned it a lot in my previous posts.
What I want to talk about though is a way to solve this problem. Quit a simple way - just force an order on your transaction writes and you’re guaranteed not to deadlock: if both T1 and T2 write to a then b (lexicographical order) there won’t be any deadlock. Ever.
But there’s a catch. It’s not always possible to define this order, simply because you can’t or because you don’t know all your keys at the very beginning of the transaction.
Now here’s the good news: Infinispan orders the keys touched in a transaction for you. And it even defines an order so that you won’t have to do that. Actually you don’t have to anything, not even enable this feature, as it is already enabled for you by default.
Does it sound too good to be true? That’s because it’s only partially true. That is lock reordering only works if you’re using optimistic locking. For pessimistic locking you still have to do it the old way - order your locks (that’s of course if you can).
Wanna know more about it? Read this.
Expect and enjoy this feature in our next release 5.1.0.BETA5.
Tags: transactions locking deadlock detection
Wednesday, 09 November 2011
The single lock owner is a highly requested Infinispan improvement. The basic idea behind it is that, when writing to a key, locks are no longer acquired on all the nodes that own that key, but only on a single designated node (named "main owner").
Short version: if you use transactions that concurrently write to the same keys, this improvement significantly increases your system' throughput.
Long version: If you’re using Infinispan with transactions that modify the same key(s) concurrently then you can easily end up in a deadlock. A deadlock can also occur if two transaction modify the same key at the same time - which is both inefficient and counter-intuitive. Such a deadlock means that at one transaction(or both) eventually rollback but also the lock on the key is held for the duration of a lockAquistionTimout config option (defaults to 10 seconds). These deadlocks reduces the throughput significantly as transactions threads are held inactive during deadlock time. On top of that, other transactions that want to operate on that key are also delayed, potentially resulting in a cascade effect.
The only encountered performance penalty is during cluster topology changes. At that point the cluster needs to perform some additional computation (no RPC involved) to fail-over the acquired locks from previous to new owners. Another noticeable aspect is that locks are now being released asynchronously, after the transaction commits. This doesn’t add any burden to the transaction duration, but it means that locks are being held slightly longer. That’s not something to be concerned about if you’re not using transactions that compete for same locks though. We plan to benchmark this feature using Radargun benchmark tool - we’ll report back!
Tags: radargun transactions locking deadlock detection
Monday, 03 October 2011
If you ever used Infinispan in a transactional way you might be very interested in this article as it describes some very significant improvements in version 5.1 "Brahma" (released with 5.1.Beta1):
starting with this release an Infinispan cache can accessed either transactionally or non-transactionally. The mixed access mode is no longer supported (backward compatibility still maintained, see below). There are several reasons for going this path, but one of them most important result of this decision is a cleaner semantic on how concurrency is managed between multiple requestors for the same cache entry.
starting with 5.1 the supported transaction models are optimistic and pessimistic. Optimistic model is an improvement over the existing default transaction model by completely deferring lock acquisition to transaction prepare time. That reduces lock acquisition duration and increases throughput; also avoids deadlocks. With pessimistic model, cluster wide-locks are being acquired on each write and only being released after the transaction completed (see below).
It’s up to you as an user to decide weather you want to define a cache as transactional or not. By default, infinispan caches are non transactional. A cache can be made transactional by changing the transactionMode attribute:
transactionMode can only take two values: TRANSACTIONAL and NON_TRANSACTIONAL. Same thing can be also achieved programatically:
Important:for transactional caches it is required to configure a TransactionManagerLookup.
The autoCommit attribute was added in order to assure backward compatibility. If a cache is transactional and autoCommit is enabled (defaults to true) then any call that is performed outside of a transaction’s scope is transparently wrapped within a transaction. In other words Infinispan adds the logic for starting a transaction before the call and committing it after the call. So if your code accesses a cache both transactionally and non-transactionally, all you have to do when migrating to Infinispan 5.1 is mark the cache as transactional and enable autoCommit (that’s actually enabled by default, so just don’t disable it :) The autoCommit feature can be managed through configuration:
With optimistic transactions locks are being acquired at transaction prepare time and are only being held up to the point the transaction commits (or rollbacks). This is different from the 5.0 default locking model where local locks are being acquire on writes and cluster locks are being acquired during prepare time. Optimistic transactions can be enabled in the configuration file:
By default, a transactional cache is optimistic.
From a lock acquisition perspective, pessimistic transactions obtain locks on keys at the time the key is written. E.g.
When cache.put(k1,v1) returns k1 is locked and no other transaction running anywhere in the cluster can write to it. Reading k1 is still possible. The lock on k1 is released when the transaction completes (commits or rollbacks).
Pessimistic transactions can be enabled in the configuration file:
From a use case perspective, optimistic transactions should be used when there’s not a lot of contention between multiple transactions running at the same time. That is because the optimistic transactions rollback if data has changed between the time it was read and the time it was committed (writeSkewCheck). On the other hand, pessimistic transactions might be a better fit when there is high contention on the keys and transaction rollbacks are less desirable. Pessimistic transactions are more costly by their nature: each write operation potentially involves a RPC for lock acquisition.
This major transaction rework has opened the way for several other transaction related improvements:
Single node locking model is a major step forward in avoiding deadlocks and increasing throughput by only acquiring locks on a single node in the cluster, disregarding the number of redundant copies (numOwners) on which data is replicated
Lock acquisition reordering is a deadlock avoidance technique that will be used for optimistic transactions
Incremental locking is another technique for minimising deadlocks.
Stay tuned! Mircea
Tags: transactions locking deadlock detection performance
Wednesday, 10 August 2011
Besides other cool features such as Map reduce and distributed executors, Infinispan 5.0.0 "Pagoa" brings some significant improvements around transactional functionality:
transaction recovery http://community.jboss.org/docs/DOC-16646?uniqueTitle=false is now supported, with a set of tools that allow state reconciliation in case the transaction fails during 2nd phase of 2PC. This is especially useful in the case of transactions spreading over Infinispan and another resource manager, e.g. a database (distributed transactions). You can find out more on how to enable and use transaction recovery here.
Synchronization enlistment is another important feature in this release. This allows Infinispan to enlist in a transaction as an Synchronization rather than an XAResource.This enlistment allows the TransactionManager to optimize 2PC with a 1PC where only one other resource is enlisted with that transaction (last resource commit optimization). This is particularly important when using Infinispan as a 2nd level cache in Hibernate. You can read more about this feature here.
besides that several bugs were fixed particularly when it comes to the integration with a transaction manager - BIG thanks to the community for reporting and testing them!
To summarise, Infinispan can participate in a transaction in 3 ways:
as a fully fledged XAResource that supports recovery
as an XAResource, but without recovery. This is the default configuration
and as an Synchronization
In order to analyze the performance of running Infinispan in different transactional modes I’ve enhanced and used Radargun. The diagram below shows a performance comparison between running Infinispan in all the 3 modes described. The forth plot in the chart shows the performance of running Infinispan without transactions - this gives an idea about the cost of using transactions vs. raw operations.
The benchmark was run on this Radargun configuration, using Infinispan 5.0.0.CR5 configured as shown here. As a TransactionManager JBossTS 4.15.0.FINAL was used, configured with a VolatileStore as shown here. Each node was an 4-core Intel® Xeon® CPU E5640 @ 2.67GHz, with 4GB RAM.
Each transaction spread over only one put operation. The chart shows the following:
And that’s not all! During Infinispan 5.0.0 development we’ve been thinking a lot about how we can improve transactional throughput, especially in scenarios in which multiple transactions are writing on the same key. As a result we’ve come up with some improvement suggestions summarised here: please feel free to take a look and comment!
Tags: jta radargun transactions benchmarks
Tuesday, 07 June 2011
Starting with Hibernate 4.0.0.Beta1, Infinispan second level cache interacts with the transaction manager as a JTA Synchronization instead of an XA resource. Based on some testing we’ve done in JBoss AS 7, this has resulted in a huge performance increase thanks to the optimisations the transaction manager can apply to Synchronizations, which work very well when Infinispan is used as a cache rather than as a authoritative data store.
From an Infinispan configuration perspective, nothing needs changing. The Infinispan provider still uses the same base configuration by default. Transactional configuration happens within the cache provider itself and it’s here where the Infinispan is configured with Hibernate’s transaction manager and where Infinispan is configured to participate as a JTA synchronization. This is the default configuration, so from an user perspective, there’s nothing you have to do to take advantage of this new change.
However, you can always switch back to previous behaviour where Infinispan interacted as an XA resource via a dedicated Hibernate property called hibernate.cache.infinispan.use_synchronization
By default this property is set to true. If you set it false, Infinispan will interact with the transaction manager as an XA resource.
For more detailed information, check the "http://community.jboss.org/docs/DOC-14105[Using Infinispan As JPA Hibernate Second Level Cache Provider]" wiki.
Tags: transactions jpa synchronization hibernate second level cache provider