Configuring Infinispan 10.1

The cache mode that you should choose depends on the qualities and guarantees you need for your data.

The following table summarizes the primary differences between cache modes:

Even with synchronous replication, distributed caches are not linearizable. (For transactional caches, we say they do not support serialization/snapshot isolation.) We can have one thread doing a single put:

But another thread might see the values in a different order:

The reason is that read can return the value from any owner, depending on how fast the primary owner replies. The write is not atomic across all the owners — in fact, the primary commits the update only after it receives a confirmation from the backup. While the primary is waiting for the confirmation message from the backup, reads from the backup will see the new value, but reads from the primary will see the old one.

Distributed caches split entries into a fixed number of segments and assign each segment to a list of owner nodes. Replicated caches do the same, with the exception that every node is an owner.

The first node in the list of owners is the primary owner. The other nodes in the list are backup owners. When the cache topology changes, because a node joins or leaves the cluster, the segment ownership table is broadcast to every node. This allows nodes to locate keys without making multicast requests or maintaining metadata for each key.

The numSegments property configures the number of segments available. However, the number of segments cannot change unless the cluster is restarted.

Likewise the key-to-segment mapping cannot change. Keys must always map to the same segments regardless of cluster topology changes. It is important that the key-to-segment mapping evenly distributes the number of segments allocated to each node while minimizing the number of segments that must move when the cluster topology changes.

You can customize the key-to-segment mapping by configuring a KeyPartitioner or by using the Grouping API.

However, Infinispan provides the following implementations:

You might need to configure a whole node where the capacity factor is 0 for every cache, user defined caches and internal caches. When defining a zero capacity node, the node won’t hold any data. This is how you declare a zero capacity node:

However, note that this will be true for distributed caches only. If you are using replicated caches, the node will still keep a copy of the value. Use only distributed caches to make the best use of this feature.

This is how you configure hashing declaratively, via XML:

And this is how you can configure it programmatically, in Java:

Infinispan’s very dynamic nature in handling topology changes (i.e. nodes being added / removed at runtime) means that, normally, a node doesn’t wait for the presence of other nodes before starting. While this is very flexible, it might not be suitable for applications which require a specific number of nodes to join the cluster before caches are started. For this reason, you can specify how many nodes should have joined the cluster before proceeding with cache initialization. To do this, use the initialClusterSize and initialClusterTimeout transport properties. The declarative XML configuration:

The programmatic Java configuration:

The above configuration will wait for 4 nodes to join the cluster before initialization. If the initial nodes do not appear within the specified timeout, the cache manager will fail to start.

When L1 is enabled, a node will keep the result of remote reads locally for a short period of time (configurable, 10 minutes by default), and repeated lookups will return the local L1 value instead of asking the owners again.

L1 caching is not free though. Enabling it comes at a cost, and this cost is that every entry update must broadcast an invalidation message to all the nodes. L1 entries can be evicted just like any other entry when the the cache is configured with a maximum size. Enabling L1 will improve performance for repeated reads of non-local keys, but it will slow down writes and it will increase memory consumption to some degree.

Is L1 caching right for you? The correct approach is to benchmark your application with and without L1 enabled and see what works best for your access pattern.

The following topology hints can be specified:

All of the above are optional. When provided, the distribution algorithm will try to spread the ownership of each segment across as many sites, racks, and machines (in this order) as possible.

In a distributed cache, a key is allocated to a list of nodes with an opaque algorithm. There is no easy way to reverse the computation and generate a key that maps to a particular node. However, we can generate a sequence of (pseudo-)random keys, see what their primary owner is, and hand them out to the application when it needs a key mapping to a particular node.

All clustered cache modes can be configured to use asynchronous communications with the mode="ASYNC" attribute on the <replicated-cache/>, <distributed-cache>, or <invalidation-cache/> element.

With asynchronous communications, the originator node does not receive any acknowledgement from the other nodes about the status of the operation, so there is no way to check if it succeeded on other nodes.

We do not recommend asynchronous communications in general, as they can cause inconsistencies in the data, and the results are hard to reason about. Nevertheless, sometimes speed is more important than consistency, and the option is available for those cases.

The Asynchronous API allows you to use synchronous communications, but without blocking the user thread.

There is one caveat: The asynchronous operations do NOT preserve the program order. If a thread calls cache.putAsync(k, v1); cache.putAsync(k, v2), the final value of k may be either v1 or v2. The advantage over using asynchronous communications is that the final value can’t be v1 on one node and v2 on another.

Because the Cache interface extends java.util.Map, write methods like put(key, value) and remove(key) return the previous value by default.

In some cases, the return value may not be correct:

Transactional caches return the correct previous value in cases 3 and 4. However, transactional caches also have a gotcha: in distributed mode, the read-committed isolation level is implemented as repeatable-read. That means this example of "double-checked locking" won’t work:

The correct way to implement this is to use cache.getAdvancedCache().withFlags(Flag.FORCE_WRITE_LOCK).get(k).

In caches with optimistic locking, writes can also return stale previous values. Write skew checks can avoid stale previous values.

Infinispan lets you define configuration templates that you can apply to multiple cache definitions or use as the basis for complex configurations.

For example, the following configuration contains a configuration template for local caches:

Configuration templates can also inherit from other templates to extend and override settings.

The following is an example of configuration template inheritance:

You can use wildcards to match cache definitions to configuration templates.

Infinispan supports XML inclusions (XInclude) that allow you to split configuration across multiple files.

For example, the following configuration uses an XInclude:

If you want to use a schema for your included fragments, use the infinispan-config-fragment-10.1.xsd schema:

XInclude specification

Infinispan provides two abstractions for programmatic cache configuration:

An example of global configuration is to enable statistics and export them via JMX or the metrics endpoint at the Cache Manager level, as follows:

The following examples show how you can programmatically configure different settings for Infinispan caches.

In cases where multiple Infinispan Cache Managers run on the same JVM, you should uniquely identify each Cache Manager to prevent conflicts.

For example, the following examples specify "Hibernate2LC" as the cache manager name, which results in a JMX MBean named org.infinispan:type=CacheManager,name="Hibernate2LC".

Infinispan includes an MBeanServerLookup interface that you can use to register MBeans in custom MBeanServer instances.

Infinispan exposes JMX MBeans that represent manageable resources.

For a complete list of available JMX MBeans along with descriptions and available operations and attributes, see the Infinispan JMX Components documentation.

Infinispan JMX Components

Default JGroups stacks are included in infinispan-core.jar and on the classpath. You can locate the default JGroups stacks in the default-configs directory.

stack.combine modifies inherited JGroups properties.

stack.position identifies protocols to modify. If you do not specify stack.position, Infinispan defaults to the same protocol as the inherited configuration, which resets all non-specified attributes to the default values.

For example, evaluate a Gossip router for cluster discovery using a TCP stack configuration named "prod":

Use system properties with default JGroups stacks.

The following system properties apply to all JGroups stacks, including default-jgroups-udp.xml and default-jgroups-tcp.xml:

The following system properties apply to default-jgroups-ec2.xml:

The following system properties apply to default-jgroups-kubernetes.xml:

The following system properties apply to default-jgroups-google.xml:

Infinispan eviction removes entries from the data container to make space when adding new entries.

Infinispan eviction relies on two configurations:

You configure the maximum size of the data container and specify if Infinispan stores cache entries as:

When an entry is added or modified in the data container, Infinispan compares the current eviction size to the maximum size. If the current size exceeds the maximum, Infinispan evicts entries.

Eviction happens immediately in the thread that adds an entry that exceeds the maximum size.

For example, consider the following configuration:

In this case, entries are stored as objects and the data container has a maximum size of 50 entries.

If 50 entries are in the data container, and a put() request attempts to create a new entry, Infinispan performs eviction.

Strategies control how Infinispan performs eviction. You can either perform eviction manually or configure Infinispan to do one of the following:

You configure eviction in infinispan.xml as part of your cache definition.

Eviction is not enabled, which is the default configuration. Infinispan stores cache entries as objects in the data container.

Passivation persists data to cache stores when Infinispan evicts entries. You should always enable eviction if you enable passivation.

Infinispan stores cache entries as objects. Eviction is not enabled but performed manually using the evict() method.

Infinispan stores cache entries as objects. Eviction happens when there are 100 entries in the data container and Infinispan gets a request to create a new entry:

Infinispan stores cache entries as bytes. Eviction happens when the size of the data container reaches 100 bytes and Infinispan gets a request to create a new entry:

Infinispan stores cache entries as bytes in native memory. Eviction happens when there are 100 entries in the data container and Infinispan gets a request to create a new entry:

Infinispan stores cache entries as bytes in native memory. When there are 100 entries in the data container, and Infinispan gets a request to create a new entry, it throws an exception and does not allow the new entry:

You must use binary or off-heap storage memory based eviction, as in the following examples:

When you configure expiration, Infinispan stores keys with metadata that determines when entries expire.

Infinispan checks if lifespan or maximum idle metadata is set and then compares the values with the current time.

If (creation + lifespan > currentTime) or (lastUsed + maxIdle > currentTime) then Infinispan detects that the entry is expired.

Expiration occurs whenever entries are accessed or found by the expiration reaper.

For example, k1 reaches the maximum idle time and a client makes a Cache.get(k1) request. In this case, Infinispan detects that the entry is expired and removes it from the data container. The Cache.get() returns null.

Infinispan also expires entries from cache stores, but only with lifespan expiration. Maximum idle expiration does not work with cache stores. In the case of cache loaders, Infinispan cannot expire entries because loaders can only read from external storage.

Infinispan uses a reaper thread that runs periodically to detect and remove expired entries. The expiration reaper ensures that expired entries that are no longer accessed are removed.

The Infinispan ExpirationManager interface handles the expiration reaper and exposes the processExpiration() method.

In some cases, you can disable the expiration reaper and manually expire entries by calling processExpiration(); for instance, if you are using local cache mode with a custom application where a maintenance thread runs periodically.

Because maximum idle expiration relies on the last access time for cache entries, it has some limitations with clustered cache modes.

With lifespan expiration, the creation time for cache entries provides a value that is consistent across clustered caches. For example, the creation time for k1 is always the same on all nodes.

For maximum idle expiration with clustered caches, last access time for entries is not always the same on all nodes. To ensure that entries have the same relative access times across clusters, Infinispan sends touch commands to all owners when keys are accessed.

The touch commands that Infinispan send have the following considerations:

When you configure Infinispan to expire entries, you can set lifespan and maximum idle times for:

Expire all cache entries after 2 seconds:

Expire all cache entries 1 second after last access time:

Disable the expiration reaper with the interval attribute and manually expire entries 1 second after last access time:

Expire all cache entries after 5 seconds or 1 second after the last access time, whichever happens first:

The following example shows how to configure lifespan and maximum idle values when creating cache entries:

If the Infinispan configuration defines a lifespan value of 1000 for all entries, the preceding Cache.put() requests cause the entries to expire:

Add cache stores to Infinispan caches in a chain either declaratively or programmatically. Cache read operations check each cache store in the configured order until they locate a valid non-null element of data. Write operations affect all cache stores except for those that you configure as read only.

Infinispan uses a global filesystem location for saving data to persistent storage.

Infinispan servers use the $ISPN_HOME/server/data directory as the global persistent location.

If you are using Infinispan as a library embedded in custom applications, the global persistent location defaults to the user.dir system property. This system property typically uses the directory where your application starts. You should configure a global persistent location to use a suitable location.

When using file-based cache stores, you can optionally specify filesystem directories for storage. Unless absolute paths are declared, directories are always relative to the global persistent location.

For example, you configure your global persistent location as follows:

You then configure a Single File cache store that uses a path named myDataStore as follows:

In this case, the configuration results in a Single File cache store in /tmp/example/myDataStore/myCache.dat

If you attempt to set an absolute path that resides outside the global persistent location, Infinispan throws the following exception:

Passivation configures Infinispan to write entries to cache stores when it evicts those entries from memory. In this way, passivation ensures that only a single copy of an entry is maintained, either in-memory or in a cache store, which prevents unnecessary and potentially expensive writes to persistent storage.

Activation is the process of restoring entries to memory from the cache store when threads attempt to access passivated entries. For this reason, when you enable passivation, you must configure cache stores that implement both CacheWriter and CacheLoader interfaces so they can write and load entries from persistent storage.

When Infinispan evicts an entry from the cache, it notifies cache listeners that the entry is passivated then stores the entry in the cache store. When Infinispan gets an access request for an evicted entry, it lazily loads the entry from the cache store into memory and then notifies cache listeners that the entry is activated.

Only JDBC String-Based cache stores support transactional operations. If you configure caches as transactional, you should set transactional=true to keep data in persistent storage synchronized with data in memory.

For all other cache stores, Infinispan does not enlist cache loaders in transactional operations. This can result in data inconsistency if transactions succeed in modifying data in memory but do not completely apply changes to data in the cache store. In this case manual recovery does not work with cache stores.

Cache stores can organize data into hash space segments to which keys map.

Segmented stores increase read performance for bulk operations; for example, streaming over data (Cache.size, Cache.entrySet.stream), pre-loading the cache, and doing state transfer operations.

However, segmented stores can also result in loss of performance for write operations. This performance loss applies particularly to batch write operations that can take place with transactions or write-behind stores. For this reason, you should evaluate the overhead for write operations before you enable segmented stores. The performance gain for bulk read operations might not be acceptable if there is a significant performance loss for write operations.

Key Ownership

In most cases, filesystem-based cache stores are appropriate for local cache stores for data that overflows from memory because it exceeds size and/or time restrictions.

Write-Through is an cache writing mode where writes to memory and writes to cache stores are synchronous. When a client application updates a cache entry, in most cases by invoking Cache.put(), Infinispan does not return the call until it updates the cache store. This cache writing mode results in updates to the cache store concluding within the boundaries of the client thread.

The primary advantage of Write-Through mode is that the cache and cache store are updated simultaneously, which ensures that the cache store is always consistent with the cache.

However, Write-Through mode can potentially decrease performance because the need to access and update cache stores directly adds latency to cache operations.

Infinispan defaults to Write-Through mode unless you explicitly configure Write-Behind mode on cache stores.

Write-Behind

Write-Behind is an cache writing mode where writes to memory are synchronous and writes to cache stores are asynchronous. When a client application updates a cache entry, Infinispan adds the update to a modification queue and then modifies the cache store in a different thread than the client thread.

You can configure the number of threads that consume the modification queue and apply updates to the underlying cache store. The modification queue fills up if there are not enough threads to handle the updates or if the underlying cache store becomes unavailable. When this occurs, Infinispan uses Write-Through mode until the modification queue can accept new entries.

Write-Behind mode provides a performance advantage over Write-Through mode because cache operations do not need to wait for updates to the underlying cache store to complete. However, data in the cache store remains inconsistent with data in the cache until the modification queue is processed. For this reason, Write-Behind mode is suitable for cache stores with low latency, such as unshared and local filesystem-based cache stores, where the time between the write to the cache and the write to the cache store is as small as possible.

The preceding configuration example uses the fail-silently parameter to control what happens when either the cache store is unavailable or the modification queue is full.

Write-Through

ClusterCacheLoader retrieves data from other Infinispan cluster members but does not persist data. In other words, ClusterCacheLoader is not a cache store.

ClusterCacheLoader provides a non-blocking partial alternative to state transfer. ClusterCacheLoader fetches keys from other nodes on demand if those keys are not available on the local node, which is similar to lazily loading cache content.

The following points also apply to ClusterCacheLoader:

Single File cache stores, SingleFileStore, persist data to file. Infinispan also maintains an in-memory index of keys while keys and values are stored in the file. By default, Single File cache stores are segmented, which means that Infinispan creates a separate file for each segment.

Because SingleFileStore keeps an in-memory index of keys and the location of values, it requires additional memory, depending on the key size and the number of keys. For this reason, SingleFileStore is not recommended for use cases where the keys have a large size.

In some cases, SingleFileStore can also become fragmented. If the size of values continually increases, available space in the single file is not used but the entry is appended to the end of the file. Available space in the file is used only if an entry can fit within it. Likewise, if you remove all entries from memory, the single file store does not decrease in size or become defragmented.

Single File cache stores support segmentation and create a separate instance per segment, which results in multiple directories in the path you configure. Each directory is a number that represents the segment to which the data maps.

JDBC String-Based cache stores, JdbcStringBasedStore, use JDBC drivers to load and store values in the underlying database.

JdbcStringBasedStore stores each entry in its own row in the table to increase throughput for concurrent loads. JdbcStringBasedStore also uses a simple one-to-one mapping that maps each key to a String object using the key-to-string-mapper interface.

Infinispan provides a default implementation, DefaultTwoWayKey2StringMapper, that handles primitive types.

JdbcStringBasedStore uses segmentation by default and requires a column in the database table to represent the segments to which entries belong.

JPA (Java Persistence API) cache stores, JpaStore, use formal schema to persist data. Other applications can then read from persistent storage to load data from Infinispan. However, other applications should not use persistent storage concurrently with Infinispan.

When using JpaStore, you should take the following into consideration:

Remote cache stores, RemoteStore, use a remote Infinispan cluster as storage.

RemoteStore uses the Hot Rod protocol to communicate with remote Infinispan clusters.

In the following configuration examples, RemoteStore uses the remote cache named "mycache" on Infinispan servers "one" and "two":

RemoteStore supports segmentation and can publish keys and entries by segment, which makes bulk operations more efficient. However, segmentation is available with Infinispan Hot Rod protocol version 2.3 or later.

RocksDB provides key-value filesystem-based storage with high performance and reliability for highly concurrent environments.

RocksDB cache stores, RocksDBStore, use two databases. One database provides a primary cache store for data in memory; the other database holds entries that Infinispan expires from memory.

You can also specify the following RocksDB tuning parameters:

Optionally set properties in the configuration as follows:

RocksDBStore supports segmentation and creates a separate column family per segment. Segmented RocksDB cache stores improve lookup performance and iteration but slightly lower performance of write operations.

Soft-Index File cache stores, SoftIndexFileStore, provide local file-based storage.

SoftIndexFileStore is a Java implementation that uses a variant of B+ Tree that is cached in-memory using Java soft references. The B+ Tree, called Index is offloaded on the file system to a single file that is purged and rebuilt each time the cache store restarts.

SoftIndexFileStore stores data in a set of files rather than a single file. When usage of any file drops below 50%, the entries in the file are overwritten to another file and the file is then deleted.

SoftIndexFileStore persists data in a set of files that are written in an append-only method. For this reason, if you use SoftIndexFileStore on conventional magnetic disk, it does not need to seek when writing a burst of entries.

Most structures in SoftIndexFileStore are bounded, so out-of-memory exceptions do not pose a risk. You can also configure limits for concurrently open files.

By default the size of a node in the Index is limited to 4096 bytes. This size also limits the key length; more precisely the length of serialized keys. For this reason, you cannot use keys longer than the size of the node, 15 bytes. Additionally, key length is stored as "short", which limits key length to 32767 bytes. SoftIndexFileStore throws an exception if keys are longer after serialization occurs.

SoftIndexFileStore cannot detect expired entries, which can lead to excessive usage of space on the file system .

Soft-Index File cache stores support segmentation and create a separate instance per segment, which results in multiple directories in the path you configure. Each directory is a number that represents the segment to which the data maps.

You can create custom cache stores that implement one or more of the Infinispan persistent SPIs.

The following notes apply to Infinispan persistence SPI classes:

Cache stores can be segmented if they do one of the following:

A provider might choose to only implement a subset of these interfaces:

If you want to migrate your existing store to the new API or to write a new cache store implementation, the SingleFileStore provides a good starting point.

Infinispan provides the StoreMigrator.java utility that recreates data for the latest Infinispan cache store implementations.

StoreMigrator takes a cache store from a previous version of Infinispan as source and uses a cache store implementation as target.

When you run StoreMigrator, it creates the target cache with the cache store type that you define using the EmbeddedCacheManager interface. StoreMigrator then loads entries from the source store into memory and then puts them into the target cache.

StoreMigrator also lets you migrate data from one type of cache store to another. For example, you can migrate from a JDBC String-Based cache store to a Single File cache store.

StoreMigrator is available as part of the Infinispan tools library, infinispan-tools, and is included in the Maven repository.

Set properties for source and target cache stores in a migrator.properties file.

Run StoreMigrator to migrate data from one cache store to another.

In a split brain situation, each network partition will install its own JGroups view, removing the nodes from the other partition(s). We don’t have a direct way of determining whether the has been split into two or more partitions, since the partitions are unaware of each other. Instead, we assume the cluster has split when one or more nodes disappear from the JGroups cluster without sending an explicit leave message.

As mentioned in the previous section, Infinispan can’t detect whether a node left the JGroups view because of a process/machine crash, or because of a network failure: whenever a node leaves the JGroups cluster abruptly, it is assumed to be because of a network problem.

If the configured number of copies (numOwners) is greater than 1, the cluster can remain available and will try to make new replicas of the data on the crashed node. However, other nodes might crash during the rebalance process. If more than numOwners nodes crash in a short interval of time, there is a chance that some cache entries have disappeared from the cluster altogether. In this case, with the DENY_READ_WRITES or ALLOW_READS strategy enabled, Infinispan assumes (incorrectly) that there is a split brain and enters DEGRADED mode as described in the split-brain section.

The administrator can also shut down more than numOwners nodes in rapid succession, causing the loss of the data stored only on those nodes. When the administrator shuts down a node gracefully, Infinispan knows that the node can’t come back. However, the cluster doesn’t keep track of how each node left, and the cache still enters DEGRADED mode as if those nodes had crashed.

At this stage there is no way for the cluster to recover its state, except stopping it and repopulating it on restart with the data from an external source. Clusters are expected to be configured with an appropriate numOwners in order to avoid numOwners successive node failures, so this situation should be pretty rare. If the application can handle losing some of the data in the cache, the administrator can force the availability mode back to AVAILABLE via JMX.

The conflict manager is a tool that allows users to retrieve all stored replica values for a given key. In addition to allowing users to process a stream of cache entries whose stored replicas have conflicting values. Furthermore, by utilising implementations of the EntryMergePolicy interface it is possible for said conflicts to be resolved automatically.

During a partition merge the ConflictManager automatically attempts to resolve conflicts utilising the configured EntryMergePolicy, however it is also possible to manually search for/resolve conflicts as required by your application.

The code below shows how to retrieve an EmbeddedCacheManager’s ConflictManager, how to retrieve all versions of a given key and how to check for conflicts across a given cache.

Unless the cache is distributed or replicated, partition handling configuration is ignored. The default partition handling strategy is ALLOW_READ_WRITES and the default EntryMergePolicy is MergePolicies::PREFERRED_ALWAYS.

The same can be achieved programmatically:

The availability mode of a cache is exposed in JMX as an attribute in the Cache MBean. The attribute is writable, allowing an administrator to forcefully migrate a cache from DEGRADED mode back to AVAILABLE (at the cost of consistency).

The availability mode is also accessible via the AdvancedCache interface: