Infinispan server is a managed, distributed, and clusterable data grid that provides elastic scaling and high performance access to caches from multiple endpoints, such as Hot Rod and REST.

## 1. Getting Started with Infinispan Server

Quickly set up Infinispan server and learn the basics.

### 1.1. Infinispan Server Requirements

Check host system requirements for the Infinispan server.

Infinispan server requires a Java Virtual Machine and supports:

• Java 8

• Java 11

The Infinispan server distribution is an archive of Java libraries (`JAR` files), configuration files, and a `data` directory.

Procedure

Verification

1. Run the `sha1sum` command with the server download archive as the argument, for example:

``$sha1sum infinispan-server-${version}.zip``
2. Compare with the `SHA-1` checksum value on the Infinispan downloads page.

Reference

Infinispan Server README describes the contents of the server distribution.

### 1.3. Installing Infinispan Server

Extract the Infinispan server archive to any directory on your host.

Procedure

Use any extraction tool with the server archive, for example:

``$unzip infinispan-server-${version}.zip``

The resulting directory is your `$ISPN_HOME`. ### 1.4. Running Infinispan Servers Spin up Infinispan server instances that automatically form clusters. Learn how to create cache definitions to store your data. #### 1.4.1. Starting Infinispan Servers Launch Infinispan server with the startup script. Procedure 1. Open a terminal in `$ISPN_HOME`.

2. Run the `server` script.

Linux
``$bin/server.sh`` Microsoft Windows ``bin\server.bat`` The server gives you these messages when it starts: ``````INFO [org.infinispan.SERVER] (main) ISPN080004: Protocol SINGLE_PORT listening on 127.0.0.1:11222 INFO [org.infinispan.SERVER] (main) ISPN080001: Infinispan Server${version}
started in 7453ms``````

Hello Infinispan!

• Open `127.0.0.1:11222` in any browser to see the Infinispan server welcome message.

Reference

Infinispan Server README describes command line arguments for the `server` script.

#### 1.4.2. Verifying Infinispan Cluster Discovery

Infinispan servers running on the same network discover each other with the `MPING` protocol.

This procedure shows you how to use Infinispan server command arguments to start two instances on the same host and verify that the cluster view forms.

Prerequisites

Start a Infinispan server.

Procedure
1. Install and run a new Infinispan server instance.

1. Open a terminal in `$ISPN_HOME`. 2. Copy the root directory to `server2`. ``$ cp -r server server2``
2. Specify a port offset and the location of the `server2` root directory.

``$bin/server.sh -o 100 -s server2`` Verification Running servers return the following messages when new servers join clusters: ``````INFO [org.infinispan.CLUSTER] (jgroups-11,<server_hostname>) ISPN000094: Received new cluster view for channel cluster: [<server_hostname>|3] (2) [<server_hostname>, <server2_hostname>] INFO [org.infinispan.CLUSTER] (jgroups-11,<server_hostname>) ISPN100000: Node <server2_hostname> joined the cluster`````` Servers return the following messages when they join clusters: ``````INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) ISPN000078: Starting JGroups channel cluster INFO [org.infinispan.CLUSTER] (main) ISPN000094: Received new cluster view for channel cluster: [<server_hostname>|3] (2) [<server_hostname>, <server2_hostname>]`````` Reference Infinispan Server README describes command line arguments for the `server` script. #### 1.4.3. Performing Operations with the Infinispan CLI Connect to servers with the Infinispan command line interface (CLI) to access data and perform administrative functions. ##### Starting the Infinispan CLI Start the Infinispan CLI as follows: 1. Open a terminal in `$ISPN_HOME`.

2. Run the CLI.

``````$bin/cli.sh [disconnected]>`````` ##### Connecting to Infinispan Servers Do one of the following: • Run the `connect` command to connect to a Infinispan server on the default port of `11222`: ``````[disconnected]> connect [hostname1@cluster//containers/default]>`````` • Specify the location of a Infinispan server. For example, connect to a local server that has a port offset of 100: ``````[disconnected]> connect 127.0.0.1:11322 [hostname2@cluster//containers/default]>``````  Press the tab key to display available commands and options. Use the `-h` option to display help text. ##### Creating Caches from Templates Use Infinispan cache templates to add caches with recommended default settings. Procedure 1. Create a distributed, synchronous cache from a template and name it "mycache". ``[//containers/default]> create cache --template=org.infinispan.DIST_SYNC mycache``  Press the tab key after the `--template=` argument to list available cache templates. 2. Retrieve the cache configuration. ``````[//containers/default]> describe caches/mycache { "distributed-cache" : { "mode" : "SYNC", "remote-timeout" : 17500, "state-transfer" : { "timeout" : 60000 }, "transaction" : { "mode" : "NONE" }, "locking" : { "concurrency-level" : 1000, "acquire-timeout" : 15000, "striping" : false } } }`````` ##### Adding Cache Entries Add data to caches with the Infinispan CLI. Prerequisites • Create a cache named "mycache". Procedure 1. Add a key/value pair to `mycache`. ``[//containers/default]> put --cache=mycache hello world``  If the CLI is in the context of a cache, do `put k1 v1` for example: ```[//containers/default]> cd caches/mycache [//containers/default/caches/mycache]> put hello world``` 2. List keys in the cache. ``````[//containers/default]> ls caches/mycache hello`````` 3. Get the value for the `hello` key. 1. Navigate to the cache. ``[//containers/default]> cd caches/mycache`` 2. Use the `get` command to retrieve the key value. ``````[//containers/default/caches/mycache]> get hello world`````` ##### Shutting Down Infinispan Servers Use the CLI to gracefully shutdown running servers. This ensures that Infinispan passivates all entries to disk and persists state. • Use the `shutdown server` command to stop individual servers, for example: ``[//containers/default]> shutdown server server_hostname`` • Use the `shutdown cluster` command to stop all servers joined to the cluster, for example: ``[//containers/default]> shutdown cluster`` Infinispan servers log the following shutdown messages: ``````INFO [org.infinispan.SERVER] (pool-3-thread-1) ISPN080002: Infinispan Server stopping INFO [org.infinispan.CONTAINER] (pool-3-thread-1) ISPN000029: Passivating all entries to disk INFO [org.infinispan.CONTAINER] (pool-3-thread-1) ISPN000030: Passivated 28 entries in 46 milliseconds INFO [org.infinispan.CLUSTER] (pool-3-thread-1) ISPN000080: Disconnecting JGroups channel cluster INFO [org.infinispan.CONTAINER] (pool-3-thread-1) ISPN000390: Persisted state, version=<Infinispan version> timestamp=YYYY-MM-DDTHH:MM:SS INFO [org.infinispan.SERVER] (pool-3-thread-1) ISPN080003: Infinispan Server stopped INFO [org.infinispan.SERVER] (Thread-0) ISPN080002: Infinispan Server stopping INFO [org.infinispan.SERVER] (Thread-0) ISPN080003: Infinispan Server stopped`````` When you shutdown Infinispan clusters, the shutdown messages include: ``````INFO [org.infinispan.SERVER] (pool-3-thread-1) ISPN080029: Cluster shutdown INFO [org.infinispan.CLUSTER] (pool-3-thread-1) ISPN000080: Disconnecting JGroups channel cluster`````` ## 2. Setting Up Infinispan Endpoints Infinispan servers provide listener endpoints that handle requests from remote client applications. ### 2.1. Infinispan Endpoints Infinispan endpoints expose the `CacheManager` interface over different connector protocols so you can remotely access data and perform operations to manage and maintain Infinispan clusters. You can define multiple endpoint connectors on different socket bindings. #### 2.1.1. Hot Rod Hot Rod is a binary TCP client-server protocol designed to provide faster data access and improved performance in comparison to text-based protocols. Infinispan provides Hot Rod client libraries in Java, C++, C#, Node.js and other programming languages. Topology state transfer Infinispan uses topology caches to provide clients with cluster views. Topology caches contain entries that map internal JGroups transport addresses to exposed Hot Rod endpoints. When client send requests, Infinispan servers compare the topology ID in request headers with the topology ID from the cache. Infinispan servers send new topology views if client have older topology IDs. Cluster topology views allow Hot Rod clients to immediately detect when nodes join and leave, which enables dynamic load balancing and failover. In distributed cache modes, the consistent hashing algorithm also makes it possible to route Hot Rod client requests directly to primary owners. #### 2.1.2. REST Infinispan exposes a RESTful interface that allows HTTP clients to access data, monitor and maintain clusters, and perform administrative operations. You can use standard HTTP load balancers to provide clients with load balancing and failover capabilities. However, HTTP load balancers maintain static cluster views and require manual updates when cluster topology changes occur. #### 2.1.3. Memcached Infinispan provides an implementation of the Memcached text protocol for remote client access. The Infinispan Memcached server supports clustering with replicated and distributed cache modes. There are some Memcached client implementations, such as the Cache::Memcached Perl client, that can offer load balancing and failover detection capabilities with static lists of Infinispan server addresses that require manual updates when cluster topology changes occur. #### 2.1.4. Protocol Comparison Hot Rod HTTP / REST Memcached Topology-aware Y N N Hash-aware Y N N Encryption Y Y N Authentication Y Y N Conditional ops Y Y Y Bulk ops Y N N Transactions N N N Listeners Y N N Query Y Y N Execution Y N N Cross-site failover Y N N ### 2.2. Endpoint Connectors You configure Infinispan server endpoints with connector declarations that specify socket bindings, authentication mechanisms, and encryption configuration. The default endpoint connector configuration is as follows: ``````<endpoints socket-binding="default"> <hotrod-connector name="hotrod"/> <rest-connector name="rest"/> <memcached-connector socket-binding="memcached"/> </endpoints>`````` • `endpoints` contains endpoint connector declarations and defines global configuration for endpoints such as default socket bindings, security realms, and whether clients must present valid TLS certificates. • `<hotrod-connector name="hotrod"/>` declares a Hot Rod connector. • `<rest-connector name="rest"/>` declares a Hot Rod connector. • `<memcached-connector socket-binding="memcached"/>` declares a Memcached connector that uses the memcached socket binding. Reference urn:infinispan:server schema provides all available endpoint configuration. #### 2.2.1. Hot Rod Connectors Hot Rod connector declarations enable Hot Rod servers. ``````<hotrod-connector name="hotrod"> <topology-state-transfer /> <authentication> ... </authentication> <encryption> ... </encryption> </hotrod-connector>`````` • `name="hotrod"` logically names the Memcached connector. • `topology-state-transfer` tunes the state transfer operations that provide Hot Rod clients with cluster topology. • `authentication` configures SASL authentication mechanisms. • `encryption` configures TLS settings for client connections. Reference urn:infinispan:server schema provides all available Hot Rod connector configuration. #### 2.2.2. REST Connectors REST connector declarations enable REST servers. ``````<rest-connector name="rest"> <authentication> ... </authentication> <cors-rules> ... </cors-rules> <encryption> ... </encryption> </rest-connector>`````` • `name="rest"` logically names the REST connector. • `authentication` configures authentication mechanisms. • `cors-rules` specifies CORS (Cross Origin Resource Sharing) rules for cross-domain requests. • `encryption` configures TLS settings for client connections. Reference urn:infinispan:server schema provides all available REST connector configuration. #### 2.2.3. Memcached Connectors Memcached connector declarations enable Memcached servers.  Infinispan servers do not enable Memcached connectors by default. ``<memcached-connector name="memcached" socket-binding="memcached" cache="mycache" />`` • `name="memcached"` logically names the Memcached connector. • `socket-binding="memcached"` declares a unique socket binding for the Memcached connector. • `cache="mycache"` names the cache that the Memcached connector exposes. The default is `memcachedCache`. Memcached connectors expose a single cache only. To expose multiple caches through the Memcached endpoint, you must declare additional connectors. Each Memcached connector must also have a unique socket binding. Reference urn:infinispan:server schema provides all available Memcached connector configuration. ## 3. Monitoring Infinispan Servers ### 3.1. Working with Infinispan Server Logs Infinispan uses JBoss LogManager to provide configurable logging mechanisms that capture details about the environment and record cache operations for troubleshooting purposes and root cause analysis. #### 3.1.1. Infinispan Log Files Infinispan writes log messages to the following directory: `$ISPN_HOME/${infinispan.server.root}/log` `server.log` Messages in human readable format, including boot logs that relate to the server startup. Infinispan creates this file by default when you launch servers. `server.log.json` Messages in JSON format that let you parse and analyze Infinispan logs. Infinispan creates this file when you enable the JSON-FILE log handler. #### 3.1.2. Configuring Infinispan Log Properties You configure Infinispan logs with `logging.properties`, which is a standard Java properties file with the `property=value` format. Procedure 1. Open `$ISPN_HOME/${infinispan.server.root}/conf/logging.properties` with any text editor. 2. Configure logging properties as appropriate. 3. Save and close `logging.properties`. ##### Log Levels Log levels indicate the nature and severity of messages. Log level Description TRACE Provides detailed information about running state of applications. This is the most verbose log level. DEBUG Indicates the progress of individual requests or activities. INFO Indicates overall progress of applications, including lifecycle events. WARN Indicates circumstances that can lead to error or degrade performance. ERROR Indicates error conditions that might prevent operations or activites from being successful but do not prevent applications from running. FATAL Indicates events that could cause critical service failure and application shutdown. ##### Infinispan Log Categories Infinispan provides categories for INFO, WARN, ERROR, FATAL level messages that organize logs by functional area. `org.infinispan.CLUSTER` Messages specific to Infinispan clustering that include state transfer operations, rebalancing events, partitioning, and so on. `org.infinispan.CONFIG` Messages specific to Infinispan configuration. `org.infinispan.CONTAINER` Messages specific to the data container that include expiration and eviction operations, cache listener notifications, transactions, and so on. `org.infinispan.PERSISTENCE` Messages specific to cache loaders and stores. `org.infinispan.SECURITY` Messages specific to Infinispan security. `org.infinispan.SERVER` Messages specific to Infinispan servers. `org.infinispan.XSITE` Messages specific to cross-site replication operations. ##### Root Logger The root logger is `org.infinispan` and is always configured. This logger captures all messages for Infinispan log categories. ##### Log Handlers Log handlers define how Infinispan records log messages. CONSOLE Write log messages to the host standard out (`stdout`) or standard error (`stderr`) stream. Uses the `org.jboss.logmanager.handlers.ConsoleHandler` class by default. FILE Write log messages to a file. Uses the `org.jboss.logmanager.handlers.PeriodicRotatingFileHandler` class by default. JSON-FILE Write log messages to a file in JSON format. Uses the `org.jboss.logmanager.handlers.PeriodicRotatingFileHandler` class by default. ##### Log Formatters Log formatters: • Configure log handlers and define the appearance of log messages. • Are strings that use syntax based on the `java.util.logging.Formatter` class. An example is the default pattern for messages with the FILE log handler: `%d{yyyy-MM-dd HH\:mm\:ss,SSS} %-5p [%c] (%t) %s%e%n` • `%d` adds the current time and date. • `%-5p` specifies the log level. • `%c` specifies the logging category. • `%t` adds the name of the current thread. • `%s` specifies the simple log message. • `%e` adds the exception stack trace. • `%n` adds a new line. ##### Enabling and Configuring the JSON Log Handler Infinispan provides a JSON log handler to write messages in JSON format. Prerequisites Ensure that Infinispan is not running. You cannot dynamically enable log handlers. Procedure 1. Open `$ISPN_HOME/${infinispan.server.root}/conf/logging.properties` with any text editor. 2. Add `JSON-FILE` as a log handler, for example: ``logger.handlers=CONSOLE,FILE,JSON-FILE`` 3. Optionally configure the JSON log handler and formatter. 1. Use the `handler.JSON-FILE` property to configure the JSON log handler. 2. Use the `formatter.JSON-FORMATTER` property to configure the JSON log formatter. 4. Save and close `logging.properties`. When you start Infinispan, it writes each log message as a JSON map in the following file: `$ISPN_HOME/${infinispan.server.root}/log/server.log.json` #### 3.1.3. Logging Framework Infinispan uses the JBoss Logging Framework and delegates to logging providers in the following order: 1. JBoss Logging, if you are running Infinispan servers. 2. Apache Log4j, if `org.apache.log4j.LogManager` and `org.apache.log4j.Hierarchy` are on the classpath. 3. LogBack, if `ch.qos.logback.classic.Logger` is on the classpath. 4. JDK logging (`java.util.logging`), if no other logging provider is available. #### 3.1.4. Access Logs Hot Rod and REST endpoints can record all inbound client requests as log entries with the following categories: • `org.infinispan.HOTROD_ACCESS_LOG` logging category for the Hot Rod endpoint. • `org.infinispan.REST_ACCESS_LOG` logging category for the REST endpoint. ##### Enabling Access Logs Access logs for Hot Rod and REST endpoints are disabled by default. To enable either logging category, set the level to `TRACE` in the Infinispan server configuration, as in the following example: ``````<logger category="org.infinispan.HOTROD_ACCESS_LOG" use-parent-handlers="false"> <level name="TRACE"/> <handlers> <handler name="HR-ACCESS-FILE"/> </handlers> </logger>`````` ##### Access Log Properties The default format for access logs is as follows: ```%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss z}] &quot;%X{method} %m %X{protocol}&quot; %X{status} %X{requestSize} %X{responseSize} %X{duration}%n``` The preceding format creates log entries such as the following: `127.0.0.1 - [DD/MM/YYYY:HH:MM:SS] "PUT /rest/v2/caches/default/key HTTP/1.1" 404 5 77 10` Logging properties use the `%X{name}` notation and let you modify the format of access logs. The following are the default logging properties: Property Description `address` Either the `X-Forwarded-For` header or the client IP address. `user` Principal name, if using authentication. `method` Method used. `PUT`, `GET`, and so on. `protocol` Protocol used. `HTTP/1.1`, `HTTP/2`, `HOTROD/2.9`, and so on. `status` An HTTP status code for the REST endpoint. `OK` or an exception for the Hot Rod endpoint. `requestSize` Size, in bytes, of the request. `responseSize` Size, in bytes, of the response. `duration` Number of milliseconds that the server took to handle the request.  Use the header name prefixed with `h:` to log headers that were included in requests; for example, `%X{h:User-Agent}`. ### 3.2. Retrieving Server Health Statistics Monitor the health of your Infinispan clusters in the following ways: • Programmatically with `embeddedCacheManager.getHealth()` method calls. • JMX MBeans • Infinispan REST Server #### 3.2.1. Accessing the Health API via JMX Retrieve Infinispan cluster health statistics via JMX. Procedure 1. Connect to Infinispan server using any JMX capable tool such as JConsole and navigate to the following object: ``org.infinispan:type=CacheManager,name="default",component=CacheContainerHealth`` 2. Select available MBeans to retrieve cluster health statistics. #### 3.2.2. Accessing the Health API via REST Get Infinispan cluster health via the REST API. Procedure • Invoke a `GET` request to retrieve cluster health. ``GET /rest/v2/cache-managers/{cacheManagerName}/health`` Infinispan responds with a `JSON` document such as the following: ``````{ "cluster_health":{ "cluster_name":"ISPN", "health_status":"HEALTHY", "number_of_nodes":2, "node_names":[ "NodeA-36229", "NodeB-28703" ] }, "cache_health":[ { "status":"HEALTHY", "cache_name":"___protobuf_metadata" }, { "status":"HEALTHY", "cache_name":"cache2" }, { "status":"HEALTHY", "cache_name":"mycache" }, { "status":"HEALTHY", "cache_name":"cache1" } ] }``````  Get cache manager status as follows: ``GET /rest/v2/cache-managers/{cacheManagerName}/health/status`` Reference See the REST v2 (version 2) API documentation for more information. ### 3.3. Monitoring Infinispan Servers Over JMX You can monitor an Infinispan Server over JMX in two ways: • Use JConsole or VisualVM running locally as the same user. This will use a local jvmstat connection and requires no additional setup • Use JMX remoting (aka JSR-160) to connect from any host. This requires connecting through the management port (usually 9990) using a special protocol which respects the server security configuration To setup a client for JMX remoting you need to add the$ISPN_HOME/bin/client/jboss-client.jar to your client’s classpath and use one of the following service URLs:

• `service:jmx:remote-http-jmx://host:port` for plain connections through the management interface

• `service:jmx:remote-https-jmx://host:port` for TLS connections through the management interface (although this requires having the appropriate keys available)

• `service:jmx:remoting-jmx://localhost:port` for connections through the remoting interface (necessary for connecting to individual servers in a domain)

The JMX subsystem registers a service with the Remoting endpoint so that remote access to JMX can be obtained over the exposed Remoting connector. This is switched on by default in standalone mode and accessible over port 9990 but in domain mode it is switched off so it needs to be enabled. In domain mode the port will be the port of the Remoting connector for the Server instance to be monitored.

``````<subsystem xmlns="urn:jboss:domain:jmx:1.3">
<expose-resolved-model/>
<expose-expression-model/>
<remoting-connector use-management-endpoint="false"/>
</subsystem>``````

#### 3.3.1. JMX Monitoring

Management of Infinispan instances is all about exposing as much relevant statistical information that allows administrators to get a view of the state of each Infinispan instance. Taking in account that a single installation could be made up of several tens or hundreds Infinispan instances, providing clear and concise information in an efficient manner is imperative. The following sections dive into the range of management tooling that Infinispan provides.

 Any management tool that supports JMX already has basic support for Infinispan. However, custom plugins could be written to adapt the JMX information for easier consumption.
##### JMX

Over the years, JMX has become the de facto standard for management and administration of middleware and as a result, the Infinispan team has decided to standardize on this technology for the exposure of management and statistical information.

###### Understanding The Exposed MBeans

By connecting to the VM(s) where Infinispan is running with a standard JMX GUI such as JConsole or VisualVM you should find the following MBeans:

• For CacheManager level JMX statistics, without further configuration, you should see an MBean called org.infinispan:type=CacheManager,name="DefaultCacheManager" with properties specified by the CacheManager MBean .

• Using the cacheManagerName attribute in globalJmxStatistics XML element, or using the corresponding GlobalJmxStatisticsConfigurationBuilder.cacheManagerName(String cacheManagerName) call, you can name the cache manager in such way that the name is used as part of the JMX object name. So, if the name had been "Hibernate2LC", the JMX name for the cache manager would have been: org.infinispan:type=CacheManager,name="Hibernate2LC" . This offers a nice and clean way to manage environments where multiple cache managers are deployed, which follows JMX best practices .

• For Cache level JMX statistics, you should see several different MBeans depending on which configuration options have been enabled. For example, if you have configured a write behind cache store, you should see an MBean exposing properties belonging to the cache store component. All Cache level MBeans follow the same format though which is the following: `org.infinispan:type=Cache,name="${name-of-cache}(${cache-mode})",manager="${name-of-cache-manager}",component=${component-name}` where:

• ${name-of-cache} has been substituted by the actual cache name. If this cache represents the default cache, its name will be `___defaultCache`. •${cache-mode} has been substituted by the cache mode of the cache. The cache mode is represented by the lower case version of the possible enumeration values shown here.

• ${name-of-cache-manager} has been substituted by the name of the cache manager to which this cache belongs. The name is derived from the cacheManagerName attribute value in `globalJmxStatistics` element. •${component-name} has been substituted by one of the JMX component names in the JMX reference documentation .

For example, the cache store JMX component MBean for a default cache configured with synchronous distribution would have the following name: `org.infinispan:type=Cache,name="___defaultcache(dist_sync)",manager="DefaultCacheManager",component=CacheStore`

Please note that cache and cache manager names are quoted to protect against illegal characters being used in these user-defined names.

###### Enabling JMX Statistics

The MBeans mentioned in the previous section are always created and registered in the MBeanServer allowing you to manage your caches but some of their attributes do not expose meaningful values unless you take the extra step of enabling collection of statistics. Gathering and reporting statistics via JMX can be enabled at 2 different levels:

CacheManager level

The CacheManager is the entity that governs all the cache instances that have been created from it. Enabling CacheManager statistics collections differs depending on the configuration style:

• If configuring the CacheManager via XML, make sure you add the following XML under the `<cache-container />` element:

`<cache-container statistics="true"/>`
• If configuring the CacheManager programmatically, simply add the following code:

``````GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics().enable();``````
Cache level

At this level, you will receive management information generated by individual cache instances. Enabling Cache statistics collections differs depending on the configuration style:

• If configuring the Cache via XML, make sure you add the following XML under the one of the top level cache elements, such as `<local-cache />`:

``<local-cache statistics="true"/>``
• If configuring the Cache programmatically, simply add the following code:

``````ConfigurationBuilder configurationBuilder = ...
configurationBuilder.jmxStatistics().enable();``````
###### Monitoring cluster health

It is also possible to monitor Infinispan cluster health using JMX. On CacheManager there’s an additional object called `CacheContainerHealth`. It contains the following attributes:

• cacheHealth - a list of caches and corresponding statuses (HEALTHY, DEGRADED or HEALTHY_REBALANCING)

• clusterHealth - overall cluster health

• clusterName - cluster name

• freeMemoryKb - Free memory obtained from JVM runtime measured in KB

• numberOfCpus - The number of CPUs obtained from JVM runtime

• numberOfNodes - The number of nodes in the cluster

• totalMemoryKb - Total memory obtained from JVM runtime measured in KB

###### Multiple JMX Domains

There can be situations where several CacheManager instances are created in a single VM, or Cache names belonging to different CacheManagers under the same VM clash.

Using different JMX domains for multi cache manager environments should be last resort. Instead, it’s possible to name a cache manager in such way that it can easily be identified and used by monitoring tools. For example:

• Via XML:

``<cache-container statistics="true" name="Hibernate2LC"/>``
• Programmatically:

``````GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics()
.enable()
.cacheManagerName("Hibernate2LC");``````

Using either of these options should result on the CacheManager MBean name being: `org.infinispan:type=CacheManager,name="Hibernate2LC"`

For the time being, you can still set your own jmxDomain if you need to and we also allow duplicate domains, or rather duplicate JMX names, but these should be limited to very special cases where different cache managers within the same JVM are named equally.

###### Registering MBeans In Non-Default MBean Servers

Let’s discuss where Infinispan registers all these MBeans. By default, Infinispan registers them in the standard JVM MBeanServer platform . However, users might want to register these MBeans in a different MBeanServer instance. For example, an application server might work with a different MBeanServer instance to the default platform one. In such cases, users should implement the MBeanServerLookup interface provided by Infinispan so that the getMBeanServer() method returns the MBeanServer under which Infinispan should register the management MBeans. Once you have your implementation ready, simply configure Infinispan with the fully qualified name of this class. For example:

• Via XML:

``````<cache-container statistics="true">
<jmx mbean-server-lookup="com.acme.MyMBeanServerLookup" />
</cache-container>``````
• Programmatically:

``````GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics()
.enable()
.mBeanServerLookup(new com.acme.MyMBeanServerLookup());``````
###### Available MBeans

For a complete list of available MBeans, refer to the JMX reference documentation

## 4. Securing Infinispan Servers

### 4.1. Security Concepts

#### 4.1.1. Authorization

Just like embedded mode, the server supports cache authorization using the same configuration, e.g.:

``````<infinispan>
<cache-container default-cache="secured" name="secured">
<security>
<authorization>
<identity-role-mapper />
<role name="writer" permissions="WRITE" />
</authorization>
</security>
<local-cache name="secured">
<security>
</security>
</local-cache>
</cache-container>

</infinispan>``````

#### 4.1.2. Server Realms

Infinispan Server security is built around the features provided by the underlying server realm and security domains. Security Realms are used by the server to provide authentication and authorization information for both the management and application interfaces.

Security Realm configuration
``````<server xmlns="urn:jboss:domain:2.1">
...
<management>
...
<security-realm name="ApplicationRealm">
<authentication>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
<authorization>
<properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
</authorization>
</security-realm>
...
</management>
...
</server>``````

Infinispan Server comes with an add-user.sh script (add-user.bat for Windows) to ease the process of adding new user/role mappings to the above property files. An example invocation for adding a user to the ApplicationRealm with an initial set of roles:

It is also possible to authenticate/authorize against alternative sources, such as LDAP, JAAS, etc.

Bear in mind that the choice of authentication mechanism you select for the protocols limits the type of authentication sources, since the credentials must be in a format supported by the algorithm itself (e.g. pre-digested passwords for the digest algorithm)

### 4.2. Hot Rod Authentication

The Hot Rod protocol supports authentication by leveraging the SASL mechanisms. The supported SASL mechanisms (usually shortened as mechs) are:

• PLAIN - This is the most insecure mech, since credentials are sent over the wire in plain-text format, however it is the simplest to get to work. In combination with encryption (i.e. TLS) it can be used safely

• DIGEST-MD5 - This mech hashes the credentials before sending them over the wire, so it is more secure than PLAIN

• GSSAPI - This mech uses Kerberos tickets, and therefore requires the presence of a properly configured Kerberos Domain Controller (such as Microsoft Active Directory)

• EXTERNAL - This mech obtains credentials from the underlying transport (i.e. from a X.509 client certificate) and therefore requires encryption using client-certificates to be enabled.

The following configuration enables authentication against ApplicationRealm, using the DIGEST-MD5 SASL mechanism and only enables the auth QoP (see SASL Quality of Protection):

Hot Rod connector configuration
``````<hotrod-connector socket-binding="hotrod" cache-container="default">
<authentication security-realm="ApplicationRealm">
<sasl server-name="myhotrodserver" mechanisms="DIGEST-MD5" qop="auth" />
</authentication>
</hotrod-connector>``````

Notice the server-name attribute: it is the name that the server declares to incoming clients and therefore the client configuration must match. It is particularly important when using GSSAPI as it is equivalent to the Kerberos service name. You can specify multiple mechanisms and they will be attempted in order.

#### 4.2.1. SASL Quality of Protection

While the main purpose of SASL is to provide authentication, some mechanisms also support integrity and privacy protection, also known as Quality of Protection (or qop). During authentication negotiation, ciphers are exchanged between client and server, and they can be used to add checksums and encryption to all subsequent traffic. You can tune the required level of qop as follows:

QOP Description

auth

Authentication only

auth-int

Authentication with integrity protection

auth-conf

Authentication with integrity and privacy protection

#### 4.2.2. SASL Policies

You can further refine the way a mechanism is chosen by tuning the SASL policies. This will effectively include / exclude mechanisms based on whether they match the desired policies.

Policy Description

forward-secrecy

Specifies that the selected SASL mechanism must support forward secrecy between sessions. This means that breaking into one session will not automatically provide information for breaking into future sessions.

pass-credentials

Specifies that the selected SASL mechanism must require client credentials.

no-plain-text

Specifies that the selected SASL mechanism must not be susceptible to simple plain passive attacks.

no-active

Specifies that the selected SASL mechanism must not be susceptible to active (non-dictionary) attacks. The mechanism might require mutual authentication as a way to prevent active attacks.

no-dictionary

Specifies that the selected SASL mechanism must not be susceptible to passive dictionary attacks.

no-anonymous

Specifies that the selected SASL mechanism must not accept anonymous logins.

Each policy’s value is either "true" or "false". If a policy is absent, then the chosen mechanism need not have that characteristic (equivalent to setting the policy to "false"). One notable exception is the no-anonymous policy which, if absent, defaults to true, thus preventing anonymous connections.

 It is possible to have mixed anonymous and authenticated connections to the endpoint, delegating actual access logic to cache authorization configuration. To do so, set the no-anonymous policy to false and turn on cache authorization.

The following configuration selects all available mechanisms, but effectively only enables GSSAPI, since it is the only one that respects all chosen policies:

Hot Rod connector policies
``````<hotrod-connector socket-binding="hotrod" cache-container="default">
<authentication security-realm="ApplicationRealm">
<sasl server-name="myhotrodserver" mechanisms="PLAIN DIGEST-MD5 GSSAPI EXTERNAL" qop="auth">
<policy>
<no-active value="true" />
<no-anonymous value="true" />
<no-plain-text value="true" />
</policy<>
</sasl>
</authentication>
</hotrod-connector>``````

#### 4.2.3. Using GSSAPI/Kerberos

If you want to use GSSAPI/Kerberos, setup and configuration differs. First we need to define a Kerberos login module using the security domain subsystem:

Security domain configuration
``````<system-properties>
<property name="java.security.krb5.conf" value="/tmp/infinispan/krb5.conf"/>
<property name="java.security.krb5.debug" value="true"/>
<property name="jboss.security.disable.secdomain.option" value="true"/>
</system-properties>

<security-domain name="infinispan-server" cache-type="default">
<authentication>
<module-option name="debug" value="true"/>
<module-option name="storeKey" value="true"/>
<module-option name="refreshKrb5Config" value="true"/>
<module-option name="useKeyTab" value="true"/>
<module-option name="doNotPrompt" value="true"/>
<module-option name="keyTab" value="/tmp/infinispan/infinispan.keytab"/>
<module-option name="principal" value="HOTROD/localhost@INFINISPAN.ORG"/>
</authentication>
</security-domain>``````

Next we need to modify the Hot Rod connector

Hot Rod connector configuration
``````<hotrod-connector socket-binding="hotrod" cache-container="default">
<authentication security-realm="ApplicationRealm">
<sasl server-name="infinispan-server" server-context-name="infinispan-server" mechanisms="GSSAPI" qop="auth" />
</authentication>
</hotrod-connector>``````

### 4.3. Hot Rod and REST encryption (TLS/SSL)

Both Hot Rod and REST protocols support encryption using SSL/TLS with optional TLS/SNI support (Server Name Indication). To set this up you need to create a keystore using the keytool application which is part of the JDK to store your server certificate. Then add a <server-identities> element to your security realm:

Security Realm configuration for SSL
``````<security-realm name="ApplicationRealm">
<server-identities>
<ssl>
</ssl>
</server-identities>
</security-realm>``````
 When using SNI support there might be multiple Security Realms configured.

It is also possible to generate development certificates on server startup. In order to do this, just specify `generate-self-signed-certificate-host` in the keystore element as shown below:

Generating Keystore automatically
``````<security-realm name="ApplicationRealm">
<server-identities>
<ssl>
</ssl>
</server-identities>
</security-realm>``````
 There are three basic principles that you should remember when using automatically generated keystores: They shouldn’t be used on a production environment They are generated when necessary (e.g. while obtaining the first connection from the client) They contain also certificates so they might be used in a Hot Rod client directly

Next modify the <hotrod-connector> and/or <rest-connector> elements in the endpoint subsystem to require encryption. Optionally add SNI configuration:

Hot Rod connector SSL configuration
``````<hotrod-connector socket-binding="hotrod" cache-container="local">
<encryption security-realm="ApplicationRealm" require-ssl-client-auth="false">
<sni host-name="domain1" security-realm="Domain1ApplicationRealm" />
<sni host-name="domain2" security-realm="Domain2ApplicationRealm" />
</encryption>
</hotrod-connector>
<rest-connector socket-binding="rest" cache-container="local">
<encryption security-realm="ApplicationRealm" require-ssl-client-auth="false">
<sni host-name="domain1" security-realm="Domain1ApplicationRealm" />
<sni host-name="domain2" security-realm="Domain2ApplicationRealm" />
</encryption>
</rest-connector>``````
 To configure the client In order to connect to the server using the Hot Rod protocol, the client needs a trust store containing the public key of the server(s) you are going to connect to, unless the key was signed by a Certification Authority (CA) trusted by the JRE.
``````ConfigurationBuilder clientBuilder = new ConfigurationBuilder();
clientBuilder
.host("127.0.0.1")
.port(11222)
.security()
.ssl()
.enabled(true)
.sniHostName("domain1")
.trustStoreFileName("truststore_client.jks")
remoteCacheManager = new RemoteCacheManager(clientBuilder.build());``````

Additionally, you might also want to enable client certificate authentication (and optionally also allow the use of the EXTERNAL SASL mech to authenticate and authorize clients). To enable this you will need the security realm on the server to be able to trust incoming client certificates by adding a trust store:

``````<security-realm name="ApplicationRealm">
<authentication>
</authentication>
<server-identities>
<ssl>
</ssl>
</server-identities>
</security-realm>``````

And then tell the connector to require a client certificate:

``````<hotrod-connector socket-binding="hotrod" cache-container="local">
<encryption security-realm="ApplicationRealm" require-ssl-client-auth="true" />
</hotrod-connector>``````

The client, at this point, will also need to specify a keyStore which contains its certificate on top of the trustStore which trusts the server certificate.

## 5. Single Port

Single-Port is a special type of router connector which allows exposing multiple protocols over the same TCP port. This approach is very convenient because it reduces the number of ports required by a server, with advantages in security, configuration and management. Protocol switching is handled in three ways:

• HTTP/1.1 Upgrade header: initiate an HTTP/1.1 connection and send an `Upgrade: protocol` header where protocol is the name assigned to the desired endpoint.

• TLS/ALPN: protocol selection is performed based on the SNI specified by the client.

• Hot Rod header detection: if a Hot Rod endpoint is present in the router configuration, then any attempt to send a Hot Rod header will be detected and the protocol will be switched automatically.

 The initial implementation supports only HTTP/1.1, HTTP/2 and Hot Rod protocols. The Memcached protocol is not supported.

### 5.1. Single-Port router

Internally, Single-Port is based on the same router component used to enable multi-tenancy, and therefore it shares the same configuration.

``````<!-- TLS/ALPN negotiation -->
<router-connector name="router-ssl" single-port-socket-binding="rest-ssl">
<single-port security-realm="SSLRealm1">
<hotrod name="hotrod" />
<rest name="rest" />
</single-port>
</router-connector>
<router-connector name="router" single-port-socket-binding="rest">
<single-port>
<hotrod name="hotrod" />
<rest name="rest" />
</single-port>
</router-connector>``````

With the configuration above, the Single-Port Router will operate on `rest` and `rest-ssl` socket bindings. The router named `router` should typically operate on port `8080` and will use HTTP/1.1 Upgrade (also known as cleartext upgrade) procedure. The other router instance (called `router-ssl`) should typically operate on port `8443` and will use TLS/ALPN.

#### 5.1.1. Testing the Single-Port router

A tool such as `curl` can be used to access cache using both cleartext upgrade or TLS/ALPN. Here’s an example:

``$curl -v -k --http2-prior-knowledge https://127.0.0.1:8443/rest/v2/caches/default/test`` The `--http2-prior-knowledge` can be exchanged with `--http2` switch allowing to control how the switch procedure is being done (via Plain-Text Upgrade or TLS/ALPN). ### 5.2. Hot Rod Protocol Detection The single-port router has built-in automatic detection of Hot Rod messages which trigger a transparent "upgrade" to the Hot Rod protocol. This means that no changes are required on the client side to connect to a single-port endpoint. It also means that older clients will also be able to function seamlessly. #### 5.2.1. TLS/ALPN protocol selection Another supported way to select the protocol is to use TLS/ALPN which uses the Application-Layer Protocol Negotiation spec. This feature requires that you have configured your endpoint to enable TLS. ##### Enabling ALPN If you are using JDK 9 or greater, ALPN is supported by default. However, if you are using JDK 8, you will need to use Netty’s BoringSSL library, which leverages native libraries to enable ALPN. 1. Add Netty dependencies. ``````<dependencyManagement> <dependency> <groupId>io.netty</groupId> <artifactId>netty-bom</artifactId> <!-- Pulled from Infinispan BOM --> <version>${version.netty}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<!-- The version is defined in Netty BOM -->
</dependency>``````
2. Configure your trust store accordingly:

``````ConfigurationBuilder builder = new ConfigurationBuilder()

builder.security().ssl().enable()
.trustStoreFileName("truststore.pkcs12")

RemoteCacheManager remoteCacheManager = new RemoteCacheManager(builder.build());
RemoteCache<String, String> cache = remoteCacheManager.getCache("default"");``````

## 6. Remotely Executing Server-Side Tasks

Define and add tasks to Infinispan servers that you can invoke from the Infinispan command line interface, REST API, or from Hot Rod clients.

You can implement tasks as custom Java classes or define scripts in languages such as JavaScript.

Infinispan server tasks are classes that extend the `org.infinispan.tasks.ServerTask` interface and generally include the following method calls:

`setTaskContext()`

Allows access to execution context information including task parameters, cache references on which tasks are executed, and so on. In most cases, implementations store this information locally and use it when tasks are actually executed.

`getName()`

`getExecutionMode()`

Returns the execution mode for tasks.

• `TaskExecutionMode.ONE_NODE` only the node that handles the request executes the script. Although scripts can still invoke clustered operations.

• `TaskExecutionMode.ALL_NODES` Infinispan uses clustered executors to run scripts across nodes. For example, server tasks that invoke stream processing need to be executed on a single node because stream processing is distributed to all nodes.

`call()`

Computes a result. This method is defined in the `java.util.concurrent.Callable` interface and is invoked with server tasks.

 Server task implementations must adhere to service loader pattern requirements. For example, implementations must have a zero-argument constructors.

The following `HelloTask` class implementation provides an example task that has one parameter:

``````package example;

@Override
this.ctx = ctx;
}

@Override
public String call() throws Exception {
String name = (String) ctx.getParameters().get().get("name");
return "Hello " + name;
}

@Override
public String getName() {
}

}``````

#### 6.1.2. Deploying Server Tasks to Infinispan Servers

Prerequisites

Stop any running Infinispan servers. Infinispan does not support runtime deployment of custom classes.

Procedure

2. Add a `META-INF/services/org.infinispan.tasks.ServerTask` file that contains the fully qualified names of server tasks, for example:

``example.HelloTask``
3. Copy the JAR file to the `$ISPN_HOME/server/lib` directory of your Infinispan server. 4. Add your classes to the deserialization whitelist in your Infinispan configuration. Alternatively set the whitelist using system properties. ### 6.2. Creating Server Scripts Create custom scripts and add them to Infinispan servers. #### 6.2.1. Server Scripts Infinispan server scripting is based on the `javax.script` API and is compatible with any JVM-based ScriptEngine implementation. Nashorn is the default JDK ScriptEngine and provides JavaScript capabilities. Hello World Script Example The following script provides a simple example that runs on a single Infinispan server, has one parameter, and uses JavaScript: ``````// mode=local,language=javascript,parameters=[greetee] "Hello " + greetee`````` When you run the preceding script, you pass a value for the `greetee` parameter and Infinispan returns `"Hello${value}"`.

Script metadata are `property=value` pairs that you add to comments in the first lines of scripts, such as the following example:

``````// name=test, language=javascript
// mode=local, parameters=[a,b,c]``````
• Use comment styles that match the scripting language (`//`, `;;`, `#`).

• Separate `property=value` pairs with commas.

• Separate values with single (') or double (") quote characters.

Property Description

`mode`

Defines the exection mode and has the following values:

`local` only the node that handles the request executes the script. Although scripts can still invoke clustered operations.

`distributed` Infinispan uses clustered executors to run scripts across nodes.

`language`

Specifies the ScriptEngine that executes the script.

`extension`

Specifies filename extensions as an alternative method to set the ScriptEngine.

`role`

Specifies roles that users must have to execute scripts.

`parameters`

Specifies an array of valid parameter names for this script. Invocations which specify parameters not included in this list cause exceptions.

`datatype`

Optionally sets the MediaType (MIME type) for storing data as well as parameter and return values. This property is useful for remote clients that support particular data formats only.

Currently you can set only `text/plain; charset=utf-8` to use the String UTF-8 format for data.

##### Script Bindings

Infinispan exposes internal objects as bindings for script execution.

Binding Description

`cache`

Specifies the cache against which the script is run.

`marshaller`

Specifies the marshaller to use for serializing data to the cache.

`cacheManager`

Specifies the `cacheManager` for the cache.

`scriptingManager`

Specifies the instance of the script manager that runs the script. You can use this binding to run other scripts from a script.

##### Script Parameters

Infinispan lets you pass named parameters as bindings for running scripts.

Parameters are `name,value` pairs, where `name` is a string and `value` is any value that the marshaller can interpret.

The following example script has two parameters, `multiplicand` and `multiplier`. The script takes the value of `multiplicand` and multiplies it with the value of `multiplier`.

``````// mode=local,language=javascript
multiplicand * multiplier``````

When you run the preceding script, Infinispan responds with the result of the expression evaluation.

#### 6.2.2. Adding Scripts to Infinispan Servers

Use the command line interface to add scripts to Infinispan servers.

Prerequisites

Infinispan servers store scripts in the `___script_cache` cache. If you enable cache authorization, users must have the `___script_manager` role to access `___script_cache`.

Procedure
1. Define scripts as required.

For example, create a file named `multiplication.js` that runs on a single Infinispan server, has two parameters, and uses JavaScript to multiply a given value:

``````// mode=local,language=javascript
multiplicand * multiplier``````
2. Open a CLI connection to Infinispan and use the `task` command to upload your scripts as in the following example:

`[//containers/default]> task upload --file=multiplication.js multiplication`
3. Verify that your scripts are available.

```[//containers/default]> ls tasks
multiplication```

#### 6.2.3. Programmatically Creating Scripts

Add scripts with the Hot Rod `RemoteCache` interface as in the following example:

``````RemoteCache<String, String> scriptCache = cacheManager.getCache("___script_cache");
scriptCache.put("multiplication.js",
"// mode=local,language=javascript\n" +
"multiplicand * multiplier\n");``````

### 6.3. Running Server-Side Tasks and Scripts

Execute tasks and custom scripts on Infinispan servers.

#### 6.3.1. Running Tasks and Scripts

Use the command line interface to run tasks and scripts on Infinispan servers.

Prerequisites
• Open a CLI connection to Infinispan.

Procedure

Use the `task` command to run tasks and scripts on Infinispan servers, as in the following examples:

• Execute a script named `multipler.js` and specify two parameters:

```[//containers/default]> task exec multipler.js -Pmultiplicand=10 -Pmultiplier=20
200.0```
• Execute a task named `@@cache@names` to retrieve a list of all available caches:

```//containers/default]> task exec @@cache@names

#### 6.3.2. Programmatically Running Scripts

Call the `execute()` method to run scripts with the Hot Rod `RemoteCache` interface, as in the following example:

``````RemoteCache<String, Integer> cache = cacheManager.getCache();
// Create parameters for script execution.
Map<String, Object> params = new HashMap<>();
params.put("multiplicand", 10);
params.put("multiplier", 20);
// Run the script with the parameters.
Object result = cache.execute("multiplication.js", params);``````

Call the `execute()` method to run tasks with the Hot Rod `RemoteCache` interface, as in the following example:

``````// Add configuration for a locally running server.
ConfigurationBuilder builder = new ConfigurationBuilder();

// Connect to the server.
RemoteCacheManager cacheManager = new RemoteCacheManager(builder.build());

// Retrieve the remote cache.
RemoteCache<String, String> cache = cacheManager.getCache();

Map<String, String> parameters = new HashMap<>();
parameters.put("name", "developer");

System.out.println(greet);``````

## 7. Adding Custom Marshaller Bridges

Infinispan provides two marshalling bridges for marshalling client/server requests using the Kryo and Protostuff libraries. To utilise either of these marshallers, you simply place the dependency of the marshaller you require in your client pom. Custom schemas for object marshalling must then be registered with the selected library using the library’s api on the client or by implementing a RegistryService for the given marshaller bridge.

### 7.1. Protostuff

``````<dependency>
<groupId>org.infinispan</groupId>
<artifactId>infinispan-marshaller-protostuff</artifactId>
<!-- Replace ${version.infinispan} with the version of Infinispan that you're using. --> <version>${version.infinispan}</version>
</dependency>``````

To register custom Protostuff schemas in your own code, you must register the custom schema with Protostuff before any marshalling begins. This can be achieved by simply calling:

``RuntimeSchema.register(ExampleObject.class, new ExampleObjectSchema());``

Or, you can implement a service provider for the `SchemaRegistryService.java` interface, placing all Schema registrations in the `register()` method. Implementations of this interface are loaded via Java’s ServiceLoader api, therefore the full path of the implementing class(es) should be provided in a `META-INF/services/org/infinispan/marshaller/protostuff/SchemaRegistryService` file within your deployment jar.

### 7.2. Kryo

``````<dependency>
<groupId>org.infinispan</groupId>
<artifactId>infinispan-marshaller-kryo</artifactId>
<!-- Replace ${version.infinispan} with the version of Infinispan that you're using. --> <version>${version.infinispan}</version>
</dependency>``````

To register custom Kryo serializer in your own code, you must register the custom serializer with Kryo before any marshalling begins. This can be achieved by implementing a service provider for the `SerializerRegistryService.java` interface, placing all serializer registrations in the `register(Kryo)` method; where serializers should be registered with the supplied `Kryo` object using the Kryo api. e.g. `kryo.register(ExampleObject.class, new ExampleObjectSerializer())`. Implementations of this interface are loaded via Java’s ServiceLoader api, therefore the full path of the implementing class(es) should be provided in a `META-INF/services/org/infinispan/marshaller/kryo/SerializerRegistryService` file within your deployment jar.

### 7.3. Storing deserialized objects

When using the Protostuff/Kryo bridges in caches configured with application/x-java-object as MediaType (storing POJOs instead of binary content) it is necessary for the class files of all custom objects to be placed on the classpath of the server. To achieve this, you should place a jar containing all of their custom classes on the server’s classpath.

When utilising a custom marshaller, it is also necessary for the marshaller and it’s runtime dependencies to be on the server’s classpath. To aid with this step we have created a "bundle" jar for each of the bridge implementations which includes all of the runtime class files required by the bridge and underlying library. Therefore, it is only necessary to include this single jar on the server’s classpath.

 Jar files containing custom classes must be placed in the same module/directory as the custom marshaller bundle so that the marshaller can load them. i.e. if you register the marshaller bundle in `modules/system/layers/base/org/infinispan/main/modules.xml`, then you must also register your custom classes here.

### 7.4. Registering Custom Schemas/Serializers

Custom serializers/schemas for the Kryo/Protostuff marshallers must be registered via their respective service interfaces in order to store deserialized objects. To achieve this, it is necessary for a JAR that contains the service provider to be registered in the same directory or module as the marshaller bundle and custom classes.

 It is not necessary for the service provider implementation to be provided in the same JAR as the user’s custom classes. However, the JAR that contains the provider must be in the same directory/module as the marshaller and custom class JAR files.

Upgrade Infinispan without downtime or data loss. You can perform rolling upgrades for Infinispan servers to start using a more recent version of Infinispan.

 This section explains how to upgrade Infinispan servers, see the appropriate documentation for your Hot Rod client for upgrade procedures.

From a high-level, you do the following to perform rolling upgrades:

1. Set up a target cluster. The target cluster is the Infinispan version to which you want to migrate data. The source cluster is the Infinispan deployment that is currently in use. After the target cluster is running, you configure all clients to point to it instead of the source cluster.

2. Synchronize data from the source cluster to the target cluster.

### 8.2. Setting Up Target Clusters

1. Start the target cluster with unique network properties or a different JGroups cluster name to keep it separate from the source cluster.

2. Configure a `RemoteCacheStore` on the target cluster for each cache you want to migrate from the source cluster.

`RemoteCacheStore` settings
• `remote-server` must point to the source cluster via the `outbound-socket-binding` property.

• `remoteCacheName` must match the cache name on the source cluster.

• `hotrod-wrapping` must be `true` (enabled).

• `shared` must be `true` (enabled).

• `purge` must be `false` (disabled).

• `passivation` must be `false` (disabled).

• `protocol-version` matches the Hot Rod protocol version of the source cluster.

Example `RemoteCacheStore` Configuration
``````<distributed-cache>
<remote-store cache="MyCache" socket-timeout="60000" tcp-no-delay="true" protocol-version="2.5" shared="true" hotrod-wrapping="true" purge="false" passivation="false">
<remote-server outbound-socket-binding="remote-store-hotrod-server"/>
</remote-store>
</distributed-cache>
...
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}"> ... <outbound-socket-binding name="remote-store-hotrod-server"> <remote-destination host="198.51.100.0" port="11222"/> </outbound-socket-binding> ... </socket-binding-group>`````` 3. Configure the target cluster to handle all client requests instead of the source cluster: 1. Configure all clients to point to the target cluster instead of the source cluster. 2. Restart each client node. The target cluster lazily loads data from the source cluster on demand via `RemoteCacheStore`. ### 8.3. Synchronizing Data from Source Clusters 1. Call the `synchronizeData()` method in the `TargetMigrator` interface. Do one of the following on the target cluster for each cache that you want to migrate: JMX Invoke the `synchronizeData` operation and specify the `hotrod` parameter on the `RollingUpgradeManager` MBean. CLI ``$ bin/cli.sh --connect controller=127.0.0.1:9990 -c "/subsystem=datagrid-infinispan/cache-container=clustered/distributed-cache=MyCache:synchronize-data(migrator-name=hotrod)"``

Data migrates to all nodes in the target cluster in parallel, with each node receiving a subset of the data.

Use the following parameters to tune the operation:

• `read-batch` configures the number of entries to read from the source cluster at a time. The default value is `10000`.

• `write-threads` configures the number of threads used to write data. The default value is the number of processors available.

For example:

`synchronize-data(migrator-name=hotrod, read-batch=100000, write-threads=3)`

2. Disable the `RemoteCacheStore` on the target cluster. Do one of the following:

JMX

Invoke the `disconnectSource` operation and specify the `hotrod` parameter on the `RollingUpgradeManager` MBean.

CLI
``\$ bin/cli.sh --connect controller=127.0.0.1:9990 -c "/subsystem=datagrid-infinispan/cache-container=clustered/distributed-cache=MyCache:disconnect-source(migrator-name=hotrod)"``
3. Decommission the source cluster.