The Infinispan Operator provides operational intelligence and reduces management complexity for deploying Infinispan on Kubernetes and Red Hat OpenShift.

1. Installing Infinispan Operator

Install Infinispan Operator into a Kubernetes namespace to create and manage Infinispan clusters.

1.1. Installing Infinispan Operator on Red Hat OpenShift

Create subscriptions to Infinispan Operator on OpenShift so you can install different Infinispan versions and receive automatic updates.

Prerequisites
  • Access to OperatorHub running on OpenShift. Some OpenShift environments, such as OpenShift Container Platform, can require administrator credentials.

  • An OpenShift project for Infinispan Operator.

Procedure
  1. Log in to the OpenShift Web Console.

  2. Navigate to OperatorHub.

  3. Find and select Infinispan Operator.

  4. Select Install and continue to Create Operator Subscription.

  5. Specify options for your subscription.

    Installation Mode

    Install Infinispan Operator into a specific OpenShift namespace. You cannot install Infinispan Operator into multiple namespaces.

    Update Channel

    Subscribe to updates for Infinispan Operator versions.

    Approval Strategies

    When new Infinispan versions become available, you can install updates manually or let Infinispan Operator install them automatically.

    Automatic updates apply to Infinispan Operator first and then for each Infinispan node. Infinispan Operator updates clusters one node at a time, gracefully shutting down each node and then bringing it back online with the updated version before going on to the next node.

  6. Select Subscribe to install Infinispan Operator.

1.2. Installing Infinispan Operator on Kubernetes

Install Infinispan Operator on Kubernetes from OperatorHub.io.

Prerequisites
  • OKD 3.11 or later.

  • Kubernetes 1.11 or later.

  • Administrator access to the Kubernetes cluster.

Procedure
  1. Navigate to the Infinispan Operator entry on OperatorHub.io.

  2. Follow the instructions to install Infinispan Operator into your Kubernetes cluster.

1.3. Building and Installing Infinispan Operator Manually

Manually build and install Infinispan Operator from the GitHub repository.

Procedure

1.4. Infinispan Upgrades

Infinispan Operator automatically upgrades Infinispan clusters when new versions become available.

To upgrade Infinispan clusters, Infinispan Operator checks the version of the image for Infinispan nodes. If Infinispan Operator determines that a new version of the image is available, it gracefully shuts down all nodes, applies the new image, and then restarts the nodes.

Infinispan Operator requires the Operator Lifecycle Manager to perform cluster upgrades.

2. Getting Started with Infinispan Operator

Infinispan Operator lets you create, configure, and manage Infinispan clusters.

Prerequisites
  • Install Infinispan Operator.

  • Have an oc or a kubectl client.

2.1. Infinispan Custom Resource (CR)

Infinispan Operator adds a new Custom Resource (CR) of type Infinispan that lets you handle Infinispan clusters as complex units on Kubernetes.

Infinispan Operator watches for Infinispan Custom Resources (CR) that you use to instantiate and configure Infinispan clusters and manage Kubernetes resources, such as StatefulSets and Services. In this way, the Infinispan CR is your primary interface to Infinispan on Kubernetes.

The minimal Infinispan CR is as follows:

apiVersion: infinispan.org/v1 (1)
kind: Infinispan (2)
metadata:
  name: example-infinispan (3)
spec:
  replicas: (4)
1 Declares the Infinispan API version.
2 Declares the Infinispan CR.
3 Names the Infinispan cluster.
4 Specifies the number of nodes in the Infinispan cluster.

2.2. Creating Infinispan Clusters

Use Infinispan Operator to create clusters of two or more Infinispan nodes.

Procedure
  1. Specify the number of Infinispan nodes in the cluster with spec.replicas in your Infinispan CR.

    For example, create a cr_minimal.yaml file as follows:

    $ cat > cr_minimal.yaml<<EOF
    apiVersion: infinispan.org/v1
    kind: Infinispan
    metadata:
      name: example-infinispan
    spec:
      replicas: 2
    EOF
  2. Apply your Infinispan CR.

    $ kubectl apply -f cr_minimal.yaml
  3. Watch Infinispan Operator create the Infinispan nodes.

    $ kubectl get pods -w
    
    NAME                        READY  STATUS              RESTARTS   AGE
    example-infinispan-1        0/1    ContainerCreating   0          4s
    example-infinispan-2        0/1    ContainerCreating   0          4s
    example-infinispan-3        0/1    ContainerCreating   0          5s
    infinispan-operator-0       1/1    Running             0          3m
    example-infinispan-3        1/1    Running             0          8s
    example-infinispan-2        1/1    Running             0          8s
    example-infinispan-1        1/1    Running             0          8s
Next Steps

Try changing the value of replicas: and watching Infinispan Operator scale the cluster up or down.

2.3. Verifying Infinispan Clusters

Review log messages to ensure that Infinispan nodes receive clustered views.

Procedure
  • Do either of the following:

    • Retrieve the cluster view from logs.

      $ kubectl logs example-infinispan-0 | grep ISPN000094
      
      INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|0] (1) [example-infinispan-0]
      
      INFO  [org.infinispan.CLUSTER] (jgroups-3,example-infinispan-0) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|1] (2) [example-infinispan-0, example-infinispan-1]
    • Retrieve the Infinispan CR for Infinispan Operator.

      $ kubectl get infinispan -o yaml

      The response indicates that Infinispan pods have received clustered views:

      conditions:
          - message: 'View: [example-infinispan-0, example-infinispan-1]'
            status: "True"
            type: wellFormed

Use kubectl wait with the wellFormed condition for automated scripts.

$ kubectl wait --for condition=wellFormed --timeout=240s infinispan/example-infinispan

3. Setting Up Infinispan Services

Use Infinispan Operator to create clusters of either Cache Service or Data Grid Service nodes.

3.1. Service Types

Services are stateful applications, based on the Infinispan server image, that provide flexible and robust in-memory data storage.

Cache Service

Use Cache Service if you want a volatile, low-latency data store with minimal configuration. Cache Service nodes:

  • Automatically scale to meet capacity when data storage demands go up or down.

  • Synchronously distribute data to ensure consistency.

  • Replicates each entry in the cache across the cluster.

  • Store cache entries off-heap and use eviction for JVM efficiency.

  • Ensure data consistency with a default partition handling configuration.

Because Cache Service nodes are volatile you lose all data when you apply changes to the cluster with the Infinispan CR or update the Infinispan version.

Data Grid Service

Use Data Grid Service if you want to:

  • Back up data across global clusters with cross-site replication.

  • Create caches with any valid configuration.

  • Add file-based cache stores to save data in the persistent volume.

  • Use Infinispan search and other advanced capabilities.

3.2. Creating Cache Service Nodes

By default, Infinispan Operator creates Infinispan clusters with Cache Service nodes.

Procedure
  1. Create an Infinispan CR.

    apiVersion: infinispan.org/v1
    kind: Infinispan
    metadata:
      name: example-infinispan
    spec:
      replicas: 2
      service:
        type: Cache (1)
    1 Creates nodes Cache Service nodes. This is the default for the Infinispan CR.
  2. Apply your Infinispan CR to create the cluster.

3.2.1. Configuring Automatic Scaling

If you create clusters with Cache Service nodes, Infinispan Operator can automatically scale nodes up or down based on memory usage for the default cache.

Infinispan Operator monitors default caches on Cache Service nodes. As you add data to the cache, memory usage increases. When it detects that the cluster needs additional capacity, Infinispan Operator creates new nodes rather than eviciting entries. Likewise, if it detects that memory usage is below a certain threshold, Infinispan Operator shuts down nodes.

Automatic scaling works with the default cache only. If you plan to add other caches to your cluster, you should not include the autoscale field in your Infinispan CR. In this case you should use eviction to control the size of the data container on each node.

Procedure
  1. Add the spec.autoscale resource to your Infinispan CR to enable automatic scaling.

  2. Configure memory usage thresholds and number of nodes for your cluster with the autoscale field.

    spec:
      ...
      service:
        type: Cache
      autoscale:
        maxMemUsagePercent: 70 (1)
        maxReplicas: 5 (2)
        minMemUsagePercent: 30 (3)
        minReplicas: 2 (4)
    1 Configures the maximum threshold, as a percentage, for memory usage on each node. When Infinispan Operator detects that any node in the cluster reaches the threshold, it creates a new node if possible. If Infinispan Operator cannot create a new node then it performs eviction when memory usage reaches 100 percent.
    2 Defines the maximum number of number of nodes for the cluster.
    3 Configures the minimum threshold, as a percentage, for memory usage across the cluster. When Infinispan Operator detects that memory usage falls below the minimum, it shuts down nodes.
    4 Defines the minimum number of number of nodes for the cluster.
  3. Apply the changes.

3.2.2. Configuring the Number of Owners

The number of owners controls how many copies of each cache entry are replicated across your Infinispan cluster. The default for Cache Service nodes is two, which duplicates each entry to prevent data loss.

Procedure
  1. Specify the number of owners with the spec.service.replicationFactor resource in your Infinispan CR as follows:

    spec:
      ...
      service:
        type: Cache
        replicationFactor: 3 (1)
    1 Configures three replicas for each cache entry.
  2. Apply the changes.

3.2.3. Cache Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan (1)
spec:
  replicas: 4 (2)
  service:
    type: Cache (3)
    replicationFactor: 2 (4)
  autoscale: (5)
    maxMemUsagePercent: 70
    maxReplicas: 5
    minMemUsagePercent: 30
    minReplicas: 2
  security:
    endpointSecretName: endpoint-identities (6)
    endpointEncryption: (7)
        type: Secret
        certSecretName: tls-secret
  container: (8)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m"
    memory: 1Gi
  logging: (9)
    categories:
      org.infinispan: trace
      org.jgroups: trace
  expose: (10)
    type: LoadBalancer
1 Names the Infinispan cluster.
2 Specifies the number of nodes in the cluster.
3 Creates Cache Service clusters. This is the default for Infinispan CR.
4 Configures the number of replicas for each cache entry across the cluster.
5 Enables and configures automatic cluster scaling.
6 Adds an authentication secret with user credentials.
7 Adds a custom encryption secret for secure connections.
8 Allocates resources to nodes.
9 Configures logging.
10 Configures services for external traffic.

3.3. Creating Data Grid Service Nodes

To use custom cache definitions along with Infinispan capabilities such as cross-site replication, create clusters of Data Grid Service nodes.

Procedure
  1. Specify DataGrid as the value for spec.service.type in your Infinispan CR.

    apiVersion: infinispan.org/v1
    kind: Infinispan
    metadata:
      name: example-infinispan
    spec:
      replicas: 2
      service:
        type: DataGrid

    You cannot change the spec.service.type field after you create nodes. To change the service type, you must delete the existing nodes and create new ones.

  2. Configure nodes with any other Data Grid Service resources.

  3. Apply your Infinispan CR to create the cluster.

3.3.1. Data Grid Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan (1)
spec:
  replicas: 6 (2)
  service:
    type: DataGrid (3)
    container:
      storage: 2Gi (4)
    sites: (5)
      local:
      name: NYC
      expose:
        type: LoadBalancer
      locations:
      - name: azure
        url: openshift://api.azure.host:6443
        secretName: azure-token
      - name: aws
        url: openshift://api.aws.host:6443
        secretName: aws-token
  security:
    endpointSecretName: endpoint-identities (6)
    endpointEncryption: (7)
        type: Secret
        certSecretName: tls-secret
  container: (8)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "1000m"
    memory: 1Gi
  logging: (9)
    categories:
      org.infinispan: debug
      org.jgroups: debug
      org.jgroups.protocols.TCP: error
      org.jgroups.protocols.relay.RELAY2: fatal
  expose: (10)
    type: LoadBalancer
1 Names the Infinispan cluster.
2 Specifies the number of nodes in the cluster.
3 Creates Data Grid Service clusters.
4 Configures size of the persistent volume.
5 Provides connection information for backup locations.
6 Adds an authentication secret with user credentials.
7 Adds a custom encryption secret for secure connections.
8 Allocates resources to nodes.
9 Configures logging.
10 Configures services for external traffic.

4. Adjusting Container Specifications

You can allocate CPU and memory resources, specify JVM options, and configure storage for Infinispan nodes.

4.1. JVM, CPU, and Memory Resources

spec:
  ...
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary" (1)
    cpu: "1000m" (2)
    memory: 1Gi (3)
1 Specifies JVM options.
2 Allocates host CPU resources to node, measured in CPU units.
3 Allocates host memory resources to nodes, measured in bytes.

When Infinispan Operator creates Infinispan clusters, it uses spec.container.cpu and spec.container.memory to:

  • Ensure that Kubernetes has sufficient capacity to run the Infinispan node. By default Infinispan Operator requests 512Mi of memory and 0.5 cpu from the Kubernetes scheduler.

  • Constrain node resource usage. Infinispan Operator sets the values of cpu and memory as resource limits.

Garbage collection logging

By default, Infinispan Operator does not log garbage collection (GC) messages. You can optionally add the following JVM options to direct GC messages to stdout:

extraJvmOpts: "-Xlog:gc*:stdout:time,level,tags"

4.2. Storage Resources

By default, Infinispan Operator allocates 1Gi for storage for both Cache Service and Data Grid Service nodes. You can configure storage size for Data Grid Service nodes but not Cache Service nodes.

spec:
  ...
  service:
    type: DataGrid
    container:
      storage: 2Gi (1)
1 Configures the storage size for Data Grid Service nodes.
Persistent Volume Claims

Infinispan Operator mounts persistent volumes at:
/opt/infinispan/server/data

Persistent volume claims use the ReadWriteOnce (RWO) access mode.

5. Stopping and Starting Infinispan Clusters

Stop and start Infinispan clusters with Infinispan Operator.

Cache definitions

Both Cache Service and Data Grid Service store permanent cache definitions in persistent volumes so they are still available after cluster restarts.

Data

Data Grid Service nodes can write all cache entries to persistent storage during cluster shutdown if you add a file-based cache store.

5.1. Shutting Down Infinispan Clusters

Shutting down Cache Service nodes removes all data in the cache. For Data Grid Service nodes, you should configure the storage size for Data Grid Service nodes to ensure that the persistent volume can hold all your data.

If the available container storage is less than the amount of memory available to Data Grid Service nodes, Infinispan writes the following exception to logs and data loss occurs during shutdown:

WARNING: persistent volume size is less than memory size. Graceful shutdown may not work.
Procedure
  • Set the value of replicas to 0 and apply the changes.

spec:
  replicas: 0

5.2. Restarting Infinispan Clusters

Complete the following procedure to restart Infinispan clusters after shutdown.

Prerequisites

For Data Grid Service nodes, you must restart clusters with the same number of nodes before shutdown. For example, you shut down a cluster of 6 nodes. When you restart that cluster, you must specify 6 as the value for spec.replicas.

This allows Infinispan to restore the distribution of data across the cluster. When all nodes in the cluster are running, you can then add or remove nodes.

You can find the correct number of nodes for Infinispan clusters as follows:

$ kubectl get infinispan example-infinispan -o=jsonpath='{.status.replicasWantedAtRestart}'
Procedure
  • Set the value of spec.replicas to the appropriate number of nodes for your cluster, for example:

    spec:
      replicas: 6

6. Configuring Network Access to Infinispan

Expose Infinispan clusters so you can access Infinispan Console, the Infinispan command line interface (CLI), REST API, and Hot Rod endpoint.

6.1. Getting the Service for Internal Connections

By default, Infinispan Operator creates a service that provides access to Infinispan clusters from clients running in Kubernetes.

This internal service has the same name as your Infinispan cluster, for example:

metadata:
  name: example-infinispan
Procedure
  • Check that the internal service is available as follows:

    $ kubectl get services
    
    NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)
    example-infinispan ClusterIP   192.0.2.0        <none>        11222/TCP
Reference

6.2. Exposing Infinispan Through Load Balancers

Use a load balancer service to make Infinispan clusters available to clients running outside Kubernetes.

To access Infinispan with unencrypted Hot Rod client connections you must use a load balancer service.

Procedure
  1. Include spec.expose in your Infinispan CR.

  2. Specify LoadBalancer as the service type with spec.expose.type.

    spec:
      ...
      expose:
        type: LoadBalancer (1)
        nodePort: 30000 (2)
    1 Exposes Infinispan on the network through a load balancer service on port 11222.
    2 Optionally defines a node port to which the load balancer service forwards traffic.
  3. Apply the changes.

  4. Verify that the -external service is available.

    $ kubectl get services | grep external
    
    NAME                         TYPE            CLUSTER-IP    EXTERNAL-IP   PORT(S)
    example-infinispan-external  LoadBalancer    192.0.2.24    hostname.com  11222/TCP

6.3. Exposing Infinispan Through Node Ports

Use a node port service to expose Infinispan clusters on the network.

Procedure
  1. Include spec.expose in your Infinispan CR.

  2. Specify NodePort as the service type with spec.expose.type.

    spec:
      ...
      expose:
        type: NodePort (1)
        nodePort: 30000 (2)
    1 Exposes Infinispan on the network through a node port service.
    2 Defines the port where Infinispan is exposed. If you do not define a port, the platform selects one.
  3. Apply the changes.

  4. Verify that the -external service is available.

    $ kubectl get services | grep external
    
    NAME                         TYPE            CLUSTER-IP       EXTERNAL-IP   PORT(S)
    example-infinispan-external  NodePort        192.0.2.24       <none>        11222:30000/TCP

6.4. Exposing Infinispan Through Routes

Use a Kubernetes Ingress or an OpenShift Route with passthrough encryption to make Infinispan clusters available on the network.

Procedure
  1. Include spec.expose in your Infinispan CR.

  2. Specify Route as the service type with spec.expose.type.

  3. Optionally add a hostname with spec.expose.host.

    spec:
      ...
      expose:
        type: Route (1)
        host: www.example.org (2)
    1 Exposes Infinispan on the network through a Kubernetes Ingress or OpenShift Route.
    2 Optionally specifies the hostname where Infinispan is exposed.
  4. Apply the changes.

  5. Verify that the is available.

    $ kubectl get ingress
    
    NAME                 CLASS    HOSTS   ADDRESS   PORTS   AGE
    example-infinispan   <none>   *                 80      73s

7. Configuring Authentication

Application users must authenticate with Infinispan clusters. Infinispan Operator generates default credentials or you can add your own.

7.1. Default Credentials

Infinispan Operator generates base64-encoded default credentials stored in an authentication secret named example-infinispan-generated-secret

Username Description

developer

Default application user.

operator

Internal user that interacts with Infinispan clusters.

7.2. Retrieving Credentials

Get credentials from authentication secrets to access Infinispan clusters.

Procedure
  • Retrieve credentials from authentication secrets, as in the following example:

    $ kubectl get secret example-infinispan-generated-secret

    Base64-decode credentials.

    $ kubectl get secret example-infinispan-generated-secret \
    -o jsonpath="{.data.identities\.yaml}" | base64 --decode
    
    credentials:
    - username: developer
      password: dIRs5cAAsHIeeRIL
    - username: operator
      password: uMBo9CmEdEduYk24

7.3. Identities File

You can define credentials for Infinispan application users and the operator user in an identities.yaml file. All users that you add to identities.yaml can access Infinispan clusters and data

identities.yaml must include the operator user.

Identities File Example
credentials:
- username: testuser
  password: testpassword
- username: operator
  password: supersecretoperatorpassword

7.4. Adding Custom Credentials

Configure access to Infinispan cluster endpoints with custom credentials.

Procedure
  1. Define custom credentials in an identities.yaml file.

  2. Create an authentication secret from identities.yaml as follows:

    $ kubectl create secret generic --from-file=identities.yaml connect-secret
  3. Specify the authentication secret with spec.security.endpointSecretName in your Infinispan CR and then apply the changes.

    spec:
      ...
      security:
        endpointSecretName: connect-secret (1)
    1 Specifies the name of the authentication secret that contains your credentials.
Verification

Modifying spec.security.endpointSecretName triggers a cluster restart. You can watch the Infinispan cluster as Infinispan Operator applies changes:

$ kubectl get pods -w

When the Infinispan cluster is running, verify your custom credentials as follows:

  1. Open a remote shell to a Infinispan node, for example:

    $ kubectl exec -it example-infinispan-0 -- /bin/bash
  2. Check that credentials exist in the users.properties file.

    $ cat /opt/infinispan/server/conf/users.properties
  3. Exit the remote shell.

    $ exit

8. Securing Infinispan Connections

Encrypt connections between clients and Infinispan nodes with Red Hat OpenShift service certificates or custom TLS certificates.

8.1. Encryption with Red Hat OpenShift Service Certificates

Infinispan Operator automatically generates TLS certificates that are signed by the Red Hat OpenShift service CA. Infinispan Operator then stores the certificates and keys in a secret so you can retrieve them and use with remote clients.

If the Red Hat OpenShift service CA is available, Infinispan Operator adds the following spec.security.endpointEncryption configuration to the Infinispan CR:

spec:
  ...
  security:
    endpointEncryption:
      type: Service
      certServiceName: service.beta.openshift.io (1)
      certSecretName: example-infinispan-cert-secret (2)
1 Specifies the Red Hat OpenShift Service.
2 Names the secret that contains a service certificate, tls.crt, and key, tls.key, in PEM format. If you do not specify a name, Infinispan Operator uses <cluster_name>-cert-secret.

Service certificates use the internal DNS name of the Infinispan cluster as the common name (CN), for example:

Subject: CN = example-infinispan.mynamespace.svc

For this reason, service certificates can be fully trusted only inside OpenShift. If you want to encrypt connections with clients running outside OpenShift, you should use custom TLS certificates.

Service certificates are valid for one year and are automatically replaced before they expire.

8.1.1. Retrieving TLS Certificates

Get TLS certificates from encryption secrets to create client trust stores.

  • Retrieve tls.crt from encryption secrets as follows:

$ kubectl get secret example-infinispan-cert-secret \
-o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.crt

8.2. Using Custom TLS Certificates

Use custom PKCS12 keystore or TLS certificate/key pairs to encrypt connections between clients and Infinispan clusters.

Prerequisites

Create either a keystore or certificate secret. See:

Procedure
  1. Add the encryption secret to your OpenShift namespace, for example:

    $ kubectl apply -f tls_secret.yaml
  2. Specify the encryption secret with spec.security.endpointEncryption in your Infinispan CR and then apply the changes.

    spec:
      ...
      security:
        endpointEncryption: (1)
                type: Secret (2)
                certSecretName: tls-secret (3)
    1 Encrypts traffic to and from Infinispan endpoints.
    2 Configures Infinispan to use secrets that contain encryption certificates.
    3 Names the encryption secret.

8.2.1. Certificate Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
data:
    tls.key:  "LS0tLS1CRUdJTiBQUk ..." (1)
    tls.crt: "LS0tLS1CRUdJTiBDRVl ..." (2)
1 Adds a base64 encoded TLS key.
2 Adds a base64 encoded TLS certificate.

8.2.2. Keystore Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
stringData:
    alias: server (1)
    password: password (2)
data:
    keystore.p12:  "MIIKDgIBAzCCCdQGCSqGSIb3DQEHA..." (3)
1 Specifies an alias for the keystore.
2 Specifies a password for the keystore.
3 Adds a base64 encoded keystore.

9. Configuring Cross-Site Replication

Set up global Infinispan clusters to back up data across sites.

9.1. Cross-Site Replication with Infinispan Operator

If you have Infinispan clusters running in separate locations, use Infinispan Operator to connect them so you can back up data across sites.

For example, in the following illustration, Infinispan Operator manages a Infinispan cluster at a data center in New York City, NYC. At another data center in London, LON, Infinispan Operator also manages a Infinispan cluster.

xsite ispn

Infinispan Operator uses a Kubernetes API to establish a secure connection between the OpenShift Container Platform clusters in NYC and LON. Infinispan Operator then creates a cross-site replication service so Infinispan clusters can back up data across locations.

Each Infinispan cluster has one site master node that coordinates all backup requests. Infinispan Operator identifies the site master node so that all traffic through the cross-site replication service goes to the site master.

If the current site master node goes offline then a new node becomes site master. Infinispan Operator automatically finds the new site master node and updates the cross-site replication service to forward backup requests to it.

9.2. Applying Cluster Roles for Cross-Site Replication

During OLM installation, Infinispan Operator sets up cluster roles required for cross-site replication. If you install Infinispan Operator manually, you must complete this procedure to set up those cluster roles.

Procedure
  • Install clusterrole.yaml and clusterrole_binding.yaml as follows:

$ kubectl apply -f deploy/clusterrole.yaml
$ kubectl apply -f deploy/clusterrole_binding.yaml

9.3. Creating Minikube Site Access Secrets

If you run Infinispan Operator in Minikube, you should create secrets that contain the files that allow different instances of Minikube to authenticate with each other.

Procedure
  1. Create secrets on each site that contain ca.crt, client.crt, and client.key from your Minikube installation.

    For example, do the following on LON:

    kubectl create secret generic site-a-secret \
        --from-file=certificate-authority=/opt/minikube/.minikube/ca.crt \
        --from-file=client-certificate=/opt/minikube/.minikube/client.crt \
        --from-file=client-key=/opt/minikube/.minikube/client.key

9.4. Creating Service Account Tokens

Generate service account tokens on each OpenShift cluster that acts as a backup location. Clusters use these tokens to authenticate with each other so Infinispan Operator can create a cross-site replication service.

Procedure
  1. Log in to an OpenShift cluster.

  2. Create a service account.

    For example, create a service account at LON:

    $ oc create sa lon
    serviceaccount/lon created
  3. Add the view role to the service account with the following command:

    $ oc policy add-role-to-user view system:serviceaccount:<namespace>:lon
  4. Repeat the preceding steps on your other OpenShift clusters.

9.5. Exchanging Service Account Tokens

After you create service account tokens on your OpenShift clusters, you add them to secrets on each backup location. For example, at LON you add the service account token for NYC. At NYC you add the service account token for LON.

Prerequisites
  • Get tokens from each service account.

    Use the following command or get the token from the OpenShift Web Console:

    $ oc sa get-token lon
    
    eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
Procedure
  1. Log in to an OpenShift cluster.

  2. Add the service account token for a backup location with the following command:

    $ oc create secret generic <token-name> --from-literal=token=<token>

    For example, log in to the OpenShift cluster at NYC and create a lon-token secret as follows:

    $ oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
  3. Repeat the preceding steps on your other OpenShift clusters.

9.6. Configuring Infinispan Clusters for Cross-Site Replication

Configure Infinispan clusters as backup locations so that they can communicate over a dedicated JGroups transport channel for replicating data.

Prerequisites
  • Create secrets that contain service account tokens for each backup location.

  • Ensure that Kubernetes project names match.

    To perform cross-site replication, Infinispan Operator requires Infinispan clusters to have the same name and run in matching namespaces.

    For example, you create a cluster at LON in a project named xsite-cluster. The cluster at NYC must also run in a project named xsite-cluster.

Procedure
  1. Create an Infinispan CR for each Infinispan cluster.

  2. Specify a matching name for each Infinispan cluster with metadata.name`.

  3. Specify the name of the local site with spec.service.sites.local.name.

  4. Provide the name, URL, and secret for each Infinispan cluster that acts as a backup location with spec.service.sites.locations.

    The following are example Infinispan CR definitions for LON and NYC:

    • LON

      apiVersion: infinispan.org/v1
      kind: Infinispan
      metadata:
        name: example-infinispan
      spec:
        replicas: 3
        service:
          type: DataGrid
          sites:
            local:
              name: LON
              expose:
                type: LoadBalancer
            locations:
              - name: LON
                url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443
                secretName: lon-token
              - name: NYC
                url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443
                secretName: nyc-token
    • NYC

      apiVersion: infinispan.org/v1
      kind: Infinispan
      metadata:
        name: example-infinispan
      spec:
        replicas: 2
        service:
          type: DataGrid
          sites:
            local:
              name: NYC
              expose:
                type: LoadBalancer
            locations:
              - name: NYC
                url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443
                secretName: nyc-token
              - name: LON
                url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443
                secretName: lon-token
  5. Adjust logging levels for cross-site replication as follows:

    ...
      logging:
        categories:
          org.jgroups.protocols.TCP: error
          org.jgroups.protocols.relay.RELAY2: fatal

    The preceding configuration decreases logging for JGroups TCP and RELAY2 protocols to reduce excessive messages about cluster backup operations, which can result in a large number of log files that use container storage.

  6. Configure nodes with any other Data Grid Service resources.

  7. Apply the Infinispan CRs.

  8. Check node logs to verify that Infinispan clusters form a cross-site view, for example:

    $ kubectl logs example-infinispan-0 | grep x-site
    
    INFO  [org.infinispan.XSITE] (jgroups-5,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC]
    INFO  [org.infinispan.XSITE] (jgroups-7,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC, LON]
Next steps

If your clusters have formed a cross-site view, you can start adding backup locations to caches.

9.6.1. Cross-Site Replication Resources

spec:
  ...
  service:
    type: DataGrid (1)
    sites:
      local:
        name: LON (2)
        expose:
          type: LoadBalancer (3)
      locations: (4)
      - name: LON (5)
        url: openshift://api.site-a.devcluster.openshift.com:6443 (6)
        secretName: lon-token (7)
      - name: NYC
        url: openshift://api.site-b.devcluster.openshift.com:6443
        secretName: nyc-token
  logging:
    categories:
      org.jgroups.protocols.TCP: error (8)
      org.jgroups.protocols.relay.RELAY2: fatal (9)
1 Specifies Data Grid Service. Infinispan supports cross-site replication with Data Grid Service clusters only.
2 Names the local site for a Infinispan cluster.
3 Defines the externally exposed service.
  • Use NodePort for local clusters on the same network.

  • Use LoadBalancer for independent OpenShift clusters.

4 Provides connection information for all backup locations.
5 Specifies a backup location that matches .spec.service.sites.local.name.
6 Specifies a backup location.
  • Use minikube:// if the backup location is a Minikube instance.

  • Use openshift:// if the backup location is an OpenShift cluster. You should specify the URL of the Kubernetes API.

7 Specifies the access secret for a site.

This secret contains different authentication objects, depending on your Kubernetes environment.

8 Logs error messages for the JGroups TCP protocol.
9 Logs fatal messages for the JGroups RELAY2 protocol.

10. Creating Infinispan Caches

Add cache definitions that configure how Infinispan stores your data. You can create caches from configuration templates and valid cache definitions in XML format through Infinispan Console, Infinispan CLI, Hot Rod clients, or Infinispan Operator.

Prerequisites

Before you start creating Infinispan caches, ensure that you have:

  • Infinispan credentials. By default Infinispan requires authentication.

  • Network access to your Infinispan cluster.

    Examples in this section use $SERVICE_HOSTNAME to denote the service or route that provides access to your Infinispan cluster.

10.1. Creating Caches with Infinispan Console

Access the console to create caches from templates or valid configuration in XML or JSON format.

Procedure
  1. Access the console from any browser at $SERVICE_HOSTNAME:11222

  2. Log in with your Infinispan credentials.

  3. Select Go to the console.

  4. Open the Data Container view.

  5. Select Create Cache and then add a cache.

  6. Return to the Data Container view and verify your Infinispan cache.

10.2. Creating Caches with Infinispan CLI

Use the command line interface (CLI) to create caches on your Infinispan cluster.

Prerequisites
  • Download Infinispan Server.

    The CLI is available as part of the server distribution, which you can run on your local host to establish remote connections to Infinispan clusters on OpenShift.

    You can also open a remote shell to a Infinispan node and access the CLI from there. However, doing this means you must create configuration files on each node when adding caches.

    $ kubectl exec -it example-infinispan-0 -- /bin/bash
Procedure
  1. Create a CLI connection to your Infinispan cluster.

    $ bin/cli.sh -c https://$SERVICE_HOSTNAME:11222/ --trustall
  2. Enter your Infinispan credentials when prompted.

  3. Add cache definitions with the create cache command.

    • Add a cache definition from an XML or JSON file with the --file option.

      [//containers/default]> create cache --file=configuration.xml mycache
    • Add a cache definition from a template with the --template option.

      [//containers/default]> create cache --template=org.infinispan.DIST_SYNC mycache

      Press the tab key after the --template= argument to list available cache templates.

  4. Verify the cache exists with the ls command.

    [//containers/default]> ls caches
    mycache
  5. Retrieve the cache configuration with the describe command.

    [//containers/default]> describe caches/mycache

10.3. Creating Caches with Hot Rod Java Clients

Configure Hot Rod Java clients to connect to Infinispan clusters running on OpenShift.

Procedure
  • Configure Hot Rod Java clients with a hotrod-client.properties file in the application classpath or with the ConfigurationBuilder class.

    Hot Rod clients running outside of OpenShift must use BASIC client intelligence.

Hot Rod client properties
# Connection
infinispan.client.hotrod.server_list=$SERVICE_HOSTNAME:11222

# Client intelligence
infinispan.client.hotrod.client_intelligence=BASIC

# Authentication
infinispan.client.hotrod.use_auth=true
infinispan.client.hotrod.auth_username=developer
infinispan.client.hotrod.auth_password=$PASSWORD
infinispan.client.hotrod.auth_server_name=$CLUSTER_NAME
infinispan.client.hotrod.sasl_properties.javax.security.sasl.qop=auth
infinispan.client.hotrod.sasl_mechanism=DIGEST-MD5

# Encryption
infinispan.client.hotrod.sni_host_name=$SERVICE_HOSTNAME
# Path to the TLS certificate.
# Clients automatically generate trust stores from certificates.
infinispan.client.hotrod.trust_store_path=tls.crt

# Add configuration for specific caches
infinispan.client.hotrod.cache.my-cache.template_name=org.infinispan.DIST_SYNC
infinispan.client.hotrod.cache.another-cache.configuration=<infinispan><cache-container><distributed-cache name=\"another-cache\"/></cache-container></infinispan>
infinispan.client.hotrod.cache.my-other-cache.configuration_uri=file:/path/to/configuration.xml
ConfigurationBuilder
import org.infinispan.client.hotrod.DefaultTemplate;
import org.infinispan.client.hotrod.RemoteCache;
import org.infinispan.client.hotrod.RemoteCacheManager;
import org.infinispan.client.hotrod.configuration.ClientIntelligence;
import org.infinispan.client.hotrod.configuration.ConfigurationBuilder;
import org.infinispan.client.hotrod.configuration.SaslQop;
import org.infinispan.client.hotrod.impl.ConfigurationProperties;
...

ConfigurationBuilder builder = new ConfigurationBuilder();
      builder.addServer()
               .host("$SERVICE_HOSTNAME")
               .port(ConfigurationProperties.DEFAULT_HOTROD_PORT)
             .security().authentication()
               .username("username")
               .password("password")
               .realm("default")
               .saslQop(SaslQop.AUTH)
               .saslMechanism("DIGEST-MD5")
             .ssl()
               .sniHostName("$SERVICE_HOSTNAME")
               .trustStorePath("/path/to/tls.crt");
      builder.clientIntelligence(ClientIntelligence.BASIC);
      builder.remoteCache("my-cache")
               .templateName(DefaultTemplate.DIST_SYNC);
      builder.remoteCache("another-cache")
               .configuration("<infinispan><cache-container><distributed-cache name=\"another-cache\"><encoding media-type=\"application/x-protostream\"/></distributed-cache></cache-container></infinispan>");
      try (RemoteCacheManager cacheManager = new RemoteCacheManager(builder.build())) {
         // Get a remote cache that does not exist.
         // Rather than return null, create the cache from a template.
         RemoteCache<String, String> cache = cacheManager.getCache("my-cache");
         /// Store a value.
         cache.put("hello", "world");
         // Retrieve the value and print it.
         System.out.printf("key = %s\n", cache.get("hello"));

10.4. Adding Caches to Cache Service Nodes

Cache Service nodes include a default cache configuration with recommended settings. This default cache lets you start using Infinispan without the need to create caches.

Because the default cache provides recommended settings, you should create caches only as copies of the default. If you want multiple custom caches you should create Data Grid Service nodes instead of Cache Service nodes.

Procedure
  • Access the Infinispan Console and provide a copy of the default configuration in XML or JSON format.

  • Use the Infinispan CLI to create a copy from the default cache as follows:

    [//containers/default]> create cache --template=default mycache

10.4.1. Default Cache Configuration

The default cache for Cache Service nodes is as follows:

<infinispan>
  <cache-container>
    <distributed-cache name="default" (1)
                       mode="SYNC" (2)
                       owners="2"> (3)
      <memory storage="OFF_HEAP" (4)
              max-size="<maximum_size_in_bytes>" (5)
              when-full="REMOVE" /> (6)
      <partition-handling when-split="ALLOW_READ_WRITES" (7)
                          merge-policy="REMOVE_ALL"/> (8)
    </distributed-cache>
  </cache-container>
</infinispan>
1 Names the cache instance as "default".
2 Uses synchronous distribution for storing data across the cluster.
3 Configures two replicas of each cache entry on the cluster.
4 Stores cache entries as bytes in native memory (off-heap).
5 Defines the maximum size for the data container in bytes. Infinispan Operator calculates the maximum size when it creates nodes.
6 Evicts cache entries to control the size of the data container. You can enable automatic scaling so that Infinispan Operator adds nodes when memory usage increases instead of removing entries.
7 Names a conflict resolution strategy that allows read and write operations for cache entries, even if segment owners are in different partitions.
8 Specifies a merge policy that removes entries from the cache when Infinispan detects conflicts.

10.5. Creating Caches with Infinispan Operator

To create caches with Infinispan Operator, you use Cache CRs to add caches from templates or XML configuration.

Creating caches with Infinispan Operator is available as a technology preview.

Technology Preview features or capabilities are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

When using Cache CRs, the following rules apply:

  • Cache CRs apply to Data Grid Service nodes only.

  • You can create a single cache for each Cache CR.

  • If your Cache CR contains both a template and an XML configuration, Infinispan Operator uses the template.

  • If you edit caches in the OpenShift Web Console, the changes are reflected through the user interface but do not take effect on the Infinispan cluster. You cannot edit caches. To change cache configuration, you must first delete the cache through the console or CLI and then re-create the cache.

  • Deleting Cache CRs in the OpenShift Web Console, does not remove caches from Infinispan clusters. You must delete caches through the console or CLI.

10.5.1. Adding Credentials to Create Caches

Infinispan Operator must authenticate with Data Grid Service clusters to create caches. You add credentials to a secret so that Infinispan Operator can access your cluster when creating caches.

The following procedure explains how to add credentials to a new secret. If you have a custom secret that contains credentials, you can use that instead of creating a new one.

Procedure
  1. Define a Secret object type that provides valid user credentials for accessing Data Grid Service clusters in a StringData map.

    For example, create an basic-auth.yaml file that provides credentials for the developer user as follows:

    apiVersion: v1
    stringData:
      username: developer (1)
      password: G8ZdJvSaY3lOOwfM (2)
    kind: Secret
    metadata:
      name: basic-auth (3)
    type: Opaque
    1 Names a user that can create caches.
    2 Specifies the password that corresponds to the user.
    3 Specifies a name for the secret.
  2. Create a secret from the file, as in the following example:

    $ kubectl apply -f basic-auth.yaml
Using Custom Credentials Secrets

Infinispan Operator requires that credentials exist as values for the username and password keys in a secret. If you have a custom secret that contains Infinispan credentials, but uses different key names, you can override those names in your Cache CR.

For example, you have a secret named "my-credentials" that holds a list of Infinispan users and their passwords as follows:

stringData:
  app_user1: spock
  app_user1_pw: G8ZdJvSaY3lOOwfM
  app_user2: jim
  app_user2_pw: zTzz2gVyyF4JsYsH
Procedure
  • In your Cache CR, override custom key names with username and password as follows:

spec:
  adminAuth:
    username:
      key: app_user1 (1)
      name: my-credentials (2)
    password:
      key: app_user1_pw (3)
      name: my-credentials
1 Overrides the app_user1 key name with username.
2 Specifies the name of your custom credentials secret.
3 Overrides the app_user1_pw key name with password.

10.5.2. Creating Infinispan Caches from Templates

Complete the following steps to create caches on Data Grid Service clusters using cache configuration templates.

Prerequisites
  • Create a secret that contains valid user credentials for accessing Infinispan clusters.

  • Identify the cache configuration template you want to use for your cache. You can find a list of available configuration templates in Infinispan Console.

Procedure
  1. Create a Cache CR that specifies the name of the template you want to use.

    For example, the following CR creates a cache named "mycache" that uses the org.infinispan.DIST_SYNC cache configuration template:

    apiVersion: infinispan.org/v2alpha1
    kind: Cache
    metadata:
      name: mycachedefinition (1)
    spec:
      adminAuth: (2)
        secretName: basic-auth
      clusterName: example-infinispan (3)
      name: mycache (4)
      templateName: org.infinispan.DIST_SYNC (5)
    1 Names the Cache CR.
    2 Specifies a secret that provides credentials with username and password keys or an override for custom credentials secrets.
    3 Specifies the name of the target Infinispan cluster where you want Infinispan Operator to create the cache.
    4 Names the Infinispan cache instance.
    5 Specifies the infinispan.org cache configuration template to create the cache.
  2. Apply the Cache CR, for example:

    $ kubectl apply -f mycache.yaml
    cache.infinispan.org/mycachedefinition created

10.5.3. Creating Infinispan Caches from XML

Complete the following steps to create caches on Data Grid Service clusters using valid infinispan.xml cache definitions.

Prerequisites
  • Create a secret that contains valid user credentials for accessing Infinispan clusters.

Procedure
  1. Create a Cache CR that contains the XML cache definition you want to create.

    apiVersion: infinispan.org/v2alpha1
    kind: Cache
    metadata:
      name: mycachedefinition (1)
    spec:
      adminAuth: (2)
        secretName: basic-auth
      clusterName: example-infinispan (3)
      name: mycache (4)
      template: <infinispan><cache-container><distributed-cache name="mycache" mode="SYNC"><persistence><file-store/></persistence></distributed-cache></cache-container></infinispan> (5)
    1 Names the Cache CR.
    2 Specifies a secret that provides credentials with username and password keys or an override for custom credentials secrets.
    3 Specifies the name of the target Infinispan cluster where you want Infinispan Operator to create the cache.
    4 Names the cache on the Infinispan cluster.
    5 Specifies the XML cache definition to create the cache. Note that the name attribute is ignored. Only spec.name applies to the resulting cache.
  2. Apply the Cache CR, for example:

    $ kubectl apply -f mycache.yaml
    cache.infinispan.org/mycachedefinition created

10.6. Adding Backup Locations to Caches

When you configure Infinispan clusters to perform cross-site replication, you can add backup locations to your cache configurations.

Procedure
  1. Create cache configurations with identical names for each site.

    Cache configurations at each site can use different cache modes and backup strategies. Infinispan replicates data based on cache names.

  2. Configure backup locations to go offline automatically with the take-offline element.

    1. Set the amount of time, in milliseconds, before backup locations go offline with the min-wait attribute.

  3. Define any other valid cache configuration.

  4. Add backup locations to the named cache on all sites in the global cluster.

    For example, if you add LON as a backup for NYC you should add NYC as a backup for LON.

The following configuration examples show backup locations for caches:

  • NYC

    <infinispan>
      <cache-container>
        <distributed-cache name="customers">
          <encoding media-type="application/x-protostream"/>
          <backups>
            <backup site="LON" strategy="SYNC">
              <take-offline min-wait="120000"/>
            </backup>
          </backups>
        </distributed-cache>
      </cache-container>
    </infinispan>
  • LON

    <infinispan>
      <cache-container>
        <replicated-cache name="customers">
          <encoding media-type="application/x-protostream"/>
          <backups>
            <backup site="NYC" strategy="ASYNC" >
                <take-offline min-wait="120000"/>
              </backup>
          </backups>
        </replicated-cache>
      </cache-container>
    </infinispan>

10.6.1. Performance Considerations with Taking Backup Locations Offline

Backup locations can automatically go offline when remote sites become unavailable. This prevents nodes from attempting to replicate data to offline backup locations, which can have a performance impact on your cluster because it results in error.

You can configure how long to wait before backup locations go offline. A good rule of thumb is one or two minutes. However, you should test different wait periods and evaluate their performance impacts to determine the correct value for your deployment.

For instance when OpenShift terminates the site master pod, that backup location becomes unavailable for a short period of time until Infinispan Operator elects a new site master. In this case, if the minimum wait time is not long enough then the backup locations go offline. You then need to bring those backup locations online and perform state transfer operations to ensure the data is in sync.

Likewise, if the minimum wait time is too long, node CPU usage increases from failed backup attempts which can lead to performance degradation.

10.7. Adding Persistent Cache Stores

You can add Single File cache stores to Data Grid Service nodes to save data to the persistent volume.

You configure cache stores as part of your Infinispan cache definition with the persistence element as follows:

<persistence>
   <file-store/>
</persistence>

Infinispan then creates a Single File cache store, .dat file, in the /opt/infinispan/server/data directory.

Procedure
  • Add a cache store to your cache configurations as follows:

    <infinispan>
      <cache-container>
        <distributed-cache name="customers" mode="SYNC">
          <encoding media-type="application/x-protostream"/>
          <persistence>
            <file-store/>
          </persistence>
        </distributed-cache>
      </cache-container>
    </infinispan>
Reference

11. Monitoring Infinispan with Prometheus

Infinispan exposes a metrics endpoint that provides statistics and events to Prometheus.

11.1. Setting Up Prometheus

Set up Prometheus so it can authenticate with and monitor Infinispan clusters.

Prerequisites
  • Install the Prometheus Operator.

  • Create a running Prometheus instance.

Procedure
  1. Add an authentication secret to your Prometheus namespace.

    This secret allows Prometheus to authenticate with your Infinispan cluster. You can find Infinispan credentials in the authentication secret in your Infinispan Operator namespace.

    apiVersion: v1
    stringData:
      username: developer (1)
      password: dIRs5cAAsHIeeRIL (2)
    kind: Secret
    metadata:
      name: basic-auth
    type: Opaque
    1 Specifies an application user. developer is the default application user.
    2 Specifies a corresponding password.
  2. Create a service monitor instance that configures Prometheus to monitor your Infinispan cluster.

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      labels:
        k8s-app: prometheus
      name: datagrid-monitoring (1)
      namespace: infinispan-monitoring (2)
    spec:
      endpoints:
        - basicAuth:
            username:
              key: username
              name: basic-auth (3)
            password:
              key: password
              name: basic-auth (4)
          interval: 30s
          scheme: https (5)
          tlsConfig:
            insecureSkipVerify: true
            serverName: certificate-CN (6)
      namespaceSelector:
        matchNames:
          - infinispan (7)
      selector:
        matchLabels:
          app: infinispan-service
          clusterName: cluster-name (8)
    1 Names the service monitor.
    2 Specifies your Prometheus namespace.
    3 Specifies the name of the authentication secret that has Infinispan credentials.
    4 Specifies the name of the authentication secret that has Infinispan credentials.
    5 Specifies that Infinispan endpoints use encryption. If you do not use TLS, remove spec.endpoints.scheme.
    6 Specifies the Common Name (CN) of the TLS certificate for Infinispan encryption. If you use an OpenShift service certificate, the CN matches the metadata.name resource for your Infinispan cluster. If you do not use TLS, remove spec.endpoints.tlsConfig.
    7 specifies the Infinispan Operator namespace.
    8 specifies the name of the Infinispan cluster.

12. Monitoring Infinispan Logs

Set logging categories to different message levels to monitor, debug, and troubleshoot Infinispan clusters.

12.1. Configuring Infinispan Logging

Procedure
  1. Specify logging configuration with spec.logging in your Infinispan CR and then apply the changes.

    spec:
      ...
      logging: (1)
        categories: (2)
          org.infinispan: debug (3)
          org.jgroups: debug
    1 Configures Infinispan logging.
    2 Adds logging categories.
    3 Names logging categories and levels.

    The root logging category is org.infinispan and is INFO by default.

  2. Retrieve logs from Infinispan nodes as required.

    $ kubectl logs -f $POD_NAME

12.2. Log Levels

Log levels indicate the nature and severity of messages.

Log level Description

trace

Provides detailed information about running state of applications. This is the most verbose log level.

debug

Indicates the progress of individual requests or activities.

info

Indicates overall progress of applications, including lifecycle events.

warn

Indicates circumstances that can lead to error or degrade performance.

error

Indicates error conditions that might prevent operations or activities from being successful but do not prevent applications from running.

13. Reference

Find information about Infinispan services and clusters that you create with Infinispan Operator.

13.1. Image Resource

spec:
  image: infinispan/server:latest (1)
1 Lets you specify a Infinispan image to use.

13.2. Network Services

Internal service
  • Allow Infinispan nodes to discover each other and form clusters.

  • Provide access to Infinispan endpoints from clients in the same Kubernetes namespace.

Service Port Protocol Description

<cluster_name>

11222

TCP

Internal access to Infinispan endpoints

<cluster_name>-ping

8888

TCP

Cluster discovery

External service

Provides access to Infinispan endpoints from clients outside Kubernetes or in different namespaces.

You must create the external service with Infinispan Operator. It is not available by default.

Service Port Protocol Description

<cluster_name>-external

11222

TCP

External access to Infinispan endpoints.

Cross-site service

Allows Infinispan to back up data between clusters in different locations.

Service Port Protocol Description

<cluster_name>-site

7900

TCP

JGroups RELAY2 channel for cross-site communication.