The Infinispan Operator provides operational intelligence and reduces management complexity for deploying Infinispan on Kubernetes clusters.

1. Manually Installing Infinispan Operator

You can manually install Infinispan Operator instead of using the OperatorHub.io.

Prerequisites
  • OKD 3.11 or later.

  • Kubernetes 1.11 or later.

  • Administrator access to the Kubernetes cluster.

1.1. Creating Infinispan Operator Resources

Add the custom resource definition and role-based access control (RBAC) resources for Infinispan Operator.

Procedure
  1. Apply the custom resource definition.

    $ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/crd.yaml
  2. Install RBAC resources.

    $ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/rbac.yaml

1.2. Deploying Infinispan Operator

Manually deploying Infinispan Operator to Kubernetes involves applying the yaml template that defines metadata and container specifications.

Procedure
  • Apply the Infinispan Operator template as follows:

$ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/operator.yaml

2. Getting Started with Infinispan Operator

You create, configure, and manage Infinispan clusters with Infinispan Operator by defining custom resource objects that specify Infinispan pod configuration.

Prerequisites
  • Either install Infinispan Operator from OperatorHub.io or install it manually.

  • Have an oc or a kubectl client in your $PATH.

    This documentation provides examples with the oc client. If you’re using a kubectl client, adapt the command examples where necessary.

2.1. Using Default Infinispan Operator Resources

Infinispan provides default resource definitions.

$ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/cr/minimal/cr_minimal.yaml

2.2. Infinispan Custom Resources

Infinispan Operator adds a resource type and custom resource definition (CRD) with recommended defaults for Infinispan pods.

You configure Infinispan pods dynamically by changing defaults and adding custom resources.

The minimum resources to define a Infinispan cluster are as follows:

apiVersion: infinispan.org/v1 (1)
kind: Infinispan (2)
metadata:
  name: (3)
spec:
  replicas: (4)
1 declares the version.
2 sets the resource type that ispn_operator} uses to handle the lifecycle of Infinispan clusters.
3 names the Infinispan cluster.
4 sets the number of pods in the Infinispan cluster.

2.3. Creating Infinispan Clusters

Use Infinispan Operator to create clusters of two or more Infinispan pods.

Procedure
  1. Create a custom resource that sets replicas: 3.

    For example, create a cr_minimal.yaml file as follows:

    $ cat > cr_minimal.yaml<<EOF
    apiVersion: infinispan.org/v1
    kind: Infinispan
    metadata:
      name: example-infinispan
    spec:
      replicas: 3
    EOF
  2. Apply your custom resource definition.

    $ oc apply -f cr_minimal.yaml
  3. Watch Infinispan Operator create the Infinispan pods.

    $ oc get pods -w
    
    NAME                        READY  STATUS              RESTARTS   AGE
    example-infinispan-1        0/1    ContainerCreating   0          4s
    example-infinispan-2        0/1    ContainerCreating   0          4s
    example-infinispan-3        0/1    ContainerCreating   0          5s
    infinispan-operator-0       1/1    Running             0          3m
    example-infinispan-3        1/1    Running             0          8s
    example-infinispan-2        1/1    Running             0          8s
    example-infinispan-1        1/1    Running             0          8s
Next Steps

Try changing the value of replicas: and watching Infinispan Operator scale the cluster up or down.

2.4. Verifying Infinispan Clusters

Review log messages to ensure that Infinispan pods receive clustered views.

Procedure
  • Do either of the following:

    • Retrieve the cluster view from pod logs.

      $ oc logs example-infinispan-0 | grep ISPN000094
      
      INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|0] (1) [example-infinispan-0]
      
      INFO  [org.infinispan.CLUSTER] (jgroups-3,{example_crd_name-0) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|1] (2) [example-infinispan-0, example-infinispan-1]
    • Retrieve the custom resource type for Infinispan Operator.

      $ oc get infinispan -o yaml

      The response indicates that Infinispan pods have received clustered views:

      conditions:
          - message: 'View: [example-infinispan-0, example-infinispan-1]'
            status: "True"
            type: wellFormed

Use oc wait with the wellFormed condition for automated scripts.

$ oc wait --for condition=wellFormed --timeout=240s infinispan/example-infinispan

3. Creating Infinispan Services

Infinispan services are stateful applications that provide flexible and robust in-memory data storage.

3.1. Cache Service

Cache service provides a volatile, low-latency data store that dramatically increases application response rates.

Cache service pods:

  • Synchronously distribute data across the cluster to ensure consistency.

  • Maintain single copies of cache entries to reduce size. You can replicate data for more resiliency.

  • Store cache entries off-heap and use eviction for JVM efficiency.

You can create multiple cache definitions with Cache service but only as copies of the default configuration.

Default cache configuration for Cache service pods is as follows:

<distributed-cache name="default" mode="SYNC" owners="1">
   <memory>
    <off-heap eviction="MEMORY" strategy="REMOVE"/>
   </memory>
  <partition-handling when-split="DENY_READ_WRITES" merge-policy="REMOVE_ALL" />
</distributed-cache>

3.2. Data Grid Service

Data Grid service provides a configurable Infinispan server distribution for Kubernetes.

  • Use with advanced capabilities like cross-site replication as well as indexing and querying.

  • Remotely access Data Grid service clusters from Hot Rod or REST clients and dynamically create caches using any Infinispan cache mode and configuration.

    Infinispan does not provide default caches for Data Grid service pods. However, you can use cache configuration templates to get started.

3.3. Creating Infinispan Services

Define the .spec.service.type resource to create Cache service and Data Grid service pods with Infinispan Operator.

By default, Infinispan Operator creates Infinispan clusters with Cache service pods.

Procedure
  • Set the service type for Infinispan pods in your custom resource definition and then apply the changes.

For example, create Data Grid service clusters as follows:

spec:
  ...
  service:
    type: DataGrid

You cannot change the .spec.service.type resource after you create Infinispan clusters.

For example, if you create a cluster of Cache service pods, you cannot change the service type to Data Grid service. In this case you must create a new cluster with Data Grid service pods in a different Kubernetes namespace.

3.3.1. Cache Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan (1)
spec:
  image: infinispan/server:latest
  replicas: 4 (2)
  service:
    type: Cache (3)
  security:
    endpointSecretName: endpoint-identities (4)
    endpointEncryption: (5)
        type: secret
        certSecretName: tls-secret
  container: (6)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m"
    memory: 1Gi
  logging: (7)
    categories:
      org.infinispan: trace
      org.jgroups: trace
  expose: (8)
    type: LoadBalancer
1 names the Infinispan cluster.
2 specifies the number of pods in the cluster.
3 creates Cache service pods.
4 adds an authentication secret with user credentials.
5 adds a custom encryption secret for secure connections.
6 allocates resources to pods.
7 configures logging.
8 configures services for external traffic.

3.3.2. Data Grid Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan (1)
spec:
  image: infinispan/server:latest
  replicas: 6 (2)
  service:
    type: DataGrid (3)
    container:
      storage: 2Gi (4)
    sites: (5)
      local:
        expose:
          type: LoadBalancer
      locations:
      - name: azure
        url: openshift://api.azure.host:6443
        secretName: azure-identities
      - name: aws
        url: openshift://api.aws.host:6443
        secretName: aws-identities
  security:
    endpointSecretName: endpoint-identities (6)
    endpointEncryption: (7)
        type: secret
        certSecretName: tls-secret
  container: (8)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "1000m"
    memory: 1Gi
  logging: (9)
    categories:
      org.infinispan: debug
      org.jgroups: debug
  expose: (10)
    type: LoadBalancer
1 names the Infinispan cluster.
2 specifies the number of pods in the cluster.
3 creates Data Grid service pods.
4 configures size of the persistent volume for pod storage.
5 provides connection information for backup locations.
6 adds an authentication secret with user credentials.
7 adds a custom encryption secret for secure connections.
8 allocates resources to pods.
9 configures logging.
10 configures services for external traffic.

4. Stopping and Starting Infinispan Services

Gracefully shut down Infinispan clusters to avoid data loss.

Cache configuration

Both Cache service and Data Grid service pods store permanent cache definitions in persistent volumes so they are still available after cluster restarts.

Data

Data Grid service pods write all cache entries to persistent storage during cluster shutdown. However, you must configure caches with persistence.

You should configure the storage size for Data Grid service pods to ensure that the persistent volume can hold all your data.

If the available container storage is less than the amount of memory available to Data Grid service pods, Infinispan writes an exception to logs and data loss occurs during shutdown.

4.1. Gracefully Shutting Down Infinispan Clusters

  • Set the value of replicas to 0 and apply the changes.

spec:
  replicas: 0

4.2. Restarting Infinispan Clusters

  • Set the value of replicas to the same number of pods that were in the cluster before you shut it down.

For example, you shut down a cluster of 6 pods. When you restart the cluster, you must set:

spec:
  replicas: 6

This allows Infinispan to restore the distribution of data across the cluster. When all pods in the cluster are running, you can then add or remove pods.

5. Configuring Pod Resources

You can allocate CPU and memory resources, specify JVM options, and configure storage for Infinispan pods.

5.1. JVM, CPU, and Memory Resources

spec:
  ...
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary" (1)
    cpu: "1000m" (2)
    memory: 1Gi (3)
1 specifies JVM options.
2 allocates host CPU resources to pods, measured in CPU units.
3 allocates host memory resources to pods, measured in bytes.

When Infinispan Operator creates Infinispan pods, it uses spec.container.cpu and spec.container.memory to:

  • Ensure that Kubernetes nodes have sufficient capacity to run the pods. The Infinispan Operator uses memory, but only half the value of cpu, as resource requests to the Kubernetes scheduler.

  • Constrain pod resource usage. Infinispan Operator sets the values of cpu and memory as resource limits.

You cannot modify JVM options or CPU and memory resources after you create Infinispan pods. In this case you must delete the existing Infinispan pods and create new ones.

Garbage collection logging

By default, Infinispan Operator does not log garbage collection (GC) messages. You can optionally add the following JVM options to direct GC messages to stdout:

extraJvmOpts: "-Xlog:gc*:stdout:time,level,tags"

5.2. Storage Resources

spec:
  ...
  service:
    type: DataGrid
    container:
      storage: 2Gi (1)
1 configures the storage size for Data Grid service pods.

By default, Infinispan Operator allocates 1Gi for storage for both Cache service and Data Grid service pods. You can configure storage size only for Data Grid service pods.

Persistence

Data Grid service pods let you configure caches with persistent storage, as in the following example:

<persistence>
    <file-store />
</persistence>

5.2.1. Persistent Volume Claims

Infinispan Operator creates persistent volume claims for pods at:
/opt/infinispan/server/data

Persistent volume claims use the ReadWriteOnce (RWO) access mode.

6. Setting Up Network Services

Infinispan Operator creates services that allow you to connect to Infinispan clusters.

6.1. Internal Service

By default, Infinispan Operator creates a service that provides access to Infinispan clusters within Kubernetes.

The internal service has the same name that you give to your Infinispan cluster.

For example:

metadata:
  name: example-infinispan

Results in an internal service named example-infinispan.

You can verify the internal service is available as follows:

$ oc get services

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)
example-infinispan ClusterIP   192.0.2.0        <none>        11222/TCP

Infinispan Operator also creates an internal -ping service. See Network Services.

6.2. Creating External Services

To connect to Infinispan clusters from clients running outside Kubernetes, create external services with Infinispan Operator.

Procedure
  1. Modify your custom resource definition to configure an external service and then apply the changes.

    spec:
      ...
      expose: (1)
        type: LoadBalancer (2)
    1 exposes an external service at port 11222.
    2 specifies one of the following service resource types:
    • NodePort for local Kubernetes clusters.

    • LoadBalancer for Kubernetes clusters that use a load balancer service to handle external network traffic.

  2. Verify that the -external service is available.

    $ oc get services | grep external
    
    NAME                         TYPE            CLUSTER-IP       EXTERNAL-IP   PORT(S)
    example-infinispan-external  LoadBalancer    192.0.2.24       <none>        11222/TCP

6.3. Retrieving Cluster IP Addresses for Services

Get the cluster IP address for a service that provides access to Infinispan.

For example, get the cluster IP for the internal service as follows:

$ oc get service example-infinispan -o jsonpath={.spec.clusterIP}

6.4. Creating Routes to Services

In some cases, such as running local Kubernetes clusters, you need to create routes to access Infinispan clusters from clients running outside Kubernetes.

You create routes to services. For convenience, you should create a route to the default service instead of creating an external service and then creating a route to that.

Procedure
  • If you are not using TLS encryption, expose the default service as in the following example:

    $ oc expose svc/example-infinispan
  • If you are using TLS encryption, create a route with passthrough encryption as in the following example:

    $ oc create route passthrough --service=example-infinispan

6.5. Retrieving Hostnames for Routes

Get the hostname for a route that exposes a service for accessing Infinispan.

$ oc get route example-infinispan -o jsonpath={.spec.host}

7. Configuring Authentication

Application users must authenticate with Infinispan clusters. Infinispan Operator generates default credentials or you can add your own.

7.1. Default Credentials

Infinispan Operator generates base64-encoded default credentials stored in an authentication secret named example-infinispan-generated-secret

Username Description

developer

Default application user.

operator

Internal user that interacts with Infinispan clusters.

7.2. Retrieving Credentials

Get credentials from authentication secrets to access Infinispan clusters.

Procedure
  • Retrieve credentials from authentication secrets, as in the following example:

    $ oc get secret example-infinispan-generated-secret -n my_namespace

    Base64-decode credentials.

    $ oc get secret example-infinispan-generated-secret \
    -o jsonpath="{.data.identities\.yaml}" | base64 --decode
    
    credentials:
    - username: developer
      password: dIRs5cAAsHIeeRIL
    - username: operator
      password: uMBo9CmEdEduYk24

7.3. Using Custom Authentication Secrets

Add custom credentials to an authentication secret.

Prerequisites

Create a custom resource definition for an authentication secret. See Authentication Secrets.

Procedure
  1. Add the authentication secret to your OpenShift namespace, for example:

    $ oc apply -f connect_secret.yaml
  2. Modify your custom resource definition to include the authentication secret and then apply the changes.

    spec:
      ...
      security:
        endpointSecretName: connect-secret (1)
    1 specifies the authentication secret.

7.3.1. Authentication Secrets

apiVersion: v1
kind: Secret
metadata:
  name: connect-secret (1)
type: Opaque (2)
stringData:
  identities.yaml: |- (3)
    credentials (4)
    - username: (5)
      password: (6)
1 names the authentication secret.
2 authentication secrets must be Opaque.
3 contains application users in YAML format.
4 lists credentials for application users.

You must include the operator user and specify a password. Infinispan Operator requires these credentials to interact with Infinispan clusters.

8. Securing Infinispan Connections

Encrypt connections between clients and Infinispan pods with Red Hat OpenShift service certificates or custom TLS certificates.

8.1. Using Red Hat OpenShift Service Certificates

Infinispan Operator automatically generates TLS certificates signed by the Red Hat OpenShift service CA.

Service certificates use the internal DNS name of the Infinispan cluster as the common name (CN), for example:

Subject: CN = example-infinispan.mynamespace.svc

For this reason, service certificates can be fully trusted only inside OpenShift. If you want to encrypt connections with clients running outside OpenShift, you should use custom TLS certificates.

If the Red Hat OpenShift service CA is available, Infinispan Operator automatically generates a certificate, tls.crt, and key, tls.key, in PEM format.

Certificates are valid for one year and are automatically replaced before they expire.

Infinispan Operator stores service certificates and keys in a secret named -cert-secret that is prefixed with the Infinispan cluster name.

For example:

metadata:
  name: example-infinispan

Results in example-infinispan-cert-secret.

8.2. Retrieving TLS Certificates

Get TLS certificates from encryption secrets to create client trust stores.

  • Retrieve tls.crt from encryption secrets as follows:

$ oc get secret example-infinispan-cert-secret \
-o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt

8.3. Using Custom TLS Certificates

Use custom PKCS12 keystore or TLS certificate/key pairs to encrypt connections between clients and Infinispan clusters.

Prerequisites

Create a custom resource definition for either a keystore or certificate secret. See:

Procedure
  1. Add the encryption secret to your OpenShift namespace, for example:

    $ oc apply -f tls_secret.yaml
  2. Modify your custom resource definition to include the encryption secret and then apply the changes.

    spec:
      ...
      security:
        endpointEncryption: (1)
                type: secret (2)
                certSecretName: tls-secret (3)
    1 encrypts traffic to and from Infinispan endpoints.
    2 configures Infinispan to use secrets that contain encryption certificates.
    3 names the encryption secret.

8.3.1. Certificate Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
data:
    tls.key:  "LS0tLS1CRUdJTiBQUk ..." (1)
    tls.crt: "LS0tLS1CRUdJTiBDRVl ..." (2)
1 adds a base64 encoded TLS key.
2 adds a base64 encoded TLS certificate.

8.3.2. Keystore Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
stringData:
    alias: server (1)
    password: password (2)
data:
    keystore.p12:  "MIIKDgIBAzCCCdQGCSqGSIb3DQEHA..." (3)
1 specifies an alias for the keystore.
2 specifies a password for the keystore.
3 adds a base64 encoded keystore.

9. Monitoring Infinispan with Prometheus

Infinispan exposes a metrics endpoint that provides statistics and events to Prometheus.

9.1. Setting Up Prometheus

Set up Prometheus so it can authenticate with and monitor Infinispan clusters.

Prerequisites
  • Install the Prometheus Operator.

  • Create a running Prometheus instance.

Procedure
  1. Add an authentication secret to your Prometheus namespace.

    This secret allows Prometheus to authenticate with your Infinispan cluster. You can find Infinispan credentials in the authentication secret in your Infinispan Operator namespace.

    apiVersion: v1
    stringData:
      username: developer (1)
      password: dIRs5cAAsHIeeRIL (2)
    kind: Secret
    metadata:
      name: basic-auth
    type: Opaque
    1 specifies an application user. developer is the default application user.
    2 specifies a corresponding password.
  2. Create a service monitor instance that configures Prometheus to monitor your Infinispan cluster.

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      labels:
        k8s-app: prometheus
      name: datagrid-monitoring (1)
      namespace: infinispan-monitoring (2)
    spec:
      endpoints:
        - basicAuth:
            username:
              key: username
              name: basic-auth (3)
            password:
              key: password
              name: basic-auth (4)
          interval: 30s
          port: ispn (5)
          scheme: https (6)
          tlsConfig:
            insecureSkipVerify: true
            serverName: certificate-CN (7)
      namespaceSelector:
        matchNames:
          - infinispan (8)
      selector:
        matchLabels:
          app: infinispan-service
          clusterName: cluster-name (9)
    1 names the service monitor.
    2 specifies your Prometheus namespace.
    3 specifies the name of the authentication secret that has Infinispan credentials.
    4 specifies the name of the authentication secret that has Infinispan credentials.
    5 specifies the port name for Infinispan endpoints.
    6 specifies that Infinispan endpoints use encryption. If you do not use TLS, remove spec.endpoints.scheme.
    7 specifies the Common Name (CN) of the TLS certificate for Infinispan encryption. If you use an OpenShift service certificate, the CN matches the metadata.name resource for your Infinispan cluster. If you do not use TLS, remove spec.endpoints.tlsConfig.
    8 specifies the Infinispan Operator namespace.
    9 specifies the name of the Infinispan cluster.

10. Connecting to Infinispan Clusters

Connect to Infinispan via the REST or Hot Rod endpoints. You can then remotely create and modify cache definitions and store data across Infinispan clusters.

The examples in this section use ${SERVICE_HOSTNAME} to denote the service or hostname that provides access to your Infinispan cluster.

Clients running in Kubernetes can specify the name of the internal service that Infinispan Operator creates.

Clients running outside Kubernetes should specify hostnames according to the type of external service and provider. For example, if using a load balancer service on AWS, the service hostname could be:

.status.loadBalancer.ingress[0].hostname

On GCP or Azure, hostnames might be as follows:

.status.loadBalancer.ingress[0].ip

10.1. Invoking the Infinispan REST API

You can invoke the Infinispan REST API with any appropriate HTTP client.

For convenience, the following examples show how to invoke the REST API with curl using unencrypted connections. It is beyond the scope of this document to describe how to configure HTTP clients to use encryption.

Use the oc rsh command to open a remote shell to a Infinispan pod where you can run curl commands, for example:

$ oc rsh {example_crd_name}
Procedure
  1. Cache service provides a default cache instance, but Data Grid service does not. Before you can store data with Data Grid service clusters, you must create a cache as in the following example:

    $ curl -X POST \
      -u ${USERNAME}:${PASSWORD} \
      ${SERVICE_HOSTNAME}:11222/rest/v2/caches/default
      ...
      < HTTP/1.1 200 OK
      ...
  2. Put an entry in the cache.

    $ curl -X POST \
      -u ${USERNAME}:${PASSWORD} \
      -H 'Content-type: text/plain' -d 'world' \
      ${SERVICE_HOSTNAME}:11222/rest/v2/caches/default/hello
      ...
      < HTTP/1.1 204 No Content
  3. Verify the entry.

    $ curl -X GET \
      -u ${USERNAME}:${PASSWORD} \
      ${SERVICE_HOSTNAME}:11222/rest/v2/caches/default/hello/
      ...
      < HTTP/1.1 200 OK
      ...
      world

10.2. Configuring Hot Rod Clients

Configure Hot Rod Java clients to connect to Infinispan clusters.

Configuration objects
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.addServer()
  //Connection
	.host("${SERVICE_HOSTNAME}").port(11222) (1)
  //Client intelligence
	.clientIntelligence(ClientIntelligence.BASIC) (2)
	.security()
        //Authentication
        .authentication().enable()
          .username("${USERNAME}") (3)
          .password("${PASSWORD}") (4)
          .serverName("${CLUSTER_NAME}") (5)
          .saslQop(SaslQop.AUTH) (6)
          .saslMechanism("DIGEST-MD5") (7)
        //Encryption
        .ssl()
          .sniHostName("${SERVICE_HOSTNAME}") (8)
          .trustStorePath​(tls.crt); (9)
Hot Rod client properties
# Connection
infinispan.client.hotrod.server_list=${SERVICE_HOSTNAME}:11222 (1)

# Client intelligence
infinispan.client.hotrod.client_intelligence=BASIC (2)

# Authentication
infinispan.client.hotrod.use_auth=true
infinispan.client.hotrod.auth_username=${USERNAME} (3)
infinispan.client.hotrod.auth_password=${PASSWORD} (4)
infinispan.client.hotrod.auth_server_name=${CLUSTER_NAME} (5)
infinispan.client.hotrod.sasl_properties.javax.security.sasl.qop=auth (6)
infinispan.client.hotrod.sasl_mechanism=DIGEST-MD5 (7)

# Encryption
infinispan.client.hotrod.sni_host_name=${SERVICE_HOSTNAME} (8)
infinispan.client.hotrod.trust_store_path=tls.crt (9)
1 adds the cluster IP of a service or the hostname of a route.
2 configures client intelligence. Hot Rod clients running outside Kubernetes can use BASIC intelligence only.
3 sets the username of an application user.
4 sets the password for the application user.
5 matches the name you gave to your Infinispan cluster.
6 uses the authentication only SASL Quality of Protection (QoP) setting.
7 uses DIGEST-MD5 as the authentication mechanism.
8 sets the Server Name Indication (SNI) for Infinispan clusters.
9 sets the path to the TLS certificate if using encryption with clients running outside Kubernetes. Hot Rod clients automatically generate trust stores from the TLS certificates.

11. Monitoring Infinispan Logs

Set logging categories to different message levels to monitor, debug, and troubleshoot Infinispan pods.

11.1. Configuring Infinispan Logging

Procedure
  1. Modify your custom resource definition to include logging resources and then apply the changes.

    spec:
      ...
      logging: (1)
        categories: (2)
          org.infinispan: debug (3)
          org.jgroups: debug
    1 configures Infinispan logging.
    2 adds logging categories.
    3 names logging categories and levels.

    The root logging category is org.infinispan and is INFO by default.

  2. Retrieve logs from Infinispan pods as required.

    $ oc logs -f ${POD_NAME}

11.2. Log Levels

Log levels indicate the nature and severity of messages.

Log level Description

trace

Provides detailed information about running state of applications. This is the most verbose log level.

debug

Indicates the progress of individual requests or activities.

info

Indicates overall progress of applications, including lifecycle events.

warn

Indicates circumstances that can lead to error or degrade performance.

error

Indicates error conditions that might prevent operations or activities from being successful but do not prevent applications from running.

12. Configuring Cross-Site Replication

Set up cross-site replication to back up data between Infinispan clusters running in different locations.

For example, you use Infinispan Operator to manage a Infinispan cluster at a data center in London, LON. At another data center in New York City, NYC, you also use Infinispan Operator to manage a Infinispan cluster. In this case, you can add LON and NYC as backup locations for each other.

Prerequisites

12.1. Creating Minikube Site Access Secrets

If you run Infinispan Operator in Minikube, you should create secrets that contain the files that allow different instances of Minikube to authenticate with each other.

Procedure
  1. Create secrets on each site that contain ca.crt, client.crt, and client.key from your Minikube installation.

    For example, do the following on SiteB:

    kubectl create secret generic site-a-secret \
        --from-file=certificate-authority=/opt/minikube/.minikube/ca.crt \
        --from-file=client-certificate=/opt/minikube/.minikube/client.crt \
        --from-file=client-key=/opt/minikube/.minikube/client.key

12.2. Creating Service Account Tokens

Traffic between independent OpenShift installations occurs through a Kubernetes API. OpenShift Container Platform clusters use tokens to authenticate with and access the API.

To enable cross-site replication between Infinispan clusters you must add tokens to the namespace on each site. For example, LON needs a secret with the token for NYC. NYC also needs a secret with the token for LON.

Procedure
  1. Create service account tokens on each OpenShift instance.

    For example, on LON create a token for NYC as follows:

    $ oc create sa lon
    serviceaccount/lon created
  2. Find the token in the list of secrets in your OpenShift namespace.

    The following example shows two lon tokens:

    $ oc get secrets
    
    NAME                       TYPE
    ...
    lon-token-8j6d8         kubernetes.io/service-account-token
    lon-token-g2brv         kubernetes.io/service-account-token
  3. Retrieve a token to allow backup locations to authenticate with the local OpenShift cluster.

    The following example shows an encoded string in the token field:

    $ oc describe secret lon-token-8j6d8
    
    Name:         lon-token-8j6d8
    Namespace:    local-operators
    Labels:       <none>
    Annotations:  kubernetes.io/created-by: openshift.io/create-dockercfg-secrets
                  kubernetes.io/service-account.name: lon
                  kubernetes.io/service-account.uid: c22bc8d6-ffd8-11e9-adb6-0a8e23c49339
    
    Type:  kubernetes.io/service-account-token
    
    Data
    ====
    ca.crt:          4634 bytes
    namespace:       15 bytes
    service-ca.crt:  5750 bytes
    token:           eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
  4. Create secrets that contain service account tokens for the backup locations.

    1. Log in to OpenShift Container Platform at NYC.

    2. Add the service account token to a lon-token secret.

      oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
    3. Repeat the preceding steps to create a nyc-token secret on LON.

After you add service account tokens to each backup location, the OpenShift instances can authenticate with each other so that Infinispan clusters can form cross-site views.

12.3. Adding Backup Locations to Infinispan Clusters

Configure Infinispan clusters as backup locations so that they can communicate over a dedicated JGroups transport channel for replicating data.

Procedure
  1. Create custom resources for Infinispan at each site.

    For example, create lon.yaml to configure LON and nyc.yaml to configure NYC. Both configurations must include the following resource definitions:

    • .spec.service.sites.local names the local site for Infinispan clusters.

    • .spec.service.sites.locations provides the location of all site masters. Infinispan pods use this information to connect with each other and form cross-site views.

  2. Configure Infinispan clusters at each site, for example:

    1. Log in to OpenShift Container Platform at LON.

    2. Apply the custom resources for LON.

      $ oc apply -f lon.yaml
    3. Log in to OpenShift Container Platform at NYC.

    4. Apply the custom resources for NYC.

      $ oc apply -f nyc.yaml
  3. Verify that Infinispan clusters form a cross-site view.

    For example, do the following on LON:

    $ oc logs example-infinispan-0 | grep x-site
    
    INFO  [org.infinispan.XSITE] (jgroups-5,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC]
    INFO  [org.infinispan.XSITE] (jgroups-7,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC, LON]

12.3.1. Cross-Site Replication Resources

spec:
  ...
  service:
    type: DataGrid (1)
    sites:
      local:
        name: LON (2)
        expose:
          type: LoadBalancer (3)
      locations: (4)
      - name: LON (5)
        url: openshift://api.site-a.devcluster.openshift.com:6443 (6)
        secretName: lon-token (7)
      - name: NYC
        url: openshift://api.site-b.devcluster.openshift.com:6443
        secretName: nyc-token
1 specifies the DataGrid service. Infinispan supports cross-site replication with the DataGrid service only.
2 names the local site for a Infinispan cluster.
3 defines the externally exposed service.
  • Use NodePort for local clusters on the same network. For example, if you use Minikube for testing purposes.

  • Use LoadBalancer for independent OpenShift clusters.

4 provides connection information for all backup locations.
5 name of a backup location that matches the .spec.service.sites.local.name resource.
6 specifies a backup location.
  • Use minikube:// if the backup location is a Minikube instance.

  • Use openshift:// if the backup location is an OpenShift cluster. You should specify the URL of the Kubernetes API.

7 specifies the access secret for a site.

This secret contains different authentication objects, depending on your Kubernetes environment.

13. Reference

Find useful information for Infinispan clusters that you create with Infinispan Operator.

13.1. Network Services

Internal service
  • Allow Infinispan nodes to discover each other and form clusters.

  • Provide access to Infinispan endpoints from clients in the same Kubernetes namespace.

Service Port Protocol Description

${cluster_name}

11222

TCP

Internal access to Infinispan endpoints

${cluster_name}-ping

8888

TCP

Cluster discovery

External service

Provides access to Infinispan endpoints from clients outside Kubernetes or in different namespaces.

You must create the external service with Infinispan Operator. It is not available by default.

Service Port Protocol Description

${cluster_name}-external

11222

TCP

External access to Infinispan endpoints.

Cross-site service

Allows Infinispan to back up data between clusters in different locations.

Service Port Protocol Description

${name}-sites

7900

TCP

JGroups RELAY2 channel for cross-site communication.

13.2. Infinispan Operator Upgrades

Infinispan Operator upgrades Infinispan when new versions become available.

To upgrade Infinispan clusters, Infinispan Operator checks the version of the image for Infinispan pods. If Infinispan Operator determines that a new version of the image is available, it gracefully shuts down all pods, applies the new image, and then restarts the pods.

Infinispan Operator requires the Operator Lifecycle Manager to perform cluster upgrades.