The Infinispan Operator provides operational intelligence and reduces management complexity for deploying Infinispan on Kubernetes clusters.

1. Installing Infinispan Operator

Install Infinispan Operator into a Kubernetes namespace to create and manage Infinispan clusters.

Prerequisites
  • OKD 3.11 or later.

  • Kubernetes 1.11 or later.

  • Administrator access to the Kubernetes cluster.

1.1. Installing from the OperatorHub

Install Infinispan Operator from OperatorHub.io.

Procedure
  1. Navigate to the Infinispan Operator entry on OperatorHub.io.

  2. Follow the instructions to install Infinispan Operator into your Kubernetes cluster.

1.2. Building and Installing Infinispan Operator Manually

Manually build and install Infinispan Operator from the GitHub repository.

Procedure

2. Getting Started with Infinispan Operator

Infinispan Operator lets you create, configure, and manage Infinispan clusters.

Prerequisites
  • Install Infinispan Operator.

  • Have an oc or a kubectl client.

2.1. Infinispan Custom Resource (CR)

Infinispan Operator adds a new Custom Resource (CR) of type Infinispan that lets you handle Infinispan clusters as complex units on Kubernetes.

You configure Infinispan clusters running on Kubernetes by modifying the Infinispan CR.

The minimal Infinispan CR for Infinispan clusters is as follows:

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: (1)
spec:
  replicas: (2)
1 Names the Infinispan cluster.
2 Sets the number of nodes in the Infinispan cluster.

2.2. Creating Infinispan Clusters

Use Infinispan Operator to create clusters of two or more Infinispan nodes.

Procedure
  1. Specify the number of Infinispan nodes in the cluster with spec.replicas in your Infinispan CR.

    For example, create a cr_minimal.yaml file as follows:

    $ cat > cr_minimal.yaml<<EOF
    apiVersion: infinispan.org/v1
    kind: Infinispan
    metadata:
      name: example-infinispan
    spec:
      replicas: 2
    EOF
  2. Apply your Infinispan CR.

    $ kubectl apply -f cr_minimal.yaml
  3. Watch Infinispan Operator create the Infinispan nodes.

    $ kubectl get pods -w
    
    NAME                        READY  STATUS              RESTARTS   AGE
    example-infinispan-1        0/1    ContainerCreating   0          4s
    example-infinispan-2        0/1    ContainerCreating   0          4s
    example-infinispan-3        0/1    ContainerCreating   0          5s
    infinispan-operator-0       1/1    Running             0          3m
    example-infinispan-3        1/1    Running             0          8s
    example-infinispan-2        1/1    Running             0          8s
    example-infinispan-1        1/1    Running             0          8s
Next Steps

Try changing the value of replicas: and watching Infinispan Operator scale the cluster up or down.

2.3. Verifying Infinispan Clusters

Review log messages to ensure that Infinispan nodes receive clustered views.

Procedure
  • Do either of the following:

    • Retrieve the cluster view from logs.

      $ kubectl logs example-infinispan-0 | grep ISPN000094
      
      INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|0] (1) [example-infinispan-0]
      
      INFO  [org.infinispan.CLUSTER] (jgroups-3,{example_crd_name-0) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|1] (2) [example-infinispan-0, example-infinispan-1]
    • Retrieve the Infinispan CR for Infinispan Operator.

      $ kubectl get infinispan -o yaml

      The response indicates that Infinispan pods have received clustered views:

      conditions:
          - message: 'View: [example-infinispan-0, example-infinispan-1]'
            status: "True"
            type: wellFormed

Use kubectl wait with the wellFormed condition for automated scripts.

$ kubectl wait --for condition=wellFormed --timeout=240s infinispan/example-infinispan

3. Creating Infinispan Services

Infinispan services are stateful applications that provide flexible and robust in-memory data storage.

3.1. Cache Service

Cache Service provides a volatile, low-latency data store that dramatically increases application response rates.

Cache Service nodes:

  • Synchronously distribute data across the cluster to ensure consistency.

  • Maintain single copies of cache entries to reduce size.

  • Store cache entries off-heap and use eviction for JVM efficiency.

  • Ensure data consistency with a default partition handling configuration.

You can create multiple cache definitions with Cache Service but only as copies of the default configuration.

If you update Cache Service nodes with the Infinispan CR or update the version, you lose all data in the cache.

3.1.1. Cache Configuration

Cache Service nodes use the following cache configuration:

<distributed-cache name="default" (1)
                   mode="SYNC" (2)
                   owners="1"> (3)
   <memory>
    <off-heap eviction="MEMORY" (4)
              strategy="REMOVE"/> (5)
   </memory>
   <partition-handling when-split="ALLOW_READ_WRITES" (6)
                       merge-policy="REMOVE_ALL"/> (7)
</distributed-cache>
1 Names the cache instance as "default".
2 Uses synchronous distribution for storing data across the cluster.
3 Configures one replica for each cache entry across the cluster.
4 Stores cache entries as bytes in native memory (off-heap).
5 Removes old entries to make space when adding new entries.
6 Specifies a conflict resolution strategy that allows read and write operations for cache entries even if segment owners are in different partitions.
7 Specifies a merge policy that removes entries from the cache when Infinispan detects conflicts.

3.2. Data Grid Service

Data Grid Service provides a configurable Infinispan server distribution for Kubernetes.

  • Use with advanced capabilities like cross-site replication as well as indexing and querying.

  • Remotely access Data Grid Service clusters from Hot Rod or REST clients and dynamically create caches using any Infinispan cache mode and configuration.

    Infinispan does not provide default caches for Data Grid Service nodes. However, you can use cache configuration templates to get started.

3.3. Creating Services

Define the .spec.service.type resource to create Cache Service and Data Grid Service nodes with Infinispan Operator.

By default, Infinispan Operator creates Infinispan clusters configured as a Cache Service.

Procedure
  • Specify the service type for Infinispan clusters with spec.service.type in your Infinispan CR and then apply the changes.

For example, create Data Grid Service clusters as follows:

spec:
  ...
  service:
    type: DataGrid

You cannot change .spec.service.type after you create Infinispan clusters.

For example, if you create a cluster of Cache Service nodes, you cannot change the service type to Data Grid Service. In this case you must create a new cluster with Data Grid Service nodes in a different Kubernetes namespace.

3.3.1. Cache Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan (1)
spec:
  image: infinispan/server:latest
  replicas: 4 (2)
  service:
    type: Cache (3)
  security:
    endpointSecretName: endpoint-identities (4)
    endpointEncryption: (5)
        type: secret
        certSecretName: tls-secret
  container: (6)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m"
    memory: 1Gi
  logging: (7)
    categories:
      org.infinispan: trace
      org.jgroups: trace
  expose: (8)
    type: LoadBalancer
1 Names the Infinispan cluster.
2 Specifies the number of nodes in the cluster.
3 Creates Cache Service clusters.
4 Adds an authentication secret with user credentials.
5 Adds a custom encryption secret for secure connections.
6 Allocates resources to nodes.
7 Configures logging.
8 Configures services for external traffic.

3.3.2. Data Grid Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan (1)
spec:
  image: infinispan/server:latest
  replicas: 6 (2)
  service:
    type: DataGrid (3)
    container:
      storage: 2Gi (4)
    sites: (5)
      local:
        expose:
          type: LoadBalancer
      locations:
      - name: azure
        url: openshift://api.azure.host:6443
        secretName: azure-identities
      - name: aws
        url: openshift://api.aws.host:6443
        secretName: aws-identities
  security:
    endpointSecretName: endpoint-identities (6)
    endpointEncryption: (7)
        type: secret
        certSecretName: tls-secret
  container: (8)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "1000m"
    memory: 1Gi
  logging: (9)
    categories:
      org.infinispan: debug
      org.jgroups: debug
  expose: (10)
    type: LoadBalancer
1 Names the Infinispan cluster.
2 Specifies the number of nodes in the cluster.
3 Creates Data Grid Service clusters.
4 Configures size of the persistent volume.
5 Provides connection information for backup locations.
6 Adds an authentication secret with user credentials.
7 Adds a custom encryption secret for secure connections.
8 Allocates resources to nodes.
9 Configures logging.
10 Configures services for external traffic.

4. Stopping and Starting Infinispan Services

Gracefully shut down Infinispan clusters to avoid data loss.

Cache configuration

Both Cache Service and Data Grid Service store permanent cache definitions in persistent volumes so they are still available after cluster restarts.

Data

Data Grid Service nodes can write all cache entries to persistent storage during cluster shutdown if you add cache stores.

You should configure the storage size for Data Grid Service nodes to ensure that the persistent volume can hold all your data.

If the available container storage is less than the amount of memory available to Data Grid Service nodes, Infinispan writes an exception to logs and data loss occurs during shutdown.

4.1. Gracefully Shutting Down Infinispan Clusters

  • Set the value of replicas to 0 and apply the changes.

spec:
  replicas: 0

4.2. Restarting Infinispan Clusters

  • Set the value of spec.replicas to the same number of nodes that were in the cluster before you shut it down.

For example, you shut down a cluster of 6 nodes. When you restart the cluster, you must set:

spec:
  replicas: 6

This allows Infinispan to restore the distribution of data across the cluster. When all nodes in the cluster are running, you can then add or remove nodes.

5. Adjusting Container Specifications

You can allocate CPU and memory resources, specify JVM options, and configure storage for Infinispan nodes.

5.1. JVM, CPU, and Memory Resources

spec:
  ...
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary" (1)
    cpu: "1000m" (2)
    memory: 1Gi (3)
1 Specifies JVM options.
2 Allocates host CPU resources to node, measured in CPU units.
3 Allocates host memory resources to nodes, measured in bytes.

When Infinispan Operator creates Infinispan clusters, it uses spec.container.cpu and spec.container.memory to:

  • Ensure that Kubernetes has sufficient capacity to run the Infinispan node. By default Infinispan Operator requests 512Mi of memory and 0.5 cpu from the Kubernetes scheduler.

  • Constrain node resource usage. Infinispan Operator sets the values of cpu and memory as resource limits.

Garbage collection logging

By default, Infinispan Operator does not log garbage collection (GC) messages. You can optionally add the following JVM options to direct GC messages to stdout:

extraJvmOpts: "-Xlog:gc*:stdout:time,level,tags"

5.2. Storage Resources

spec:
  ...
  service:
    type: DataGrid
    container:
      storage: 2Gi (1)
1 Configures the storage size for Data Grid Service nodes.

By default, Infinispan Operator allocates 1Gi for storage for both Cache Service and Data Grid Service nodes. You can configure storage size only for Data Grid Service nodes.

Persistence

Data Grid Service lets you configure Single File cache stores for data persistence:

<persistence>
    <file-store />
</persistence>

5.2.1. Persistent Volume Claims

Infinispan Operator mounts persistent volumes at:
/opt/infinispan/server/data

Persistent volume claims use the ReadWriteOnce (RWO) access mode.

6. Creating Network Services

Network services provide access to Infinispan clusters for client connections.

6.1. Getting the Service for Internal Connections

By default, Infinispan Operator creates a service that provides access to Infinispan clusters from clients running in Kubernetes.

This internal service has the same name as your Infinispan cluster, for example:

metadata:
  name: example-infinispan
Procedure
  • Check that the internal service is available as follows:

    $ kubectl get services
    
    NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)
    example-infinispan ClusterIP   192.0.2.0        <none>        11222/TCP
Reference

6.2. Exposing Infinispan to External Clients

Expose Infinispan clusters to clients running outside Kubernetes with external services.

Procedure
  • Specify an external service type with spec.expose.type in your Infinispan CR and then apply the changes.

    spec:
      ...
      expose: (1)
        type: LoadBalancer (2)
    1 exposes an external service at port 11222.
    2 specifies either a LoadBalancer or NodePort service resource type.
LoadBalancer

Use for Kubernetes clusters where a load balancer service is available to handle external network traffic. You can then use the URL for the load balancer service for client connections.

To access Infinispan with unencrypted Hot Rod client connections you must use a load balancer service.

NodePort

Use for local Kubernetes clusters.

Verification
  • Check that the -external service is available.

$ kubectl get services | grep external

NAME                         TYPE            CLUSTER-IP       EXTERNAL-IP   PORT(S)
example-infinispan-external  LoadBalancer    192.0.2.24       <none>        11222/TCP
Reference

7. Configuring Authentication

Application users must authenticate with Infinispan clusters. Infinispan Operator generates default credentials or you can add your own.

7.1. Default Credentials

Infinispan Operator generates base64-encoded default credentials stored in an authentication secret named example-infinispan-generated-secret

Username Description

developer

Default application user.

operator

Internal user that interacts with Infinispan clusters.

7.2. Retrieving Credentials

Get credentials from authentication secrets to access Infinispan clusters.

Procedure
  • Retrieve credentials from authentication secrets, as in the following example:

    $ kubectl get secret example-infinispan-generated-secret

    Base64-decode credentials.

    $ kubectl get secret example-infinispan-generated-secret \
    -o jsonpath="{.data.identities\.yaml}" | base64 --decode
    
    credentials:
    - username: developer
      password: dIRs5cAAsHIeeRIL
    - username: operator
      password: uMBo9CmEdEduYk24

7.3. Identities File

You can define credentials for Infinispan application users and the operator user in an identities.yaml file. All users that you add to identities.yaml can access Infinispan clusters and data

identities.yaml must include the operator user.

Identities File Example
credentials:
- username: testuser
  password: testpassword
- username: operator
  password: supersecretoperatorpassword

7.4. Adding Custom Credentials

Configure access to Infinispan cluster endpoints with custom credentials.

Procedure
  1. Define custom credentials in an identities.yaml file.

  2. Create an authentication secret from identities.yaml as follows:

    $ kubectl create secret generic --from-file=identities.yaml connect-secret
  3. Specify the authentication secret with spec.security.endpointSecretName in your Infinispan CR and then apply the changes.

    spec:
      ...
      security:
        endpointSecretName: connect-secret (1)
    1 Specifies the name of the authentication secret that contains your credentials.
Verification

Modifying spec.security.endpointSecretName triggers a cluster restart. You can watch the Infinispan cluster as Infinispan Operator applies changes:

$ kubectl get pods -w

When the Infinispan cluster is running, verify your custom credentials as follows:

  1. Open a remote shell to a Infinispan node, for example:

    $ kubectl exec -it example-infinispan-0 -- /bin/bash
  2. Check that credentials exist in the users.properties file.

    $ cat /opt/infinispan/server/conf/users.properties
  3. Exit the remote shell.

    $ exit

8. Securing Infinispan Connections

Encrypt connections between clients and Infinispan nodes with Red Hat OpenShift service certificates or custom TLS certificates.

8.1. Using Red Hat OpenShift Service Certificates

Infinispan Operator automatically generates TLS certificates signed by the Red Hat OpenShift service CA. You can use these certificates to encrypt remote client connections.

Procedure
  • Set the following spec.security.endpointEncryption configuration in your Infinispan CR and then apply the changes.

    spec:
      ...
      security:
        endpointEncryption:
          type: service
          certServiceName: service.beta.openshift.io (1)
          certSecretName: example-infinispan-cert-secret (2)
1 Specifies the Red Hat OpenShift Service.
2 Specifies the name of the secret where Infinispan Operator stores service certificates and keys.

Infinispan Operator stores in a secret named -cert-secret that is prefixed with the Infinispan cluster name, for example:

metadata:
  name: example-infinispan

The preceding cluster name results in a secret named example-infinispan-cert-secret.

8.1.1. Red Hat OpenShift Service Certificates

If the Red Hat OpenShift service CA is available, Infinispan Operator automatically generates a certificate, tls.crt, and key, tls.key, in PEM format.

Service certificates use the internal DNS name of the Infinispan cluster as the common name (CN), for example:

Subject: CN = example-infinispan.mynamespace.svc

For this reason, service certificates can be fully trusted only inside OpenShift. If you want to encrypt connections with clients running outside OpenShift, you should use custom TLS certificates.

Certificates are valid for one year and are automatically replaced before they expire.

8.1.2. Retrieving TLS Certificates

Get TLS certificates from encryption secrets to create client trust stores.

  • Retrieve tls.crt from encryption secrets as follows:

$ kubectl get secret example-infinispan-cert-secret \
-o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.crt

8.2. Using Custom TLS Certificates

Use custom PKCS12 keystore or TLS certificate/key pairs to encrypt connections between clients and Infinispan clusters.

Prerequisites

Create either a keystore or certificate secret. See:

Procedure
  1. Add the encryption secret to your OpenShift namespace, for example:

    $ kubectl apply -f tls_secret.yaml
  2. Specify the encryption secret with spec.security.endpointEncryption in your Infinispan CR and then apply the changes.

    spec:
      ...
      security:
        endpointEncryption: (1)
                type: secret (2)
                certSecretName: tls-secret (3)
    1 encrypts traffic to and from Infinispan endpoints.
    2 configures Infinispan to use secrets that contain encryption certificates.
    3 names the encryption secret.

8.2.1. Certificate Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
data:
    tls.key:  "LS0tLS1CRUdJTiBQUk ..." (1)
    tls.crt: "LS0tLS1CRUdJTiBDRVl ..." (2)
1 Adds a base64 encoded TLS key.
2 Adds a base64 encoded TLS certificate.

8.2.2. Keystore Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
stringData:
    alias: server (1)
    password: password (2)
data:
    keystore.p12:  "MIIKDgIBAzCCCdQGCSqGSIb3DQEHA..." (3)
1 Specifies an alias for the keystore.
2 Specifies a password for the keystore.
3 Adds a base64 encoded keystore.

9. Monitoring Infinispan with Prometheus

Infinispan exposes a metrics endpoint that provides statistics and events to Prometheus.

9.1. Setting Up Prometheus

Set up Prometheus so it can authenticate with and monitor Infinispan clusters.

Prerequisites
  • Install the Prometheus Operator.

  • Create a running Prometheus instance.

Procedure
  1. Add an authentication secret to your Prometheus namespace.

    This secret allows Prometheus to authenticate with your Infinispan cluster. You can find Infinispan credentials in the authentication secret in your Infinispan Operator namespace.

    apiVersion: v1
    stringData:
      username: developer (1)
      password: dIRs5cAAsHIeeRIL (2)
    kind: Secret
    metadata:
      name: basic-auth
    type: Opaque
    1 specifies an application user. developer is the default application user.
    2 specifies a corresponding password.
  2. Create a service monitor instance that configures Prometheus to monitor your Infinispan cluster.

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      labels:
        k8s-app: prometheus
      name: datagrid-monitoring (1)
      namespace: infinispan-monitoring (2)
    spec:
      endpoints:
        - basicAuth:
            username:
              key: username
              name: basic-auth (3)
            password:
              key: password
              name: basic-auth (4)
          interval: 30s
          scheme: https (5)
          tlsConfig:
            insecureSkipVerify: true
            serverName: certificate-CN (6)
      namespaceSelector:
        matchNames:
          - infinispan (7)
      selector:
        matchLabels:
          app: infinispan-service
          clusterName: cluster-name (8)
    1 names the service monitor.
    2 specifies your Prometheus namespace.
    3 specifies the name of the authentication secret that has Infinispan credentials.
    4 specifies the name of the authentication secret that has Infinispan credentials.
    5 specifies that Infinispan endpoints use encryption. If you do not use TLS, remove spec.endpoints.scheme.
    6 specifies the Common Name (CN) of the TLS certificate for Infinispan encryption. If you use an OpenShift service certificate, the CN matches the metadata.name resource for your Infinispan cluster. If you do not use TLS, remove spec.endpoints.tlsConfig.
    7 specifies the Infinispan Operator namespace.
    8 specifies the name of the Infinispan cluster.

10. Connecting to Infinispan Clusters

Connect to Infinispan via the REST or Hot Rod endpoints. You can then remotely create and modify cache definitions and store data across Infinispan clusters.

The examples in this section use $SERVICE_HOSTNAME to denote the service that provides access to your Infinispan cluster.

Clients running in Kubernetes can specify the name of the internal service that Infinispan Operator creates.

Clients running outside Kubernetes should specify hostnames according to the type of external service and provider. For example, if using a load balancer service on AWS, the service hostname could be:

.status.loadBalancer.ingress[0].hostname

On GCP or Azure, hostnames might be as follows:

.status.loadBalancer.ingress[0].ip

10.1. Invoking the Infinispan REST API

You can invoke the Infinispan REST API with any appropriate HTTP client.

For convenience, the following examples show how to invoke the REST API with curl using unencrypted connections. It is beyond the scope of this document to describe how to configure HTTP clients to use encryption.

Procedure
  1. If necessary, open a remote shell to a Infinispan node, for example:

    $ kubectl exec -it example-infinispan-0 -- /bin/bash
  2. Cache Service provides a default cache instance, but Data Grid Service does not. Before you can store data with Data Grid Service clusters, you must create a cache as in the following example:

    $ curl -X POST \
      -u $USERNAME:$PASSWORD \
      $SERVICE_HOSTNAME:11222/rest/v2/caches/default
      ...
      < HTTP/1.1 200 OK
      ...
  3. Put an entry in the cache.

    $ curl -X POST \
      -u $USERNAME:$PASSWORD \
      -H 'Content-type: text/plain' -d 'world' \
      $SERVICE_HOSTNAME:11222/rest/v2/caches/default/hello
      ...
      < HTTP/1.1 204 No Content
  4. Verify the entry.

    $ curl -X GET \
      -u $USERNAME:$PASSWORD \
      $SERVICE_HOSTNAME:11222/rest/v2/caches/default/hello/
      ...
      < HTTP/1.1 200 OK
      ...
      world

10.2. Configuring Hot Rod Clients

Configure Hot Rod Java clients to connect to Infinispan clusters.

Hot Rod client ConfigurationBuilder
import org.infinispan.client.hotrod.configuration.ConfigurationBuilder;

ConfigurationBuilder builder = new ConfigurationBuilder();
builder.addServer()
       //Connection
       .host("$SERVICE_HOSTNAME").port(11222)
       //Client intelligence
       //External clients can use `BASIC` intelligence only.
       .clientIntelligence(ClientIntelligence.BASIC)
       .security()
          //Authentication
          .authentication().enable()
            //Application user credentials.
            //The default username is developer.
            .username("developer")
            .password("$PASSWORD")
            .serverName("$CLUSTER_NAME")
            .saslQop(SaslQop.AUTH)
            .saslMechanism("DIGEST-MD5")
          //Encryption
          .ssl()
            .sniHostName("$SERVICE_HOSTNAME")
            //Path to the TLS certificate.
            //Clients automatically generate trust stores from certificates.
            .trustStorePath​(tls.crt);
Hot Rod client properties
# Connection
infinispan.client.hotrod.server_list=$SERVICE_HOSTNAME:11222

# Client intelligence
# External clients can use `BASIC` intelligence only.
infinispan.client.hotrod.client_intelligence=BASIC

# Authentication
infinispan.client.hotrod.use_auth=true
# Application user credentials.
# The default username is developer.
infinispan.client.hotrod.auth_username=developer
infinispan.client.hotrod.auth_password=$PASSWORD
infinispan.client.hotrod.auth_server_name=$CLUSTER_NAME
infinispan.client.hotrod.sasl_properties.javax.security.sasl.qop=auth
infinispan.client.hotrod.sasl_mechanism=DIGEST-MD5

# Encryption
infinispan.client.hotrod.sni_host_name=$SERVICE_HOSTNAME
# Path to the TLS certificate.
# Clients automatically generate trust stores from certificates.
infinispan.client.hotrod.trust_store_path=tls.crt

11. Creating Infinispan Caches

Add cache definitions that configure how Infinispan stores your data.

11.1. Creating Caches for Data Grid Service Clusters

You can create caches on Data Grid Service clusters using either infinispan.org cache configuration templates or from valid XML cache definitions.

You can create only one cache at a time using a Cache CR that specifies either a template or an XML definition. If your Cache CR contains both a template and an XML definition, Infinispan Operator uses the template.

11.1.1. Adding Credentials Secrets for Cache Creation

Infinispan Operator must authenticate with Data Grid Service clusters to create caches. You add credentials to a secret so that Infinispan Operator can access your cluster when creating caches.

The following procedure explains how to add credentials to a new secret. If you have a custom secret that contains credentials, you can use that instead of creating a new one.

Procedure
  1. Define a Secret object type that provides valid user credentials for accessing Data Grid Service clusters in a StringData map.

    For example, create an auth_secret.yaml file that provides credentials for the developer user as follows:

    apiVersion: v1
    stringData:
      username: developer (1)
      password: G8ZdJvSaY3lOOwfM (2)
    kind: Secret
    metadata:
      name: basic-auth (3)
    type: Opaque
    1 Names a user that can create caches.
    2 Specifies the password that corresponds to the user.
    3 Specifies a name for the secret.
  2. Create a secret from the file, as in the following example:

    $ kubectl create secret generic --from-file=auth_secret.yaml basic-auth
Using Custom Credentials Secrets

Infinispan Operator requires that credentials exist as values for the username and password keys in a secret. If you have a custom secret that contains Infinispan credentials, but uses different key names, you can override those names in your Cache CR.

For example, you have a secret named "my-credentials" that holds a list of Infinispan users and their passwords as follows:

stringData:
  app_user1: spock
  app_user1_pw: G8ZdJvSaY3lOOwfM
  app_user2: jim
  app_user2_pw: zTzz2gVyyF4JsYsH
Procedure
  • In your Cache CR, override custom key names with username and password as follows:

spec:
  adminAuth:
    username:
      key: app_user1 (1)
      name: my-credentials (2)
    password:
      key: app_user1_pw (3)
      name: my-credentials
1 Overrides the app_user1 key name with username.
2 Specifies the name of your custom credentials secret.
3 Overrides the app_user1_pw key name with password.

11.1.2. Creating Infinispan Caches from Templates

Complete the following steps to create caches on Data Grid Service clusters using infinispan.org cache configuration templates.

Prerequisites
  • Create a secret that contains valid user credentials for accessing Infinispan clusters.

  • Identify the cache configuration template you want to use for your cache. You can retrieve all available configuration templates with the following REST call:

    GET /rest/v2/cache-managers/default/cache-configs/templates
Procedure
  1. Create a Cache CR that specifies the name of the template you want to use.

    For example, the following CR creates a cache named "mycache" that uses the org.infinispan.DIST_SYNC cache configuration template:

    apiVersion: infinispan.org/v2alpha1
    kind: Cache
    metadata:
      name: mycachedefinition (1)
    spec:
      adminAuth: (2)
        secretName: basic-auth
      clusterName: example-infinispan (3)
      name: mycache (4)
      templateName: org.infinispan.DIST_SYNC (5)
    1 Names the Cache CR.
    2 Specifies a secret that provides credentials with username and password keys or an override for custom credentials secrets.
    3 Specifies the name of the target Infinispan cluster where you want Infinispan Operator to create the cache.
    4 Names the Infinispan cache instance.
    5 Specifies the infinispan.org cache configuration template to create the cache.
  2. Apply the Cache CR, for example:

    $ kubectl apply -f mycache.yaml
    cache.infinispan.org/mycachedefinition created

11.1.3. Creating Infinispan Caches from XML

Complete the following steps to create caches on Data Grid Service clusters using valid infinispan.xml cache definitions.

Prerequisites
  • Create a secret that contains valid user credentials for accessing Infinispan clusters.

Procedure
  1. Create a Cache CR that contains the XML cache definition you want to create.

    apiVersion: infinispan.org/v2alpha1
    kind: Cache
    metadata:
      name: mycachedefinition (1)
    spec:
      adminAuth: (2)
        secretName: basic-auth
      clusterName: example-infinispan (3)
      name: mycache (4)
      template: <infinispan><cache-container><distributed-cache/></cache-container></infinispan> (5)
    1 Names the Cache CR.
    2 Specifies a secret that provides credentials with username and password keys or an override for custom credentials secrets.
    3 Specifies the name of the target Infinispan cluster where you want Infinispan Operator to create the cache.
    4 Names the Infinispan cache instance.
    5 Specifies the XML cache definition to create the cache. Note that the name attribute is ignored. Only spec.name applies to the resulting cache.
  2. Apply the Cache CR, for example:

    $ kubectl apply -f mycache.yaml
    cache.infinispan.org/mycachedefinition created

11.2. Creating Caches for Cache Service Clusters

Cache Service clusters provide a default cache configuration. You can create multiple caches but only as copies of the default configuration.

Procedure
  • Invoke the following REST call:

    POST /rest/v2/caches/$CACHENAME?template=default

In test or development environments, you can do this as follows:

  1. Open a remote shell to a Infinispan node, for example:

    $ kubectl exec -it example-infinispan-0 -- /bin/bash
  2. Create a cache named "mycache" as follows:

    $ curl -X POST -u $USERNAME:$PASSWORD $ClusterIP:11222/rest/v2/caches/mycache?template=default
  3. Exit the remote shell.

    $ exit

11.3. Verifying Infinispan Caches

After you create Infinispan caches, complete the following procedure to verify they exist.

Procedure
  • Invoke the following REST call to list all available caches:

    GET $SERVICE_HOSTNAME:11222/rest/v2/caches/

In test or development environments, you can do this as follows:

  1. Open a remote shell to a Infinispan node, for example:

    $ kubectl exec -it example-infinispan-0 -- /bin/bash
  2. Get the list of available caches as follows:

    $ curl -u $USERNAME:$PASSWORD $ClusterIP:11222/rest/v2/caches/
  3. Exit the remote shell.

    $ exit

12. Monitoring Infinispan Logs

Set logging categories to different message levels to monitor, debug, and troubleshoot Infinispan clusters.

12.1. Configuring Infinispan Logging

Procedure
  1. Specify logging configuration with spec.logging in your Infinispan CR and then apply the changes.

    spec:
      ...
      logging: (1)
        categories: (2)
          org.infinispan: debug (3)
          org.jgroups: debug
    1 configures Infinispan logging.
    2 adds logging categories.
    3 names logging categories and levels.

    The root logging category is org.infinispan and is INFO by default.

  2. Retrieve logs from Infinispan nodes as required.

    $ kubectl logs -f $POD_NAME

12.2. Log Levels

Log levels indicate the nature and severity of messages.

Log level Description

trace

Provides detailed information about running state of applications. This is the most verbose log level.

debug

Indicates the progress of individual requests or activities.

info

Indicates overall progress of applications, including lifecycle events.

warn

Indicates circumstances that can lead to error or degrade performance.

error

Indicates error conditions that might prevent operations or activities from being successful but do not prevent applications from running.

13. Configuring Cross-Site Replication

Set up cross-site replication to back up data between Infinispan clusters running in different locations.

For example, you use Infinispan Operator to manage a Infinispan cluster at a data center in London, LON. At another data center in New York City, NYC, you also use Infinispan Operator to manage a Infinispan cluster. In this case, you can add LON and NYC as backup locations for each other.

Prerequisites

13.1. Infinispan Cluster and Project Naming

Infinispan Operator expects Infinispan clusters in each site to have the same cluster names and be running in matching namespaces.

For example, in the LON site you create a Infinispan cluster with metadata.name: mydatagrid in a Kubernetes project named "my-xsite". In this case you must create Infinispan clusters in other backup locations, such as NYC, with identical names in matching namespaces.

In effect, you must create Infinispan cluster names and Kubernetes namespaces at each backup location that mirror one another.

13.2. Creating Minikube Site Access Secrets

If you run Infinispan Operator in Minikube, you should create secrets that contain the files that allow different instances of Minikube to authenticate with each other.

Procedure
  1. Create secrets on each site that contain ca.crt, client.crt, and client.key from your Minikube installation.

    For example, do the following on LON:

    kubectl create secret generic site-a-secret \
        --from-file=certificate-authority=/opt/minikube/.minikube/ca.crt \
        --from-file=client-certificate=/opt/minikube/.minikube/client.crt \
        --from-file=client-key=/opt/minikube/.minikube/client.key

13.3. Creating Service Account Tokens

Traffic between independent OpenShift installations occurs through a Kubernetes API. OpenShift Container Platform clusters use tokens to authenticate with and access the API.

To enable cross-site replication between Infinispan clusters you must add tokens to the namespace on each site. For example, LON needs a secret with the token for NYC. NYC also needs a secret with the token for LON.

Procedure
  1. Create service accounts on each OpenShift instance.

    For example, create a service account on LON as follows:

    $ oc create sa lon
    serviceaccount/lon created
  2. Add the view role to service accounts.

    For example, if your Infinispan cluster runs in the "my-xsite" namespace, add the view role to the service account on LON as follows:

    $ oc policy add-role-to-user view system:serviceaccount:my-xsite:lon
  3. Retrieve tokens from each service account.

    The following example shows the service account token for LON:

    $ oc sa get-token lon
    
    eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
  4. Create secrets that contain service account tokens for the backup locations.

    1. Log in to OpenShift Container Platform at NYC.

    2. Add the service account token to a lon-token secret.

      oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
    3. Repeat the preceding steps to create a nyc-token secret on LON.

After you add service account tokens to each backup location, the OpenShift instances can authenticate with each other so that Infinispan clusters can form cross-site views.

13.4. Adding Backup Locations to Infinispan Clusters

Configure Infinispan clusters as backup locations so that they can communicate over a dedicated JGroups transport channel for replicating data.

Procedure
  1. Configure Infinispan clusters at each site with the Infinispan CR as necessary.

    For example, create lon.yaml to configure LON and nyc.yaml to configure NYC. Both configurations must include the following:

    • .spec.service.sites.local names the local site for Infinispan clusters.

    • .spec.service.sites.locations provides the location of all site masters. Infinispan nodes use this information to connect with each other and form cross-site views.

  2. Instantiate Infinispan clusters at each site, for example:

    1. Apply the Infinispan CR for LON.

      $ kubectl apply -f lon.yaml
    2. Log in to OpenShift Container Platform at NYC.

    3. Apply the Infinispan CR for NYC.

      $ kubectl apply -f nyc.yaml
  3. Verify that Infinispan clusters form a cross-site view.

    For example, do the following on LON:

    $ kubectl logs example-infinispan-0 | grep x-site
    
    INFO  [org.infinispan.XSITE] (jgroups-5,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC]
    INFO  [org.infinispan.XSITE] (jgroups-7,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC, LON]

13.4.1. Cross-Site Replication Resources

spec:
  ...
  service:
    type: DataGrid (1)
    sites:
      local:
        name: LON (2)
        expose:
          type: LoadBalancer (3)
      locations: (4)
      - name: LON (5)
        url: openshift://api.site-a.devcluster.openshift.com:6443 (6)
        secretName: lon-token (7)
      - name: NYC
        url: openshift://api.site-b.devcluster.openshift.com:6443
        secretName: nyc-token
1 Specifies Data Grid Service. Infinispan supports cross-site replication with Data Grid Service clusters only.
2 Names the local site for a Infinispan cluster.
3 Defines the externally exposed service.
  • Use NodePort for local clusters on the same network.

  • Use LoadBalancer for independent OpenShift clusters.

4 Provides connection information for all backup locations.
5 Specifies a backup location that matches .spec.service.sites.local.name.
6 Specifies a backup location.
  • Use minikube:// if the backup location is a Minikube instance.

  • Use openshift:// if the backup location is an OpenShift cluster. You should specify the URL of the Kubernetes API.

7 Specifies the access secret for a site.

This secret contains different authentication objects, depending on your Kubernetes environment.

14. Reference

Find useful information for Infinispan clusters that you create with Infinispan Operator.

14.1. Network Services

Internal service
  • Allow Infinispan nodes to discover each other and form clusters.

  • Provide access to Infinispan endpoints from clients in the same Kubernetes namespace.

Service Port Protocol Description

$ClusterName

11222

TCP

Internal access to Infinispan endpoints

$ClusterName-ping

8888

TCP

Cluster discovery

External service

Provides access to Infinispan endpoints from clients outside Kubernetes or in different namespaces.

You must create the external service with Infinispan Operator. It is not available by default.

Service Port Protocol Description

$ClusterName-external

11222

TCP

External access to Infinispan endpoints.

Cross-site service

Allows Infinispan to back up data between clusters in different locations.

Service Port Protocol Description

$NAME-sites

7900

TCP

JGroups RELAY2 channel for cross-site communication.

14.2. Infinispan Operator Upgrades

Infinispan Operator upgrades Infinispan when new versions become available.

To upgrade Infinispan clusters, Infinispan Operator checks the version of the image for Infinispan nodes. If Infinispan Operator determines that a new version of the image is available, it gracefully shuts down all nodes, applies the new image, and then restarts the nodes.

Infinispan Operator requires the Operator Lifecycle Manager to perform cluster upgrades.