The Infinispan Operator provides operational intelligence and reduces management complexity for deploying Infinispan on Kubernetes clusters.

1. Manually Deploying the Infinispan Operator

You can manually deploy the Infinispan Operator as an alternative to installing it from OperatorHub.io.

Prerequisites
  • OKD 3.11 or later.

  • Kubernetes 1.11 or later.

  • Administrator access to the Kubernetes cluster.

1.1. Creating Infinispan Operator Resources

Add the custom resource definition and role-based access control (RBAC) resources for the Infinispan Operator.

Procedure
  1. Apply the custom resource definition.

    $ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/crd.yaml
  2. Install RBAC resources.

    $ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/rbac.yaml

1.2. Deploying the Infinispan Operator

Manually deploying the Infinispan Operator to Kubernetes involves applying the yaml template that defines metadata and container specifications.

Procedure
  • Apply the Infinispan Operator template as follows:

$ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/operator.yaml

2. Creating Infinispan Clusters

You create, configure, and manage Infinispan clusters with the Infinispan Operator by defining custom resource objects in yaml format and applying them to Infinispan pods. Get started here and learn the basics for using the Infinispan Operator.

Prerequisites

2.1. Using Default Infinispan Operator Resources

Infinispan provides default resource definitions that are available in GitHub.

Procedure
  • Run oc apply -f to apply default Infinispan Operator resource definitions.

    For example, to use the default minimal Infinispan cluster:

    $ oc apply -f https://raw.githubusercontent.com/infinispan/infinispan-operator/master/deploy/cr/minimal/cr_minimal.yaml

2.2. Infinispan Custom Resources

The Infinispan Operator adds a resource type and custom resource definition (CRD) with recommended defaults for Infinispan pods.

You configure Infinispan pods dynamically by changing defaults and adding custom resources.

The minimum resources to define a Infinispan cluster are as follows:

apiVersion: infinispan.org/v1 (1)
kind: Infinispan (2)
metadata:
  name: (3)
spec:
  replicas: (4)
1 declares the version.
2 sets the resource type that the Infinispan Operator uses to handle the lifecycle of Infinispan clusters.
3 names the Infinispan cluster.
4 sets the number of pods in the Infinispan cluster.

2.3. Spinning Up Infinispan Clusters

Use the Infinispan Operator to create clusters of two or more Infinispan pods.

Procedure
  1. Create a custom resource that sets replicas: 3.

    For example, create a cr_minimal.yaml file as follows:

    $ cat > cr_minimal.yaml<<EOF
    apiVersion: infinispan.org/v1
    kind: Infinispan
    metadata:
      name: example-infinispan
    spec:
      replicas: 3
    EOF
  2. Apply your custom resources.

    $ oc apply -f cr_minimal.yaml
  3. Watch the Infinispan Operator create the Infinispan pods.

    $ oc get pods -w
    
    NAME                        READY  STATUS              RESTARTS   AGE
    example-infinispan-1        0/1    ContainerCreating   0          4s
    example-infinispan-2        0/1    ContainerCreating   0          4s
    example-infinispan-3        0/1    ContainerCreating   0          5s
    infinispan-operator-0       1/1    Running             0          3m
    example-infinispan-3        1/1    Running             0          8s
    example-infinispan-2        1/1    Running             0          8s
    example-infinispan-1        1/1    Running             0          8s
Next Steps

Try changing the value of replicas: and watching the Infinispan Operator scale the cluster up or down.

2.4. Verifying that Infinispan Pods Receive Clustered Views

Review log messages to verify that Infinispan pods successfully form clusters.

Procedure
  • Do either of the following:

    • Retrieve the cluster view from pod logs.

      $ oc logs example-infinispan-0 | grep ISPN000094
      
      INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|0] (1) [example-infinispan-0]
      
      INFO  [org.infinispan.CLUSTER] (jgroups-3,{example_crd_name-0) \
      ISPN000094: Received new cluster view for channel infinispan: \
      [example-infinispan-0|1] (2) [example-infinispan-0, example-infinispan-1]
    • Retrieve the custom resource type for the Infinispan Operator.

      $ oc get infinispan -o yaml

      The response indicates that Infinispan pods have received clustered views:

      conditions:
          - message: 'View: [example-infinispan-0, example-infinispan-1]'
            status: "True"
            type: wellFormed

Use oc wait with the wellFormed condition for automated scripts.

$ oc wait --for condition=wellFormed --timeout=240s infinispan/example-infinispan

2.5. Shutting Down and Restarting Infinispan Clusters

Gracefully shut down your Infinispan clusters before restarting them.

Cache service pods store permanent cache definitions in persistent volumes so they survive between cluster restarts.

Data Grid service pods store permanent cache definitions and also persist your data during cluster shutdown.

You should configure the storage size for Data Grid service pods to ensure that the persistent volume can hold all your data.

If the available container storage is less than the amount of memory available to Data Grid service pods, Infinispan writes an exception to logs and data loss occurs during shutdown.

Procedure
  • Set the value of replicas to 0 and apply your custom resources.

    spec:
      replicas: 0
  • To restart Infinispan clusters, set the value of replicas to the same number of pods that were in the cluster before you shut it down.

    For example, you shut down a cluster of 6 pods. When you restart the cluster, you must set:

    spec:
      replicas: 6

    This allows Infinispan to restore the distribution of data across the cluster. When all pods in the cluster are running, you can then scale it up or down.

3. Creating Infinispan Services with the Infinispan Operator

Infinispan services are stateful applications that provide flexible and robust in-memory data storage.

3.1. Cache Service

Cache service provides a volatile, low-latency data store that dramatically increases application response rates.

Cache service pods:

  • Synchronously distribute data across the cluster to ensure consistency.

  • Maintain single copies of cache entries to reduce size. You can replicate data for more resiliency.

  • Store cache entries off-heap and use eviction for JVM efficiency.

You can create multiple cache definitions with Cache service but only as copies of the default configuration.

Default cache configuration for Cache service pods is as follows:

<distributed-cache name="default" mode="SYNC" owners="1">
   <memory>
    <off-heap eviction="MEMORY" strategy="REMOVE"/>
   </memory>
  <partition-handling when-split="DENY_READ_WRITES" merge-policy="REMOVE_ALL" />
</distributed-cache>

3.2. Data Grid Service

Data Grid service provides a configurable Infinispan server distribution for Kubernetes.

  • Use with advanced capabilities like cross-site replication as well as indexing and querying.

  • Remotely access Data Grid service clusters from Hot Rod or REST clients and dynamically create caches using any Infinispan cache mode and configuration.

    Infinispan does not provide default caches for Data Grid service pods. However, you can use cache configuration templates to get started.

3.3. Creating Infinispan Services

Define the .spec.service.type resource to create Cache service and Data Grid service pods with the Infinispan Operator.

By default, the Infinispan Operator creates Infinispan clusters with Cache service pods.

Procedure
  1. Create a custom resource that sets the service type for Infinispan pods.

    For example, create Data Grid service clusters as follows:

    spec:
      ...
      service:
        type: DataGrid
  2. Specify additional resources for your pods as required.

  3. Apply the custom resources with oc apply -f.

  4. Use oc get pods -w to watch the Infinispan Operator apply the custom resources.

You cannot change the .spec.service.type resource after you create Infinispan clusters.

For example, if you create a cluster of Cache service pods, you cannot change the service type to Data Grid service. In this case you must create a new cluster with Data Grid service pods in a different Kubernetes namespace.

3.3.1. Cache Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  image: infinispan/server:latest
  replicas: 4
  service:
    type: Cache (1)
  security: (2)
    endpointSecretName: endpoint-identities
    endpointEncryption:
        type: secret
        certSecretName: tls-secret
  container: (3)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m"
    memory: 1Gi
  logging: (4)
    categories:
      org.infinispan: trace
      org.jgroups: trace
  expose: (5)
    type: LoadBalancer
1 creates Cache service pods.
2 encrypts Infinispan endpoint traffic. See Securing Infinispan Endpoints.
3 allocates resources to pods. See JVM, CPU, and Memory Resources.
4 configures logging. See Monitoring Infinispan Logs.
5 configures services for external traffic. See Making Infinispan Available to External Traffic.

3.3.2. Data Grid Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  image: infinispan/server:latest
  replicas: 6
  service:
    type: DataGrid (1)
    container:
      storage: 2Gi (2)
    sites: (3)
      local:
        expose:
          type: LoadBalancer
      backups:
      - name: google
        url: xsite://google.host:23456
      - name: azure
        url: openshift://api.azure.host:6443
        secretName: azure-identities
      - name: aws
        url: openshift://api.aws.host:6443
        secretName: aws-identities
  security: (4)
    endpointSecretName: endpoint-identities
    endpointEncryption:
      type: service
      certServiceName: service.beta.openshift.io
      certSecretName: served-tls-secret
  container: (5)
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "1000m"
    memory: 1Gi
  logging: (6)
    categories:
      org.infinispan: debug
      org.jgroups: debug
  expose: (7)
    type: LoadBalancer
1 creates Data Grid service pods.
2 configures the size of the persistent volume. See Storage Resources.
3 defines local and backup sites. See Configuring Cross-Site Replication.
4 encrypts Infinispan endpoint traffic. See Securing Infinispan Endpoints.
5 allocates resources to pods. See JVM, CPU, and Memory Resources.
6 configures logging. See Monitoring Infinispan Logs.
7 configures services for external traffic. See Making Infinispan Available to External Traffic.

4. Configuring Infinispan Pod Resources

You can allocate CPU and memory resources, specify JVM options, and configure storage for Infinispan pods.

4.1. JVM, CPU, and Memory Resources

spec:
  ...
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary" (1)
    cpu: "1000m" (2)
    memory: 1Gi (3)
1 specifies JVM options.
2 allocates CPU resources.
3 allocates memory resources.

You cannot pass JVM options or allocate CPU and memory after you create Infinispan pods. In this case you must delete the existing Infinispan pods and create new ones.

4.2. Storage Resources

spec:
  ...
  service:
    type: DataGrid
    container:
      storage: 2Gi (1)
1 configures the storage size for Data Grid service pods.

By default, the Infinispan Operator allocates 1Gi for storage for both Cache service and Data Grid service pods. You can configure storage size only for Data Grid service pods.

4.2.1. Persistent Volume Claims

The Infinispan Operator creates persistent volume claims for pods at:
/opt/infinispan/server/data

Persistent volume claims use the ReadWriteOnce (RWO) access mode.

5. Making Infinispan Available to External Traffic

To access Infinispan from outside Kubernetes, you must configure an external service.

5.1. Exposing External Services

Procedure
  1. Create a custom resource that exposes an external service.

  2. Apply the custom resources, for example:

    $ oc apply -f my_service.yaml
  3. Verify that the ${cluster_name}-external service is available.

    $ oc get services | grep external
    
    NAME                         TYPE            CLUSTER-IP       EXTERNAL-IP   PORT(S)
    example-infinispan-external  LoadBalancer    192.0.2.24       <none>        11222/TCP

    The preceding example shows an external service named example-infinispan-external of type LoadBalancer.

5.2. External Service Resources

spec:
  ...
  expose: (1)
    type: LoadBalancer (2)
1 exposes an external service at port 11222.
2 specifies one of the following service resource types:
NodePort

Suitable for locally running Kubernetes clusters.

LoadBalancer

Suitable for Kubernetes clusters where a load balancer service is available to handle external network traffic.

6. Configuring Authentication

Application users must authenticate with Infinispan pods to access data.

The Infinispan Operator generates default credentials and stores them in secrets. You can use the default credentials or add custom authentication secrets.

6.1. Retrieving Credentials

Get base64-encoded credentials from the authentication secret.

Default credentials

example-infinispan-generated-secret is the default authentication secret.
developer is the default application user.
operator is an internal user that interacts with Infinispan clusters.

Procedure
  • Retrieve credentials from the default authentication secret as follows:

    $ oc get secret example-infinispan-generated-secret -n my_namespace

    Do the following to base64 decode credentials:

    $ oc get secret example-infinispan-generated-secret -n my_namespace \
    -o jsonpath="{.data.identities\.yaml}" | base64 -D
    
    credentials:
    - username: developer
      password: dIRs5cAAsHIeeRIL
    - username: operator
      password: uMBo9CmEdEduYk24

6.2. Using Custom Authentication Secrets

Create a custom authentication secret and configure Infinispan pods to use it.

Procedure
  1. Create an authentication secret.

  2. Add the authentication secret to your OpenShift namespace.

    $ oc apply -f connect_secret.yaml
  3. Define a custom resource for the authentication secret.

    spec:
      ...
      security:
        endpointSecretName: connect-secret (1)
    1 specifies the authentication secret.
  4. Apply the custom resource with oc apply -f.

  5. Use oc get pods -w to watch the Infinispan Operator apply the custom resources.

6.2.1. Credentials Secrets

apiVersion: v1
kind: Secret
metadata:
  name: connect-secret (1)
type: Opaque (2)
stringData:
  identities.yaml: |- (3)
    credentials (4)
    - username: (5)
      password: (6)
1 names the authentication secret.
2 authentication secrets must be Opaque.
3 contains application users in YAML format.
4 lists credentials for application users.
  • You must include the operator user and specify a password. The Infinispan Operator requires these credentials to interact with Infinispan clusters.

  • Do not use example passwords from this documentation. You should always replace example credentials with ones that conform to the relevant security guidelines.

7. Securing Infinispan Endpoints

Configure Infinispan pods to encrypt network traffic between clients and endpoints with TLS certificates.

You can generate Red Hat OpenShift service certificates or use custom TLS certificates.

7.1. Generating Service Certificates for Encryption

The Infinispan Operator can generate Red Hat OpenShift service certificates that:

  • Consist of a certificate, tls.crt, and key, tls.key, in PEM format that OpenShift stores in a secret.

  • Are signed by the OpenShift CA, are valid for one year, and are automatically replaced before expiration.

Procedure
  1. Define custom resources to use service certificates for encryption.

    spec:
      ...
      security:
        endpointEncryption: (1)
                type: service (2)
                certServiceName: service.beta.openshift.io (3)
                certSecretName: tls-secret (4)
    1 encrypts traffic to and from Infinispan endpoints.
    2 configures Infinispan to service certificates.
    3 adds the OpenShift annotation.
    4 names the encryption secret.
  2. Apply the custom resource with oc apply -f.

  3. Use oc get pods -w to watch the Infinispan Operator apply the custom resources.

  4. Retrieve tls.crt from the secret to create truststores for clients.

    $ oc get secret tls-secret -o jsonpath='{.data.tls\.crt}' |  \
    base64 -d > tls.crt

7.2. Adding TLS Certificates to Encryption Secrets

Add a PKCS12 keystore or TLS certificate/key pair to a secret and configure Infinispan to use it for endpoint encryption.

Procedure
  1. Create an encryption secret.

  2. Add the encryption secret to your OpenShift namespace.

    $ oc apply -f tls_secret.yaml
  3. Define a custom resource for the encryption secret.

    spec:
      ...
      security:
        endpointEncryption: (1)
                type: secret (2)
                certSecretName: tls-secret (3)
    1 encrypts traffic to and from Infinispan endpoints.
    2 configures Infinispan to use secrets that contain encryption certificates.
    3 names the encryption secret.
  4. Apply the custom resource with oc apply -f.

  5. Use oc get pods -w to watch the Infinispan Operator apply the custom resources.

7.2.1. Certificate Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
data:
    tls.key:  "LS0tLS1CRUdJTiBQUk ..." (1)
    tls.crt: "LS0tLS1CRUdJTiBDRVl ..." (2)
1 adds a base64 encoded TLS key.
2 adds a base64 encoded TLS certificate.

7.2.2. Keystore Secrets

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: Opaque
stringData:
    alias: server (1)
    password: password (2)
data:
    keystore.p12:  "MIIKDgIBAzCCCdQGCSqGSIb3DQEHA..." (3)
1 specifies an alias for the keystore.
2 specifies a password for the keystore.
3 adds a base64 encoded keystore.

8. Configuring Cross-Site Replication

Infinispan allows you to back up data across geographic locations. For example, you have a Infinispan cluster running on Red Hat OpenShift in a data center in New York (SiteA). You have another Infinispan cluster running on Red Hat OpenShift in a data center in San Francisco (SiteB).

Configuring cross-site replication enables the Infinispan cluster at SiteA to back up data to the Infinispan cluster at SiteB.

Prerequisites

8.1. Creating Minikube Site Access Secrets

If you run the Infinispan Operator in Minikube, you should create secrets that contain the files that allow different instances of Minikube to authenticate with each other.

Procedure
  1. Create secrets on each site that contain ca.crt, client.crt, and client.key from your Minikube installation.

    For example, do the following on SiteB:

    kubectl create secret generic site-a-secret \
        --from-file=certificate-authority=/opt/minikube/.minikube/ca.crt \
        --from-file=client-certificate=/opt/minikube/.minikube/client.crt \
        --from-file=client-key=/opt/minikube/.minikube/client.key

8.2. Creating Service Account Tokens

Traffic between independent OpenShift installations occurs through a Kubernetes API. OpenShift Container Platform clusters use tokens to authenticate with and access the API.

To enable cross-site replication between Infinispan clusters you must add tokens to the namespace on each site. For example, SiteA needs a secret with the token for SiteB. SiteB also needs a secret with the token for SiteA.

Procedure
  1. Create service account tokens on each OpenShift instance.

    For example, on SiteA create a token for SiteB as follows:

    $ oc create sa site-a
    serviceaccount/site-a created
  2. Find the token in the list of secrets in your OpenShift namespace.

    The following example shows two site-a tokens:

    $ oc get secrets
    
    NAME                       TYPE
    ...
    site-a-token-8j6d8         kubernetes.io/service-account-token
    site-a-token-g2brv         kubernetes.io/service-account-token
  3. Retrieve a token to allow backup sites to authenticate with the local OpenShift cluster.

    The following example shows an encoded string in the token field:

    $ oc describe secret site-a-token-8j6d8
    
    Name:         site-a-token-8j6d8
    Namespace:    local-operators
    Labels:       <none>
    Annotations:  kubernetes.io/created-by: openshift.io/create-dockercfg-secrets
                  kubernetes.io/service-account.name: site-a
                  kubernetes.io/service-account.uid: c22bc8d6-ffd8-11e9-adb6-0a8e23c49339
    
    Type:  kubernetes.io/service-account-token
    
    Data
    ====
    ca.crt:          4634 bytes
    namespace:       15 bytes
    service-ca.crt:  5750 bytes
    token:           eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
  4. Create secrets on each site that contain service account tokens for the corresponding backup site.

    1. Log in to OpenShift Container Platform at SiteB.

    2. Add the service account token to a site-a-token secret.

      oc create secret generic site-a-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
    3. Repeat the preceding steps to create a site-b-token secret on SiteA.

After you add service account tokens to each backup site, the OpenShift instances can authenticate with each other so that Infinispan clusters can form cross-site views.

8.3. Adding Site Definitions to Infinispan

Configure Infinispan clusters as sites so that they can communicate over a dedicated JGroups transport channel for replicating data.

Procedure
  1. Create custom resources for Infinispan at each site.

    You must configure each site as a backup for the other. For example, you create site_a.yaml to configure SiteA as the local site and SiteB as the backup site. You must also then create site_b.yaml to configure SiteB as the local site and SiteA as the backup site. See Cross-Site Replication Resources.

  2. Configure Infinispan clusters at each site, for example:

    1. Log in to OpenShift Container Platform at SiteA.

    2. Apply the custom resources for SiteA.

      $ oc apply -f site_a.yaml
    3. Log in to OpenShift Container Platform at SiteB.

    4. Apply the custom resources for SiteB.

      $ oc apply -f site_b.yaml
  3. Verify that Infinispan clusters form a cross-site view.

    For example, do the following on SiteA:

    $ oc logs example-infinispan-0 | grep x-site
    
    INFO  [org.infinispan.XSITE] (jgroups-5,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [SiteB]
    INFO  [org.infinispan.XSITE] (jgroups-7,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [SiteB, SiteA]

8.3.1. Cross-Site Replication Resources

spec:
  ...
  service:
    type: DataGrid (1)
    sites:
      local:
        name: SiteA (2)
        expose:
          type: LoadBalancer (3)
      backups:
      - name: SiteB (4)
        url: openshift://api.site-b.devcluster.openshift.com:6443 (5)
        secretName: site-b-token (6)
1 specifies the DataGrid service. Infinispan supports cross-site replication with the DataGrid service only.
2 names the site where the Infinispan cluster runs, the local site.
3 defines the externally exposed service.
  • Use NodePort for local clusters on the same network. For example, if you use Minikube for testing purposes.

  • Use LoadBalancer for independent OpenShift clusters.

4 names the site to which the Infinispan cluster replicates data, the backup site.
5 specifies the URL to access the backup site.
  • Use minikube:// if the backup site is a Minikube instance.

  • Use openshift:// if the backup site is an OpenShift cluster. You should specify the URL of the Kubernetes API for the backup site.

6 specifies the secret to access backup sites.

This secret contains different authentication objects, depending on your Kubernetes environment.

9. Monitoring Infinispan Logs

Set logging categories to different message levels to monitor, debug, and troubleshoot Infinispan pods.

9.1. Configuring Infinispan Logging

Procedure
  1. Create custom resources to configure Infinispan logging.

  2. Apply the custom resources.

    $ oc apply -f my_logging.yaml
  3. Retrieve logs from Infinispan pods as required.

    $ oc logs -f ${POD_NAME}
Reference

Logging Resources

9.1.1. Logging Resources

spec:
  ...
  logging: (1)
    categories: (2)
      org.infinispan: debug (3)
      org.jgroups: debug
1 configures Infinispan logging.
2 adds logging categories.
3 names logging categories and levels.

The root logging category is org.infinispan and is INFO by default.

Infinispan Operator supports Infinispan logging categories. Refer to the Infinispan documentation for more information.

9.1.2. Log Levels

Log levels indicate the nature and severity of messages.

Log level Description

trace

Provides detailed information about running state of applications. This is the most verbose log level.

debug

Indicates the progress of individual requests or activities.

info

Indicates overall progress of applications, including lifecycle events.

warn

Indicates circumstances that can lead to error or degrade performance.

error

Indicates error conditions that might prevent operations or activities from being successful but do not prevent applications from running.

10. Reference

Find useful information for Infinispan clusters that you create with the Infinispan Operator.

10.1. Services

The Infinispan Operator automatically creates services to route network traffic. These services use the names that you give to your Infinispan clusters.

Internal services
  • Allow Infinispan nodes to discover each other and form clusters.

  • Provide access to Infinispan endpoints from clients in the same Kubernetes namespace.

    Service Port Protocol Description

    ${cluster_name}

    11222

    TCP

    Infinispan endpoints

    ${cluster_name}-ping

    8888

    TCP

    Cluster discovery

External services

Provide access to Infinispan endpoints from clients outside Kubernetes or in different namespaces.

You must expose external services with the Infinispan Operator. External services are not available by default.

Service Port Protocol Description

${cluster_name}-external

11222

TCP

Infinispan endpoints