Monday, 11 November 2019
One of the biggest changes in Infinispan 10 is the new server, which replaces the WildFly-based server we had been using up until 9.x.
This is the first of a series of blog posts which will describe the new server, how to use it, how to configure it and how to deploy it in your environment. More specifically, this post will focus mostly on the reasons behind the change, while the next ones will be of a more practical nature.
Infinispan has had a server implementing the Hot Rod protocol since 4.1. Originally it was just a main class which bootstrapped the server protocol. It was configured via the same configuration file used by the embedded library, it had no security and only handled Hot Rod.
Over time both a RESTful HTTP and a Memcached protocol were added and could be bootstrapped in the same way.
While the server bootstrap code was trivial, it was not going to scale to support all the things we needed (security, management, provisioning, etc). We therefore decided to build our next server on top of the very robust foundation provided by WildFly (aka, the application server previously known as JBoss AS 7), which made its first appearance in 5.3.
Integration with WildFly’s management model was not trivial but it gave us all of the things we were looking for and more, such as deployments, data sources, CLI, console, etc. It also came with a way to provision multiple nodes and manage them from a central controller, i.e. domain mode. All of these facilities however came at the cost of a lot of extra integration code to maintain as well as a larger footprint, both in terms of memory and storage use, caused by a number of subsystems which we had to carry along, even though we didn’t use them directly.
Fast-forward several versions, and the computing landscape has changed considerably: services are containerized, they are provisioned and managed via advanced orchestration tools like Kubernetes or via configuration management tools like Ansible and the model we were using was overlapping (if not altogether clashing) with the container model, where global configuration is immutable and managed externally.
With the above in mind, we have therefore decided to reboot our server implementation. During planning and development it has been known affectionately as ServerNG, but nowadays it is just the Infinispan Server. The WildFly-based server is now the legacy server.
The new server separates global configuration (clustering, endpoints, security) from the configuration of dynamic resources like caches, counters, etc. This means that global configuration can be made immutable while the ,mutable configuration is stored separately in the global persistence location. In a containerized environment you will place the persistence location onto a volume that will survive restarts.
Starting a two-node cluster using the latest version of the server image is easy:
$ docker run --name ispn1 --hostname ispn1 -e USER=admin -e PASS=admin -p 11222:11222 infinispan/server $ docker run --name ispn2 --hostname ispn2 -e USER=admin -e PASS=admin -p 11322:11222 infinispan/server
The two nodes will discover each other, as can be seen from the logs:
15:58:21,201 INFO [org.infinispan.CLUSTER] (jgroups-5,ispn-1-42736) ISPN000094: Received new cluster view for channel infinispan: [ispn-1-42736|1] (2) [ispn-1-42736, ispn-2-51789] 15:58:21,206 INFO [org.infinispan.CLUSTER] (jgroups-5,ispn-1-42736) ISPN100000: Node ispn-2-51789 joined the cluster
Next we will connect to the cluster using the CLI:
$ docker run -it --rm infinispan/server /opt/infinispan/bin/cli.sh [disconnected]> connect http://172.17.0.2:11222 Username: admin Password: ***** [ispn-1-42736@infinispan//containers/DefaultCacheManager]>
Next we will create a distributed cache and select it for future operations:
[ispn-1-42736@infinispan//containers/DefaultCacheManager]> create cache --template=org.infinispan.DIST_SYNC distcache [ispn-1-42736@infinispan//containers/DefaultCacheManager]> cache distcache [ispn-1-42736@infinispan//containers/DefaultCacheManager/caches/distcache]>
Let’s insert some data now:
[ispn-1-42736@infinispan//containers/DefaultCacheManager/caches/distcache]> put k1 v1 [ispn-1-42736@infinispan//containers/DefaultCacheManager/caches/distcache]> put k2 v2 [ispn-1-42736@infinispan//containers/DefaultCacheManager/caches/distcache]> ls k2 k1 [ispn-1-42736@infinispan//containers/DefaultCacheManager/caches/distcache]> get k1 v1
Now let’s use the RESTful API to fetch one of the entries:
$ curl --digest -u admin:admin http://localhost:11222/rest/v2/caches/distcache/k2 v2
Since we didn’t map persistent volumes to our containers, both the cache and its contents will be lost when we terminate the containers.
In the next blog post we will look at configuration and persistence in more depth.
Tuesday, 24 July 2018
The Infinispan Spark connector version 0.8 has been released and is available in Maven central and SparkPackages.
This is a maintenance only release to bring compatibility with Spark 2.3 and Infinispan 9.3.
For feedback and general help, please use the Infinispan chat.
Tags: release spark server
Wednesday, 07 March 2018
HTTP has become one of the most successful and heavily used network protocols around the world. Version 1.0 was created in 1996 and received a minor update 3 years later. But it took more than a decade to create HTTP/2 (which was approved in 2015). Why did it take so long? Well, I wouldn’t tell you all the truth if I didn’t mention an experimental protocol, called SPDY. SPDY was primarily focused on improving performance. The initial results were very promising and inside Google’s lab, the developers measured 55% speed improvement. This work and experience was converted into HTTP/2 proposal back in 2012. A few years later, we can all use HTTP/2 (sometimes called h2) along with its older brother - HTTP/1.1.
HTTP/1.1 is a text-based protocol. Sometimes this is very convenient, since you can use low level tools, such as Telnet, for hacking. But it doesn’t work very well for transporting large, binary payloads. HTTP/2 solves this problem by using a completely redesigned architecture. Each HTTP message (a request or a response) consists of one or more frames. A frame is the smallest portion of data travelling through a TCP connection. A set of messages is aggregated into a, so called stream.
HTTP/2 allows to lower the number of physical connections between the server and the client by multiplexing logical connections into one TCP connection. Streams allow the server to recognize, which frame belongs to which conversation.
There are two ways for starting an HTTP/2 conversation.
The first one, and the most commonly used one, is TLS/ALPN. During TLS handshake the server and the client negotiate protocol for further communication. Unfortunately JDK below 9 doesn’t support it by default (there are a couple of workarounds but please refer to your favorite HTTP client’s manual to find some suggestions).
The second one, much less popular, is so called plain text upgrade. During HTTP/1.1 conversation, the client issues an HTTP/1.1 Upgrade header and proposes new conversation protocol. If the server agrees, they start using it. If not, they stick with HTTP/1.1.
The good news is that Infinispan supports both those upgrade paths. Thanks to the ALPN Hack Engine (the credit goes to Stuart Douglas from the Wildfly Team), we support TLS/ALPN without any bootstrap classpath modification.
Infinispan’s REST server already supports plain text upgrades out of the box. TLS/ALPN however, requires additional configuration since the server needs to use a Keystore. In order to make it even more convenient, we support generating keystores automatically when needed. Here’s an example showing how to configure a security realm:
The next step is to bind the security realm to a REST endpoint:
You may also use one of our configuration examples. The easiest way to get it working is to use our Docker image:
Let’s explain a couple of things from the command above:
-e "APP_USER=test" - This is a user name we will be used for REST authentication.
-e "APP_PASS=test" - Corresponding password.
../../docs/examples/configs/standalone-rest-ssl.xml - Here is a ready-to-go configuration with REST and TLS/ALPN support
Unfortunately, HTTP/2 functionality has been broken in 9.2.0.Final. But we promise to fix it as soon as we can :) Please use 9.1.5.Final in the meantime.
Curl is one of my favorite tools. It’s very simple, powerful, and… it supports HTTP/2. Assuming that you already started Infinispan server using
docker run command, you can put something into the cache:
Once, it’s there, let’s try to get it back:
Let’s analyze CURL switches one by one:
-k - Ignores certificate validation. All automatically generated certificates and self-signed and not trusted by default.
-v - Debug logging.
-u test:test - Username and password for authentication.
-d test - This is the payload when invoking HTTP POST.
-H “Accept: text/plain” - This tells the server what type of data we’d like to get in return.
I hope you enjoyed this small tutorial about HTTP/2. I highly encourage you to have a look at the links below to learn some more things about this topic. You may also measure the performance of your app when using HTTP/1.1 and HTTP/2. You will be surprised!
Tags: docker server http/2 rest
Sunday, 10 September 2017
In recent years Software as a Service concept has gained a lot of traction. I’m pretty sure you’ve used it many times before. Let’s take a look at a practical example and explain what’s going on behind the scenes.
Imagine a very simple photo album application hosted within the cloud. Upon the first usage you are asked to create an account. Once you sign up, a new tenant is created for you in the application with all necessary details and some dedicated storage just for you. Starting from this point you can start using the album - download and upload photos.
The software provider that created the photo album application can also celebrate. They have a new client! But with a new client the system needs to increase its capacity to ensure it can store all those lovely photos. There are also other concerns - how to prevent leaking photos and other data from one account into another? And finally, since all the content will be transferred through the Internet, how to secure transmission?
As you can see, multi-tenancy is not that easy as it would seem. The good news is that if it’s properly configured and secured, it might be beneficial both for the client and for the software provider.
Let’s think again about our photo album application for a moment. Whenever a new client signs up we need to create a new account for him and dedicate some storage. Translating that into Infinispan concepts this would mean creating a new CacheContainer. Within a CacheContainer we can create multiple Caches for user details, metadata and photos. You might be wondering why creating a new Cache is not sufficient? It turns out that when a Hot Rod client connects to a cluster, it connects to a CacheContainer exposed via a Hot Rod Endpoint. Such a client has access to all Caches. Considering our example, your friends could possibly see your photos. That’s definitely not good! So we need to create a CacheContainer per tenant. Before we introduced Multi-tenancy, you could expose each CacheContainer using a separate port (using separate Hot Rod Endpoint for each of them). In many scenarios this is impractical because of proliferation of ports. For this reason we introduced the Router concept. It allows multiple clients to access their own CacheContainers through a single endpoint and also prevents them from accessing data which doesn’t belong to them. The final piece of the puzzle is transmitting sensitive data through an unsecured channel such as the Internet. The use of TLS encryption solves this problem. The final outcome should look like the following:
The Router component on the diagram above is responsible for recognizing data from each client and redirecting it to the appropriate Hot Rod endpoint. As the name implies, the router inspects incoming traffic and reroutes it to the appropriate underlying CacheContainer. To do this it can use two different strategies depending on the protocol: TLS/SNI for the Hot Rod protocol, matching each server certificate to a specific cache container and path prefixes for REST. The SNI strategy detects the SNI Host Name (which is used as tenant) and also requires TLS certificates to match. By creating proper trust stores we can match which tenant can access which CacheContainers. URL path prefix is very easy to understand, but it is also less secure unless you enable authentication. For this reason it should not be used in production unless you know what you are doing (the SNI strategy for the REST endpoint will be implemented in the near future). Each client has its own unique REST path prefix that needs to be used for accessing the data (e.g. http://127.0.0.1:8080/rest/client1/fotos/2).
Confused? Let’s clarify this with an example.
The first step is to generate proper key/trust stores for the server and client:
The next step is to configure the server. The snippet below shows only the most important parts:
Let’s analyze the most critical lines:
7, 15 - We need to add generated key stores to the server identities
25, 30 - It is highly recommended to use separate CacheContainers
38, 39 - A Hot Rod connector (but without socket binding) is required to provide proper mapping to CacheContainer. You can also use many useful settings on this level (like ignored caches or authentication).
42 - Router definition which binds into default Hot Rod and REST ports.
44 - 46 - The most important bit which states that only a client using SSLRealm1 (which uses trust store corresponding to client_1_server_keystore.jks) and TLS/SNI Host name client-1 can access Hot Rod endpoint named multi-tenant-hotrod-1 (which points to CacheContainer multi-tenancy-1).
Hint: You might be interested in looking at our previous blog posts about hosting Infinispan on OpenShift. You may find them at the bottom of the page.
So far we’ve learned how to create and configure a new CacheContainer per tenant. But we also need to remember that system capacity needs to be increased with each new tenant. OpenShift is a perfect tool for scaling the system up and down. The configuration we created in the previous step almost matches our needs but needs some tuning.
As we mentioned earlier, we need to encrypt transport between the client and the server. The main disadvantage is that OpenShift Router will not be able to inspect it and take routing decisions. A passthrough Route fits perfectly in this scenario but requires creating TLS/SNI Host Names as Fully Qualified Application Names. So if you start OpenShift locally (using oc cluster up) the tenant names will look like the following: client-1-fotoalbum.192.168.0.17.nip.io.
We also need to think how to store generated key stores. The easiest way is to use Secrets:
Finally, a full DeploymentConfiguration:
If you’re interested in playing with the demo by yourself, you might find a working example here. It mainly targets OpenShift but the concept and configuration are also applicable for local deployment.
Tags: security hotrod server multi-tenancy rest
Monday, 03 April 2017
The Infinispan Spark connector offers seamless integration between Apache Spark and Infinispan Servers. Apart from supporting Infinispan 9.0.0.Final and Spark 2.1.0, this release brings many usability improvements, and support for another major Spark API.
The connector no longer uses a java.util.Properties object to hold configuration, that’s now duty of org.infinispan.spark.config.ConnectorConfiguration, type safe and both Java and Scala friendly:
The previous version introduced the possibility of filtering an InfinispanRDD by providing a Query instance, that required going through the QueryDSL which in turn required a properly configured remote cache.
It’s now possible to simply use an Ickle query string:
Support for reading from a Cache with protobuf encoding was present in the previous connector version, but now it’s possible to also write using protobuf encoding and also have protobuf schema registration automatically handled.
To see this in practice, consider an arbitrary non-Infinispan based RDD<Integer, Hotel> where Hotel is given by:
In order to write this RDD to Infinispan it’s just a matter of doing:
Internally the connector will trigger the auto-generation of the .proto file and message marshallers related to the configured entity(ies) and will handle registration of schemas in the server prior to writing.
The Splitter is the interface responsible to create one or more partitions from a Infinispan cache, being each partition related to one or more segments. The Infinispan Spark connector now can be created using a custom implementation of Splitter allowing for different data partitioning strategies during the job processing.
Scala 2.10 support was removed, Scala 2.11 is currently the only supported version. Scala 2.12 support will follow https://issues.apache.org/jira/browse/SPARK-14220
It is possible to configure the InfinispanInputDStream with an extra boolean parameter to receive the current cache state as events.
The Infinispan Spark connector now ships with support for Spark’s Dataset API, with support for pushing down predicates, similar to rdd.filterByQuery. The entry point of this API is the Spark session:
To create an Infinispan based Dataframe, the "infinispan" data source need to be used, along with the usual connector configuration:
From here it’s possible to use the untyped API, for example:
or execute SQL queries by setting a view:
In both cases above, the predicates and the required columns will be converted to an Infinispan Ickle filter, thus filtering data at the source rather than at Spark processing phase.
Tags: spark server
Tuesday, 21 March 2017
In the latest 9.0.0.CR3 version, the Infinispan REST endpoint is secured by default, and in order to facilitate remote access, the Docker image has some changes related to the security.
The image now creates a default user login upon start; this user can be changed via environment variables if desired:
You can check if the settings are in place by manipulating data via REST. Trying to do a curl without credentials should lead to a 401 response:
So make sure to always include the credentials from now on when interacting with the Rest endpoint! If using curl, this is the syntax:
And that’s all for this post. To find out more about the Infinispan Docker image, check the documentation, give it a try and let us know if you have any issues or suggestions!
Tags: docker security server rest
Monday, 05 December 2016
In the previous post we showed how to manipulate the Infinispan Docker container configuration at both runtime and boot time.
Before diving into multi-host Docker usage, in this post we’ll explore how to create multi-container Docker applications involving Infinispan with the help of Docker Compose.
For this we’ll look at a typical scenario of an Infinispan server backed by an Oracle database as a cache store.
All the code for this sample can be found on github.
In order to have a cache with persistence with Oracle, we need to do some configuration: configure the driver in the server, create the data source associated with the driver, and configure the cache itself with JDBC persistence.
Let’s take a look at each of those steps:
The driver (ojdbc6.jar) should be downloaded and placed in the 'driver' folder of the sample project.
The module.xml declaration used to make it available on the server is as follows:
The data source is configured in the "datasource" element of the server configuration file as shown below:
and inside the "datasource/drivers" element, we need to declare the driver:
From now on, without using Docker we’d be ready to download and install Oracle following the specific instructions for your OS, then download the Infinispan Server, edit the configuration files, copy over the driver jar, figure out how to launch the database and server, taking care not to have any port conflicts.
If it sounds too much work, it’s because it really is. Wouldn’t it be nice to have all these wired together and launched with a simple command line? Let’s take a look at the Docker way next.
Docker Compose is a tool part of the Docker stack to facilitate configuration, execution and management of related Docker containers.
By describing the application aspects in a single yaml file, it allows centralized control of the containers, including custom configuration and parameters, and it also allows runtime interactions with each of the exposed services.
Our Docker Compose file to assemble the application is given below:
It contains two services:
one called oracle that uses the wnameless/oracle-xe-11g Docker image, with an environment variable to allow remote connections.
another one called *infinispan* that uses version 8.2.5.Final of the Infinispan Server image. It is launched with a custom command pointing to the changed configuration file and it also mounts two volumes in the container: one for the driver and its module.xml and another for the folder holding our server xml configuration.
To start the application, just execute
To inspect the status of the containers:
To follow the Infinispan server logs, use:
Infinispan usually starts faster than the database, and since the server waits until the database is ready (more on that later), keep an eye in the log output for "Infinispan Server 8.2.5.Final (WildFly Core 2.0.10.Final) started". After that, both Infinispan and Oracle are properly initialized.
Let’s insert a value using the Rest endpoint from Infinispan and verify it was saved to the Oracle database:
To check the Oracle database, we can attach to the container and use Sqlplus:
When dealing with dependent containers in Docker based environments, it’s highly recommended to make the connection obtention between parties robust enough so that the fact that one dependency is not totally initialized doesn’t cause the whole application to fail when starting.
Although Compose does have a depends_on instruction, it simply starts the containers in the declared order but it has no means to detected when a certain container is fully initialized and ready to serve requests before launching a dependent one.
One may be tempted to simply write some glue script to detect if a certain port is open, but that does not work in practice: the network socket may be opened, but the background service could still be in transient initialization state.
The recommended solution for this it to make whoever depends on a service to retry periodically until the dependency is ready. On the Infinispan + Oracle case, we specifically configured the data source with retries to avoid failing at once if the database is not ready:
When starting the application via Compose you’ll notice that Infinispan print some WARN with connection exceptions until Oracle is available: don’t panic, this is expected!
Docker Compose is a powerful and easy to use tool to launch applications involving multiple containers: in this post it allowed to start Infinispan plus Oracle with custom configurations with a single command. It’s also a handy tool to have during development and testing phase of a project, specially when using/evaluating Infinispan with its many possible integrations.
Tags: compose jdbc docker persistence server modules oracle cache store
Friday, 28 October 2016
In the previous post we introduced the improved Docker image for Infinispan and showed how to run it with different parameters in order to create standalone, clustered and domain mode servers.
This post will show how to address more advanced configuration changes than swapping the JGroups stack, covering cases like creating extra caches or using a pre-existent configuration file.
Since the Infinispan server is based on Wildfly, it also supports the Command Line Interface (CLI) to change configurations at runtime.
Let’s consider an example of a custom indexed cache with Infinispan storage. In order to configure it, we need 4 caches, one cache to hold our data, called testCache and other three caches to hold the indexes: LuceneIndexesMetadata, LuceneIndexesData and LuceneIndexesLocking.
This is normally achieved by adding this piece of configuration to the server xml:
This is equivalent to the following script:
To apply it to the server, save the script to a file, and run:
where CONTAINER is the id of the running container.
Everything that is applied using the CLI is automatically persisted in the server, and to check what the script produced, use the command to dump the config to a local file called config.xml.
Check the file config.xml: it should contain all four caches created via the CLI.
Most of the time changing configuration at runtime is sufficient, but it may be desirable to run the server with an existent xml, or change configurations that cannot be applied without a restart. For those cases, the easier option is to mount a volume in the Docker container and start the container with the provided configuration.
This can be achieved with Docker’s volume support. Consider an xml file called customConfig.xml located on a local folder /home/user/config. The following command:
will create a volume inside the container at the /opt/jboss/infinispan-server/standalone/configuration/extra/ directory, with the contents of the local folder /home/user/config.
The container is then launched with the entrypoint extra/customConfig, which means it will use a configuration named customConfig located under the extra folder relative to where the configurations are usually located at /opt/jboss/infinispan-server/standalone/configuration.
Tags: docker server configuration cli
Wednesday, 20 July 2016
The Infinispan Docker image has been improved, making it easier to run Infinispan Servers in clustered, domain and standalone modes, with different protocol stacks.
In this blog post we’ll show a few usage scenarios and how to combine it with the jgroups-gossip image to create Infinispan Server clusters in docker based environments.
==== Getting started
By default the container runs in clustered mode, and to start a node simply execute:
Bringing a second container will cause it to form a cluster.The membership can be verified by running a command directly in the newly launched container:
==== Using a different JGroups stack
The command above creates a cluster with the default JGroups stack (UDP), but it’s possible to pick another one provided it’s supported by the server. For example, to use TCP:
==== Running on cloud environments
We recently dockerized the JGroups Gossip Router to be used as an alternative discovery mechanism in environments where multicast is not enabled, such as cloud environments.
Employing a gossip router will enable discovery via TCP, where the router acts as a registry: each member will register itself in this registry upon start and also discover other members.
The gossip router container can be launched by:
Take note of the address where the router will bind to, it’s needed by the Infinispan nodes. The address can be easily obtained by:
Finally we can now launch our cluster specifying the tcp-gossip stack with the location of the gossip router:
==== Launching Standalone mode
Passing an extra parameter allows to run a server in standalone (non-clustered) mode:
==== Server Management Console in Domain mode
Domain mode is a special case of clustered mode (and currently a requirement to use the Server Management Console), that involves launching a domain controller process plus one or more host controller processes. The domain controller does not hold data, it is used as a centralized management process that can replicate configuration and provision servers on the host controllers.
Running a domain controller is easily achievable with a parameter:
Once the domain controller is running, it’s possible to start one or more host controllers. In the default configuration, each host controller has two Infinispan server instances:
The command line interface can be used to verify the hosts managed in the domain:
It should output all the host names that are part of the domain, including the master (domain controller):
To get access to the Management console, use credentials admin/admin and go to port 9990 of the domain controller, for example: http://172.17.0.2:9990/
The image is built on Dockerhub shortly after each Infinispan release (stable and unstable), and the improvements presented in this post are available for Infinispan 9.0.0.Alpha3 and Infinispan 8.2.3.Final. As a reminder, make sure to pick the right version when launching containers:
The image was created to be flexible and easy to use, but if something is not working for you or if you have any suggestions to improve it, please report it at https://github.com/jboss-dockerfiles/infinispan/issues/
Tags: docker console domain mode server jgroups
Wednesday, 09 December 2015
The connector allows the Infinispan Server to become a data source for Apache Spark, for both batch jobs and stream processing, including read and write.
In this release, the highlight is the addition of two new operators to the RDD that support filtering using native capabilities of Infinispan. The first one is filterByQuery:
The second operator was introduced to replace the previous configuration based filter factory name, and was extended to support arbitrary parameters:
The connector has also been updated to be compatible with Spark 1.5.2 and Infinispan 8.1.0.Final.
For more details including full list of changes and download info please visit the Connectors Download section. The project Github contains up-to-date info on how to get started with the connector, also make sure to try the included docker based demo. To report any issue or to request new features, use the new dedicated issue tracker. We’d love to get your feedback!
Tags: spark server