Reactive Big Data on OpenShift In-Memory Data Grids

Before we start with any of the demos, it’s necessary to run some set up steps:

    1. Check out git repository:

        git clone https://github.com/galderz/swiss-transport-datagrid

    2. Install OpenShift Origin or Minishift to get an OpenShift environment running in your own 

        machine. I decided to use OpenShift Origin, so the instructions below are tailored for that 

        environment, but similar instructions could be used with Minishift.

    3. Install Anaconda for Python 3, this is required to run Jupyter notebook for plotting.

Once the set up is complete, it’s time to talk about the demos before we run them.

Both demos shown below work with the same application domain: swiss rail transport systems. In this domain, we differentiate between physical stations, trains, station boards which are located in stations, and finally stops, which are individual entries in station boards.

The first demo is about working with real-time data from station boards around the country and presenting a centralised dashboard of delayed trains around the country. The following diagrams shows how the following components interact with each other to achieve this:

Infinispan, which provides the in-memory data grid storage, and Vert.x, which provides the glue for the centralised delayed dashboard to interact with Infinispan, all run within OpenShift cloud. 

Within the cloud, the Injector verticle cycles through station board data and injects it into Infinispan. Also within the cloud, a Vert.x verticle that uses Infinispan’s Continuous Query to listen for station board entries that are delayed, and these are pushed into the Vert.x event bus, which in turn, via a SockJS bridge, get consumed via WebSockets from the dashboard. The centralised dashboards is written with JavaFX and runs outside the cloud.

To run the demo, do the following:

    1. Start OpenShift Origin if you’ve not already done so:

        oc cluster up --public-hostname=127.0.0.1

    2. Deploy all the OpenShift cloud components:

        cd ~/swiss-transport-datagrid

        ./deploy-all.sh

    3. Open the OpenShift console and verify that all pods are up.

    4. Load github repository into your favourite IDE and run

        delays.query.continuous.fx.FxApp Java FX application. This will load the

        centralised dashboard. Within seconds delayed trains will start appearing. For example:

The first demo is focused on how you can use Infinispan for doing offline analytics. In particular, this demo tries to answer the following question:

Q. What is the time of the day when there is the biggest ratio of delayed trains?

Once again, this demo runs on top of OpenShift cloud, uses Infinispan as in-memory data grid for storage and Vert.x for glueing components together.

To answer this question, Infinispan data grid will be loaded with 3 weeks worth of data from station boards using a Vert.x verticle. Once the data is loaded, the Jupyter notebook will invoke an HTTP restful endpoint which will invoke an Vert.x verticle called AnalyticsVerticle. 

This verticle will invoke a remote server task which will use Infinispan Distributed Java Streams to calculate the two pieces of information required to answer the question: per hour, how many trains are going through the system, and out of those, how many are delayed.

An important aspect to bear in mind about this server tasks is that it will only be executed in one of the nodes in the cluster. It does not matter which one. In turn, this node will will ship the lambdas required to do the computation to each of the nodes so that they can executed against their local data. The other nodes will reply with the results and the node where the server task was invoked will aggregate the results.

The results will be sent back to the originating invoker, the Jupyter notebook which will plot the results. The following diagrams shows how the following components interact with each other to achieve this:

Here is the demo step-by-step guide:

    1. Start OpenShift Origin and deploy all components as shown in previous demo.

    2. Start the Jupyter notebook:

        cd ~/swiss-transport-datagrid/analytics/analytics-jupyter

        ~/anaconda/bin/jupyter notebook

    3.  Once the notebook opens, click open live-demo.ipynb notebook and execute each of the cells in order. You should end up seeing a plot like this:

So, the answer to the question:

Q. What is the time of the day when there is the biggest ratio of delayed trains?

https://www.blogger.com/blogger.g?blogID=5717179571414330874https://www.blogger.com/blogger.g?blogID=5717179571414330874[]https://www.blogger.com/blogger.g?blogID=5717179571414330874

is 2am! That’s because last connecting trains of the day wait for each other to avoid leaving passengers stranded.

This has been a summary of the demos that I presented at Great Indian Developer Summit with the intention of getting you running these demos as quickly as possible. The repository contains more detailed information of these demos. If there’s anything unclear or any of the instructions above are not working, please let us know!

Once again, a very special thanks to Alexandre Masselot for being the inspiration for these demos. Merci @Alex!!

Over the next few months we will be enhancing the demo and hopefully we’ll be able to do some more live demonstrations at other conferences.

Cheers,

Galder