Friday, 07 April 2017
In Memory Data Grid Patterns Demos from Devoxx France!
Devoxx France 2017 was a blast!! Emmanuel and I would like to thank all attendees to our in-memory data grids patterns talk. The room was full and we thoroughly enjoyed the experience!
During the talk we presented a couple of small demos that showcased some in-memory data grid use cases. The demos are located here, but I thought it’d be useful to provide some step-by-step here so that you can get them running as quickly as possible.
Before we start with any of the demos, it’s necessary to run some set up steps:
1. Check out git repository:
2. Download Infinispan Server 9.0.0.Final and at the same level as the git repository.
3. Go into the datagrid-patterns directory, start the servers and wait until they’ve started:
cd datagrid-patterns ./run-servers.sh
4. Install Anaconda for Python 3, this is required to run Jupyter notebook for plotting.
5. Install Maven 3.
Once the set up is complete, it’s time to start with the individual demos.
Both demos shown below work with the same application domain: rail transport systems. In this domain, we differentiate between physical stations, trains, station boards which are located in stations, and finally stops, which are individual entries in station boards.
Analytics Demo
The first demo is focused on how you can use Infinispan for doing offline analytics. In particular, this demo tries to answer the following question:
Q. What is the time of the day when there is the biggest ratio of delayed trains?
To answer this question, Infinispan data grid will be loaded with 3 weeks worth of data from station boards. Once the data is loaded, we will execute a remote server task which will use Infinispan Distributed Java Streams to calculate the two pieces of information required to answer the question: per hour, how many trains are going through the system, and out of those, how many are delayed.
An important aspect to bear in mind about this server tasks is that it will only be executed in one of the nodes in the cluster. It does not matter which one. In turn, this node will will ship the lambdas required to do the computation to each of the nodes so that they can executed against their local data. The other nodes will reply with the results and the node where the server task was invoked will aggregate the results.
Then, these results are sent back to the client, which in turn, stores the results as JSON in an intermediate cache. Once the results are in place, we will use a Jupyter notebook to read those results and plot the result.
Let’s see these steps in action:
1. First, we need to install the server tasks in the running servers above:
cd datagrid-patterns/analytics
mvn clean install package -am -pl analytics-server
mvn wildfly:deploy -pl analytics-server
2. Open the datagrid-pattern repo with your favourite IDE and run delays.java.stream.InjectApp class located in analytics/analytics-server project. This command will inject the data into the cache. On my environment, it takes between 1 and 2 minutes.
3. With the data loaded, we need to run the remote task that will calculate the total number of trains per hour and how many of those are delayed. To do that, execute delays.java.stream.AnalyticsApp class located in analytics/analytics-server project from your IDE.
4. You can verify that the results have been calculating by going to the following address:
5. With the results in place, it’s time to start the Jupyter notebook:
cd datagrid-patterns/analytics/analytics-jupyter
~/anaconda/bin/jupyter notebook
6. Once the notebook opens, click open live-demo.ipynb notebook and execute each of the cells in order. You should end up seeing a plot like this:
So, the answer to the question:
Q. What is the time of the day when there is the biggest ratio of delayed trains?
is 2am! That’s because last connecting trains of the day wait for each other to avoid leaving passengers stranded.
Real Time Demo
The second demo that we presented uses the same application domain as above, but this time we’re trying to use our data grid as a way of storing the station board state of each station at a given point in time. So, the idea is to use Infinispan as an in memory data grids for working with real time data.
So, what can we do with this type of data? In our demo, we will create a centralised dashboard of delayed trains around the country. To do that, we will take advantage of Infinispan’s Continuous Query functionality which allows us to find those station boards which contain stops that are delayed, and as new delayed trains appeared these will be pushed to our dashboard.
To run this demo, keep the same servers running as above and do the following:
-
Run delays.query.continuous.FxApp application located in real-time project inside the datagrid-patterns demo. This app will inject some live station board data and will launch a JavaFX dashboard that shows delayed trains as they appear. It should look something like this:
Conclusion
This has been a summary of the demos that we run in our talk at Devoxx France with the intention of getting you running these demos as quickly as possible. The repository contains more detailed information of these demos. If there’s anything unclear or any of the instructions above are not working, please let us know!
Thanks to Emmanuel Bernard for partnering with me for this Devoxx France talk and for the continuous feedback while developing the demos. Thanks as well to Tristan Tarrant for the input in the demos and many thanks to all Devoxx France attendees who attended our talk :)
A very special thanks to Alexandre Masselot whose "Swiss Transport in Real Time: Tribulations in the Big Data Stack" talk at Soft-Shake 2016 was the inspiration for these demos. @Alex, thanks a lot for sharing the demos and data with me and the rest of the community!!
In a just a few weeks I’ll be at Great Indian Developer Summit presenting these demos and much more! Stay tuned :)
Cheers,
Galder
Tags: conference devoxx demo streams query
Wednesday, 02 November 2016
Infinispan coming to Devoxx Morocco!
Tomorrow Thursday, 3rd November, I’ll be speaking at the Devoxx Morocco conference (Casablanca, Morocco) about writing apps in functional reactive programming style with Infinispan, Elm and Node.js. If you’re interested in the topic and live in the area, make sure you come to my talk!
To find out more, head to the Devoxx Morocco site, where you can find exact details about the rest of the programme, location…etc.
Cheers,
Galder
Tags: conference devoxx
Wednesday, 21 November 2012
Back from Devoxx
This year - as always - Devoxx was a great conference! Not only the quality of the presentation (including quite some delivered by my fellows from Red Hat) but also the chance to meet and discuss with industry experts. And the Hackergarten was one of the this great networking places. We had a long chat with Marius Bogoevici on a nice extension of the Ticket Monster Tutorial for Infinispan, so that users looking for an sample integration application would have a good starting point. Also Duncan Doyle contributed a nice demo of the cross-site replication functionality added in Infinispan 5.2 and Guillaume Scheibel implemented a Mongo DB cache store extension: awesome stuff! All in all great conference and very good chance for us to get in touch with our community! Cheers, Mircea
Tags: conference devoxx event
Monday, 19 November 2012
Infinispan @ Devoxx 2012 Hackergarten
Just came back from Devoxx, and once again it didn’t let me down! Great conference, with awesome talks, and fantastic networking opportunities (hackergarten, party…etc). As far as Infinispan is concerned, I joined the Hackergarten on Tuesday morning to try lure some contributors into our project :).
One of the most promising opportunities came from Alex Soto, who’s founder and lead of NoSQLUnit, which is a JUnit extension that helps you write NoSQL unit tests. He already has support for a number of NoSQL engines and he was explaining me the challenges of supporting multiple engines. We also discussed the possibility of supporting Infinispan as well :D.
During the Hackergarten I also met Andrés Almiray, which is the Griffon founder, and we briefly discussed the possibility of adding integrating Infinispan’s Hot Rod client with Griffon clients. Unfortunately we didn’t have time to get into some coding, but since he lives close-by and he organises Hackergartens in Basel, I might pop in next time around and sit down with him to work on this integration :).
Can’t wait for next Devoxx!!
Cheers, Galder
Tags: conference devoxx event hackergarten
Tuesday, 13 November 2012
Devoxx here we come!
If you’re interested in data grids and especially in general and measuring data grids performance in particular then come to my presentation at this year’s Devoxx.
Manik Surtani is getting funny (literally!) with a gig suggestively named "http://www.devoxx.com/display/DV12/Manik+Surtani[Will you Map/Reduce my cloud]" and Galder Zamarreno brings the polyglot JBoss at Devoxx.
We’ll also be around the Red Hat boot in case you want to have an Infinispan related discussion or just to say hi.
Cheers,
Mircea
Tags: presentations devoxx event
Monday, 17 October 2011
An understudy for Devoxx 2011
I won’t be able to make it to Devoxx this year, but worry not, the University talk and hands-on deep-dive on Infinispan will still go on. Pete and Mircea will be joined by Sanne Grinovero - maintainer of Infinispan’s querying capabilities, Lucene and Hibernate hacker, and committer on Hibernate OGM. Now I wish I was attending, as an audience member! :-)
Enjoy Manik
Tags: devoxx
Thursday, 01 September 2011
JavaOne 2011 and Devoxx 2011
I never got around to blogging about this when my talks were accepted for JavaOne this year, but it’s about time.
I have a conference session titled "https://oracleus.wingateweb.com/content/sessionDetail.do?SESSION_ID=23382[A Tale About Caching (JSR 107) and Data Grids (JSR 347) in Enterprise Java]" and a BoF session focused on JSR 347 titled "https://oracleus.wingateweb.com/content/sessionDetail.do?SESSION_ID=23380[Making Java EE Cloud-Friendly: JSR 347, Data Grids for the Java Platform]", which I will be delivering with fellow Infinispan developer, JBoss rockstar and overall nice guy Pete Muir.
Later on in the year, I will also be running a University talk at Devoxx in Antwerp, titled "http://www.devoxx.com/display/DV11/Real-world+deep-dive+into+Infinispan+-+the+open+source+data+grid+platform[A real-world deep-dive into Infinispan]". This too will be with Pete and Mircea Markus, another core Infinispan developer.
This will be a great chance to learn more about Infinispan, data grids, JSR 107 and JSR 347, so if you are attending these conferences, make sure you add these talks to your agenda! :-)
Cheers Manik
Tags: devoxx jsr 107 jsr 347 JavaOne
Tuesday, 09 November 2010
Infinispan Community Meetup @ Devoxx 2010
Infinispan and Seam are doing it again, and this year Arquillian will be there to keep things in check: a community social in the midst of Europe’s premier Java conference, Devoxx in Antwerp. If you plan to be at Devoxx - or just happen to be nearby - and fancy an evening of in-depth conversation about three open source projects set to rock the enterprise, come on by and partake in some of the finest beer in the world with the project core developers. Demos, design discussions, architectural questions are all fair game. Join our pilgrimage to Kulminator at 20:00 on Monday the 15th of November (near the Groenplaats Station).
And a quick note, just because open source is all about Free as in Speech doesn’t mean we can’t have free beer. Yes, we’re sponsoring beer as well as other project-related goodies, but these are limited, first come first served.
Details Where: Kulminator, close to Groenplaats Station, Antwerp When: Monday the 15th of November, 2010 What time: 20:00
Hope to see many of you there!
Cheers Manik
Tags: meetup devoxx community
Thursday, 30 September 2010
Want to learn about Infinispan at Devoxx 2010?
Yes, I will be speaking at the Devoxx conference this year. Devoxx - probably the biggest Java/Middleware conference outside of JavaOne - is always awesome with a strong tech focus.
I’ll be talking about Infinispan as a NoSQL solution, so this should be interesting to anyone trying to build a data service around Infinispan.
Check out the details here. See you in Antwerp!
Cheers Manik
Tags: data-as-a-service devoxx infinispan
Thursday, 07 January 2010
Video on Devoxx 09 presentation
So the people at Devoxx have recorded my presentation on Infinispan and the future of Data Grids at Devoxx 2009, and have uploaded it for online viewing. For those of you who were not able to attend the talk, the recording can be accessed on http://links.infinispan.org/devoxx.
Enjoy!
Manik
Tags: presentations devoxx video
