Friday, 02 November 2018

Infinispan triple connector release!

Infinispan Spark, Infinispan Hadoop and Infinispan Kafka have a new fresh release each!

Infinispan Spark 0.9

 

The native Apache Spark connector now supports Infinispan 9.4.x and Spark 2.3.2, and it exposes Infinispan’s new transcoding capabilities, enabling the InfinispanRDD and InfinispanDStream to operate with multiple data formats. For more details see the documentation.

Infinispan Hadoop 0.3

The connector that allows accessing Infinispan using standard Input/OutputFormat interfaces now offers compatibility with Infinispan 9.4.x and has been certified to run with the Hadoop 3.1.1 runtime. For more details about this connector, see the user manual. Also make sure to check the docker based demos: Infinispan + Yarn and Infinispan + Apache Flink.

Infinispan Kafka 0.3

 

Last but not least, the Infinispan Kafka connector was upgraded to work with the latest Kafka (2.0.x) and Infinispan releases (9.4.x). Many thanks to Andrea Cosentino for contributing this integration.

image

Posted by Gustavo on 2018-11-02
Tags: release kafka spark hadoop

Tuesday, 24 July 2018

Infinispan Spark connector 0.8 released

The Infinispan Spark connector version 0.8 has been released and is available in Maven central and SparkPackages.

This is a maintenance only release to bring compatibility with Spark 2.3 and Infinispan 9.3.

For more information about the connector, please consult the documentation and also try the docker based sample.

For feedback and general help, please use the Infinispan chat.

Posted by Gustavo on 2018-07-24
Tags: release spark server

Thursday, 29 March 2018

Infinispan Spark connector 0.7 released!

A new version of connector that integrates Infinispan and Apache Spark has just been released!

This release brings compatibility with Infinispan 9.2.x and Spark 2.3.0.

Also included a new feature that allows to create and delete caches on demand, passing custom configurations when required. For more details, please consult the documentation.

To quickly try the connector, make sure to check the Twitter demo, and for any issues or suggestions, please report them on our JIRA.

Cheers!

Posted by Gustavo on 2018-03-29
Tags: release spark

Tuesday, 28 November 2017

Back from Madrid JUG and Codemotion Madrid!!

We’ve just come back from our trip to Spain and first of all, we’d like to thank everyone who attended our talks and workshops at Madrid Java User Group and Codemotion Madrid as well organisers and sponsors who made it possible!

image

image

We had a very hectic schedule, which started with a Red Hat double bill for Madrid JUG. Thomas Segismont started the evening with Vert.x talk and Galder followed up with a talk on how to do data analytics using Infinispan-based data grids.

In the data analytics talk, Galder focused on how to use distributed Java Streams to do analytics and also showed how to use Infinispan Spark connector when Java Streams are not enough. The distributed Java Streams demo he ran can be found here. The most relevant files of that demo are:

Galder also demonstrated how to use Infinispan Spark connector by showing the Twitter example. The slides from this talk (in Spanish) can be found here:

Next day on Friday, Galder gave a talk at Codemotion Madrid on working with streaming data with Infinispan, Vert.x and OpenShift. For the first time he was running it all on top of Google Cloud, so he could finally free up my laptop from running the demos and take advantage of the power of a cloud provider!

The demo can be found here where you can also find instructions on how to run it on top of Google Cloud. If you want to follow the same steps he followed during the talk, live coding instructions are here. The slides from this talk (in Spanish) can be found here:

Finally on Saturday we delivered the Streaming Data workshop at Codemotion Madrid. Once again, basing our workshop on top of Virtual Box still caused us some issues, but people managed to get through it. We have some plans for next year to avoid the need of Virtual Box, stay tuned!

imagehttp://3.bp.blogspot.com/-vhDHZjc2CX0/Wh0t4gzxt3I/AAAAAAAAFDY/J_1l7ze26-gYTcp3XSu_S7AeZs2zFbM4QCK4BGAYYCw/s1600/DPfG2yYX0AAH682.jpeg[image]

imagehttp://3.bp.blogspot.com/-hKcqGpbsNno/Wh0t4QnaYaI/AAAAAAAAFDI/k8OvIG8-VpMvZPhHRKVjyADMBdASgnBEQCK4BGAYYCw/s1600/DPfG2yWXUAEOdoY.jpg[image]

image

We have added more detailed instructions on how to run the workshop in your office or at home, so if you’re interested in going through it, make sure you check these steps and let us know how they work for you:

image

This trip to Madrid wraps up a very intense year in terms of promoting Infinispan! Next month we’ll be doing a recap of the talks, videos…​etc so that you can catch up with them in case you missed any of them :)

Katia & Galder Un saludo!

Posted by Galder Zamarreño on 2017-11-28
Tags: conference openshift spark google

Tuesday, 21 November 2017

Infinispan Spark connector 0.6 is out!

This is a small release to align with Infinispan 9.2.x and Spark 2.2, and it also has an improvement to the Java API related to filterByQuery.

For a full list of changes, please refer to the release notes. Make sure to also check the improved documentation and the Twitter demo.

Enjoy!

Posted by Gustavo on 2017-11-21
Tags: spark release

Monday, 03 April 2017

Infinispan Spark connector 0.5 released!

The Infinispan Spark connector offers seamless integration between Apache Spark and Infinispan Servers. Apart from supporting Infinispan 9.0.0.Final and Spark 2.1.0, this release brings many usability improvements, and support for another major Spark API.

Configuration changes

The connector no longer uses a java.util.Properties object to hold configuration, that’s now duty of org.infinispan.spark.config.ConnectorConfiguration, type safe and both Java and Scala friendly:

 

Filtering by query String

The previous version introduced the possibility of filtering an InfinispanRDD by providing a Query instance, that required going through the QueryDSL which in turn required a properly configured remote cache.

It’s now possible to simply use an Ickle query string:

Improved support for Protocol Buffers

Support for reading from a Cache with protobuf encoding was present in the previous connector version, but now it’s possible to also write using protobuf encoding and also have protobuf schema registration automatically handled.

To see this in practice, consider an arbitrary non-Infinispan based RDD<Integer, Hotel> where Hotel is given by:

In order to write this RDD to Infinispan it’s just a matter of doing:

Internally the connector will trigger the auto-generation of the .proto file and message marshallers related to the configured entity(ies) and will handle registration of schemas in the server prior to writing.

Splitter is now pluggable

The Splitter is the interface responsible to create one or more partitions from a Infinispan cache, being each partition related to one or more segments. The Infinispan Spark connector now can be created using a custom implementation of Splitter allowing for different data partitioning strategies during the job processing.

Goodbye Scala 2.10

Scala 2.10 support was removed, Scala 2.11 is currently the only supported version. Scala 2.12 support will follow https://issues.apache.org/jira/browse/SPARK-14220

 

Streams with initial state

It is possible to configure the InfinispanInputDStream with an extra boolean parameter to receive the current cache state as events.

 

Dataset support

The Infinispan Spark connector now ships with support for Spark’s Dataset API, with support for pushing down predicates, similar to rdd.filterByQuery. The entry point of this API is the Spark session:

To create an Infinispan based Dataframe, the "infinispan" data source need to be used, along with the usual connector configuration:

From here it’s possible to use the untyped API, for example:

or execute SQL queries by setting a view:

In both cases above, the predicates and the required columns will be converted to an Infinispan Ickle filter, thus filtering data at the source rather than at Spark processing phase.

For the full list of changes see the release notes. For more information about the connector, the official documentation is the place to go. Also check the twitter data processing sample and to report bugs or request new features use the project JIRA.

Posted by Gustavo on 2017-04-03
Tags: spark server

Monday, 07 March 2016

Infinispan related presentations on DevConf.cz 2016

DevConf.cz (Developer Conference) is a free annual conference for all Linux and JBoss Community Developers, Admins and Linux users organized by Red Hat Czech Republic in cooperation with the Fedora and JBoss communities. See more details on DevConf’s homepage.

Since it’s JBoss related conference, Infinispan cannot be missing there! We had two Infinispan related presentantions. First of them was presented by Jiří Holuša focusing on new features in Infinispan 8. The presentation also covers very gentle introduction to Infinispan in general.

The second one, named "From Big Data towards Fast Data",  was given by Vojtěch Juránek and talked about latest trends in Big Data world. The presentation goes through the introduction about Big Data, mentions the problems with it and comes up with the term Fast Data. Some approaches are presented, how to solve this Fast Data phenomenon, ending up with new Infinispan Spark connector as one of the possible solutions.

Please, see the links below.

Presentation "Infinispan 8 - keeping up with the latest trends" Video: https://www.youtube.com/watch?v=r0__NEgldzI Slides: bit.ly/1oxAuCM * Presentation "From Big Data towards Fast Data"* Video: https://www.youtube.com/watch?v=TVXeXM2g7So Slides: bit.ly/1PrQ0ZK

Cheers, Jiří 

Posted by Unknown on 2016-03-07
Tags: conference presentations spark infinispan 8

Monday, 01 February 2016

'Infinispan 8 Tour' video available

image

The recording of the talk presented last Wednesday in London is available! Thanks to everyone who joined and to the London JBUG organizers for the awesome new venue!

Cheers, Gustavo

Posted by Gustavo on 2016-02-01
Tags: functional presentations spark hadoop streams video

Wednesday, 09 December 2015

Infinispan Spark connector 0.2 released!

The connector allows the Infinispan Server to become a data source for Apache Spark, for both batch jobs and stream processing, including read and write.

In this release, the highlight is the addition of two new operators to the RDD that support filtering using native capabilities of Infinispan. The first one is filterByQuery:

The second operator was introduced to replace the previous configuration based filter factory name, and was extended to support arbitrary parameters:

The connector has also been updated to be compatible with Spark 1.5.2 and Infinispan 8.1.0.Final.

For more details including full list of changes and download info please visit the Connectors Download section.  The project Github contains up-to-date info on how to get started with the connector, also make sure to try the included docker based demo. To report any issue or to request new features, use the new dedicated issue tracker. We’d love to get your feedback!

Posted by Gustavo on 2015-12-09
Tags: spark server

Monday, 17 August 2015

Infinispan Spark connector 0.1 released!

Dear users,

The Infinispan connector for Apache Spark has just been made available as a Spark Package!

What is it?

The Infinispan Spark connector allows tight integration with Apache Spark, allowing Spark jobs to be run against data stored in the Infinispan Server, exposing any cache as an RDD, and also writing data from any key/value RDD to a cache. It’s also possible to create a DStream backed by cache events and to save any key-value DStream to a cache.

The minimum version required is Infinispan 8.0.0.Beta3.

Giving it a spin with Docker

A handy docker image that contains an Infinispan cluster co-located with an Apache Spark standalone cluster is the fastest way to try the connector. Start by launching the container that hosts the Spark Master:

And then run as many worker nodes as you want:

Using the shell

The Apache Spark shell is a convenient way to quickly run jobs in an interactive fashion. Taking advantage of the fact that Spark is already installed in the docker containers (and thus the shell), let’s attach to the master:

Once inside, a Spark shell can be launched by:

That’s all it’s needed. The shell grabs the Infinispan connector and its dependencies from spark-packages.org and exposes them in the classpath.

Generating data and writing to Infinispan

Let’s obtain a list of words from the Linux dictionary, and generate 1k random 4-word phrases. Paste the commands in the shell:

From the phrases, we’ll create a key value RDD (Long, String):

To save to Infinispan:

Obtaining facts about data

To be able to explore data in the cache, the first step is to create an infinispan RDD:

As an example job, let’s calculate a histogram showing the distribution of word lengths in the phrases. This is simply a sequence of transformations expressed by:

This pipeline yields:

2 chars words: 10 occurrences 3 chars words: 37 occurrences 4 chars words: 133 occurrences 5 chars words: 219 occurrences 6 chars words: 373 occurrences 7 chars words: 428 occurrences 8 chars words: 510 occurrences 9 chars words: 508 occurrences 10 chars words: 471 occurrences 11 chars words: 380 occurrences 12 chars words: 309 occurrences 13 chars words: 238 occurrences

…​

Now let’s find similar words using the Levenshtein distance algorithm. For that we need to define a function that will calculate the edit distance between two strings. As usual, paste in the shell:

Empowered by the Levenshtein distance implementation, we need another function that given a word, will find in the cache similar words according to the provided maximum edit distance:

Sample usage:

Where to go from here

And that concludes this first post on Infinispan-Spark integration. Be sure to check the Twitter demo for non-shell usages of the connector, including Java and Scala API.

And it goes without saying, your feedback is much appreciated! :)

Posted by Gustavo on 2015-08-17
Tags: release spark

News

Tags

JUGs alpha as7 asymmetric clusters asynchronous beta c++ cdi chat clustering community conference configuration console data grids data-as-a-service database devoxx distributed executors docker event functional grouping and aggregation hotrod infinispan java 8 jboss cache jcache jclouds jcp jdg jpa judcon kubernetes listeners meetup minor release off-heap openshift performance presentations product protostream radargun radegast recruit release release 8.2 9.0 final release candidate remote query replication queue rest query security spring streams transactions vert.x workshop 8.1.0 API DSL Hibernate-Search Ickle Infinispan Query JP-QL JSON JUGs JavaOne LGPL License NoSQL Open Source Protobuf SCM administration affinity algorithms alpha amazon annotations announcement archetype archetypes as5 as7 asl2 asynchronous atomic maps atomic objects availability aws beer benchmark benchmarks berkeleydb beta beta release blogger book breizh camp buddy replication bugfix c# c++ c3p0 cache benchmark framework cache store cache stores cachestore cassandra cdi cep certification cli cloud storage clustered cache configuration clustered counters clustered locks codemotion codename colocation command line interface community comparison compose concurrency conference conferences configuration console counter cpp-client cpu creative cross site replication csharp custom commands daas data container data entry data grids data structures data-as-a-service deadlock detection demo deployment dev-preview devnation devoxx distributed executors distributed queries distribution docker documentation domain mode dotnet-client dzone refcard ec2 ehcache embedded query equivalence event eviction example externalizers failover faq final fine grained flags flink full-text functional future garbage collection geecon getAll gigaspaces git github gke google graalvm greach conf gsoc hackergarten hadoop hbase health hibernate hibernate ogm hibernate search hot rod hotrod hql http/2 ide index indexing india infinispan infinispan 8 infoq internationalization interoperability interview introduction iteration javascript jboss as 5 jboss asylum jboss cache jbossworld jbug jcache jclouds jcp jdbc jdg jgroups jopr jpa js-client jsr 107 jsr 347 jta judcon kafka kubernetes lambda language leveldb license listeners loader local mode lock striping locking logging lucene mac management map reduce marshalling maven memcached memory migration minikube minishift minor release modules mongodb monitoring multi-tenancy nashorn native near caching netty node.js nodejs nosqlunit off-heap openshift operator oracle osgi overhead paas paid support partition handling partitioning performance persistence podcast presentations protostream public speaking push api putAll python quarkus query quick start radargun radegast react reactive red hat redis rehashing releaase release release candidate remote remote events remote query replication rest rest query roadmap rocksdb ruby s3 scattered cache scripting second level cache provider security segmented server shell site snowcamp spark split brain spring spring boot spring-session stable standards state transfer statistics store store by reference store by value streams substratevm synchronization syntax highlighting testing tomcat transactions uneven load user groups user guide vagrant versioning vert.x video videos virtual nodes vote voxxed voxxed days milano wallpaper websocket websockets wildfly workshop xsd xsite yarn zulip

back to top