Friday, 09 February 2018

RESTful queries coming to Infinispan 9.2

One of the interesting features in the upcoming Infinispan 9.2 release is the possibility to execute queries over the REST endpoint, enabling users to take advantage of the easy-to-use and expressiveness of the Ickle query language, that combines a subset of JP-QL with full-text features. You can learn more info about Ickle in a previous post.

Besides exposing query over REST, Infinispan 9.2 also adds support for mapping between JSON and Protobuf formats, allowing an efficient storage in binary format while exposing queries, reading and writing content as JSON documents.

To illustrate those new capabilities, this post will walk you through a sample app from scratch!

Sample app

Running the server

We start by running the Infinispan Server 9.2.0.CR2 (the latest release candidate):

This will get you a fresh instance of Infinispan running, with login and password 'user' and the REST port 8080 mapped to localhost. TIP: if you run more than one container, they’ll form a cluster automatically.

Creating an indexed cache

Next step is to create an indexed cache called 'pokemon'. We make use of the CLI (Command Line Interface) to create this cache. In the future, with ISPN-8529, we’ll also be able to create cache with arbitrary configuration using REST, but for now we execute a CLI recipe:

Creating the schema

In order to be able to query, we need to define a protobuf schema for our data. The schema follows the Protobuf 2 format (Protobuf 3 support is coming) and allows for extensions to define indexing properties (analyzers, storage, etc).

Here’s how it looks like:

The protobuf schema can contain some comments on top of fields and messages with "annotations" to control indexing. Hibernate Search users will recognize some of those pseudo annotations we are using here: they resemble closely their counterpart.

Registering the schema

Once we have our schema, we can easily register it via REST:

Populating the cache

We’re now ready to put some data in the cache. As mentioned earlier, ingesting can be done by sending JSON documents directly. Once Infinispan receives those documents, it will convert them to protobuf, index and store them.

In order to match a particular inbound document to an entity in the schema, Infinispan uses a special meta field called _type that must be provided in the document. Here’s an example of a JSON document that conforms to our schema:

Writing the document is easy:

we can retrieve content by key as JSON:

Querying

The new query endpoint can be called with an "action" parameter named "search", after the cache name. The simplest query, which returns all data can be done with:

http://localhost:8080/rest/pokemon?action=search&query=from

If you do not want to return all the fields, use a Select clause:

http://localhost:8080/rest/pokemon?action=search&query=Select

Pagination can be controlled with the offset, max_results URL parameters:

http://localhost:8080/rest/pokemon?action=search&query=from

Grouping is also possible:

http://localhost:8080/rest/pokemon?action=search&query=select

Example of a query result:

http://localhost:8080/rest/pokemon?action=search&query=select

Results:

Conclusion

Infinispan 9.2 makes it easier to quickly ingest and query datasets using the ubiquitous JSON format, without sacrificing type safety and storage size.

By storing Protobuf, this will also enable other clients like the Hot Rod C#/C++ clients to query, read and write data simultaneously with REST clients.

The full source code for the demo, along with instructions on how to populate the whole dataset can be found at Github.

Finally, please try out this new feature in your own dataset and let us know how it goes!

Posted by Gustavo on 2018-02-09
Tags: rest query Protobuf JSON query

Friday, 09 January 2015

Infinispan 7.1.0.Beta1

Dear Infinispan community,

We’re proud to announce the first Beta release of Infinispan 7.1.0.

Infinispan brings the following major improvements:

Near-Cache support for Remote HotRod caches
Annotation-based generation of ProtoBuf serializers which removes the need to write the schema files by hand and greatly improves usability of Remote Queries
Cluster Listener Event Batching, which coalesces events for better performance
Cluster- and node-wide aggregated statistics
Vast improvements to the indexing performance
Support for domain mode and the security vault in the server
Further improvements to the Partition Handling with many stability fixes and the removal of the Unavailable mode: a cluster can now be either Available or Degraded.

Of course there’s also the usual slew of bug fixes, performance and memory usage improvements and documentation cleanups.

Feel free to join us and shape the future releases on our forums, our mailing lists or our #infinispan IRC channel.

For a complete list of features and bug fixes included in this release please refer to the release notes. Visit our downloads section to find the latest release.

Thanks to everyone for their involvement and contribution!

Posted by Tristan Tarrant on 2015-01-09
Tags: release near caching domain mode performance Protobuf indexing annotations

Tuesday, 19 November 2013

Infinispan 6.0.0.Final is out!

Dear Infinispan community,

We’re pleased to announce the final release of Infinispan 6.0 "Infinium". As announced, this is the first Infinispan stable version to be released under the terms of Apache License v2.0.

This release brings some highly demanded features besides many stability enhancements and bug fixes:

Support for remote query. It is now possible for the HotRod clients to query an Infinispan grid using a new expressive query DSL. This querying functionality is built on top of Apache Lucene and Google Protobuf and lays the foundation for storing information and querying an Infinispan server in a language neutral manner. The Java HotRod client has already been enhanced to support this, the soon-to-be announced C++ HotRod client will also contain this functionality (initially for write/read, then full blown querying).
C HotRod client. Allows C applications to read and write information from an Infinispan server. This is a fully fledged HotRod client that is topology (level 2) and consistent hash aware (level 3) and will be released in the following days. Some features (such as Remote Query and SSL support) will be developed during the next iteration so that it maintains feature parity with its Java counterpart.
Better persistence integration. We’ve revisited the entire cache loader API and we’re quite pleased with the result: the new Persistence API brought by Infinispan 6.0 supports parallel iteration of the stored entries, reduces the overall serialization overhead and also is aligned with the JSR-107 specification, which makes implementations more portable.
A more efficient FileCacheStore implementation. This file store is built with efficiency in mind: it outperforms the existing file store with up to 2 levels of magnitude. This comes at a cost though, as keys need to be kept in memory. Thanks to Karsten Blees for contributing this!
Support for heterogeneous clusters. Up to this release, every member of the cluster owns an equal share of the cluster’s data. This doesn’t work well if one machine is more powerful than the other cluster participants. This functionality allows specifying the amount of data, compared with the average, held by a particular machine.
A new set of usage and performance statistics developed within the scope of the CloudTM projecthttps://issues.jboss.org/browse/ISPN-3234[].
JCache (JSR-107) implementation upgrade. First released in Infinispan 5.3.0, the standard caching support is now upgraded to version 1.0.0-PFD.

For a complete list of features included in this release please refer to the release notes.

The user documentation for this release has been revamped and migrated to the new website - we think it looks much better and hope you’ll like it too!

This release has spread over a period of 5 months: a sustained effort from the core development team, QE team and our growing community - a BIG thanks to everybody involved! Please visit our downloads section to find the latest release. Also if you have any questions please check our forums, our mailing lists or ping us directly on IRC.

Cheers,

Adrian

Posted by Unknown on 2013-11-19
Tags: hotrod persistence jsr 107 jcache Protobuf remote query query

Thursday, 26 September 2013

Embedded and remote queries in Infinispan 6.0.0.Beta1

If you’re following Infinispan’s mailing lists you’ve probably caught a glimpse of the new developments in the Query land: a new DSL, remote querying via Hot Rod client, a new marshaller based on Google’s Protobuf. Time to unveil these properly!

==== The new Query DSL

Starting with version 6.0 Infinispan offers a new (experimental) way of running queries against your cached entities based on a simple filtering DSL. The aim of the new DSL is to simplify the way you write queries and to be agnostic of the underlying query mechanism(s) making it possible to provide alternative query engines in the future besides Lucene and still being able to use the same query language/API. The previous Hibernate Search & Lucene based approach is still in place and will continue to be supported and in fact the new DSL is currently implemented right on top of it. The future will surely bring index-less searching based on map-reduce and possibly other new cool search technologies.

Running DSL-based queries in embedded mode is almost identical to running the existing Lucene-based queries. All you need to do is have infinispan-query-dsl.jar and infinispan-query.jar in your classpath (besides Infinispan and its dependecies), enable indexing for your caches, annotate your POJO cache values and your’re ready.

ConfigurationBuilder cfg = new ConfigurationBuilder();
cfg.indexing().enable();

DefaultCacheManager cacheManager = new DefaultCacheManager(cfg.build());

Cache cache = cacheManager.getCache();

____Alternatively, indexing (and everything else) can also be configured via XML configuration, as already described in the user guide, so we’ll not delve into details here.

Your Hibernate Search annotated entity might look like this.

import org.hibernate.search.annotations.*;
...

@Indexed
public class User {

    @Field(store = Store.YES, analyze = Analyze.NO)
    private String name;

    @Field(store = Store.YES, analyze = Analyze.NO, indexNullAs = Field.DEFAULT_NULL_TOKEN)
    private String surname;

    @IndexedEmbedded(indexNullAs = Field.DEFAULT_NULL_TOKEN)
    private List addresses;

    // .. the rest omitted for brevity
}

___Running a DSL based query involves obtaining a _https://github.com/infinispan/infinispan/blob/6.0.0.Beta1/query-dsl/src/main/java/org/infinispan/query/dsl/QueryFactory.java[QueryFactory] from the (cache scoped) SearchManager and then constructing the query as follows:

import org.infinispan.query.Search;
import org.infinispan.query.dsl.QueryFactory;
import org.infinispan.query.dsl.Query;
...

QueryFactory qf = Search.getSearchManager(cache).getQueryFactory();

Query q = qf.from(User.class)
    .having("name").eq("John")
    .toBuilder().build();

List list = q.list();

assertEquals(1, list.size());
assertEquals("John", list.get(0).getName());
assertEquals("Doe", list.get(0).getSurname());

___That’s it! I’m sure this raised your curiosity as to what the DSL is actually capable of so you might want to look at the list of supported filter operators in _https://github.com/infinispan/infinispan/blob/6.0.0.Beta1/query-dsl/src/main/java/org/infinispan/query/dsl/FilterConditionEndContext.java[FilterConditionEndContext]. Combining multiple conditions with boolean operators, including sub-conditions, is also possible:

Query q = qf.from(User.class)
    .having("name").eq("John")
    .and().having("surname").eq("Doe")
    .and().not(qf.having("address.street").like("%Tanzania%").or().having("address.postCode").in("TZ13", "TZ22"))
    .toBuilder().build();

The DSL is pretty nifty right now and will surely be expanded in the future based on your feedback. It also provides support for result pagination, sorting, projections, embedded objects, all demonstrated in QueryDslConditionsTest which I encourage you to look at until the proper user guide is published. Still, this is not a relational database, so keep in mind that all queries are written in the scope of the single targeted entity (and its embedded entities). There are no joins (yet), no correlated subqueries, no grouping or aggregations.

Moving further, probably the most exciting thing about the new DSL is using it remotely via the Hot Rod client. But to make this leap we first had to adopt a common format for storing our cache entries and marshalling them over the wire that would also be cross-language and robust enough to support evolving object schemas. But probably most of all, this format had to have a schema rather than just being an opaque blob otherwise indexing and searching are meaningless. Enter Protocol Buffers.

The Protobuf marshaller

Configuring the RemoteCacheManager of the Java Hot Rod client to use it is straight forward: __

import org.infinispan.client.hotrod.configuration.ConfigurationBuilder;
...

ConfigurationBuilder clientBuilder = new ConfigurationBuilder();
clientBuilder.addServer()
    .host("127.0.0.1").port(11234)
    .marshaller(new ProtoStreamMarshaller());

___Now you’ll be able to store and get from the remote cache your _User instaces encoded in protobuf format provided that:

a Protobuf type was declared for your entity in a .proto file which was then compiled into a .protobin binary descriptor
the binary descriptor was registered with your RemoteCacheManager's ProtoStreamMarshaller instance like this: __

ProtoStreamMarshaller.getSerializationContext(remoteCacheManager)
    .registerProtofile("my-test-schema.protobin");

__3. a per-entity marshaller was registered:

ProtoStreamMarshaller.getSerializationContext(remoteCacheManager)
    .registerMarshaller(User.class, new UserMarshaller());

___Steps 2 and 3 are closely tied to the way Protosteam library works, which is pretty straight forward but cannot be detailed here. Having a look at our _UserMarshaller sample should clear this up.

Keeping your objects stored in protobuf format has the benefit of being able to consume them with compatible clients written in other languages. But if this does not sound enticing enough probably the fact they can now be easily indexed should be more appealing.

Remote querying via the Hot Rod client

Given a RemoteCacheManager configured as previously described the next steps to enable remote query over its caches are:

add the DSL jar to client’s classpath, infinispan-remote-query-server.jar to server’s classpath and infinispan-remote-query-client.jar to both
enable indexing in your cache configuration - same as for embedded mode
register your protobuf binary descriptor by invoking the 'registerProtofile' method of the server’s ProtobufMetadataManager MBean (one instance per EmbeddedCacheManager)

All data placed in cache now is being indexed without the need to annotate your entities for Hibernate Search. In fact these classes are only meaningful to the Java client and do not even exist on the server.

Running the queries over the Hot Rod client is now very similar to embedded mode. The DSL is in fact the same. The only part that is slightly different is how you obtain the QueryFactory:

import org.infinispan.client.hotrod.Search;
import org.infinispan.query.dsl.QueryFactory;
import org.infinispan.query.dsl.Query;
...

remoteCache.put(2, new User("John", "Doe", 33));

QueryFactory qf = Search.getQueryFactory(remoteCache);

Query query = qf.from(User.class)
    .having("name").eq("John")
    .toBuilder().build();

List list = query.list();
assertEquals(1, list.size());
assertEquals("John", list.get(0).getName());
assertEquals("Doe", list.get(0).getSurname());

Voila! The end of our journey for today! Stay tuned, keep an eye on Infinispan Query and please share your comments with us.

Posted by Unknown on 2013-09-26
Tags: protostream hotrod lucene Protobuf remote query hibernate search embedded query Infinispan Query DSL