Infinispan 14 OpenTelemetry tracing integration
Dear Infinispan community,
With the Infinispan 14 development release 04, we started to support tracing with OpenTelemetry.
If configured, Infinispan Server produces cache events tracing spans and sends them to a remote tracing collector server, such as Jaeger or Zipkin.
Moreover, if a Java client application with the HotRot or the Rest client produces some tracing spans, these spans can be correlated as parent spans of the corresponding spans events produced by the Infinispan Server.
Set up tracing on Infinispan Server
The new version of Infinispan Server comes with a gRPC OpenTelemetry Protocol (OTLP) Exporter, which is now supported by the majority of tracing servers.
For instance, with the newer Jaeger server versions, you can enable data collection through the OTLP protocol with the following option:
./jaeger-all-in-one --collector.otlp.enabledThe server opens a port to import gRPC OTLP tracing data at port 4317.
Configure tracing on the Infinispan Server by setting system properties or environment variables:
infinispan.tracing.enabled=true
otel.service.name=infinispan-server-service
otel.exporter.otlp.endpoint=http://localhost:4317
otel.metrics.exporter=noneThe first property is Infinispan specific and enables the tracing capability of the Infinispan Server. The further properties belong to the OpenTelemetry SDK Autoconfigure project that Infinispan uses to configure the tracing exporter. In this case, OTLP gRPC Exporter protocol is used and the server runs on the same machine as the Infinispan Server.
Starting the server with these parameters:
export JAVA_OPTS="-Dinfinispan.tracing.enabled=true -Dotel.service.name=infinispan-server-service -Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dotel.metrics.exporter=none"
./server.shThe following log is produced when the server starts:
(ForkJoinPool.commonPool-worker-2) [org.infinispan.server.core.telemetry.TelemetryServiceFactory] ISPN000952: OpenTelemetry instance loaded: OpenTelemetrySdk{...This indicates that the OpenTelemetrySdk is correctly configured.
Tracing from a HotRod client application
Any OpenTelemetry tracing context present on HotRot client applications will be automatically propagated by the new Hot Rod v4 client to the server tracing context.
For instance, for a client that defines some tracing spans containing cache operations, such as the following:
public class MyRestClient {
public void putSomeValues(RemoteCache cache) {
Span span = tracer.spanBuilder("sub-bulk-1").setSpanKind(SpanKind.CLIENT).startSpan();
// put the span into the current Context
try (Scope scope = span.makeCurrent()) {
cache.put(1, "A");
cache.put(2, "B");
cache.put(3, "C");
} catch (Throwable throwable) {
span.setStatus(StatusCode.ERROR, "Something bad happened!");
span.recordException(throwable);
throw throwable;
} finally {
span.end(); // Cannot set a span after this call
}
}
}The client span sub-bulk-1 will be correlated to any related server spans, in this case the three put operations.
Opening the Jaeger console, we can see that client and server spans are correctly aggregated:
You can find a complete application example here: https://github.com/fax4ever/infinispan-play/tree/main/tracing-hotrod-client
Tracing from a REST client application
You can achieve the same with a REST client by putting manually in the HTTP headers the requests to provide information about the current tracing context using a standard OpenTelemetry instance of W3CTraceContextPropagator.
public class MyRestClient {
public void putSomeValues(RestCacheClient cache) {
Span span = tracer.spanBuilder("sub-bulk-1").setSpanKind(SpanKind.CLIENT).startSpan();
// put the span into the current Context
try (Scope scope = span.makeCurrent()) {
putSomeEntries(cache);
} catch (Throwable throwable) {
span.setStatus(StatusCode.ERROR, "Something bad happened!");
span.recordException(throwable);
throw throwable;
} finally {
span.end(); // Cannot set a span after this call
}
}
private void putSomeEntries(RestCacheClient cache) {
Map<String, String> contextMap = getContextMap();
CompletableFuture[] futures = new CompletableFuture[3];
futures[0] = cache.put("1", MediaType.TEXT_PLAIN.toString(),
RestEntity.create(MediaType.TEXT_PLAIN, "A"), contextMap).toCompletableFuture();
futures[1] = cache.put("2", MediaType.TEXT_PLAIN.toString(),
RestEntity.create(MediaType.TEXT_PLAIN, "B"), contextMap).toCompletableFuture();
futures[2] = cache.put("3", MediaType.TEXT_PLAIN.toString(),
RestEntity.create(MediaType.TEXT_PLAIN, "C"), contextMap).toCompletableFuture();
CompletableFuture.allOf(futures).join();
}
public static Map<String, String> getContextMap() {
HashMap<String, String> result = new HashMap<>();
// Inject the request with the *current* Context, which contains our current Span.
W3CTraceContextPropagator.getInstance().inject(Context.current(), result,
(carrier, key, value) -> carrier.put(key, value));
return result;
}
}Opening the Jaeger console, you can see that client and server spans are correctly aggregated:
You can find a complete application example here: https://github.com/fax4ever/infinispan-play/tree/main/tracing-rest-client
Get it, Use it, Ask us!
We’re hard at work on new features, improvements and fixes, so watch this space for more announcements!Please, download and test the latest release.
The source code is hosted on GitHub. If you need to report a bug or request a new feature, look for a similar one on our JIRA issues tracker. If you don’t find any, create a new issue.
If you have questions, are experiencing a bug or want advice on using Infinispan, you can use GitHub discussions. We will do our best to answer you as soon as we can.
The Infinispan community uses Zulip for real-time communications. Join us using either a web-browser or a dedicated application on the Infinispan chat.
Fabio Massimo Ercoli
Member of the Infinispan @core team at Red Hat. Accountable for indexing, query, serialization, transcoding, metrics and tracing for the hybrid cloud. Former Hibernate @core team, working on NoSql & searching. Former Red Hat consultant, working on intense data applications and solutions. An open source enthusiast and continuous learner.
