Add instrumentation for Nebulex, a distributed cache library. This
library provides solid telemetry support for this initial
implementation.
Caching implementation is mostly based on in-memory storage (like ETS)
and RPC calls for distribution (via OTP libraries, like :erpc). AFAICT,
there is not much specifics for how to translate into Semantic
Attributes: those caches are not quite a DB, except maybe for the one
which implements the storage; the RPC can't be reliably captured
either.
Given the above constraints, this initial implementation instruments the
library via custom attributes (namespaced as `nebulex.*`). It's not 100%
clear the behaviour of OTel for actual distributed caches - from my
tests, that may create some orphan spans. I think that's fine as first
release.
Nebulex follow the patterns of Ecto, so this instrumentation follows a
similar pattern of OpentelemetryEcto. It does include a `setup_all/1`
function for convenience, that leverages the :init events Nebulex emit
on process start.
Co-authored-by: Tristan Sloughter <t@crashfast.com>
Initial approach follows Ecto instrumentation, recording spans for all
Redix `[:redix, :pipeline, :stop]` events.
The command sanitization is inspired-by and adapted from [Java
instrumentation][1], from where I've also copied the actual commands and
what configuration should they follow.
Network attributes are tracked via a "sidecar" process, which keeps
track of connection attributes also via `telemetry`. This extra bit of
bookkeeping is needed as command events doesn't include that piece of
information, unfortunately.
[1]: b2bc41453b/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/db/RedisCommandSanitizer.java
By default a new trace is automatically started when a job is processed
by monitoring these events:
* `[:oban, :job, :start]` — at the point a job is fetched from the database and will execute
* `[:oban, :job, :stop]` — after a job succeeds and the success is recorded in the database
* `[:oban, :job, :exception]` — after a job fails and the failure is recorded in the database
To also record a span when a job is created and to link traces together
`Oban.insert/2` has to be replaced by `OpentelemetryOban.insert/2`.
Before:
```elixir
%{id: 1, in_the: "business", of_doing: "business"}
|> MyApp.Business.new()
|> Oban.insert()
```
After:
```elixir
%{id: 1, in_the: "business", of_doing: "business"}
|> MyApp.Business.new()
|> OpentelemetryOban.insert()
```
Co-authored-by: Tristan Sloughter <t@crashfast.com>
Andrew Rosa
Ricardo Paiva
Indrek Juhkam
Bryan Naegele
Sergey Kanzhelev