telemetry handler detaches automatically on error

Telemetry handlers should be designed to run efficiently over and over again without making a noticeable performance impact on your production system. But what happens if an occurs? Your monitoring should have no bearing on the success of your business logic.

So what happens if your monitoring has an error? With telemetry, it's ok, the offending handler is just detached, never to run again. Business logic unaffected.

Here's a test that demonstrates the handler being removed.

handlerFn = fn _, _, _, _ ->
  raise "something"
end

:telemetry.attach(
  :bad_handler,
  [:something_happened],
  handlerFn,
  :no_config
)

# the handler is in ets
assert [[[:something_happened]]] =
         :ets.match(:telemetry_handler_table, {:handler, :bad_handler, :"$1", :_, :_})

:telemetry.execute([:something_happened], %{}, %{})

# the handler is gone!
assert [] = :ets.match(:telemetry_handler_table, {:handler, :bad_handler, :"$1", :_, :_})

You can see the try/catch block in the telemetry source code here.