* Add prom metrics for webhook dispatch.
1. success/failure/drop counter.
2. queue length histogram.
- doing this only for resource_url_notifier which does a per room
queue as queue length is easily available. This is the one
currently in use.
- worker pool based notifier will need a change in `core` package to
return a queue length when enqueuing.
* promauto goodness
* Make ResourceURLNotifier configurable.
Also change default max age to 10s.
* generated protobuf
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Per resource webhook queue.
With a worker based approach, an event on a resource (like a room) not
being handled could affect other resources (other rooms or ingress or
egress).
To prevent that, do a per resource queue with two kinds of limits
- age of event: when queuing, if the event at the front of the queue
exceeds a certain limit (default: 5 seconds), the event is dropped
- depth of queue: drop event if the number of queued events exceeds some
level (default: 200)
To reap the per resource queue, specific end events are tracked and the
resource is added to the reap queue and reaped after a configurable
timeout (default: 5 min). This is to ensure that events happening just
after end event (for example: participant_left happening after
room_finished) are handled without re-creating a new resource.
Also, deprecating `num_dropped` from proto.
Some other bits of notes/challenges:
- Keeping the old URL notifier also. Also, the default notifier is that.
Can change to the new one before merging this PR if this approach
looks good.
- Using string compares of event name to detect end. Not great.
- Needs a sweeper to reap the resource queue. I think 5 min default
timeout to reap should be okay, but maybe there are things that can
cause re-creation of a resource queue after reaping and those could
leak? Needs more analysis.
- Storing `context.Context` in resource queue so that it can be used
during posting to webhook from resource queue go routine. Storing that
is not great, but we need it for analytics key.
* generated protobuf
* missed files
* generated protobuf
* fix test in data race
* generate
* generated protobuf
* paul feedback
* fixes
* race
* more Paul feedback
* make resource notifier the default
* generated protobuf
* stop
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Webhook analytics event.
NOTE: Need a frostbyte73/core PR to merge and tag a release
PR pending: https://github.com/frostbyte73/core/pull/5
* sync submit
* generated protobuf
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Add sample rate and drift to RTP stats
* make tests a bit more robust
* generated protobuf
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
