Multiple websocket subscribers to Kafka topic using Cowboy and Brod

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple websocket subscribers to Kafka topic using Cowboy and Brod

Noah Silverman
Hi,

I'm building an application where people can use websocket connections to receive streaming data from Kafka topics. 

Currently, I use Cowboy 2.5 to manage the websockets.  It works great!

However, I can only have ONE websocket subscribe to a Kafka topic using brod.  When The second person connects and subscribes, brod returns an error: {error,{already_subscribed_by,<0.1374.0>}}

In the Cowboy websocket handler, I handle subscription requests with a simple brod subscribe call:
brod:subscribe(api_brod_client, self(), Topic, 0, [])


Perhaps that's not the optimal way.

I did read that brod has a higher level call:  brod_topic_subscriber:start_link.  However, that wants a module it can callback with an init.  Since there is already a websocket process running from Cowboy, I don't want broad to start it again.

does anybody know of a simple way to have each websocket process just subscribe to the Kafka topic.  The brod_topic_subscriber might work, if there is a way to just feed the mssages back to the cowboy process calling it.

Thoughts?

--
Noah




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

publickey - noah@helios.ai - 0xBCEFA873.asc (4K) Download Attachment
signature.asc (875 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Multiple websocket subscribers to Kafka topic using Cowboy and Brod

Grigory Fateyev-2
Hello, Noah!

I'm not sure about brod, but I think the first arg of brod:subscribe/5 [1] should be dynamic and a unique per client. It can be username or user id.



пт, 16 нояб. 2018 г. в 16:41, Noah Silverman <[hidden email]>:
Hi,

I'm building an application where people can use websocket connections to receive streaming data from Kafka topics. 

Currently, I use Cowboy 2.5 to manage the websockets.  It works great!

However, I can only have ONE websocket subscribe to a Kafka topic using brod.  When The second person connects and subscribes, brod returns an error: {error,{already_subscribed_by,<0.1374.0>}}

In the Cowboy websocket handler, I handle subscription requests with a simple brod subscribe call:
brod:subscribe(api_brod_client, self(), Topic, 0, [])


Perhaps that's not the optimal way.

I did read that brod has a higher level call:  brod_topic_subscriber:start_link.  However, that wants a module it can callback with an init.  Since there is already a websocket process running from Cowboy, I don't want broad to start it again.

does anybody know of a simple way to have each websocket process just subscribe to the Kafka topic.  The brod_topic_subscriber might work, if there is a way to just feed the mssages back to the cowboy process calling it.

Thoughts?

--
Noah



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions