Quantcast

RabbitMQ message consumption

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RabbitMQ message consumption

Kareem Hepburn
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!

--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Alexey Lebedeff-2

Hi Kareem,

You don't need a secondary database. Unless you consume in a 'no-ack'
mode, RabbitMQ expects an explicit 'ack' message from you for every
message that it sends to you. This the message will not be sent to other
consumers - unless there is an explicit reject or a connection
loss. Also you should look into QoS parameter to prevent overwhelming
your consumer.

Best,
Alexey

Kareem Hepburn writes:

> Is it possible with RabbitMQ to consume a message without it being deleted
> and be deleted manually?
>
> Because let's say I consumed a message and started to process it and half
> way through processing it the application fails. I want to be able to
> re-process that message again. Is it possible to not delete the message and
> delete it manually after my process has successfully completed?
>
> To avoid duplication, I plan to use a secondary database, either Redis or
> Memcache to store the ID of the message to ensure that it's not being
> processed by more than one application.
>
> Thanks!
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Sölvi Páll Ásgeirsson
In reply to this post by Kareem Hepburn
I think you want to use RabbitMQ's message acknowledgement.

The general idea is that you:
- Consume a message
- Process it
- Acknowledge to RabbitMQ that you've processed it, which causes it to be deleted from the queue.


If you're using the Erlang RabbitMQ client, you can do something like this when subscribing:

  #'basic.consume_ok'{} = amqp_channel:subscribe(Channel, #'basic.consume'{queue = Queue, no_ack = false}, self()),

Then, when you've handled your message, you call:

    amqp_channel:cast(Channel, #'basic.ack'{delivery_tag = Tag}).

Hope this helps

Cheers
Sölvi


On Fri, May 12, 2017 at 11:55 AM, Kareem Hepburn <[hidden email]> wrote:
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!

--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Bastien Chamagne
In reply to this post by Kareem Hepburn

Hi Kareem, you should ack only when your application have processed the message. See https://www.rabbitmq.com/confirms.html


On 12/05/2017 13:55, Kareem Hepburn wrote:
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!

--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Kareem Hepburn
In reply to this post by Sölvi Páll Ásgeirsson
It most certainly does! Thank you!

On Fri, May 12, 2017 at 7:42 AM, Sölvi Páll Ásgeirsson <[hidden email]> wrote:
I think you want to use RabbitMQ's message acknowledgement.

The general idea is that you:
- Consume a message
- Process it
- Acknowledge to RabbitMQ that you've processed it, which causes it to be deleted from the queue.


If you're using the Erlang RabbitMQ client, you can do something like this when subscribing:

  #'basic.consume_ok'{} = amqp_channel:subscribe(Channel, #'basic.consume'{queue = Queue, no_ack = false}, self()),

Then, when you've handled your message, you call:

    amqp_channel:cast(Channel, #'basic.ack'{delivery_tag = Tag}).

Hope this helps

Cheers
Sölvi


On Fri, May 12, 2017 at 11:55 AM, Kareem Hepburn <[hidden email]> wrote:
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!

--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions





--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Kareem Hepburn
In reply to this post by Bastien Chamagne
What about reject? The thing is, if there are multiple nodes of a worker that's consuming the message I want it that if a worker already read the message, it will not be possible to for it to be delivered to the other consumers so I am guessing ack will do this for me and reject will requeue it if my process fails.

Am I correct on this?

On Fri, May 12, 2017 at 7:56 AM, Bastien CHAMAGNE <[hidden email]> wrote:

Hi Kareem, you should ack only when your application have processed the message. See https://www.rabbitmq.com/confirms.html


On 12/05/2017 13:55, Kareem Hepburn wrote:
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!

--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Bastien Chamagne

Once a consumer reads a message, that message goes from "READY" to "UNACKED". It's not deleted that fast! As long as it's in "UNACKED" state, it will not be delivered to anyone else. Then once your consumer have finished processing the message, it must send an ACK that will remove the message from rabbitMQ. If your consumer crashes midway through, there will be no ACK and rabbitMQ knows that it can be redistributed (message is "READY" again).


On 12/05/2017 15:16, Kareem Hepburn wrote:
What about reject? The thing is, if there are multiple nodes of a worker that's consuming the message I want it that if a worker already read the message, it will not be possible to for it to be delivered to the other consumers so I am guessing ack will do this for me and reject will requeue it if my process fails.

Am I correct on this?

On Fri, May 12, 2017 at 7:56 AM, Bastien CHAMAGNE <[hidden email]> wrote:

Hi Kareem, you should ack only when your application have processed the message. See https://www.rabbitmq.com/confirms.html


On 12/05/2017 13:55, Kareem Hepburn wrote:
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!

--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions
--

kareemgan

Kareem Gan / Software Engineer [hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Kareem Hepburn
Gotcha. Thanks! I was using Reject earlier but I guess Ack is sufficient to what I need to do.

On Fri, May 12, 2017 at 8:24 AM, Bastien CHAMAGNE <[hidden email]> wrote:

Once a consumer reads a message, that message goes from "READY" to "UNACKED". It's not deleted that fast! As long as it's in "UNACKED" state, it will not be delivered to anyone else. Then once your consumer have finished processing the message, it must send an ACK that will remove the message from rabbitMQ. If your consumer crashes midway through, there will be no ACK and rabbitMQ knows that it can be redistributed (message is "READY" again).


On 12/05/2017 15:16, Kareem Hepburn wrote:
What about reject? The thing is, if there are multiple nodes of a worker that's consuming the message I want it that if a worker already read the message, it will not be possible to for it to be delivered to the other consumers so I am guessing ack will do this for me and reject will requeue it if my process fails.

Am I correct on this?

On Fri, May 12, 2017 at 7:56 AM, Bastien CHAMAGNE <[hidden email]> wrote:

Hi Kareem, you should ack only when your application have processed the message. See https://www.rabbitmq.com/confirms.html


On 12/05/2017 13:55, Kareem Hepburn wrote:
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!

--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions
--

kareemgan

Kareem Gan / Software Engineer [hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.




--

kareemgan

Kareem Gan / Software Engineer
[hidden email]

Skype Facebook Twitter Google Plus Linkedin Skype

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Nathaniel Waisbrot
If the connection drops and there are un-acked messages, Rabbit will reject them all (returning them to a "ready" state). If you set up your supervision tree such that an error while working on a message takes out the process that was talking to Rabbit, this can be convenient.

If your Rabbit consumer is just handing off messages and you don't plan for it to die itself, you will need to track messages you've handed off and explicitly reject those that went to a dead consumer.



On May 12, 2017, at 9:31 AM, Kareem Hepburn <[hidden email]> wrote:

Gotcha. Thanks! I was using Reject earlier but I guess Ack is sufficient to what I need to do.

On Fri, May 12, 2017 at 8:24 AM, Bastien CHAMAGNE <[hidden email]> wrote:

Once a consumer reads a message, that message goes from "READY" to "UNACKED". It's not deleted that fast! As long as it's in "UNACKED" state, it will not be delivered to anyone else. Then once your consumer have finished processing the message, it must send an ACK that will remove the message from rabbitMQ. If your consumer crashes midway through, there will be no ACK and rabbitMQ knows that it can be redistributed (message is "READY" again).


On 12/05/2017 15:16, Kareem Hepburn wrote:
What about reject? The thing is, if there are multiple nodes of a worker that's consuming the message I want it that if a worker already read the message, it will not be possible to for it to be delivered to the other consumers so I am guessing ack will do this for me and reject will requeue it if my process fails.

Am I correct on this?

On Fri, May 12, 2017 at 7:56 AM, Bastien CHAMAGNE <[hidden email]> wrote:

Hi Kareem, you should ack only when your application have processed the message. See https://www.rabbitmq.com/confirms.html


On 12/05/2017 13:55, Kareem Hepburn wrote:
Is it possible with RabbitMQ to consume a message without it being deleted and be deleted manually?

Because let's say I consumed a message and started to process it and half way through processing it the application fails. I want to be able to re-process that message again. Is it possible to not delete the message and delete it manually after my process has successfully completed?

To avoid duplication, I plan to use a secondary database, either Redis or Memcache to store the ID of the message to ensure that it's not being processed by more than one application.

Thanks!


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RabbitMQ message consumption

Jesper Louis Andersen-2
In reply to this post by Kareem Hepburn
On Fri, May 12, 2017 at 3:40 PM Kareem Hepburn <[hidden email]> wrote:
Gotcha. Thanks! I was using Reject earlier but I guess Ack is sufficient to what I need to do.


When you reject a message, you can choose if it should be requeued or not. Here it is good to build your application such that it knows if the message is likely to succeed later or not. If you can't parse the message, say, then it will never work. If your database is currently down, you can try later, for instance. Think the difference between HTTP 4xx errors (client side) or 5xx errors (server side).

You want to make the discrimination because otherwise a single "poisonous" message will get sent to all your workers and that can take all the workers down by infecting them one by one. Granted, Erlang makes that harder by restarting. Yet, I've seen such problems in production. I've also seen what it does to a Python system with no supervisor restarts. It aint' pretty :)
 

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...