FIFO queue

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

FIFO queue

Roberto Ostinelli
Dear list,
I need to use a FIFO queue in Erlang, disk-backed.

Additions to the queue can be quite heavy (3000 / sec), so it's a write intensive operation.

Any ideas on best practices?

Thank you ^^_
r.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: FIFO queue

Roberto Ostinelli
To be a little more extensive: I'm familiar with the queue module, but it'd need an additional layer to write stuff to disk.

Therefore, I'm currently considering using an ordered_set mnesia table, with os:timestamp() values as the primary key.

Just wanted to avoid reinventing the wheel, so if something is out there... please let me know.

Best,
r.



On Wed, Jun 8, 2016 at 5:15 PM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
I need to use a FIFO queue in Erlang, disk-backed.

Additions to the queue can be quite heavy (3000 / sec), so it's a write intensive operation.

Any ideas on best practices?

Thank you ^^_
r.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: FIFO queue

dmkolesnikov
Hello,

You can look on my project, see branch on-disk-overflow. This is most recent best effort queue I am using in prod (optimized for read/write)

https://github.com/fogfish/esq

-Dmitry
Sent from my iPhone

On 08 Jun 2016, at 18:19, Roberto Ostinelli <[hidden email]> wrote:

To be a little more extensive: I'm familiar with the queue module, but it'd need an additional layer to write stuff to disk.

Therefore, I'm currently considering using an ordered_set mnesia table, with os:timestamp() values as the primary key.

Just wanted to avoid reinventing the wheel, so if something is out there... please let me know.

Best,
r.



On Wed, Jun 8, 2016 at 5:15 PM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
I need to use a FIFO queue in Erlang, disk-backed.

Additions to the queue can be quite heavy (3000 / sec), so it's a write intensive operation.

Any ideas on best practices?

Thank you ^^_
r.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: FIFO queue

Richard Jonas
Hi Roberto,

when we needed to implement something like that I stole the idea from Joe's repo (fast term storage), which is simply a raw (processless) binary file with buffer.
So you can implement something custom if you like. Also there is a good opportunity for using quickcheck. :)

For enterprise ready solutions consider integrate bitcask: https://github.com/basho/bitcask
I have never done that, but it is proven solution.

Richard


On Wed, Jun 8, 2016 at 5:59 PM, <[hidden email]> wrote:
Hello,

You can look on my project, see branch on-disk-overflow. This is most recent best effort queue I am using in prod (optimized for read/write)

https://github.com/fogfish/esq

-Dmitry
Sent from my iPhone

On 08 Jun 2016, at 18:19, Roberto Ostinelli <[hidden email]> wrote:

To be a little more extensive: I'm familiar with the queue module, but it'd need an additional layer to write stuff to disk.

Therefore, I'm currently considering using an ordered_set mnesia table, with os:timestamp() values as the primary key.

Just wanted to avoid reinventing the wheel, so if something is out there... please let me know.

Best,
r.



On Wed, Jun 8, 2016 at 5:15 PM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
I need to use a FIFO queue in Erlang, disk-backed.

Additions to the queue can be quite heavy (3000 / sec), so it's a write intensive operation.

Any ideas on best practices?

Thank you ^^_
r.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Richard Jonas
Erlang Solutions Hungary Kft

Address:
  Riverpark Office K.32
  Közraktár street 32. 3/1.
  1093 Budapest
  Hungary
Phone/fax:
  +36-1-7000-654

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: FIFO queue

Roberto Ostinelli
Thank you Dmitry, Ulf and Richard!
These seem all interesting possibilities and I will dive into them.

Best,
r.

On Thu, Jun 9, 2016 at 10:10 AM, Richard Jonas <[hidden email]> wrote:
Hi Roberto,

when we needed to implement something like that I stole the idea from Joe's repo (fast term storage), which is simply a raw (processless) binary file with buffer.
So you can implement something custom if you like. Also there is a good opportunity for using quickcheck. :)

For enterprise ready solutions consider integrate bitcask: https://github.com/basho/bitcask
I have never done that, but it is proven solution.

Richard


On Wed, Jun 8, 2016 at 5:59 PM, <[hidden email]> wrote:
Hello,

You can look on my project, see branch on-disk-overflow. This is most recent best effort queue I am using in prod (optimized for read/write)

https://github.com/fogfish/esq

-Dmitry
Sent from my iPhone

On 08 Jun 2016, at 18:19, Roberto Ostinelli <[hidden email]> wrote:

To be a little more extensive: I'm familiar with the queue module, but it'd need an additional layer to write stuff to disk.

Therefore, I'm currently considering using an ordered_set mnesia table, with os:timestamp() values as the primary key.

Just wanted to avoid reinventing the wheel, so if something is out there... please let me know.

Best,
r.



On Wed, Jun 8, 2016 at 5:15 PM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
I need to use a FIFO queue in Erlang, disk-backed.

Additions to the queue can be quite heavy (3000 / sec), so it's a write intensive operation.

Any ideas on best practices?

Thank you ^^_
r.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Richard Jonas
Erlang Solutions Hungary Kft

Address:
  Riverpark Office K.32
  Közraktár street 32. 3/1.
  1093 Budapest
  Hungary
Phone/fax:
  <a href="tel:%2B36-1-7000-654" value="+3617000654" target="_blank">+36-1-7000-654


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions