ets+dets

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

ets+dets

Vlad Dumitrescu-2
Hi all,

I need to implement a simple key-value store that should be persistent (on disk) but have a fast in-memory cache for the recently used entries (possibly with a configurable policy) and with little concurrent access. I was thinking of an ets+dets combo, but it may be tricky to get the details right. Is there something like this out there? Would it be better to just use mnesia? I'd like to keep it lean, but if third-party databases fit the bill, I will look at them.

best regards,
Vlad


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: ets+dets

Lloyd R. Prentice-2
Hi Vlad,

My understanding is that dets lookups do involve disk-lookup latencies. But maybe ETSHandler will provide the functionality you need.

https://github.com/zsoci/ETSHandler

All the best,

Lloyd

Sent from my iPad

> On May 9, 2017, at 4:00 PM, Vlad Dumitrescu <[hidden email]> wrote:
>
> Hi all,
>
> I need to implement a simple key-value store that should be persistent (on disk) but have a fast in-memory cache for the recently used entries (possibly with a configurable policy) and with little concurrent access. I was thinking of an ets+dets combo, but it may be tricky to get the details right. Is there something like this out there? Would it be better to just use mnesia? I'd like to keep it lean, but if third-party databases fit the bill, I will look at them.
>
> best regards,
> Vlad
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: ets+dets

Vlad Dumitrescu-2
Hi Lloyd,

That is an interesting tool, I might be able to divert it from its original usage. Thanks!

regards,
Vlad



On Wed, May 10, 2017 at 1:35 AM, Lloyd R. Prentice <[hidden email]> wrote:
Hi Vlad,

My understanding is that dets lookups do involve disk-lookup latencies. But maybe ETSHandler will provide the functionality you need.

https://github.com/zsoci/ETSHandler

All the best,

Lloyd

Sent from my iPad

> On May 9, 2017, at 4:00 PM, Vlad Dumitrescu <[hidden email]> wrote:
>
> Hi all,
>
> I need to implement a simple key-value store that should be persistent (on disk) but have a fast in-memory cache for the recently used entries (possibly with a configurable policy) and with little concurrent access. I was thinking of an ets+dets combo, but it may be tricky to get the details right. Is there something like this out there? Would it be better to just use mnesia? I'd like to keep it lean, but if third-party databases fit the bill, I will look at them.
>
> best regards,
> Vlad
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: ets+dets

Anders Ramsell
In reply to this post by Vlad Dumitrescu-2
Vlad Dumitrescu wrote:

> Hi all,
>
> I need to implement a simple key-value store that should be persistent
> (on disk) but have a fast in-memory cache for the recently used
> entries (possibly with a configurable policy) and with little
> concurrent access. I was thinking of an ets+dets combo, but it may be
> tricky to get the details right. Is there something like this out
> there? Would it be better to just use mnesia? I'd like to keep it
> lean, but if third-party databases fit the bill, I will look at them.
>
In our system we have a few cases that "combine" ets and dets in roughly
this way:
- on startup the process fills the ets table with all contents of the
dets table
- write is done in both ets and dets
- read is done from ets only

Perhaps not so good if there are a lot of writes but that is not the
case where we do this.

/Anders
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: ets+dets

Vlad Dumitrescu-2
On Thu, May 11, 2017 at 10:01 PM, Anders Ramsell <[hidden email]> wrote:
Vlad Dumitrescu wrote:
Hi all,

I need to implement a simple key-value store that should be persistent (on disk) but have a fast in-memory cache for the recently used entries (possibly with a configurable policy) and with little concurrent access. I was thinking of an ets+dets combo, but it may be tricky to get the details right. Is there something like this out there? Would it be better to just use mnesia? I'd like to keep it lean, but if third-party databases fit the bill, I will look at them.

In our system we have a few cases that "combine" ets and dets in roughly this way:
- on startup the process fills the ets table with all contents of the dets table
- write is done in both ets and dets
- read is done from ets only

Perhaps not so good if there are a lot of writes but that is not the case where we do this.
 
Hi!

That is a straight use, but the point in my case would be to be able to not keep everything in memory at the same time. A two-level cache, I think is the right concept. Thanks!

best regards,
Vlad



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: ets+dets

Adam Rutkowski-3
In reply to this post by Vlad Dumitrescu-2
On Tue, May 9, 2017, at 22:00, Vlad Dumitrescu wrote:

I was thinking of an ets+dets combo, but it may be tricky to get the details right. Is there something like this out there? Would it be better to just use mnesia? 

Hi Vlad,

Apologies if this sounds blunt, but isn't mnesia in fact an ETS+DETS combo with lots of tricky details?

With DETS alone I often get the tables corrupted, erlang-history is the prime example -- `rm ~/.erlang-hist.nonode@nohost` is my weekly operation. 

What other properties are you looking for? Why _not_ mnesia in your case?

Cheers

/A.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: ets+dets

Leonard Boyce-2
On Thu, May 11, 2017 at 4:32 PM, Adam Rutkowski <[hidden email]> wrote:

> On Tue, May 9, 2017, at 22:00, Vlad Dumitrescu wrote:
>
> I was thinking of an ets+dets combo, but it may be tricky to get the details
> right. Is there something like this out there? Would it be better to just
> use mnesia?
>
>
> Hi Vlad,
>
> Apologies if this sounds blunt, but isn't mnesia in fact an ETS+DETS combo
> with lots of tricky details?
>
> With DETS alone I often get the tables corrupted, erlang-history is the
> prime example -- `rm ~/.erlang-hist.nonode@nohost` is my weekly operation.
>
> What other properties are you looking for? Why _not_ mnesia in your case?
>
> Cheers
>
> /A.
>

Hi Adam,

It seems to be a common misconception that mnesia disc_copies behaves like this.
I used to think so myself. In fact it keeps all data in memory and
flushes to disc,
so it does not meet Vlad's requirements for minimal in-memory cache, with disk
storage.

As was pointed out to me when I asked about it in '15, from the docs:
"If a table is of type disc_copies at a certain node, the entire table
is resident in RAM memory and on disc."

Leonard

>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: ets+dets

Vlad Dumitrescu-2
In reply to this post by Adam Rutkowski-3
On Thu, May 11, 2017 at 10:32 PM, Adam Rutkowski <[hidden email]> wrote:
On Tue, May 9, 2017, at 22:00, Vlad Dumitrescu wrote:

I was thinking of an ets+dets combo, but it may be tricky to get the details right. Is there something like this out there? Would it be better to just use mnesia? 

Hi Vlad,

Apologies if this sounds blunt, but isn't mnesia in fact an ETS+DETS combo with lots of tricky details?

With DETS alone I often get the tables corrupted, erlang-history is the prime example -- `rm ~/.erlang-hist.nonode@nohost` is my weekly operation. 

What other properties are you looking for? Why _not_ mnesia in your case?

 Hi Adam,

No problems. There are of course additional requirements, but the main issue with mnesia is that I would like to be able to build the data store from fragments that are precomputed inependently from each other, without needing to import them or something like that. These fragments can be dets tables or even simple files (one per entry), with the in-memory table just keeping references to them and sometimes computing and adding new ones.

best regards,
Vlad


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions