Mnesia vs. ets+dets

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Mnesia vs. ets+dets

Charles Hixson-2
I'm going to be running on a single node, at least for the foreseeable
future, if that matters.

I expect the data to grow to be too large to fit into RAM, OTOH, only a
small subset will be needed at any one time.  Also I expect active items
to have frequent updates, but most items to rarely be updated (only when
a purge is contemplated).  There doesn't seem to be any way to control
the frequency with which Mnesia synchronizes to disk...but I also don't
see any information about how often it does so.  Once every few minutes
wouldn't be a problem, but on every change would be prohibitive.  On the
plus side, Mnesia will automatically handle splitting the data between
tables (when the data grows too large to fit into one dets file).

One way of handling the problem is to operate out of an ets database
which pulls in data as needed from various dets files and occasionally
synchronizes with them.  This starts looking complex, and Mnesia is
clearly intended to solve problems *like* this.  I just can't determine
that it will handle *this* problem.  If I could set the rate of
synchronization between RAM and disk, it would be the clear best
approach...but I haven't even found any documentation on that.

OTOH, I'm quite new to Erlang  (I've just written one successful
program) so I could easily be missing something that's extremely obvious
to someone more experienced.

Any comments or suggestions?
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia vs. ets+dets

PAILLEAU Eric

Hi,
did you look at change_table_copy/3 an dump_tables/1 ? This could let you decide when data have to go on disc (timer or/and sasl overload request).
regards

« Envoyé depuis mon mobile » Eric
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia vs. ets+dets

Charles Hixson-2
In reply to this post by Charles Hixson-2

On 11/09/2014 12:56 PM, T Ty wrote:
I usually advocate using Mnesia because any schema you come up with you will later want some form of fault tolerance and as Joe Armstrong keeps harping, you can't do fault tolerance with one node.

That said, Mnesia does not automatically split data between tables. You have to explicitly setup fragmented tables for it. 

To control how often Mnesia dumps from RAM to disc see:

-mnesia dc_dump_limit Number. Controls how often disc_copies tables are dumped from memory. Tables are dumped when filesize(Log) > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but increases disk space and startup times. The default is 4.



On Sun, Nov 9, 2014 at 8:00 PM, Charles Hixson <[hidden email]> wrote:
I'm going to be running on a single node, at least for the foreseeable future, if that matters.

I expect the data to grow to be too large to fit into RAM, OTOH, only a small subset will be needed at any one time.  Also I expect active items to have frequent updates, but most items to rarely be updated (only when a purge is contemplated).  There doesn't seem to be any way to control the frequency with which Mnesia synchronizes to disk...but I also don't see any information about how often it does so.  Once every few minutes wouldn't be a problem, but on every change would be prohibitive.  On the plus side, Mnesia will automatically handle splitting the data between tables (when the data grows too large to fit into one dets file).

One way of handling the problem is to operate out of an ets database which pulls in data as needed from various dets files and occasionally synchronizes with them.  This starts looking complex, and Mnesia is clearly intended to solve problems *like* this.  I just can't determine that it will handle *this* problem.  If I could set the rate of synchronization between RAM and disk, it would be the clear best approach...but I haven't even found any documentation on that.

OTOH, I'm quite new to Erlang  (I've just written one successful program) so I could easily be missing something that's extremely obvious to someone more experienced.

Any comments or suggestions?
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia vs. ets+dets

Charles Hixson-2
Sorry, something seems to have happened to what I originally sent.  So this is essentially what got cut:

Those controls tell me what I needed to know about the original question.  Every 3 minutes to save seems reasonable, though I'd probably want to increase the number of allowed writes without flush to 100,000.  But while looking things up I read some things that seemed to mean that Mnesia wanted to either hold the entire database in memory (disc_copies) or not hold any cache (disc_only_copies).  This doesn't sound good, as what I need is a cache of active records in RAM.  I can still do this by using disc_only_copies and using it to fill an ets table which I occasionally flush to Mnesia, but this seems unexpectedly clumsy.  Am I reading the documentation incorrectly?  I really don't expect to be able to fit the entire table(s) into RAM.



On 11/09/2014 05:26 PM, Charles Hixson wrote:

On 11/09/2014 12:56 PM, T Ty wrote:
I usually advocate using Mnesia because any schema you come up with you will later want some form of fault tolerance and as Joe Armstrong keeps harping, you can't do fault tolerance with one node.

....



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia vs. ets+dets

PAILLEAU Eric
In reply to this post by Charles Hixson-2
I'm afraid you found the answer by yourself.
You have to dump on disk older entries from RAM, but this need to code it.
by the way, coding in Erlang is cheap.
if so, do a module and share !

« Envoyé depuis mon mobile » Eric
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions