mnesia lookup weirdness

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

mnesia lookup weirdness

Chandru-4
Hi,

One of my production nodes running R12B-5 is exhibiting some strange behaviour.

mnesia:dirty_read(gtp_tid, mnesia:dirty_first(gtp_tid)).

gives me an empty list. The table has 39K entries. How do I find out what is happening? I tried using ets operations directly and get the same result (which is what mnesia is doing behind the scenes anyway). I get the same empty result using a transaction to read. Any clues?

96> ets:info(gtp_tid).
[{memory,213857900},
 {owner,<0.741.0>},
 {name,gtp_tid},
 {size,39088},
 {node,node@server},
 {named_table,true},
 {type,set},
 {keypos,2},
 {protection,public}]

The same operation works ok on other tables.

cheers
Chandru


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: mnesia lookup weirdness

Chandru-4
For the record, I've found out what was causing this.

A process was regularly executing ets:safe_fixtable(true) on this table, and not finishing off with ets:safe_fixtable(false).

This meant that the behaviour of mnesia:dirty_first was affected, and was also causing increased memory consumption.

Thanks to Håkan and Sverker for helping me find the root cause.

cheers
Chandru

On 1 July 2011 14:44, Chandru <[hidden email]> wrote:
Hi,

One of my production nodes running R12B-5 is exhibiting some strange behaviour.

mnesia:dirty_read(gtp_tid, mnesia:dirty_first(gtp_tid)).

gives me an empty list. The table has 39K entries. How do I find out what is happening? I tried using ets operations directly and get the same result (which is what mnesia is doing behind the scenes anyway). I get the same empty result using a transaction to read. Any clues?

96> ets:info(gtp_tid).
[{memory,213857900},
 {owner,<0.741.0>},
 {name,gtp_tid},
 {size,39088},
 {node,node@server},
 {named_table,true},
 {type,set},
 {keypos,2},
 {protection,public}]

The same operation works ok on other tables.

cheers
Chandru



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: mnesia lookup weirdness

Sverker Eriksson
I have to confess...

ets:first is not supposed to behave like that, returning already deleted
keys,
even if the table is fixated. You can of course have a race with a
deleting process,
but the key should have existed in the table some time during the
execution of ets:first().

I think you suffer from this bug, fixed in R13B01:

http://www.erlang.org/download/otp_src_R13B01.readme:

    OTP-8040  Various fixes in ETS: ets:first could return a deleted key in
              a fixated table. ets:lookup could return objects out of order
              if a deleted object was re-inserted into a fixed bag.
              ets:delete_object could fail to delete duplicate objects in a
              duplicate_bag.


The irony being that you probably wouldn't had found the bug of
the leaking ets:safe_fixtable now if ets:first had behaved like it should.

Leaking ets:safe_fixtable can be a nasty thing as it might result in
gradual performance degradation,
both memory and cpu wise. Deleted objects are not deallocated and the
table is not allowed
to adjust its size (rehash) until it is "unfixated".

/Sverker, Erlang/OTP


Chandru wrote:

> For the record, I've found out what was causing this.
>
> A process was regularly executing ets:safe_fixtable(true) on this table, and
> not finishing off with ets:safe_fixtable(false).
>
> This meant that the behaviour of mnesia:dirty_first was affected, and was
> also causing increased memory consumption.
>
> Thanks to Håkan and Sverker for helping me find the root cause.
>
> cheers
> Chandru
>
> On 1 July 2011 14:44, Chandru <[hidden email]> wrote:
>
>  
>> Hi,
>>
>> One of my production nodes running R12B-5 is exhibiting some strange
>> behaviour.
>>
>> mnesia:dirty_read(gtp_tid, mnesia:dirty_first(gtp_tid)).
>>
>> gives me an empty list. The table has 39K entries. How do I find out what
>> is happening? I tried using ets operations directly and get the same result
>> (which is what mnesia is doing behind the scenes anyway). I get the same
>> empty result using a transaction to read. Any clues?
>>
>> 96> ets:info(gtp_tid).
>> [{memory,213857900},
>>  {owner,<0.741.0>},
>>  {name,gtp_tid},
>>  {size,39088},
>>  {node,node@server},
>>  {named_table,true},
>>  {type,set},
>>  {keypos,2},
>>  {protection,public}]
>>
>> The same operation works ok on other tables.
>>
>> cheers
>> Chandru
>>
>>
>>    
>
>  


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: mnesia lookup weirdness

Chandru-4
On 14 July 2011 10:36, Sverker Eriksson <[hidden email]> wrote:
I have to confess...

ets:first is not supposed to behave like that, returning already deleted keys,
even if the table is fixated. You can of course have a race with a deleting process,
but the key should have existed in the table some time during the execution of ets:first().

I think you suffer from this bug, fixed in R13B01:

http://www.erlang.org/download/otp_src_R13B01.readme:

  OTP-8040  Various fixes in ETS: ets:first could return a deleted key in
             a fixated table. ets:lookup could return objects out of order
             if a deleted object was re-inserted into a fixed bag.
             ets:delete_object could fail to delete duplicate objects in a
             duplicate_bag.


The irony being that you probably wouldn't had found the bug of
the leaking ets:safe_fixtable now if ets:first had behaved like it should.

Leaking ets:safe_fixtable can be a nasty thing as it might result in gradual performance degradation,
both memory and cpu wise. Deleted objects are not deallocated and the table is not allowed
to adjust its size (rehash) until it is "unfixated".


Yes, you are right about this. The reason I stumbled upon this was because the erlang node was gradually eating up all available memory and crashing. I could find no obvious culprit, until I came across this!

Chandru



Chandru wrote:
For the record, I've found out what was causing this.

A process was regularly executing ets:safe_fixtable(true) on this table, and
not finishing off with ets:safe_fixtable(false).

This meant that the behaviour of mnesia:dirty_first was affected, and was
also causing increased memory consumption.

Thanks to Håkan and Sverker for helping me find the root cause.

cheers
Chandru

On 1 July 2011 14:44, Chandru <[hidden email]> wrote:

 
Hi,

One of my production nodes running R12B-5 is exhibiting some strange
behaviour.

mnesia:dirty_read(gtp_tid, mnesia:dirty_first(gtp_tid)).

gives me an empty list. The table has 39K entries. How do I find out what
is happening? I tried using ets operations directly and get the same result
(which is what mnesia is doing behind the scenes anyway). I get the same
empty result using a transaction to read. Any clues?

96> ets:info(gtp_tid).
[{memory,213857900},
 {owner,<0.741.0>},
 {name,gtp_tid},
 {size,39088},
 {node,node@server},
 {named_table,true},
 {type,set},
 {keypos,2},
 {protection,public}]

The same operation works ok on other tables.

cheers
Chandru


   

 





_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions