Mnesia deleting log file

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Mnesia deleting log file

ARUN P

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

Dan Gudmundsson-3
You should get 'node_not_running' when mnesia is not running (stopped or have crashed) so something is very strange.
Don't you get other mnesia log entries?

Try to enable debug printouts with 
mnesia:start([{debug, debug}]). or
erl -mnesia debug debug

/Dan

On Wed, Nov 22, 2017 at 5:16 AM Arun <[hidden email]> wrote:

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

Michael.K.Schmidt-2
In reply to this post by ARUN P

Hi Arun,

 

If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data).  We fight this issue on embedded devices a lot.

 

Look “Configuration Parameters” section here:

http://erlang.org/doc/man/mnesia.html#id69584

 

You will want to set dump_log_update_in_place to false.  This will make it write out the table to a new file, and rename it the very end.  

 

One other option to look at is dump_log_time_threshold to change how often items are flushed to disk.  For configuration data, you can likely set this to a few seconds.

 

These options can be set via the sys.config file as well

 

Mike

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Arun
Sent: Tuesday, November 21, 2017 10:16 PM
To: [hidden email]
Subject: [erlang-questions] Mnesia deleting log file

 

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

Jesper Louis Andersen-2
This thread makes we wonder about the following section in the Mnesia documentation:

"If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority."

In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should.

With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question.

[1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole.



On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt <[hidden email]> wrote:

Hi Arun,

 

If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data).  We fight this issue on embedded devices a lot.

 

Look “Configuration Parameters” section here:

http://erlang.org/doc/man/mnesia.html#id69584

 

You will want to set dump_log_update_in_place to false.  This will make it write out the table to a new file, and rename it the very end.  

 

One other option to look at is dump_log_time_threshold to change how often items are flushed to disk.  For configuration data, you can likely set this to a few seconds.

 

These options can be set via the sys.config file as well

 

Mike

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Arun
Sent: Tuesday, November 21, 2017 10:16 PM
To: [hidden email]
Subject: [erlang-questions] Mnesia deleting log file

 

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

Michael.K.Schmidt-2

I agree—setting dump_log_in_place to false makes extremely unlikely for mnesia to fail in this way.  Both JFFS2 and UBIFS ensure that rename is atomic, even when power is pulled.

 

For desktop applications, the culprit is likely the VM being killed, not the whole system crashing.  Thus the rename was either executed or not;  either way we have a complete DAT file. 

 

There is always a chance of things like this happening:  http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but this shouldn’t happen on most hardware.

 

Note that the default for dump_log_time_threshold is 3 minutes.  Speed vs Safety is always a trade-off, but many people will be surprised to lose data in a crash 2 minutes after saving.

 

 

From: Jesper Louis Andersen [mailto:[hidden email]]
Sent: Wednesday, November 22, 2017 7:54 AM
To: Michael Schmidt <[hidden email]>
Cc: Arun <[hidden email]>; [hidden email]
Subject: Re: [erlang-questions] Mnesia deleting log file

 

This thread makes we wonder about the following section in the Mnesia documentation:

"If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority."

In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should.

With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question.

[1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole.

 

 

On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt <[hidden email]> wrote:

Hi Arun,

 

If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data).  We fight this issue on embedded devices a lot.

 

Look “Configuration Parameters” section here:

http://erlang.org/doc/man/mnesia.html#id69584

 

You will want to set dump_log_update_in_place to false.  This will make it write out the table to a new file, and rename it the very end.  

 

One other option to look at is dump_log_time_threshold to change how often items are flushed to disk.  For configuration data, you can likely set this to a few seconds.

 

These options can be set via the sys.config file as well

 

Mike

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Arun
Sent: Tuesday, November 21, 2017 10:16 PM
To:
[hidden email]
Subject: [erlang-questions] Mnesia deleting log file

 

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

ARUN P

Even after running the application for 2 days I do a power off of the device, I'm facing this problem.
Currently, the value of "dump_log_write_threshold" configuration is set to 1.

Anyhow, I'll try to reproduce the problem by setting the "dump_log_update_in_place" to false and
intimate the status.


On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote:

I agree—setting dump_log_in_place to false makes extremely unlikely for mnesia to fail in this way.  Both JFFS2 and UBIFS ensure that rename is atomic, even when power is pulled.

 

For desktop applications, the culprit is likely the VM being killed, not the whole system crashing.  Thus the rename was either executed or not;  either way we have a complete DAT file. 

 

There is always a chance of things like this happening:  http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but this shouldn’t happen on most hardware.

 

Note that the default for dump_log_time_threshold is 3 minutes.  Speed vs Safety is always a trade-off, but many people will be surprised to lose data in a crash 2 minutes after saving.

 

 

From: Jesper Louis Andersen [[hidden email]]
Sent: Wednesday, November 22, 2017 7:54 AM
To: Michael Schmidt [hidden email]
Cc: Arun [hidden email]; [hidden email]
Subject: Re: [erlang-questions] Mnesia deleting log file

 

This thread makes we wonder about the following section in the Mnesia documentation:

"If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority."

In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should.

With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question.

[1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole.

 

 

On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt <[hidden email]> wrote:

Hi Arun,

 

If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data).  We fight this issue on embedded devices a lot.

 

Look “Configuration Parameters” section here:

http://erlang.org/doc/man/mnesia.html#id69584

 

You will want to set dump_log_update_in_place to false.  This will make it write out the table to a new file, and rename it the very end.  

 

One other option to look at is dump_log_time_threshold to change how often items are flushed to disk.  For configuration data, you can likely set this to a few seconds.

 

These options can be set via the sys.config file as well

 

Mike

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Arun
Sent: Tuesday, November 21, 2017 10:16 PM
To:
[hidden email]
Subject: [erlang-questions] Mnesia deleting log file

 

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

ARUN P
In reply to this post by Michael.K.Schmidt-2

Even after setting the "dump_log_update_in_place" configuration to false, the data is not getting retained
after a power outage. Any more configurations that can be tried.

Regards,
Arun P


On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote:

I agree—setting dump_log_in_place to false makes extremely unlikely for mnesia to fail in this way.  Both JFFS2 and UBIFS ensure that rename is atomic, even when power is pulled.

 

For desktop applications, the culprit is likely the VM being killed, not the whole system crashing.  Thus the rename was either executed or not;  either way we have a complete DAT file. 

 

There is always a chance of things like this happening:  http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but this shouldn’t happen on most hardware.

 

Note that the default for dump_log_time_threshold is 3 minutes.  Speed vs Safety is always a trade-off, but many people will be surprised to lose data in a crash 2 minutes after saving.

 

 

From: Jesper Louis Andersen [[hidden email]]
Sent: Wednesday, November 22, 2017 7:54 AM
To: Michael Schmidt [hidden email]
Cc: Arun [hidden email]; [hidden email]
Subject: Re: [erlang-questions] Mnesia deleting log file

 

This thread makes we wonder about the following section in the Mnesia documentation:

"If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority."

In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should.

With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question.

[1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole.

 

 

On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt <[hidden email]> wrote:

Hi Arun,

 

If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data).  We fight this issue on embedded devices a lot.

 

Look “Configuration Parameters” section here:

http://erlang.org/doc/man/mnesia.html#id69584

 

You will want to set dump_log_update_in_place to false.  This will make it write out the table to a new file, and rename it the very end.  

 

One other option to look at is dump_log_time_threshold to change how often items are flushed to disk.  For configuration data, you can likely set this to a few seconds.

 

These options can be set via the sys.config file as well

 

Mike

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Arun
Sent: Tuesday, November 21, 2017 10:16 PM
To:
[hidden email]
Subject: [erlang-questions] Mnesia deleting log file

 

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

Jesper Louis Andersen-2
I'm curious: are the data written as part of a mnesia transaction?


On Thu, Nov 23, 2017 at 12:22 PM Arun <[hidden email]> wrote:

Even after setting the "dump_log_update_in_place" configuration to false, the data is not getting retained
after a power outage. Any more configurations that can be tried.

Regards,
Arun P



On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote:

I agree—setting dump_log_in_place to false makes extremely unlikely for mnesia to fail in this way.  Both JFFS2 and UBIFS ensure that rename is atomic, even when power is pulled.

 

For desktop applications, the culprit is likely the VM being killed, not the whole system crashing.  Thus the rename was either executed or not;  either way we have a complete DAT file. 

 

There is always a chance of things like this happening:  http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but this shouldn’t happen on most hardware.

 

Note that the default for dump_log_time_threshold is 3 minutes.  Speed vs Safety is always a trade-off, but many people will be surprised to lose data in a crash 2 minutes after saving.

 

 

From: Jesper Louis Andersen [[hidden email]]
Sent: Wednesday, November 22, 2017 7:54 AM
To: Michael Schmidt [hidden email]
Cc: Arun [hidden email]; [hidden email]
Subject: Re: [erlang-questions] Mnesia deleting log file

 

This thread makes we wonder about the following section in the Mnesia documentation:

"If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority."

In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should.

With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question.

[1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole.

 

 

On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt <[hidden email]> wrote:

Hi Arun,

 

If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data).  We fight this issue on embedded devices a lot.

 

Look “Configuration Parameters” section here:

http://erlang.org/doc/man/mnesia.html#id69584

 

You will want to set dump_log_update_in_place to false.  This will make it write out the table to a new file, and rename it the very end.  

 

One other option to look at is dump_log_time_threshold to change how often items are flushed to disk.  For configuration data, you can likely set this to a few seconds.

 

These options can be set via the sys.config file as well

 

Mike

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Arun
Sent: Tuesday, November 21, 2017 10:16 PM
To:
[hidden email]
Subject: [erlang-questions] Mnesia deleting log file

 

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Mnesia deleting log file

Michael.K.Schmidt-2
In reply to this post by ARUN P

Can you provide more details?  Does this happen if you kill the VM?  Or are you pulling power on the machine?  If it’s the later case, what filesystem are you using?

 

Also, please try setting dump_log_time_threshold to 500 or so.  This will flush it within a half second.  I don’t think you want dump_log_write_threshold set to 1—this will cause the file to be repeatedly written.  The goal is to cache the writes in RAM so that a single write to the FS happens.

 

From: Arun [mailto:[hidden email]]
Sent: Thursday, November 23, 2017 5:22 AM
To: Michael Schmidt <[hidden email]>; Jesper Louis Andersen <[hidden email]>
Cc: [hidden email]
Subject: Re: [erlang-questions] Mnesia deleting log file

 

Even after setting the "dump_log_update_in_place" configuration to false, the data is not getting retained
after a power outage. Any more configurations that can be tried.

Regards,
Arun P

On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote:

I agree—setting dump_log_in_place to false makes extremely unlikely for mnesia to fail in this way.  Both JFFS2 and UBIFS ensure that rename is atomic, even when power is pulled.

 

For desktop applications, the culprit is likely the VM being killed, not the whole system crashing.  Thus the rename was either executed or not;  either way we have a complete DAT file. 

 

There is always a chance of things like this happening:  http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but this shouldn’t happen on most hardware.

 

Note that the default for dump_log_time_threshold is 3 minutes.  Speed vs Safety is always a trade-off, but many people will be surprised to lose data in a crash 2 minutes after saving.

 

 

From: Jesper Louis Andersen [[hidden email]]
Sent: Wednesday, November 22, 2017 7:54 AM
To: Michael Schmidt
[hidden email]
Cc: Arun
[hidden email]; [hidden email]
Subject: Re: [erlang-questions] Mnesia deleting log file

 

This thread makes we wonder about the following section in the Mnesia documentation:

"If a power failure occurs during the dump, this can cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia copies the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files are renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files becomes much smaller with this strategy. However, the actual dumping of the transaction log becomes considerably slower. The system designer must decide whether speed or safety is the higher priority."

In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS crash might leave you with a temporary file and the target file, hardlinked as if ln() was called, but the move itself is atomic, even under OS crashes. Of course, this assumes you correctly wrote your temporary file, fsync()'ed the file and then performed the rename() as you should.

With that information in mind, what is the scenario which can make mnesia fail if the dump_log_update_in_place is set to false? There may be a window, but the operating system cannot be the provider of this window at all, unless it is incorrectly implemented or configured[1]. Hence my question.

[1] The primary concern are write caches on disks which can have a write hole unless they have proper (working!) battery backup of their cache or are able to write their memory to a small solid state part of the chip. Or if you are using ZFS where copy-on-write semantics completely eliminates the write hole.

 

 

On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt <[hidden email]> wrote:

Hi Arun,

 

If you happen to reboot when Mnesia is writing out its logfile, it can become corrupt (and lose data).  We fight this issue on embedded devices a lot.

 

Look “Configuration Parameters” section here:

http://erlang.org/doc/man/mnesia.html#id69584

 

You will want to set dump_log_update_in_place to false.  This will make it write out the table to a new file, and rename it the very end.  

 

One other option to look at is dump_log_time_threshold to change how often items are flushed to disk.  For configuration data, you can likely set this to a few seconds.

 

These options can be set via the sys.config file as well

 

Mike

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Arun
Sent: Tuesday, November 21, 2017 10:16 PM
To:
[hidden email]
Subject: [erlang-questions] Mnesia deleting log file

 

Dear all,

I've a program written in erlang which uses Mnesia application as the database application. I've a table created by name "configuration_table" which stores

certain configurations that need to be persistent. Occasionally, whenever I restart my program the following error is thrown by mnesia and I end up losing

all the persistent configurations.

I've searched about this problem in the erlang documentation and all it tells is "Node not running". What could be the probable cause for this problem

and how do I fix it?

Mnesia('[hidden email]'): Data may be missing, Corrupt logfile deleted: "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
'test[hidden email]'}


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________



______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions