locking caveats in file module

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

locking caveats in file module

Max Lapshin-2
We have been hunting a performance bug on customer installation during some time and have found that group of processes (about 300) each hour (when new archive file is opened) got locked with following stacktrace:

[{gen,do_call,4,[{file,"gen.erl"},{line,168}]},

 {gen_server,call,3,[{file,"gen_server.erl"},{line,208}]},

 {file,call,2,[{file,"file.erl"},{line,1487}]},

 {filelib,do_is_dir,2,[{file,"filelib.erl"},{line,130}]},

 {filelib,ensure_dir,1,[{file,"filelib.erl"},{line,231}]},

 {dvr_hour_storage,append1,2,...



Just a friendly reminder that most of file access functions are called via singleton process.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: locking caveats in file module

dmkolesnikov
Hello,

Have you tried a raw file access?

- Dmitry

> On May 18, 2016, at 9:37 AM, Max Lapshin <[hidden email]> wrote:
>
> We have been hunting a performance bug on customer installation during some time and have found that group of processes (about 300) each hour (when new archive file is opened) got locked with following stacktrace:
>
> [{gen,do_call,4,[{file,"gen.erl"},{line,168}]},
>
>  {gen_server,call,3,[{file,"gen_server.erl"},{line,208}]},
>
>  {file,call,2,[{file,"file.erl"},{line,1487}]},
>
>  {filelib,do_is_dir,2,[{file,"filelib.erl"},{line,130}]},
>
>  {filelib,ensure_dir,1,[{file,"filelib.erl"},{line,231}]},
>
>  {dvr_hour_storage,append1,2,...
>
>
>
>
>
> Just a friendly reminder that most of file access functions are called via singleton process.
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: locking caveats in file module

Michael Truog
In reply to this post by Max Lapshin-2
If this is related to the async thread pool, you could increase the number of async threads.  Since the async threads have no shared job queue, long filesystem blocking can cause async threads to accumulate jobs, and the only option is to increase the number of threads (despite the CPU/core count) if you have variable latency with filesystem use (which causes async threads to get clogged up unpredictably).

On 05/17/2016 11:37 PM, Max Lapshin wrote:
We have been hunting a performance bug on customer installation during some time and have found that group of processes (about 300) each hour (when new archive file is opened) got locked with following stacktrace:

[{gen,do_call,4,[{file,"gen.erl"},{line,168}]},

 {gen_server,call,3,[{file,"gen_server.erl"},{line,208}]},

 {file,call,2,[{file,"file.erl"},{line,1487}]},

 {filelib,do_is_dir,2,[{file,"filelib.erl"},{line,130}]},

 {filelib,ensure_dir,1,[{file,"filelib.erl"},{line,231}]},

 {dvr_hour_storage,append1,2,...



Just a friendly reminder that most of file access functions are called via singleton process.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: locking caveats in file module

Max Lapshin-2
async thread pool cannot help filelib:ensure_dir, because this library method is using only global singleton file server.

Customer has mounted NFS and thus file_server_1 is blocked waiting for lost UDP packet.


We have moved almost all file:read_file_info to explicit opening port and calling not via singleton, but via separate port, but filelib:ensure_dir is using singleton process.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions