Information on file:advise/4

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Information on file:advise/4

Valentin Micic-5
Hi,

Would it be possible for someone to shed a bit more light on various posix_file_advise() options in file:advise/4 function?

For example, I could guess that 'will_need' may ask OS to buffer the file, whilst 'don't_need' may have an opposite effect.
I could also guess that 'sequential' should instruct OS to provide for a larger working I/O buffer than in a case of 'random'.

But what about 'no_resue' or 'normal'?

Finally, it would be good to know when one should reference these functions? Immediately after opening the file, or when required.

Thanks in advance.

V/


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Information on file:advise/4

Jesper Louis Andersen-2
On Sun, Aug 26, 2018 at 9:23 AM Valentin Micic <[hidden email]> wrote:
Hi,

Would it be possible for someone to shed a bit more light on various posix_file_advise() options in file:advise/4 function?


It binds to the posix_fadvise(2) call If I remember correctly. Thus, the actual behavior is dependent on your operating system. FreeBSD ignores SEQUENTIAL for instance, because its default semantics autodetects sequential access. FreeBSD can also handle a WILL_NEED by async loading that data. NetBSD ignores the offset and len in some cases and applies it to the whole file. OpenBSD doesn't support the call at all it seems. Illumos supports the call, but does so in libc, detects that the parameters are valid, and then does nothing with the information. This is compliant with the specification.

In most cases, you want to carefully measure before toying with these interfaces.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Information on file:advise/4

Valentin Micic-5

On 26 Aug 2018, at 2:44 PM, Jesper Louis Andersen wrote:

It binds to the posix_fadvise(2) call If I remember correctly

Thanks Jasper, much appreciated. 

V/


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Information on file:advise/4

Max Lapshin-2
I advice to find out how to very deeply and properly monitor usage of this feature.

We had once made a release that ruined performance of flussonic disk access when I just tried to play with this feature.

Just plain linux vfs cache happened to be more efficient than our efforts to help it.

So if you want to use it, it would be good to find out how to track its performance.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Information on file:advise/4

Valentin Micic-5

On 30 Aug 2018, at 5:43 AM, Max Lapshin wrote:

> I advice to find out how to very deeply and properly monitor usage of this feature.
>
> We had once made a release that ruined performance of flussonic disk access when I just tried to play with this feature.
>
> Just plain linux vfs cache happened to be more efficient than our efforts to help it.
>
> So if you want to use it, it would be good to find out how to track its performance.

Thanks for sharing, Max.

As a matter of fact, if memory serves me correctly, it may be someone from my company (amongst many others) that have asked for this feature a few years back.
At the time, we have been loading a number of reasonably big files into the RAM on Linux OS. We noticed that a considerable amount of RAM was utilized by OS for caching of files, which we no longer needed.
As there was no on-demand support for release of this cache from Erlang, we had to write a C code and call it against the given file once it has been loaded. Quite cumbersome!
So, I am very grateful that this interface has been made available in Erlang. I just wasn't clear about semantics behind various options.

But, I do understand caveats... for a long time, my (or, shall I say our clients') operating environment was a mixed bag of Solairs and Linux. The thing about these two, for example, is that they have a diametrically different views on OS caching.
Thus, over the time, I came to a realization that it is quite difficult to relay on good OS features and still stay out of trouble in a heterogeneous environment.
The remedy for us was to reinvent a hot water from time to time, by providing our own alternative for these features, which was always easy because Erlang seems to be generous that way.

Again thank you very much for sharing your experiences.

Kind regards

V/

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions