funs versus code loading

[Sam Bobroff-3]

Hi all,

I've encountered a situation that makes my brain hurt and I was wondering if anyone could help me understand it. It's so strange that it may be a bug, if so please let me know and I'll do whatever groundwork is necessary to get it tracked down. Anyway on to the situation:

It appears that when:

* Executing a Fun,

* that contains a local function call,

* in a process running in the old code of a module that has both current and old versions loaded,

* the local function call will sometimes go to the current code and sometimes go to the old code.

It appears that the decision is based somehow on the content of the code in the current version.

Obviously an example is in order:

--- begin foo.erl ---

-module(foo).

-ifdef(BODY).

-export([start/0, loop/0]).

start() ->

        register(?MODULE, spawn(?MODULE, loop, [])).

loop() ->

        io:fwrite("loop version ~p\n", [?V]),

        receive

                _ ->

                        F = fun () -> loop() end,

                        F()

        end.

-endif.

--- end foo.erl ---

Armed with this file, we can do this:

$ erl

Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)

1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).

{ok,foo,[]}

2> foo:start().

loop version 1

true

3> compile:file("foo", [{d,'BODY'}, {d,'V',2}]).

{ok,foo,[]}

4> code:load_file(foo).

{module,foo}

5> foo ! x.

loop version 2

x

The local function call in the Fun has called from the old version into the current version of the module (not what I expected).

Run the same sequence again but this time load a module that is empty:

$ erl

Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)

1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).

{ok,foo,[]}

2> foo:start().

loop version 1

true

3> compile:file("foo", [{d,'V',2}]).

{ok,foo,[]}

4> code:load_file(foo).

{module,foo}

5> foo ! x.

loop version 1

x

The local function call in the Fun has stayed in the old version of the module (actually what I expected).

Some changes other than emptying the body of the module seem to have a similar effect but I haven't tracked down exactly what the trigger is.

My system is Ubuntu 11.04 running Erlang R14B02 and I haven't yet checked older versions of Erlang for the same behaviour.

Thanks,

Sam.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Daniel Dormont]

I'm far from an expert, but I repeated your test just for fun on R12B and got the same result. I did a few other tests as well.

Perhaps unsurprisingly, changing "loop()" to "foo:loop()" produced the same result in test 1 and an error {undef,[{foo,loop,[]}]} in test 2. That is, the old code seems to be fully discarded in that case.

Maybe more surprisingly, loading the code for a third time in test 1 results in an error "=ERROR REPORT==== 13-May-2011::00:13:18 ===
Loading of /Users/dan/foo.beam failed: not_purged
{error,not_purged}"

However, using the shell command l/1 instead of code:load_file/1 avoids this error. I'll leave it to the experts to tell us why :)

dan

On Thu, May 12, 2011 at 10:16 PM, Sam Bobroff <[hidden email]> wrote:

Hi all,

I've encountered a situation that makes my brain hurt and I was wondering if anyone could help me understand it. It's so strange that it may be a bug, if so please let me know and I'll do whatever groundwork is necessary to get it tracked down. Anyway on to the situation:

It appears that when:

* Executing a Fun,

* that contains a local function call,

* in a process running in the old code of a module that has both current and old versions loaded,

* the local function call will sometimes go to the current code and sometimes go to the old code.

It appears that the decision is based somehow on the content of the code in the current version.

Obviously an example is in order:

--- begin foo.erl ---

-module(foo).

-ifdef(BODY).

-export([start/0, loop/0]).

start() ->

        register(?MODULE, spawn(?MODULE, loop, [])).

loop() ->

        io:fwrite("loop version ~p\n", [?V]),

        receive

                _ ->

                        F = fun () -> loop() end,

                        F()

        end.

-endif.

--- end foo.erl ---

Armed with this file, we can do this:

$ erl

Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)

1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).

{ok,foo,[]}

2> foo:start().

loop version 1

true

3> compile:file("foo", [{d,'BODY'}, {d,'V',2}]).

{ok,foo,[]}

4> code:load_file(foo).

{module,foo}

5> foo ! x.

loop version 2

x

The local function call in the Fun has called from the old version into the current version of the module (not what I expected).

Run the same sequence again but this time load a module that is empty:

$ erl

Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)

1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).

{ok,foo,[]}

2> foo:start().

loop version 1

true

3> compile:file("foo", [{d,'V',2}]).

{ok,foo,[]}

4> code:load_file(foo).

{module,foo}

5> foo ! x.

loop version 1

x

The local function call in the Fun has stayed in the old version of the module (actually what I expected).

Some changes other than emptying the body of the module seem to have a similar effect but I haven't tracked down exactly what the trigger is.

My system is Ubuntu 11.04 running Erlang R14B02 and I haven't yet checked older versions of Erlang for the same behaviour.

Thanks,

Sam.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Sam Bobroff-3]

Thanks for taking a look at this :-)

On 13 May 2011 14:44, Daniel Dormont <[hidden email]> wrote:

I'm far from an expert, but I repeated your test just for fun on R12B and got the same result. I did a few other tests as well.

Perhaps unsurprisingly, changing "loop()" to "foo:loop()" produced the same result in test 1 and an error {undef,[{foo,loop,[]}]} in test 2. That is, the old code seems to be fully discarded in that case.

 

Yes, unsurprising because doing an explicit "external" call is supposed to transition into the newest code version.

Maybe more surprisingly, loading the code for a third time in test 1 results in an error "=ERROR REPORT==== 13-May-2011::00:13:18 ===
Loading of /Users/dan/foo.beam failed: not_purged
{error,not_purged}"

This isn't too surprising to me: code:load_file() does not do an implicit code:purge() and that error is normal in that case. Because there is an old version of the module loaded, you need to do code:purge(foo) before you can issue another code:load_file().

However, using the shell command l/1 instead of code:load_file/1 avoids this error. I'll leave it to the experts to tell us why :)

I can answer that one: l() does an implicit code:purge() (although it doesn't say so in it's little help description).

 

dan

On Thu, May 12, 2011 at 10:16 PM, Sam Bobroff <[hidden email]> wrote:

Hi all,

I've encountered a situation that makes my brain hurt and I was wondering if anyone could help me understand it. It's so strange that it may be a bug, if so please let me know and I'll do whatever groundwork is necessary to get it tracked down. Anyway on to the situation:

It appears that when:

* Executing a Fun,

* that contains a local function call,

* in a process running in the old code of a module that has both current and old versions loaded,

* the local function call will sometimes go to the current code and sometimes go to the old code.

It appears that the decision is based somehow on the content of the code in the current version.

Obviously an example is in order:

--- begin foo.erl ---

-module(foo).

-ifdef(BODY).

-export([start/0, loop/0]).

start() ->

        register(?MODULE, spawn(?MODULE, loop, [])).

loop() ->

        io:fwrite("loop version ~p\n", [?V]),

        receive

                _ ->

                        F = fun () -> loop() end,

                        F()

        end.

-endif.

--- end foo.erl ---

Armed with this file, we can do this:

$ erl

Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)

1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).

{ok,foo,[]}

2> foo:start().

loop version 1

true

3> compile:file("foo", [{d,'BODY'}, {d,'V',2}]).

{ok,foo,[]}

4> code:load_file(foo).

{module,foo}

5> foo ! x.

loop version 2

x

The local function call in the Fun has called from the old version into the current version of the module (not what I expected).

Run the same sequence again but this time load a module that is empty:

$ erl

Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)

1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).

{ok,foo,[]}

2> foo:start().

loop version 1

true

3> compile:file("foo", [{d,'V',2}]).

{ok,foo,[]}

4> code:load_file(foo).

{module,foo}

5> foo ! x.

loop version 1

x

The local function call in the Fun has stayed in the old version of the module (actually what I expected).

Some changes other than emptying the body of the module seem to have a similar effect but I haven't tracked down exactly what the trigger is.

My system is Ubuntu 11.04 running Erlang R14B02 and I haven't yet checked older versions of Erlang for the same behaviour.

Thanks,

Sam.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Nahuel Greco-2]

Sam, check this thread:

http://thread.gmane.org/gmane.comp.lang.erlang.general/47310

Saludos,
Nahuel Greco.



On Thu, May 12, 2011 at 11:16 PM, Sam Bobroff <[hidden email]> wrote:

> Hi all,
> I've encountered a situation that makes my brain hurt and I was wondering if
> anyone could help me understand it. It's so strange that it may be a bug, if
> so please let me know and I'll do whatever groundwork is necessary to get it
> tracked down. Anyway on to the situation:
> It appears that when:
> * Executing a Fun,
> * that contains a local function call,
> * in a process running in the old code of a module that has both current and
> old versions loaded,
> * the local function call will sometimes go to the current code and
> sometimes go to the old code.
> It appears that the decision is based somehow on the content of the code in
> the current version.
> Obviously an example is in order:
> --- begin foo.erl ---
> -module(foo).
> -ifdef(BODY).
> -export([start/0, loop/0]).
> start() ->
>         register(?MODULE, spawn(?MODULE, loop, [])).
> loop() ->
>         io:fwrite("loop version ~p\n", [?V]),
>         receive
>                 _ ->
>                         F = fun () -> loop() end,
>                         F()
>         end.
> -endif.
> --- end foo.erl ---
> Armed with this file, we can do this:
> $ erl
> Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0]
> [hipe] [kernel-poll:false]
> Eshell V5.8.3  (abort with ^G)
> 1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).
> {ok,foo,[]}
> 2> foo:start().
> loop version 1
> true
> 3> compile:file("foo", [{d,'BODY'}, {d,'V',2}]).
> {ok,foo,[]}
> 4> code:load_file(foo).
> {module,foo}
> 5> foo ! x.
> loop version 2
> x
> The local function call in the Fun has called from the old version into the
> current version of the module (not what I expected).
> Run the same sequence again but this time load a module that is empty:
> $ erl
> Erlang R14B02 (erts-5.8.3) [source] [smp:4:4] [rq:4] [async-threads:0]
> [hipe] [kernel-poll:false]
> Eshell V5.8.3  (abort with ^G)
> 1> compile:file("foo", [{d,'BODY'}, {d,'V',1}]).
> {ok,foo,[]}
> 2> foo:start().
> loop version 1
> true
> 3> compile:file("foo", [{d,'V',2}]).
> {ok,foo,[]}
> 4> code:load_file(foo).
> {module,foo}
> 5> foo ! x.
> loop version 1
> x
> The local function call in the Fun has stayed in the old version of the
> module (actually what I expected).
> Some changes other than emptying the body of the module seem to have a
> similar effect but I haven't tracked down exactly what the trigger is.
> My system is Ubuntu 11.04 running Erlang R14B02 and I haven't yet checked
> older versions of Erlang for the same behaviour.
> Thanks,
> Sam.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Sam Bobroff-3]

On 13 May 2011 16:08, Nahuel Greco <[hidden email]> wrote:

Sam, check this thread:

http://thread.gmane.org/gmane.comp.lang.erlang.general/47310

Saludos,
Nahuel Greco.

Ah yes that seems to match exactly what I've observed!

Thanks for that :-)

I agree with your (and Matthias') comments in that it should be changed in accordance with the patch!

In my opinion the current system is confusing and therefore dangerous.

Hmm... from poking around in the emulator code it looks like the "hash" is made up of three components XOR'ed together:

* the index

* the module's name

* a "uniq" value

Since during code reloading, the module name and index could easily be the same... if the uniq were the same then the emulator would be fooled into thinking it was OK to use the new version of the code... when it was not OK.

I couldn't see what system was used to set the uniq value ("old_uniq" in most of the code I saw)... does anyone know what it is?

Even without knowing the algorithm, if I can control the Fun's index, and I can see it's uniq, I can probably cause a hash collision by setting the index to a specific value.

The reason I ask is that I'm wondering if a hash collision is possible and if that has the potential to crash the emulator.

Sam.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Tony Rogvall-2]

Hi!

On 13 maj 2011, at 09.05, Sam Bobroff wrote:

On 13 May 2011 16:08, Nahuel Greco <[hidden email]> wrote:

Sam, check this thread:

http://thread.gmane.org/gmane.comp.lang.erlang.general/47310

Saludos,
Nahuel Greco.

Ah yes that seems to match exactly what I've observed!

Thanks for that :-)

I agree with your (and Matthias') comments in that it should be changed in accordance with the patch!

In my opinion the current system is confusing and therefore dangerous.

If you change your example into this:

-------------

-module(foo).

-ifdef(BODY).

-export([start/0, loop/0]).

start() ->

        register(?MODULE, spawn(?MODULE, loop, [])).

loop() ->

        io:fwrite("loop version ~p\n", [?V]),

        receive

                _ ->

                        F = fun () -> ?V,loop() end,

                        F()

        end.

-endif.

-------------

The only difference is that I put a dummy version value inside the fun. This is enough to make the

fun handle unique (in this case). In one project I used a erlang:now value to (mostly) generate a new function reference.

Hmm... from poking around in the emulator code it looks like the "hash" is made up of three components XOR'ed together:

* the index

* the module's name

* a "uniq" value

Since during code reloading, the module name and index could easily be the same... if the uniq were the same then the emulator would be fooled into thinking it was OK to use the new version of the code... when it was not OK.

I guess the new_unique (same as module:info) should/could be used some how.

I couldn't see what system was used to set the uniq value ("old_uniq" in most of the code I saw)... does anyone know what it is?

Even without knowing the algorithm, if I can control the Fun's index, and I can see it's uniq, I can probably cause a hash collision by setting the index to a specific value.

The reason I ask is that I'm wondering if a hash collision is possible and if that has the potential to crash the emulator.

Sam.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

"Have run Make so many times I dunno what's installed anymore"

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Nahuel Greco-2]

On Fri, May 13, 2011 at 4:05 AM, Sam Bobroff <[hidden email]> wrote:
> I agree with your (and Matthias') comments in that it should be changed in
> accordance with the patch!

Erlang really needs a bug tracker... this problem is very old.

Saludos,
Nahuel Greco.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions