Installing a module from code?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Installing a module from code?

John Haugeland
I've got a problem I don't fully understand.  I keep wanting to call it a bug, but it may be that I'm handling this wrongly.

During my attempt to compile and install a module from code, it seems I've managed to break the Erlang VMs I'm installing into.  My gut instinct is that my namespaced module sc.file is somehow conflicting with the real module file.  This is frustrating, because after my compile process it appears to work as expected; sc.file is treated differently than file, all package calls are resolved appropriately, etc.  But, on next start, one gets the following error:

Erlang (BEAM) emulator version 5.6.4 [smp:2] [async-threads:0]

{"init terminating in do_boot",{undef,
  [{file,path_eval,[[".","C:\Users\John Haugeland"],".erlang"]},
  {c,f_p_e,2},{init,eval_script,7},{init,do_boot,3}]}}


Crash dump was written to: erl_crash.dump
init terminating in do_boot ()

Abnormal termination

(Formatted that a little to make it fit email)

And I mean, that kind of looks like that's the Erlang VM saying "I have no function path_eval in module file", and as stupid as it sounds, the back of my mind is screaming "it's because sc.file overrides real .file".

I have replicated this on Windows Vista HP-32 5.6.4, Windows Vista HP-32 5.5.5 and Centos4-32 5.6.4.  All I have to do is run my install process, where everything appears to go kosher, then restart the Erlang VM.

So, my method of "installing" is to take the source root directory and call .compile:file() progressively on each source file.  Then I use .code:get_object_code() to get the binary and the filename, which I pass with the module name as an atom to .code:load_binary() to load the actual code (since :load.module() doesn't seem to understand package names).  I'm actually going to show my install process twice, because there's a much shorter version and a much longer version.  The shorter version is used to compile my string module, which contains utilities that the installer uses, so that it can turn around and install everything else.  As such, it's a single special case and shows my expectation of The Right Way to do this in a very short space; that way if I'm just taking a numbskulled approach someone can point out my fool fault with little effort.  Then I'm posting the long version, in case it's somehow importantly different than the string special case.

But frankly, here's how this seems to me: I think the packaged modules are conflicting at boot time with the real thing.  I can't imagine any other reason for do_boot to complain that a function that's part of the standard library is unavailable.

If for whatever reason this code is unreadable, it can be gotten at http://scutil.com/ from /src/sc/installer.erl .  I would recommend installing it on an isolated VM; calling the installer will prevent that VM from booting again.

The short one (note that get_object_code seems only to work with the "un-packaged" name, which I worry is part of the problem):

.compile:file(From++"/src/sc/string.erl"),
{_, Bin, FileName} = .code:get_object_code(string),
.code:load_binary('sc.string', FileName, Bin),

The long one:

compile_all(From, Options) ->

    ReportOnCompile = fun
        ( Module, error)                                 -> { error, Module };
        ( Module, {error, ErrorList, WarningList})       -> { error, Module, ErrorList, WarningList };
        (_Module, {ok,AtomName})                         -> AtomName;
        (_Module, {ok,AtomName, Warnings})               -> { AtomName, Warnings }
    end,

    IsFailureCase = fun
        ({ error,_Module })                         -> true;
        ({ error,_Module,_ErrorList,_WarningList }) -> true;
        (_)                                         -> false
    end,

    IsWarningCase = fun
        ({_ModuleAtom, [] })       -> false;
        ({_ModuleAtom,_Warnings }) -> true
    end,

    ToFilename = fun
        (List) when is_list(List) -> .sc.string:implode("/", [ atom_to_list(Item) || Item <- List ]) ++ ".erl";
        (Atom) when is_atom(Atom) -> atom_to_list(Atom) ++ ".erl"
    end,

    Report = [ ReportOnCompile(From ++ ToFilename(File), .compile:file(From ++ ToFilename(File), Options)) ||
        File <- contained_modules()
    ],

    { Fail, PassWarn } = .lists:partition(IsFailureCase, Report),
    { Warn, Pass     } = .lists:partition(IsWarningCase, PassWarn),

    LoadFile = fun(Passed) ->
        [LastPiece|_]      = .lists:reverse(.sc.string:explode(".", atom_to_list(Passed))),
        {_, Bin, FileName} = .code:get_object_code(list_to_atom(LastPiece)),
        .code:load_binary(Passed, FileName, Bin)
    end,

    [ LoadFile(Passed) || {Passed,[]} <- Pass ],

    { { pass, Pass }, { warn, Warn }, { fail, Fail } }.

I know it's a lot to ask, but I could use some help sorting this out; these parts of the standard library aren't well documented, and I wouldn't want to think one could break an erlang install by compiling and loading modules containing correct code under a name which should not but does conflict.  I really want there to be a stupid mistake on my part here. 

Any help would be greatly appreciated.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://www.erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Installing a module from code?

Ulf Wiger-3
John Haugeland wrote:
> My gut instinct is that my namespaced module sc.file is somehow
 > conflicting with the real module file.

A loong time ago, when I played around with packages, I noticed
that dotted file names didn't go well with embedded boot code
loading. It was the erl_prim_loader module, I think.

Even if you're not using embedded code loading, the kernel
and stdlib modules are always pre-loaded. I have no idea
if this issue was ever addressed.

http://erlang.org/pipermail/erlang-questions/2003-November/010741.html

BR,
Ulf W

--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com
_______________________________________________
erlang-questions mailing list
[hidden email]
http://www.erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Installing a module from code?

John Haugeland
Well it's good to know it isn't my imagination, at least, though it's rather concerning that this has been known since 2003.

There's actually another looming problem.  Package names aren't resolved correctly during conflict after a boot.  My current assumption is that the only reason they work correctly before a boot is because they're loaded after compiling into the runtime.

-----------------------

-module(a.foo).
-export([shared/0, unique_a/0]).

shared()   -> "I am a.foo".
unique_a() -> "I am a.foo's unique.".

-----------------------

-module(b.foo).
-export([shared/0, unique_b/0]).

shared()   -> "I am b.foo".
unique_b() -> "I am b.foo's unique.".

-----------------------

-module(c.foo).
-export([shared/0, unique_c/0]).

shared()   -> "I am c.foo".
unique_c() -> "I am c.foo's unique.".

-----------------------



If you compile each of those, they will all work as expected until you reboot the VM. 

    7> c("/scratch/a/foo.erl").
    {ok,'a.foo'}
    8> c("/scratch/b/foo.erl").
    {ok,'b.foo'}
    9> c("/scratch/c/foo.erl").
    {ok,'c.foo'}
    10> a.foo:shared().       
    "I am a.foo"
    11> b.foo:shared().
    "I am b.foo"
    12> c.foo:shared().
    "I am c.foo"



After you reboot the VM, you will no longer be able to call any of them directly.

    2> a.foo:shared().
    ** exception error: undefined function 'a.foo':shared/0
    3> b.foo:shared().
    ** exception error: undefined function 'b.foo':shared/0
    4> c.foo:shared().
    ** exception error: undefined function 'c.foo':shared/0



If you pretend there's no namespace, Erlang will clearly expose that the binary is the last one compiled.

    5> foo:shared().
    ** exception error: undefined function foo:shared/0
   
    =ERROR REPORT==== 24-Mar-2009::10:42:03 ===
    beam/beam_load.c(1035): Error loading module foo:
      module name in object code is c.foo



I suspect that the problem is that the .beam naming structure doesn't indicate package structure; x.y.z.quux is stored on disk as quux.beam, which is ambiguous.  The two natural solutions seem to me to be either to make the /lib directory structure reflect the package structure (which is sensible given that source is required to do the same) or to make the filenames something like x.y.z.quux.beam.  In both cases, backwards compatability is slightly broken, in that someone will have to recompile their stuff or suffer lots of missing errors, but if they're in the middle of upgrading the VM they probably have to do that anyway (I don't know the toolchain very well, so for all I know .beams are migrated or something.)

That packages/namespaces can collide in naming kind of defeats the purpose of packages/namespaces, and makes them relatively dangerous in that they can apparently take out the standard library.

This really should, IMO, be fixed before r13 stable.





On Tue, Mar 24, 2009 at 1:43 AM, Ulf Wiger <[hidden email]> wrote:
John Haugeland wrote:
My gut instinct is that my namespaced module sc.file is somehow
> conflicting with the real module file.

A loong time ago, when I played around with packages, I noticed
that dotted file names didn't go well with embedded boot code
loading. It was the erl_prim_loader module, I think.

Even if you're not using embedded code loading, the kernel
and stdlib modules are always pre-loaded. I have no idea
if this issue was ever addressed.

http://erlang.org/pipermail/erlang-questions/2003-November/010741.html

BR,
Ulf W

--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com



--
---
GuaranteedVPS.com - bandwidth commitments and root starting from $12.98/mo

_______________________________________________
erlang-questions mailing list
[hidden email]
http://www.erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Installing a module from code?

Kenneth Lundin
Packages was introduced many years ago as an experimental feature.
Already from the beginning we where aware of that some
functions/features  would not work
together with packages.
We have not decided if and when packages will be more than experimental.
 If we decide to support packages some day it will most probably not
work exactly as today's solution.
another alternative could be to remove the half -hearted implementation
from the distribution in order to avoid confusion.

We strongly discourage the use of packages in professional projects.
We have no plans to do anything regarding packages until the R13B release.

/Kenneth Erlang/OTP Ericsson

But packages don't work as bad as you think and the problems described
below are actually not
caused by a bug, it works just as it is intended to do. Embedded commnets below.



2009/3/24 John Haugeland <[hidden email]>:

> Well it's good to know it isn't my imagination, at least, though it's rather
> concerning that this has been known since 2003.
>
> There's actually another looming problem.  Package names aren't resolved
> correctly during conflict after a boot.  My current assumption is that the
> only reason they work correctly before a boot is because they're loaded
> after compiling into the runtime.
>
> -----------------------
>
> -module(a.foo).
> -export([shared/0, unique_a/0]).
>
> shared()   -> "I am a.foo".
> unique_a() -> "I am a.foo's unique.".
>
> -----------------------
>
> -module(b.foo).
> -export([shared/0, unique_b/0]).
>
> shared()   -> "I am b.foo".
> unique_b() -> "I am b.foo's unique.".
>
> -----------------------
>
> -module(c.foo).
> -export([shared/0, unique_c/0]).
>
> shared()   -> "I am c.foo".
> unique_c() -> "I am c.foo's unique.".
>
> -----------------------
>
>
>
> If you compile each of those, they will all work as expected until you
> reboot the VM.
>
>     7> c("/scratch/a/foo.erl").
>     {ok,'a.foo'}
The c/1 function compiles and loads the module. The module is given
with full path.
>     8> c("/scratch/b/foo.erl").
>     {ok,'b.foo'}
>     9> c("/scratch/c/foo.erl").
>     {ok,'c.foo'}
>     10> a.foo:shared().
a.foo is already loaded and the call succeeds

>     "I am a.foo"
>     11> b.foo:shared().
>     "I am b.foo"
>     12> c.foo:shared().
>     "I am c.foo"
>
>
>
> After you reboot the VM, you will no longer be able to call any of them
> directly.

I assume that reboot means starting a new instance of the VM and this time
the modules a.foo, b.foo and c.foo are not loaded.
>
>     2> a.foo:shared().
The module a.foo is not loaded and the code path is searched for a
matching module
this means that for each element E in the code path "E/a/foo.beam" is tried
If you started the new VM with the same working directory as the first
one you will have
only one file foo.beam in the working directory and that will not
match, so the module a.foo will not be loaded and thus
can not be called.


>     ** exception error: undefined function 'a.foo':shared/0
>     3> b.foo:shared().
>     ** exception error: undefined function 'b.foo':shared/0
>     4> c.foo:shared().
>     ** exception error: undefined function 'c.foo':shared/0
>
>
>
> If you pretend there's no namespace, Erlang will clearly expose that the
> binary is the last one compiled.
>
>     5> foo:shared().
>     ** exception error: undefined function foo:shared/0
>
>     =ERROR REPORT==== 24-Mar-2009::10:42:03 ===
>     beam/beam_load.c(1035): Error loading module foo:
>       module name in object code is c.foo
>
>
>
> I suspect that the problem is that the .beam naming structure doesn't
> indicate package structure; x.y.z.quux is stored on disk as quux.beam, which
> is ambiguous.  The two natural solutions seem to me to be either to make the
> /lib directory structure reflect the package structure (which is sensible
> given that source is required to do the same) or to make the filenames
> something like x.y.z.quux.beam.  In both cases, backwards compatability is
> slightly broken, in that someone will have to recompile their stuff or
> suffer lots of missing errors, but if they're in the middle of upgrading the
> VM they probably have to do that anyway (I don't know the toolchain very
> well, so for all I know .beams are migrated or something.)
>
> That packages/namespaces can collide in naming kind of defeats the purpose
> of packages/namespaces, and makes them relatively dangerous in that they can
> apparently take out the standard library.
>
> This really should, IMO, be fixed before r13 stable.
>
>
>
>
>
> On Tue, Mar 24, 2009 at 1:43 AM, Ulf Wiger <[hidden email]>
> wrote:
>>
>> John Haugeland wrote:
>>>
>>> My gut instinct is that my namespaced module sc.file is somehow
>>
>> > conflicting with the real module file.
>>
>> A loong time ago, when I played around with packages, I noticed
>> that dotted file names didn't go well with embedded boot code
>> loading. It was the erl_prim_loader module, I think.
>>
>> Even if you're not using embedded code loading, the kernel
>> and stdlib modules are always pre-loaded. I have no idea
>> if this issue was ever addressed.
>>
>> http://erlang.org/pipermail/erlang-questions/2003-November/010741.html
>>
>> BR,
>> Ulf W
>>
>> --
>> Ulf Wiger
>> CTO, Erlang Training & Consulting Ltd
>> http://www.erlang-consulting.com
>
>
>
> --
> ---
> GuaranteedVPS.com - bandwidth commitments and root starting from $12.98/mo
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://www.erlang.org/mailman/listinfo/erlang-questions