linked-in drivers once again

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Karel Van Oudheusden
Hello once again,

I have a few specific questions and a few general questions regarding
linked-in drivers. Since I am new to Erlang and C programming, I am
having the time of my life.

I have written an erlang file 'erlToeasiest.erl' that communicates with
the C file 'easiest_drv.so'.  (see attachment for the details).  The C
file 'easiest_drv.c' correctly compiles (and has kindly been checked by
Vance Shiley). I therefore mainly have questions about the
'erlToeasiest.erl' file.

Regarding the 'erlToeasiest.erl' file:

- I use the following statement:
Port = open_port({spawn, "./easiest_drv.so"}, [])
instead of using the statement:
Port = open_port({spawn, "easiest_drv"},[])
The first statement works, the latter does not. Yet I am confused
because all the examples I have seen follow the second convention.

- Then there's the issue of the empty list in the above statements. I
assume I can for instance use [{packet, 2}] for communication between my
Erlang process and the driver.  What does the empty list actually imply?

- How does the communication between the Erlang process and the driver
work precisely?  If I were to use a binary for communication, I
understand that a reference to the binary is passed. So this is in a
sense shared memory communication?
And what exactly is a binary? Do they use this principle in for instance
Java-C communication or is this unique for Erlang and if so, what is the
drawback?

- According to the "byteorder" driver code the following works (but I
don't know why):
Port ! {self(), {command, "test"}},
receive
    {Port, {data, [$l|_]}}  -> ...
    {Port, {data, [$m|_]}}-> ...
    {Port, closed}  -> ...
...
Where do the $l and $m come from?
Besides, my code contains the above but does not work. I have debugged
it and it appears that the driver is closed. ...
-----------------------------------------
And now more general questions:

- I am building a distributed (multi-tasking) application written mainly
in Erlang. However, the application contains a lot of large
lookup-tables that are consulted very often. These lookup-tables can
efficiently be implemented in different ways in an imperative (e.g. C
language) approach. Therefore, I want to use Erlang for the
multi-tasking, fault-tolerance, and control intelligence of the
language. On the other hand, I want to use C or C++ for the total memory
management of the lookup tables.

The intention is to use a linked-in driver for the C part. Is this
approach reasonable? The C code will be passive code but maybe (probably
not) in the future I will want to make it multi-tasking (posix threads
...) as well.

I know there are possibilities with the Mnesia data base but I have not
looked at that yet and for the moment I am not planning to either. I
really want to use these two different approaches (functional and
imperative) together in one application. And if I am not mistaken, this
was/is one of the main advantages of the Erlang philosophy (yet I am
having some trouble finding the documentation for this).

The Erlang process will consult the C driver often. Yet I am not
planning to use a binary for this since only a small message is sent
between the two: the Erlang process sends a key to the C data structure.
The C driver sends a value (corresponding to the key) back to the Erlang
process.

I am also assuming that in the long run I might even be able to specify
which data structures are placed where exactly in physical memory.
However, this is a C compiler problem so this shouldn't be a problem
regarding Erlang or is it?

Note that I am also assuming that the application I am talking about is
mainly a data-dominated problem. Therefore I consider Erlang
inappropriate for the data management (of the lookup tables). I am
however wandering whether at the end of the day the concurrency in
Erlang will be the most critical factor (in terms of performance) or
will it (still) be the data management (e.g. retrieving different memory
often) assuming that I am using the above stated approach?

And last but not least, if anybody has an application like the one I am
talking about in which a lot of C code interacts with Erlang code I
would be happy to check it out.


thanks a lot for your comments,
KVO.

-------------- next part --------------
An embedded message was scrubbed...
From: Van Oudheusden <voudheus>
Subject: (no subject)
Date: Wed, 27 Jun 2001 22:13:50 +0200
Size: 9946
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20010627/e0ee966e/attachment.mht>

Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Vance Shipley-2
> I have written an erlang file 'erlToeasiest.erl' that communicates with
> the C file 'easiest_drv.so'.  (see attachment for the details).  The C
> file 'easiest_drv.c' correctly compiles (and has kindly been checked by
> Vance Shiley). I therefore mainly have questions about the
> 'erlToeasiest.erl' file.

I only checked it as far as getting your init function working so that
the driver would load with erl_ddll.

> - I use the following statement:
> Port = open_port({spawn, "./easiest_drv.so"}, [])
> instead of using the statement:
> Port = open_port({spawn, "easiest_drv"},[])
> The first statement works, the latter does not. Yet I am confused
> because all the examples I have seen follow the second convention.

Why the former works is a fluke.  The latter doesn't work because you
have not initialized erlang_port to -1.  You are checking to see if
it is -1 and if not returning -1 to abort:

        if (erlang_port != (ErlDrvPort) -1)
                return((ErlDrvData) -1);

> Since I am new to Erlang and C programming ...

The reason that the fisrt example works is one of the tricky things
about debugging C programs.  Sometimes things that shouldn't work do.
When you ran this program with that calling syntax erlang_port just
happened to end up as -1.  Since you hadn't initia;ized it it could be
anything, no guarantees.  Often with uninitialized variables/pointers
things will work and then later when you make a perfectly valid code
change they stop working.  You rack your brain staring at the new code
thinking it must be bad but it isn't.  You just moved things around and
now your variable/pointer has a value that doesn't work for you.

This is one of the reasons we code in Erlang!  :)

        -Vance

Vance Shipley
Motivity Telecom Inc., Telephony Signaling Software
+1 519 579 5816
vances

> - Then there's the issue of the empty list in the above statements. I
> assume I can for instance use [{packet, 2}] for communication between my
> Erlang process and the driver.  What does the empty list actually imply?
>
> - How does the communication between the Erlang process and the driver
> work precisely?  If I were to use a binary for communication, I
> understand that a reference to the binary is passed. So this is in a
> sense shared memory communication?
> And what exactly is a binary? Do they use this principle in for instance
> Java-C communication or is this unique for Erlang and if so, what is the
> drawback?
>
> - According to the "byteorder" driver code the following works (but I
> don't know why):
> Port ! {self(), {command, "test"}},
> receive
>     {Port, {data, [$l|_]}}  -> ...
>     {Port, {data, [$m|_]}}-> ...
>     {Port, closed}  -> ...
> ...
> Where do the $l and $m come from?
> Besides, my code contains the above but does not work. I have debugged
> it and it appears that the driver is closed. ...
> -----------------------------------------
> And now more general questions:
>
> - I am building a distributed (multi-tasking) application written mainly
> in Erlang. However, the application contains a lot of large
> lookup-tables that are consulted very often. These lookup-tables can
> efficiently be implemented in different ways in an imperative (e.g. C
> language) approach. Therefore, I want to use Erlang for the
> multi-tasking, fault-tolerance, and control intelligence of the
> language. On the other hand, I want to use C or C++ for the total memory
> management of the lookup tables.
>
> The intention is to use a linked-in driver for the C part. Is this
> approach reasonable? The C code will be passive code but maybe (probably
> not) in the future I will want to make it multi-tasking (posix threads
> ...) as well.
>
> I know there are possibilities with the Mnesia data base but I have not
> looked at that yet and for the moment I am not planning to either. I
> really want to use these two different approaches (functional and
> imperative) together in one application. And if I am not mistaken, this
> was/is one of the main advantages of the Erlang philosophy (yet I am
> having some trouble finding the documentation for this).
>
> The Erlang process will consult the C driver often. Yet I am not
> planning to use a binary for this since only a small message is sent
> between the two: the Erlang process sends a key to the C data structure.
> The C driver sends a value (corresponding to the key) back to the Erlang
> process.
>
> I am also assuming that in the long run I might even be able to specify
> which data structures are placed where exactly in physical memory.
> However, this is a C compiler problem so this shouldn't be a problem
> regarding Erlang or is it?
>
> Note that I am also assuming that the application I am talking about is
> mainly a data-dominated problem. Therefore I consider Erlang
> inappropriate for the data management (of the lookup tables). I am
> however wandering whether at the end of the day the concurrency in
> Erlang will be the most critical factor (in terms of performance) or
> will it (still) be the data management (e.g. retrieving different memory
> often) assuming that I am using the above stated approach?
>
> And last but not least, if anybody has an application like the one I am
> talking about in which a lot of C code interacts with Erlang code I
> would be happy to check it out.
>
>
> thanks a lot for your comments,
> KVO.
>
>


Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Raimo Niskanen-3
Vance Shipley wrote:
>
.
.
>
> > - I use the following statement:
> > Port = open_port({spawn, "./easiest_drv.so"}, [])
> > instead of using the statement:
> > Port = open_port({spawn, "easiest_drv"},[])
> > The first statement works, the latter does not. Yet I am confused
> > because all the examples I have seen follow the second convention.
>
> Why the former works is a fluke.  

A fluke and an ugly behaviour with open_port/2. The function
open_port({spawn, Command}, []) was originally used for opening a port
that communicates with an external program - Command, and then reused
for opening a port that is an instance of a port driver - Command. So we
have a namespace clash here. If there exists a port driver named 'foo'
you cannot call an external program (port program) named 'foo' - it is
shadowed by the driver (or is it the other way around?)

What probably happens in "the former case" is that a port is opened (an
instance of the built in 'spawn' driver) that tries to execute
"./easiest_drv.so" in another unix process, which fails and the port
dies. Therefore you get a port, but it is closed when you try to use it.

/ Raimo Niskanen, Ericsson UAB, Erlang/OTP


Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Ulf Wiger-4
In reply to this post by Karel Van Oudheusden
On Wed, 27 Jun 2001, Karel Van Oudheusden wrote:

>And now more general questions:
>
>- I am building a distributed (multi-tasking) application
>written mainly in Erlang. However, the application contains a
>lot of large lookup-tables that are consulted very often. These
>lookup-tables can efficiently be implemented in different ways
>in an imperative (e.g. C language) approach. Therefore, I want
>to use Erlang for the multi-tasking, fault-tolerance, and
>control intelligence of the language. On the other hand, I want
>to use C or C++ for the total memory management of the lookup
>tables.

You should take a look at ets (Erlang Term Storage). Ets
implements efficient storage structures through built-in
functions. Currently supported access structures are:

- set (linear hash table, unique keys)
- bag (linear hash table, multiple objects per key)
- duplicate_bag (like bag, but multiple instances of each object)
- ordered_set (B+ tree)

Access times are usually in the order of 10-100 microseconds,
depending on the size of the object being read/written (data is
copied from/to the process heap). The ets tables are stored
outside the process heaps, and are not garbage collected.

Read more about it using "erl -man ets", or at
http://www.erlang.org/doc/r7b/lib/stdlib-1.9.1/doc/html/ets.html

If these access types suit your needs, then you will not gain any
performance from writing a linked in driver.

The mnesia database uses these built in access structures, and
adds transaction semantics, replication, and lots of other stuff.
It is quite easy to start with ets tables and then transition to
mnesia, if your requirements change.


>Note that I am also assuming that the application I am talking
>about is mainly a data-dominated problem. Therefore I consider
>Erlang inappropriate for the data management (of the lookup
>tables). I am however wandering whether at the end of the day
>the concurrency in Erlang will be the most critical factor (in
>terms of performance) or will it (still) be the data management
>(e.g. retrieving different memory often) assuming that I am
>using the above stated approach?

I don't know if this can be answered without looking very closely
at your particular application. Using ets, data access is really
quite fast. Concurrency is also very efficiently implemented in
Erlang. Since your application will also be distributed, I'd
expect that the strategy for task distribution, the capacity of
the communication channels, and the general nature of the tasks
being distributed could become at least as important for overall
performance.

You should plan to do some prototyping, and develop your
application incrementally. You will find out soon enough where
your real bottlenecks are. Even if you then decide that you will
have to leave Erlang for really good performance (normally, this
is not the case), your work in Erlang will still have been
extremely valuable for your understanding of the application.

/Uffe
--
Ulf Wiger                                    tfn: +46  8 719 81 95
Senior System Architect                      mob: +46 70 519 81 95
Strategic Product & System Management    ATM Multiservice Networks
Data Backbone & Optical Services Division      Ericsson Telecom AB



Reply | Threaded
Open this post in threaded view
|

ODBC issues

Martin Carlson-2
I am currently writing a proxy that uses ODBC modules. I have all of the
necessary components for ODBC installed on a slackware linux machine.  I
am however getting some strange behavior from when I try to start the
ODBC processes from within the shell. Below is the output from the shell
when I try to start the server.


(martin)5> {ok, _Pid} = odbc:start_link({local, odbc1}, [], []).
** exited: {noproc,{gen_server,call,
                               [odbc_sup,
                                {start_child,

[{local,odbc1},[{client,<0.46.0>}],[]]},
                                infinity]}} **

After executing the above command the ODBC module apears to be in the
system as I can type "o" and tab and I see that odbc is one  of the
options that is available. If anyone knows of an explanation for this
please let me know.

                    Thanks,
                    Martin Logan
                    Software Engineer, Vail Systems.



Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Per Hedeland-4
In reply to this post by Raimo Niskanen-3
Raimo Niskanen <raimo> wrote:
>
>Vance Shipley wrote:
>>
>> Why the former works is a fluke.  
>
>A fluke and an ugly behaviour with open_port/2. The function
>open_port({spawn, Command}, []) was originally used for opening a port
>that communicates with an external program - Command, and then reused
>for opening a port that is an instance of a port driver - Command.

Actually I think it's a feature (I may even be to blame for it:-) - it
allows you to change from external program to driver (which is really
just an "implementation detail", right:-) or vice versa without changing
the Erlang code - or have the same Erlang code on different OSes even
though one uses an external program and another a driver (for whatever
OS-specific reason).

> So we
>have a namespace clash here. If there exists a port driver named 'foo'
>you cannot call an external program (port program) named 'foo' - it is
>shadowed by the driver (or is it the other way around?)

Well, not quite - you can always call the external program with the full
pathname, which is often the best thing to do if it is specific to your
application - and if you don't, you have plenty of opportunity for
shadowing in your $PATH anyway...

>What probably happens in "the former case" is that a port is opened (an
>instance of the built in 'spawn' driver) that tries to execute
>"./easiest_drv.so" in another unix process, which fails and the port
>dies. Therefore you get a port, but it is closed when you try to use it.

I like that explanation better than Vance's:-) - the C semantics specify
that static variables are initialized to zero... (unless an initial
value is given, of course).

--Per Hedeland
per


Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Vance Shipley-2
> I like that explanation better than Vance's:-) - the C semantics specify
> that static variables are initialized to zero... (unless an initial
> value is given, of course).
>
> --Per Hedeland

I disagree:

$ cat > t.c
main()
{
        int var;

        printf("var=%d\n", var);
}
$ gcc -o t t.c
$ ./t
var=134484863


        -Vance



Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Vance Shipley-2
In reply to this post by Per Hedeland-4
> Raimo Niskanen <raimo> wrote:
> >
> >Vance Shipley wrote:
> >>
> >> Why the former works is a fluke.  
> >
> >A fluke and an ugly behaviour with open_port/2. The function
> >open_port({spawn, Command}, []) was originally used for opening a port
> >that communicates with an external program - Command, and then reused
> >for opening a port that is an instance of a port driver - Command.

To be clear I agree with Raimo's version of what is happening here.
While I contend that my explanation could well have been what was
happening it is obvious to me now that it was not what was happening.
In any event the error I pointed out was the real problem with the code.

        -Vance


Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Sebastian Strollo-3
In reply to this post by Vance Shipley-2
"Vance Shipley" <vances> writes:

> I disagree:
>
> $ cat > t.c
> main()
> {
>         int var;

That is an *automatic* variable, not a static one. Automatic variables
can have (as your example shows) any value, static and external are
initialized to 0.

  % cat > t.c
  static int var;
  main()
  {
      printf("var=%d\n", var);
  }
  % gcc t.c
  % ./a.out
  var=0

Although I agree with your statement that uninitialized variables is a
common cause of very confusing errors when you program in C.

/Sebastian


Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Vance Shipley-2

I stand corrected. :)

In any event he was expecting it to be -1 so he still needed to
initialize it that way to get his code working.

        -Vance

> That is an *automatic* variable, not a static one. Automatic variables
> can have (as your example shows) any value, static and external are
> initialized to 0.
>
>   % cat > t.c
>   static int var;
>   main()
>   {
>       printf("var=%d\n", var);
>   }
>   % gcc t.c
>   % ./a.out
>   var=0
>
> Although I agree with your statement that uninitialized variables is a
> common cause of very confusing errors when you program in C.
>
> /Sebastian



Reply | Threaded
Open this post in threaded view
|

linked-in drivers once again

Daniel Neri
In reply to this post by Vance Shipley-2
"Vance Shipley" <vances> writes:

> I disagree:
>
> $ cat > t.c
> main()
> {
>         int var;
>
>         printf("var=%d\n", var);
> }

Yes, but that is *not* a static variable (you'd have prefix the
declaration with the keyword "static", or move it outside main).

Uninitialised static variables are traditionally allocated in a
section of memory called BSS (i.e. not on the stack or in a register
like "var" in the code above). This section is zeroed by the OS when
the executable is loaded into memory.


Regards,
   --Daniel

--
Daniel Neri
dne