'recbuf' and the size of received data

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

'recbuf' and the size of received data

Willem Broekema
I have a problem receiving tcp packets with data larger than
1024 bytes.

The connection is declared as:

   {ok, Sock} =  gen_tcp:connect(Host, Port,
              [binary, {packet, 0}, {active, true}, {recbuf, 2048}]),

And data is received with:

   receive
     {tcp, Sock, Data} -> ...
   end

I wonder why the binary object Data is truncated to 1024 bytes,
as I declared 'recbuf' to be twice that size.  Is there another
option I should use?

Thanks for your help.


- Willem



Reply | Threaded
Open this post in threaded view
|

'recbuf' and the size of received data

Scott Lystig Fritchie-3
>>>>> "wb" == Willem Broekema <willem> writes:

wb> I have a problem receiving tcp packets with data larger than 1024
wb> bytes.

If you attempt to read N bytes from a TCP socket, you'll get at most N
bytes, but all other bets are off.  If the sender is transmitting in M
byte chunks, you may get M bytes when attempting to read N bytes
(assuming M < N), but you may not.  If you need message boundaries,
you need to use a different protocol (e.g. UDP) or you need to embed
those boundaries into the protocol you're using over TCP.

So, in your case, if recbuf is 2K, and you're gettings reads of 1K,
you'll just have to do another read and then concatenate the results.
There's no guarantee that your second read will be exactly 1K, either:
it might be 1.5K, which means you'll have to "carry over" 0.5K into
the next message you read.

-Scott
---
Scott Lystig Fritchie
Professional Governing: Is It Faked?


Reply | Threaded
Open this post in threaded view
|

'recbuf' and the size of received data

Håkan Mattsson-6
On Sun, 24 Jun 2001, Scott Lystig Fritchie wrote:

Scott> >>>>> "wb" == Willem Broekema <willem> writes:
Scott>
Scott> wb> I have a problem receiving tcp packets with data larger than 1024
Scott> wb> bytes.
Scott>
Scott> If you attempt to read N bytes from a TCP socket, you'll get at most N
Scott> bytes, but all other bets are off.  If the sender is transmitting in M
Scott> byte chunks, you may get M bytes when attempting to read N bytes
Scott> (assuming M < N), but you may not.  If you need message boundaries,
Scott> you need to use a different protocol (e.g. UDP) or you need to embed
Scott> those boundaries into the protocol you're using over TCP.

If you want to use TCP you may utilize the builtin Erlang emulator
support for TPKT (http://www.ietf.org/rfc/rfc1006.txt).

Open the socket with:

        gen_tcp:connect(Host, Port, [{packet, tpkt} | OtherOptions])

Add the TPKT header manually on the sending side:

        tpkt_send(Socket, MessageBody) when binary(MessageBody) ->
            Len = size(MessageBody) + 4,
            gen_tcp:send(Socket, [3, 0, ((Len) bsr 8) band 16#ff, (L) band 16#ff , MessageBody]).

On the receiving side, the Erlang emulator will automatically assemble
complete TPKT-packets before they are forwarded to the control
process, where you strip the header off with:

        handle_info({tcp, Socket, <<3:8, _:8, Len:16, MessageBody/binary>>}, State) ->
            decode_body(MessageBody, State)...

/H?kan

---
H?kan Mattsson
Ericsson
Computer Science Laboratory
http://www.ericsson.se/cslab/~hakan



Reply | Threaded
Open this post in threaded view
|

Seeking advice on port "pattern" (long)

Garry Hodgson-3

okay, i've been using erlang for a while now, and have learned
enough to be dangerous.  i've got my first app working nicely, but have
this nagging feeling that i'm not doing things in the most elegant
way.  so i'm seeking advice on good practice for using the port
mechanism
to attach to legacy applications.  in particular, if there are standard
patterns for how to do this cleanly, i'd appreciate some clue.
pointers to code examples would be especially helpful.

the application is a server that provides a single point of contact for
several other servers on different machines, each offering different
kinds
of network information queries.  i've got a registered allocator
process,
which maintains a list of idle and busy nodes, a list of active servers
running on those nodes, and maps a given request to a server who can
handle it, starting a fresh one if needed.  this is done via a
spawn_link
of a port server, which starts the appropriate legacy app (more on this
later).

requests come into the master, which asks the allocator for the
approriate
server, then passes the request on to that server.

the complication comes in the port process.  some of the legacy code is
multithreaded, and can potentially have a mix of long running database
queries and others that respond quickly.  each message and corresponding
response is tagged with a transaction id, so i can associate it with the
correct caller on return.  because i can't rewrite the legacy apps, i'm
communicating with simple ascii strings (no length prefix), and must
accumulate
data until i get a complete response, as some of the queries return
lengthy
replies, and don't accumulate the whole thing before replying.

so, my port module, absent some detail, is included below.  it seems to
me that
it's much more complicated than it ought to be.  spawning the separate
alarm
processes seems clumsy, vs. spawning a process per transaction with a
receive/after/end
statement.  but either way, i've got to keep a list of 'em, so i can
send the reply
to the right one once i get a complete message.

i'm (dimly) aware of the gen_server behavior, which i've avoided in
favor of rolling my own for learning purposes.  i'm open to going that
route now if that's the right thing to do.

so, that's more or less where i am.  i love this language, and am just
beginning
to feel fluent in it.  but i could use come pointers to gain
Enlightenment.  does
anyone have any sage words of advice?

thanks very much.

% ----------------- code follows ------------------

% alarm process to handle timeouts, spawned when call comes in

setalarm( Id ) ->
  spawn( port, doalarm, [ self(), 10000, Id ] ).

doalarm( Caller, Timeout, Id ) ->
  receive
    cancel ->  true
  after Timeout ->  Caller ! { timeout, Id }
  end.

% port:start()
start( Service, ExtPrg ) -> init( Service, ExtPrg ).

init( Service, ExtPrg ) ->
  Port = open_port( {spawn, ExtPrg}, [] ),
  loop( Service, Port, [], 1000, [] ).

loop( Service, Port, Pending, NextTrans, Buffered ) ->
  receive
    { call, Caller, Request, Args } ->
      Job = { NextTrans, Caller, setalarm( NextTrans ) },
      Port ! {self(), {command, encode( NextTrans, Request, Args ) } },
      loop( Service, Port, [ Job|Pending ], NextTrans + 1, Buffered );

    { Port, { data, Data } } ->
      AllData = Buffered++Data,
      case decode( AllData ) of
        { Id, Req, Result } ->
          case lists:keysearch( Id, 1, Pending ) of
            { value, { Id, Caller, Alarm } } ->
              Caller ! { ok, { Service, { Req, Result } } },
              Alarm ! cancel,
              loop( Service, Port, lists:keydelete( Id, 1, Pending ),
NextTrans, Buffered );
            false ->
              screamloudly( "Got data for nonexistent transaction ~w", [
Id ] ),
              loop( Service, Port, Pending, NextTrans, Buffered )
          end;
        { parseError, Details } ->
          io:format( "huh? got ~p ~n.  i'll wait for more...~n",
[Details] ),
          loop( Service, Port, Pending, NextTrans, AllData )
      end;

    { timeout, Id } ->
      io:format( "port: timeout on Id ~w", [ Id ] ),
      case lists:keysearch( Id, 1, Pending ) of
        { value, { Id, Caller, Alarm } } ->
          Caller ! { error, timeout },
        %  loop( Service, Port, lists:keydelete( Id, 1, Pending ),
NextTrans, [] );
          exit( { error, "Application timed out" } );
        false ->
          screamloudly( "Got timeout for nonexistent transaction ~w", [
Id ] ),
          loop( Service, Port, Pending, NextTrans, [] )
      end;

    stop ->
      Port ! { self(), close },
      receive
        {Port, closed} ->
          exit( normal )
      end
  end.




---
Garry Hodgson                   sometimes we ride on your horses
Senior Hacker                   sometimes we walk alone
Software Innovation Services    sometimes the songs that we hear
AT&T Labs                       are just songs of our own
garry


Reply | Threaded
Open this post in threaded view
|

Seeking advice on port "pattern" (long)

tobbe

Your program looks fine to me.
Two comments:

 1. You don't have to implement you own timer,
    look at the erlang man page for start_timer/3
    and cancel_timer/1.

 2. You could write your decoder in a continuation
    based style. So instead of having to do the
    append (AllData = Buffered++Data) your decoder
    returns either { Id, Req, Result } , { parseError, Details } ,
    or {needmore, Cont}. When new data arrives, you
    feed it into your continuation ( Cont(Data) ), which
    again will return any of the three possibilities above.

Cheers /Tobbe