{packet, PacketType} for little-signed packets

[Pablo Platt-2]

Hi

Is there a way I can handle {packet, PacketType} for little-endian where
Packet = <<Length:32/little-signed,Message/binary>>
Length=4+byte_size(Message)

I currently handle it manually but it'll be nice if gen_tcp could handle it for me the way {packet, PacketType}
works for big-endian byte order.

Thanks



     

[Torben Hoffmann]

Hi Pablo,

I don't quite understand what your issue is, but it might be due to my lack
of understanding of the gen_tcp details. If not I think you should rephrase
your question.

Anyway, I have a piece of advice based on your code sniplet.

If you are defining the protocol yourself I would strongly suggest that you
make Length equal to the size of the Message since it makes decoding a lot
easier. I have dealt with Q.SIG and there the length in the PDU also depends
on the size of the header which forces one to do the decoding in two steps
instead of having the option to do it in one step in the function clause
pattern match.

Cheers,
Torben

On Sat, Jun 5, 2010 at 17:39, Pablo Platt <[hidden email]> wrote:

> Hi
>
> Is there a way I can handle {packet, PacketType} for little-endian where
> Packet = <<Length:32/little-signed,Message/binary>>
> Length=4+byte_size(Message)
>
> I currently handle it manually but it'll be nice if gen_tcp could handle it
> for me the way {packet, PacketType}
> works for big-endian byte order.
>
> Thanks
>
>
>
>

--
http://www.linkedin.com/in/torbenhoffmann

[Pablo Platt-2]

Hi Torben

I'm implementing mongodbwire protocol http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol
The first 4 bytes in each message are int32 of the "total message size, including the 4 bytes of length"
and all data is little-endian: "Note that like BSON documents, all data in the mongo wire protocol is  little-endian."

Is it possible to patch gen_tcp to support BSON?
Is there a simpler solution?




________________________________
From: Torben Hoffmann <[hidden email]>
To: Pablo Platt <[hidden email]>; erlang-questions <[hidden email]>
Sent: Sun, June 6, 2010 12:35:58 AM
Subject: Re: [erlang-questions] {packet, PacketType} for little-signed packets

Hi Pablo,

I don't quite understand what your issue is, but it might be due to my lack
of understanding of the gen_tcp details. If not I think you should rephrase
your question.

Anyway, I have a piece of advice based on your code sniplet.

If you are defining the protocol yourself I would strongly suggest that you
make Length equal to the size of the Message since it makes decoding a lot
easier. I have dealt with Q.SIG and there the length in the PDU also depends
on the size of the header which forces one to do the decoding in two steps
instead of having the option to do it in one step in the function clause
pattern match.

Cheers,
Torben

On Sat, Jun 5, 2010 at 17:39, Pablo Platt <[hidden email]> wrote:

> Hi
>
> Is there a way I can handle {packet, PacketType} for little-endian where
> Packet = <<Length:32/little-signed,Message/binary>>
> Length=4+byte_size(Message)
>
> I currently handle it manually but it'll be nice if gen_tcp could handle it
> for me the way {packet, PacketType}
> works for big-endian byte order.
>
> Thanks
>
>
>
>

--
http://www.linkedin.com/in/torbenhoffmann



     

[Torben Hoffmann]

Hi Pablo,

No clue on the patching of gen_tcp.

The mongodb wire protocol is not Erlang pattern matching friendly, since you
have to record the Length, then peel off the message inside your function
instead of doing the pattern matching directly in the function clause.
Not a major issue, just annoying.

Cheers,
Torben

On Sun, Jun 6, 2010 at 00:17, Pablo Platt <[hidden email]> wrote:

> Hi Torben
>
> I'm implementing mongodb wire protocol
> http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol
> The first 4 bytes in each message are int32 of the "total message size,
> including the 4 bytes of length"
> and all data is little-endian: "Note that like BSON documents, all data in
> the mongo wire protocol is little-endian."
>
> Is it possible to patch gen_tcp to support BSON?
> Is there a simpler solution?
>
> ------------------------------
> *From:* Torben Hoffmann <[hidden email]>
> *To:* Pablo Platt <[hidden email]>; erlang-questions <
> [hidden email]>
> *Sent:* Sun, June 6, 2010 12:35:58 AM
> *Subject:* Re: [erlang-questions] {packet, PacketType} for little-signed
> packets
>
> Hi Pablo,
>
> I don't quite understand what your issue is, but it might be due to my lack
> of understanding of the gen_tcp details. If not I think you should rephrase
> your question.
>
> Anyway, I have a piece of advice based on your code sniplet.
>
> If you are defining the protocol yourself I would strongly suggest that you
> make Length equal to the size of the Message since it makes decoding a lot
> easier. I have dealt with Q.SIG and there the length in the PDU also
> depends
> on the size of the header which forces one to do the decoding in two steps
> instead of having the option to do it in one step in the function clause
> pattern match.
>
> Cheers,
> Torben
>
> On Sat, Jun 5, 2010 at 17:39, Pablo Platt <[hidden email]> wrote:
>
> > Hi
> >
> > Is there a way I can handle {packet, PacketType} for little-endian where
> > Packet = <<Length:32/little-signed,Message/binary>>
> > Length=4+byte_size(Message)
> >
> > I currently handle it manually but it'll be nice if gen_tcp could handle
> it
> > for me the way {packet, PacketType}
> > works for big-endian byte order.
> >
> > Thanks
> >
> >
> >
> >
>
>
>
>
> --
> http://www.linkedin.com/in/torbenhoffmann
>
>

--
http://www.linkedin.com/in/torbenhoffmann

[Dmitry Belyaev]

There is implementation on BSON in emongo. You can look at
http://bitbucket.org/rumataestor/emongo
It is not necessary to use {packet, ...} at all.

Dmitry Belyaev


On 06/06/2010 02:17 AM, Pablo Platt wrote:

> Hi Torben
>
> I'm implementing mongodbwire protocol http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol
> The first 4 bytes in each message are int32 of the "total message size, including the 4 bytes of length"
> and all data is little-endian: "Note that like BSON documents, all data in the mongo wire protocol is  little-endian."
>
> Is it possible to patch gen_tcp to support BSON?
> Is there a simpler solution?
>
>
>
>
> ________________________________
> From: Torben Hoffmann<[hidden email]>
> To: Pablo Platt<[hidden email]>; erlang-questions<[hidden email]>
> Sent: Sun, June 6, 2010 12:35:58 AM
> Subject: Re: [erlang-questions] {packet, PacketType} for little-signed packets
>
> Hi Pablo,
>
> I don't quite understand what your issue is, but it might be due to my lack
> of understanding of the gen_tcp details. If not I think you should rephrase
> your question.
>
> Anyway, I have a piece of advice based on your code sniplet.
>
> If you are defining the protocol yourself I would strongly suggest that you
> make Length equal to the size of the Message since it makes decoding a lot
> easier. I have dealt with Q.SIG and there the length in the PDU also depends
> on the size of the header which forces one to do the decoding in two steps
> instead of having the option to do it in one step in the function clause
> pattern match.
>
> Cheers,
> Torben
>
> On Sat, Jun 5, 2010 at 17:39, Pablo Platt<[hidden email]>  wrote:
>
>    
>> Hi
>>
>> Is there a way I can handle {packet, PacketType} for little-endian where
>> Packet =<<Length:32/little-signed,Message/binary>>
>> Length=4+byte_size(Message)
>>
>> I currently handle it manually but it'll be nice if gen_tcp could handle it
>> for me the way {packet, PacketType}
>> works for big-endian byte order.
>>
>> Thanks
>>
>>
>>
>>
>>      
>
>
>
>    

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[Pablo Platt-2]

{packet, ...} will be much faster, reduce the code and possible errors.
If someone can point me to what file need to be patched I can try to do it myself.




________________________________
From: Dmitry Belyaev <[hidden email]>
To: [hidden email]
Sent: Mon, June 7, 2010 11:48:35 AM
Subject: Re: [erlang-questions] {packet, PacketType} for little-signed packets

There is implementation on BSON in emongo. You can look at
http://bitbucket.org/rumataestor/emongo
It is not necessary to use {packet, ...} at all.

Dmitry Belyaev


On 06/06/2010 02:17 AM, Pablo Platt wrote:

> Hi Torben
>
> I'm implementing mongodbwire protocol http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol
> The first 4 bytes in each message are int32 of the "total message size, including the 4 bytes of length"
> and all data is little-endian: "Note that like BSON documents, all data in the mongo wire protocol is  little-endian."
>
> Is it possible to patch gen_tcp to support BSON?
> Is there a simpler solution?
>
>
>
>
> ________________________________
> From: Torben Hoffmann<[hidden email]>
> To: Pablo Platt<[hidden email]>; erlang-questions<[hidden email]>
> Sent: Sun, June 6, 2010 12:35:58 AM
> Subject: Re: [erlang-questions] {packet, PacketType} for little-signed packets
>
> Hi Pablo,
>
> I don't quite understand what your issue is, but it might be due to my lack
> of understanding of the gen_tcp details. If not I think you should rephrase
> your question.
>
> Anyway, I have a piece of advice based on your code sniplet.
>
> If you are defining the protocol yourself I would strongly suggest that you
> make Length equal to the size of the Message since it makes decoding a lot
> easier. I have dealt with Q.SIG and there the length in the PDU also depends
> on the size of the header which forces one to do the decoding in two steps
> instead of having the option to do it in one step in the function clause
> pattern match.
>
> Cheers,
> Torben
>
> On Sat, Jun 5, 2010 at 17:39, Pablo Platt<[hidden email]>  wrote:
>
>    
>> Hi
>>
>> Is there a way I can handle {packet, PacketType} for little-endian where
>> Packet =<<Length:32/little-signed,Message/binary>>
>> Length=4+byte_size(Message)
>>
>> I currently handle it manually but it'll be nice if gen_tcp could handle it
>> for me the way {packet, PacketType}
>> works for big-endian byte order.
>>
>> Thanks
>>
>>
>>
>>
>>      
>
>
>
>    

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]


     

[Pablo Platt-2]

I think I somehow need to add BSON to the {packet, PacketType} options
and then add a case similar to asn1 http://github.com/erlang/otp/blob/dev/erts/emulator/beam/packet_parser.c#L326
Any ideas how to add BSON to the list of options and what code should be in the case TCP_PB_BSON ?

case TCP_PB_ASN1: {
    /* TCP_PB_ASN1: handles long (4 bytes) or short length format */
    const char* tptr = ptr;
    int length;
    int nn = n;
   
    if (n < 2) goto more;
    nn--;
    if ((*tptr++ & 0x1f) == 0x1f) { /* Long tag format */
        while (nn && ((*tptr & 0x80) == 0x80)) {
            tptr++;
            nn--;
        }
        if (nn < 2) goto more;
        tptr++;
        nn--;
    }
   
    /* tptr now point to length field and nn characters remain */
    length = *tptr & 0x7f;
    if ((*tptr & 0x80) == 0x80) { /* Long length format */
        tptr++;
        nn--;
        if (nn < length) goto more;
        switch (length) {
        case 0: plen = 0; break;
        case 1: plen = get_int8(tptr); tptr += 1; break;
        case 2: plen = get_int16(tptr); tptr += 2; break;
        case 3: plen = get_int24(tptr); tptr += 3; break;
        case 4: plen = get_int32(tptr); tptr += 4; break;
        default: goto error; /* error */
        }
    }
    else {
        tptr++;
        plen = length;
    }
    hlen = (tptr-ptr);
    goto remain;
}





________________________________
From: Jachym Holecek <[hidden email]>
To: Pablo Platt <[hidden email]>
Sent: Wed, June 9, 2010 12:48:12 AM
Subject: Re: [erlang-questions] {packet, PacketType} for little-signed packets

Hello,

# Pablo Platt 2010-06-08:
> {packet, ...} will be much faster, reduce the code and possible errors.
> If someone can point me to what file need to be patched I can try to do it myself.

These seem like good places to start:

  erts/emulator/beam/packet_parser.c
  erts/emulator/drivers/common/inet_drv.c

> Subject: Re: [erlang-questions] {packet, PacketType} for little-signed packets
                                                                  ^^^^^^
Huh? Are they seriously using signed integers for message length?

Have fun,
    -- Jachym