Sending file descriptors and sockets to another beam

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Sending file descriptors and sockets to another beam

Danil Zagoskin-2
Hi!

As you may know, OTP 22 has a new low-level socket API: http://erlang.org/doc/man/socket.html
It allows to communicate over UNIX sockets,
and UNIX sockets have sendmsg feature, which has an option to pass a file descriptor.

I tried to pass the FD with a fresh OTP, and this didn't just work.
After a bit of hacking I've made a patch for socket NIF and a demo.

Example: in four different shells execute:

  • ./sendsock.escript proxy
  • ./sendsock.escript worker
  • telnet localhost 13456
  • telnet localhost 13456

See the listen and established sockets are owned by different beam.smp processes:

$ lsof -i:13456
COMMAND    PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
beam.smp 82931 stolen   32u  IPv4 0x3c2e6eba43bdf26f      0t0  TCP *:13456 (LISTEN)
beam.smp 82952 stolen   32u  IPv4 0x3c2e6eba47fa726f      0t0  TCP localhost:13456->localhost:62163 (ESTABLISHED)
beam.smp 82952 stolen   33u  IPv4 0x3c2e6eba41348e8f      0t0  TCP localhost:13456->localhost:62185 (ESTABLISHED)
telnet   82976 stolen    5u  IPv4 0x3c2e6eba4f19452f      0t0  TCP localhost:62163->localhost:13456 (ESTABLISHED)
telnet   82993 stolen    5u  IPv4 0x3c2e6eba4786052f      0t0  TCP localhost:62185->localhost:13456 (ESTABLISHED)

What this may be useful for?
  * Hot upgrades with BEAM upgrade/restart
  * Load-balancing TCP load between unreliable workers without proxying
  * Single-connection distribution (see https://github.com/stolen/webdist)
  * Maybe anything else

Have fun!

-- 
Danil Zagoskin | [hidden email]

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Sending file descriptors and sockets to another beam

Loïc Hoguin-3
That's pretty cool.

Here's one very real use case using this. Evidently the same could be
done with Erlang with this patch:

   https://blog.cloudflare.com/know-your-scm_rights/

On 24/09/2019 17:33, Danil Zagoskin wrote:

> Hi!
>
> As you may know, OTP 22 has a new low-level socket API:
> http://erlang.org/doc/man/socket.html
> It allows to communicate over UNIX sockets,
> and UNIX sockets have sendmsg feature, which has an option to pass a
> file descriptor.
>
> I tried to pass the FD with a fresh OTP, and this didn't just work.
> After a bit of hacking I've made a patch for socket NIF and a demo.
>    Patch: https://github.com/erlang/otp/pull/2400
>    Demo: https://gist.github.com/stolen/303c30d4edbb8835f9bec3fad0d75ede
>
> Example: in four different shells execute:
>
>   * |./sendsock.escript proxy|
>   * |./sendsock.escript worker|
>   * |telnet localhost 13456|
>   * |telnet localhost 13456|
>
> See the listen and established sockets are owned by different
> |beam.smp| processes:
>
> |$ lsof -i:13456 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> beam.smp 82931 stolen 32u IPv4 0x3c2e6eba43bdf26f 0t0 TCP *:13456
> (LISTEN) beam.smp 82952 stolen 32u IPv4 0x3c2e6eba47fa726f 0t0 TCP
> localhost:13456->localhost:62163 (ESTABLISHED) beam.smp 82952 stolen 33u
> IPv4 0x3c2e6eba41348e8f 0t0 TCP localhost:13456->localhost:62185
> (ESTABLISHED) telnet 82976 stolen 5u IPv4 0x3c2e6eba4f19452f 0t0 TCP
> localhost:62163->localhost:13456 (ESTABLISHED) telnet 82993 stolen 5u
> IPv4 0x3c2e6eba4786052f 0t0 TCP localhost:62185->localhost:13456
> (ESTABLISHED)|
>
>
> What this may be useful for?
>    * Hot upgrades with BEAM upgrade/restart
>    * Load-balancing TCP load between unreliable workers without proxying
>    * Single-connection distribution (see https://github.com/stolen/webdist)
>    * Maybe anything else
>
> Have fun!
>
> --
> Danil Zagoskin | [hidden email] <mailto:[hidden email]>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Sending file descriptors and sockets to another beam

Tristan Sloughter-4
In reply to this post by Danil Zagoskin-2
Oh nice, maybe this is why I could never get the systemd socket activation to work! It relies on using the file descriptor 3 for the listen socket.

On Tue, Sep 24, 2019, at 09:33, Danil Zagoskin wrote:
Hi!

As you may know, OTP 22 has a new low-level socket API: http://erlang.org/doc/man/socket.html
It allows to communicate over UNIX sockets,
and UNIX sockets have sendmsg feature, which has an option to pass a file descriptor.

I tried to pass the FD with a fresh OTP, and this didn't just work.
After a bit of hacking I've made a patch for socket NIF and a demo.

Example: in four different shells execute:

  • ./sendsock.escript proxy
  • ./sendsock.escript worker
  • telnet localhost 13456
  • telnet localhost 13456

See the listen and established sockets are owned by different beam.smp processes:

$ lsof -i:13456
COMMAND    PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
beam.smp 82931 stolen   32u  IPv4 0x3c2e6eba43bdf26f      0t0  TCP *:13456 (LISTEN)
beam.smp 82952 stolen   32u  IPv4 0x3c2e6eba47fa726f      0t0  TCP localhost:13456->localhost:62163 (ESTABLISHED)
beam.smp 82952 stolen   33u  IPv4 0x3c2e6eba41348e8f      0t0  TCP localhost:13456->localhost:62185 (ESTABLISHED)
telnet   82976 stolen    5u  IPv4 0x3c2e6eba4f19452f      0t0  TCP localhost:62163->localhost:13456 (ESTABLISHED)
telnet   82993 stolen    5u  IPv4 0x3c2e6eba4786052f      0t0  TCP localhost:62185->localhost:13456 (ESTABLISHED)

What this may be useful for?
  * Hot upgrades with BEAM upgrade/restart
  * Load-balancing TCP load between unreliable workers without proxying
  * Single-connection distribution (see https://github.com/stolen/webdist)
  * Maybe anything else

Have fun!

-- 
Danil Zagoskin | [hidden email]
_______________________________________________
erlang-questions mailing list
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions