Denial-of-service vulnerability in erlang's group.erl

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Stefan Zegenhagen
Dear all,

All input routines in lib/kernel/src/group.erl suffer from a
denial-of-service vulnerability that can easily be used by an attacker
to randomly crash erlang applications, the erlang VM or other system
processes on the same computer.

The group.erl I/O server offers line-editing functionality (using
edlin.erl) for all input. It therefore applies line-editing until a
newline character was received and the current input line is finished.
Only then the supplied end-of-input detection routines are called, that
differ for the get_chars, get_until, get_line, and get_password
commands.

Any attacker that has access to an input prompt serviced by group.erl
can send several megabytes of characters to the input prompt without a
newline in between. Thereby he can cause the Erlang VM to
     A.  consume all available memory and eventually have the Linux OOM
        (Out-Of-Memory) killer randomly kill system processes even other
        than the Erlang VM
     B. consume all available CPU power and slow the system down
     C. cause timeout errors in unrelated Erlang applications by the
        amount of garbage collection going on
     D. cause the Erlang VM to crash (was observed on a live system, but
        a crash dump could not be saved).

Currently, it is not possible to circumvent this behaviour (except by
re-implementing group.erl in any application using it). This is because
all of the end-of-input searching routines are only ever called *AFTER*
a complete, newline-terminated line has been read. Therefore, get_until
and get_chars won't help.

A meaningful solution is to allow users of group.erl to optionally limit
the maximum line length it will handle. If this maximum line length is
reached, group.erl should report an error response to the I/O request.
This would likely be implemented as a new set_opts option (although not
a global one affecting all group.erl process instances). This way, the
current behaviour would be retained and compatibility problems are not
to expect.

Please note that group.erl is *NOT EXCLUSIVELY* used by the erlang
shell. Erlang's SSH application uses group.erl as I/O server for its SSH
server implementation. Several devices may have a specific command line
interface (CLI) running on serial ports. Other uses of group.erl are
imaginable.


Kind regards,

--
Dr. Stefan Zegenhagen

arcutronix GmbH
Garbsener Landstr. 10
30419 Hannover
Germany

Tel:   +49 511 277-2734
Fax:   +49 511 277-2709
Email: stefan.zegenhagen
Web:   www.arcutronix.com

*Synchronize the Ethernet*

General Managers: Dipl. Ing. Juergen Schroeder, Dr. Josef Gfrerer -
Legal Form: GmbH, Registered office: Hannover, HRB 202442, Amtsgericht
Hannover; Ust-Id: DE257551767.

Please consider the environment before printing this message.


Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Fred Hebert
What are the risks of someone being able to gain access to group.erl,
where killing the node through OOM because of end of lines is the most
practical approach?

For the SSH module, the group.erl module is used with custom shells
implementations, and I've never seen Erlang code that allowed arbitrary
code execution without being ultimately very easy to crash in all kinds
of ways.

If you can run simple expressions such as sending messages, or can
construct binaries, chances are you can kill a node with far more ease
than you can by overflowing group.erl.

I'm asking this seriously, and not as some kind of snark. I can see the
OOM error being the result of honest mistakes, but I can't imagine
connecting to a node, seeing a shell, and saying "yes, I will overload
it by streaming data" instead of trying any of the following:

- halt(0)
- init:stop()
- c:q()
- init ! {stop,stop}
- apply(Mod, Fun, ...) from any of the previous options
- {init,[0]}:stop()
- lists:duplicate(99999999999999999999999999999, hello)
- <<0:100000000000000000/binary>>.
- << <<X,1>> || <<X:0, _/binary>> <= <<1,2,3>> >>
- os:cmd("..."),
- all sorts of binary_to_term tricks
- and so on

Which would take far less effort and far less bandwidth to accomplish.
You might also want to cripple systems by using sys:suspend/1-2 on all
the processes you can get your hand on, either through using
`processes()` or dynamically building pids from binaries.

So is there a case where input frequently makes it to group.erl
*without* the user having the ability to execute arbitrary code of some
sort if it ever succeeds? If not, then I don't think limiting the input
will accomplish much. I'm curious about the cases that would use
group.erl without wanting to evaluate input.

Regards,
Fred.

On 06/25, Stefan Zegenhagen wrote:

> Dear all,
>
> All input routines in lib/kernel/src/group.erl suffer from a
> denial-of-service vulnerability that can easily be used by an attacker
> to randomly crash erlang applications, the erlang VM or other system
> processes on the same computer.
>
> The group.erl I/O server offers line-editing functionality (using
> edlin.erl) for all input. It therefore applies line-editing until a
> newline character was received and the current input line is finished.
> Only then the supplied end-of-input detection routines are called, that
> differ for the get_chars, get_until, get_line, and get_password
> commands.
>
> Any attacker that has access to an input prompt serviced by group.erl
> can send several megabytes of characters to the input prompt without a
> newline in between. Thereby he can cause the Erlang VM to
>      A.  consume all available memory and eventually have the Linux OOM
>         (Out-Of-Memory) killer randomly kill system processes even other
>         than the Erlang VM
>      B. consume all available CPU power and slow the system down
>      C. cause timeout errors in unrelated Erlang applications by the
>         amount of garbage collection going on
>      D. cause the Erlang VM to crash (was observed on a live system, but
>         a crash dump could not be saved).
>
> Currently, it is not possible to circumvent this behaviour (except by
> re-implementing group.erl in any application using it). This is because
> all of the end-of-input searching routines are only ever called *AFTER*
> a complete, newline-terminated line has been read. Therefore, get_until
> and get_chars won't help.
>
> A meaningful solution is to allow users of group.erl to optionally limit
> the maximum line length it will handle. If this maximum line length is
> reached, group.erl should report an error response to the I/O request.
> This would likely be implemented as a new set_opts option (although not
> a global one affecting all group.erl process instances). This way, the
> current behaviour would be retained and compatibility problems are not
> to expect.
>
> Please note that group.erl is *NOT EXCLUSIVELY* used by the erlang
> shell. Erlang's SSH application uses group.erl as I/O server for its SSH
> server implementation. Several devices may have a specific command line
> interface (CLI) running on serial ports. Other uses of group.erl are
> imaginable.
>
>
> Kind regards,
>
> --
> Dr. Stefan Zegenhagen
>
> arcutronix GmbH
> Garbsener Landstr. 10
> 30419 Hannover
> Germany
>
> Tel:   +49 511 277-2734
> Fax:   +49 511 277-2709
> Email: stefan.zegenhagen
> Web:   www.arcutronix.com
>
> *Synchronize the Ethernet*
>
> General Managers: Dipl. Ing. Juergen Schroeder, Dr. Josef Gfrerer -
> Legal Form: GmbH, Registered office: Hannover, HRB 202442, Amtsgericht
> Hannover; Ust-Id: DE257551767.
>
> Please consider the environment before printing this message.
>
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs
> http://erlang.org/mailman/listinfo/erlang-bugs

Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Matthias Lang-2
In reply to this post by Stefan Zegenhagen
Hi,

Stefan Zegenhagen wrote:

> All input routines in lib/kernel/src/group.erl suffer from a
> denial-of-service vulnerability that can easily be used by an attacker

Is this a new problem or are you reporting

  http://erlang.org/pipermail/erlang-patches/2013-June/004138.html

one more time?

Matt

Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Stefan Zegenhagen
In reply to this post by Fred Hebert
Hi Fred,

> I'm asking this seriously, and not as some kind of snark. I can see the
> OOM error being the result of honest mistakes, but I can't imagine
> connecting to a node, seeing a shell, and saying "yes, I will overload
> it by streaming data" instead of trying any of the following:


I'm very happy that the discussion has started again and hope for open
ears this time.

I'm talking about the case where custom shells come into play. We have
devices where a custom command line interface (CLI) was implemented to
allow authorized customers to configure/control the device. The CLI is
available via both, SSH and a serial port.

Such a custom CLI is not uncommon in the computer world. Many databases
offer a CLI to run SQL queries, Routers offer a CLI to configure
networking, ... They all have in common that the user does not get
system privileges or a system shell prompt and is restricted to a
certain set of self-defined commands.

The user that is using our CLI has no access whatever to an erlang shell
and is restricted to executing our custom commands allowed by the CLI.
He cannot simply execute arbitrary code, he cannot simply type arbitrary
commands, he cannot arbitrarily interfere with the system.

The user has to authenticate to the device by means of username/password
(over the serial port) or whatever method SSH allows.

What sounds like a bug to me is that the user is able to crash our
system, using our CLI system and *NOT THE ERLANG SHELL*, just by sending
megabyte-long lines to the CLI (where a prompt is waiting for input),
and we can do nothing against it except reimplementing group.erl. While
we could do that for our shell implementation over the serial port
easily (we start group.erl in our code), the situation is a little more
complicated for SSH. There is no chance to replace group.erl without
additionally patching the Erlang SSH application.

Two examples:
1.) The user connects a cable to the serial port and is asked to login.
He sends megabytes of data, the device crashes reliably.
2.) A user with read-only permissions connects to our device via SSH. He
has no permission to reboot the system. By sending tons of characters
over SSH he reboots the device reliably.

To summarize:
      * I'm not talking of arbitrary code execution possiblities.
      * I'm talking of a "hardended", custom CLI implementation.
      * No regular access to the erlang shell.
      * Instead of logging on to our CLI, the user just sends
        megabyte-long lines to the device and it crashes as reliably as
        the sun rises in the morning.
      * Various misbehaviours of the software act as precursor to the
        final crash/reboot, including the ones I already mentioned:
        system slowdown, OOM killer killing non-erlang services, erlang
        crash, misbehaving erlang applications unrelated to the CLI.
      * We cannot simply amend the situation.
      * The device is unable to fulfil its designated task, the attacker
        is able to make the device "deny the service".

I would call this a bug in group.erl, and the only thing I want to
request is an optional method to circumvent the problem (that is not
esotheric or academic for us, but a real, practical problem) for the
services where this is needed (our CLI). This would most easily be
achieved using a new io:setopts() option.

How overlong lines are reported back to the application utilizing
group.erl is a matter of taste, but it is likely that some kind of
error-abort of the io_request should be done to stop further processing
of that line.


Kind regards,

--
Dr. Stefan Zegenhagen

arcutronix GmbH
Garbsener Landstr. 10
30419 Hannover
Germany

Tel:   +49 511 277-2734
Fax:   +49 511 277-2709
Email: stefan.zegenhagen
Web:   www.arcutronix.com

*Synchronize the Ethernet*

General Managers: Dipl. Ing. Juergen Schroeder, Dr. Josef Gfrerer -
Legal Form: GmbH, Registered office: Hannover, HRB 202442, Amtsgericht
Hannover; Ust-Id: DE257551767.

Please consider the environment before printing this message.


Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Stefan Zegenhagen
In reply to this post by Matthias Lang-2
Dear Matthias,

> > All input routines in lib/kernel/src/group.erl suffer from a
> > denial-of-service vulnerability that can easily be used by an attacker
>
> Is this a new problem or are you reporting
>
>   http://erlang.org/pipermail/erlang-patches/2013-June/004138.html
>
> one more time?

Yes, it is. I reposted it because the patch was rejected, but the issue
is still unsolved. It is, however, imported for us to get solved.


Kind regards,

--
Dr. Stefan Zegenhagen

arcutronix GmbH
Garbsener Landstr. 10
30419 Hannover
Germany

Tel:   +49 511 277-2734
Fax:   +49 511 277-2709
Email: stefan.zegenhagen
Web:   www.arcutronix.com

*Synchronize the Ethernet*

General Managers: Dipl. Ing. Juergen Schroeder, Dr. Josef Gfrerer -
Legal Form: GmbH, Registered office: Hannover, HRB 202442, Amtsgericht
Hannover; Ust-Id: DE257551767.

Please consider the environment before printing this message.


Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Erik Søe Sørensen-4
In reply to this post by Stefan Zegenhagen
+1 for meaningful application of the phrase "crashes reliably" :-)


Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Matthias Lang-2
In reply to this post by Stefan Zegenhagen
Hi,

While it sounds like there's a problem in group.erl which should be
fixed, my experience is that things aren't quite as bad as:

>       * We cannot simply amend the situation.

On our embedded device, we completely avoid the situation by
authenticating users before they get to the CLI implemented in Erlang.

We use 'dropbear' as the SSH server and have /etc/passwd use 'to_erl'
as the user's shell. The approach for serial ports is similar.

Once authenticated, you can reboot our system using the 'reset' command.

---

You were also concerned about linux's OOM killer.

Finding and stopping all possible ways Erlang can grab unexpected
amounts of RAM is difficult, and not just because they're no longer
unexpected once you find them. Telling linux to limit the amount
Erlang _can_ grab is much easier. You can do that with 'ulimit' or, if
starting from something like 'heart' (which you probably should be),
setrlimit().

---

Finally, if I start a server like this:

    # cat ss.erl
    -module(ss).
    -export([go/0]).

    go() ->
       crypto:start(),
       ssh:start(),
       ssh:daemon(9988, [{password, "bla"}, {ip_v6_disabled, true}]).

    # ulimit -m 12000
    # /usr/local/src/otp_src_R15B03/bin/erl
    Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:2:2] ...
    1> ss:go().
    {ok,<0.46.0>}

can you give me a program which crashes that through overflow, automatically?

(You've given a description. I'd like a program so that I can be sure
I'm doing exactly what you're doing.)

Matt

Reply | Threaded
Open this post in threaded view
|

Denial-of-service vulnerability in erlang's group.erl

Stefan Zegenhagen
Dear Matthias,

sorry for the delayed answer, I was on vacation for two weeks and read
your mail just today.


> While it sounds like there's a problem in group.erl which should be
> fixed, my experience is that things aren't quite as bad as:
>
> >       * We cannot simply amend the situation.
>
> On our embedded device, we completely avoid the situation by
> authenticating users before they get to the CLI implemented in Erlang.
>
> We use 'dropbear' as the SSH server and have /etc/passwd use 'to_erl'
> as the user's shell. The approach for serial ports is similar.
>
> Once authenticated, you can reboot our system using the 'reset' command.

Our system probably differs slightly. We keep the "to_erl" option for
the developer's shell (yes, we have a hidden serial port on the board
that's not accessible to customers). And we need much more control over
configurability and features than getty/dropbear give us. Therefore we
would like to stick to the option of implementing our application logic
in erlang with the software stacks that are already available.


> You were also concerned about linux's OOM killer.
>
> Finding and stopping all possible ways Erlang can grab unexpected
> amounts of RAM is difficult, and not just because they're no longer
> unexpected once you find them. Telling linux to limit the amount
> Erlang _can_ grab is much easier. You can do that with 'ulimit' or, if
> starting from something like 'heart' (which you probably should be),
> setrlimit().

That might be an option to stop linux from crashing other processes, but
still it's bad if the erlang VM itself crashes because that means that
the service is out of operation until everything is fully
re-initialized.


I want to underline that I do not want to start a flame war or be overly
offensive, but I'm astonished about the reluctance to fix this
particular bug which we consider a security bug due to the way it
affects our devices.


Kind regards,

--
Dr. Stefan Zegenhagen

arcutronix GmbH
Garbsener Landstr. 10
30419 Hannover
Germany

Tel:   +49 511 277-2734
Fax:   +49 511 277-2709
Email: stefan.zegenhagen
Web:   www.arcutronix.com

*Synchronize the Ethernet*

General Managers: Dipl. Ing. Juergen Schroeder, Dr. Josef Gfrerer -
Legal Form: GmbH, Registered office: Hannover, HRB 202442, Amtsgericht
Hannover; Ust-Id: DE257551767.

Please consider the environment before printing this message.