Dynamic Configuration Database in Erlang

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Dynamic Configuration Database in Erlang

Stefano Bertuola
Hi all.

I am looking for understanding how to implement a configuration database in Erlang (which allows dynamic configuration).

For example, in a Fred's blog (https://ferd.ca/erlang-otp-21-s-new-logger.html) related to Logger, he mentions about: "[a] configuration database is an opaque set of OTP processes that hold all the data in an ETS table so it's fast of access (something lager did as well), but that could very well be written using the OTP Persistent Term Storage [...]".

Does anyone have more details about this or any other implementation?

Br. Stefano
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic Configuration Database in Erlang

zxq9-2
On 2019/12/04 22:03, Stefano Bertuola wrote:
> I am looking for understanding how to implement a configuration database
> in Erlang (which allows dynamic configuration).
>
> Does anyone have more details about this or any other implementation?

Hi, Stefano.

I make a configuration manager/DB in just about any program I write that
needs configuration. Usually this takes the form of a single gen_server
that is started first at the beginning of execution (you can have it
start later, but usually your system configuration is pretty close to
the very top of your supervision tree).

You can implement a relatively naked "global dictionary" if you haven't
figured out what sort of configuration data your program needs. This
would be a simple process that exposes two functions:

-spec config(Key :: atom()) -> Value :: term().
-spec config(Key :: atom(), Value :: term()) -> ok.

This is as naive as things can possibly get and is really just a wrapper
over either a front-end for an ETS table or a gen_server that simply
keeps a config map of #{Key => Value} in its state.

(I often avoid ETS unless it becomes an actual *need* in a program,
which is somewhat rare. As long as you hide the act of accessing the
config data behind functions it doesn't matter which approach you take
to start out with because you can change things later -- just don't get
into the habit of scattering naked reads from ETS tables throughout your
code!)

While this is naive it is not a *bad* way to start playing with the idea
of a config manager within your programs (and even this naive approach
already gives you the magic ability to alter global settings on the
fly), but a config manager can be much more useful.

In client-side programs you often have a large amount of evolving state
that the user expects to persist between executions, or settings they
can change globally across the program (for example, changing a l10n
option or adding/removing a plugin in naive programs very often requires
restarting the program -- that's unsatisfactory in many cases). In
complex server-side programs there may be a huge amount of configuration
data about who to contact, what connections must be established and so on.

In those cases "config" doesn't just mean "read the config file" (though
it usually involves that), but also often means "try to acquire external
resources that may require discovery first". In this case the
configuration manager process is where you can codify the discovery
logic, default settings, config failure criteria (to cause a clean exit
with appropriate logs/warnings/alerts sent), and other config
initialization or safe/unsafe update checks. It can also be used as a
place to register processes that need to be notified when config update
events occur (imagine having a function where a process can register for
config updates "send me a notification if Key/Value is changed", for
example).

The more complex a program's configuration state becomes the more often
you'll find yourself writing specific functions to handle specific
config keys and states (or specific clauses of an exposed config/2
function that match on specific keys) because config changes have a way
over time of interacting with one another and requiring sanity checks
that are better to formalize within a config manager module than scatter
around in ad-hoc code.

I'm explaining this in prose instead of providing an example, but
hopefully this gives you some ideas. Write a config manager and play
around with the idea. Having a config manager process at the top of your
supervision tree (and offloading it from ad-hoc config management
throughout the rest of the code) can sometimes make a lot of other code
simpler to test, debug and comprehend.

-Craig
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic Configuration Database in Erlang

zxq9-2
On 2019/12/05 0:08, zxq9 wrote:
> On 2019/12/04 22:03, Stefano Bertuola wrote:
>> I am looking for understanding how to implement a
>> configuration database in Erlang (which allows dynamic configuration).
>>
>> Does anyone have more details about this or any other implementation?

A quick note on this bit here:

> You can implement a relatively naked "global dictionary" if you haven't
> figured out what sort of configuration data your program needs. This
> would be a simple process that exposes two functions:
>
> -spec config(Key :: atom()) -> Value :: term().
> -spec config(Key :: atom(), Value :: term()) -> ok.

Really the spec should be:

   -spec config(Key :: atom()) -> {ok, Value :: term()} | undefined.
   -spec config(Key :: atom(), Value :: term()) -> ok.

The alternative is to crash the system if someone requests an undefined
value (or to return the atom 'undefined' and messily match on it all the
time -- which is workable until the setting value you actually *intend*
happens to itself be 'undefined'!). You could also use 'false' in place
of 'undefined' if you happen to need some listy abstractions that
operate over config data, of course, but anyway, I think you get the idea.

I'm a big fan of {ok, Value} | {error, Reason} type return values
because they provide more options for the author of the calling code, to
include direct assertion at the place they call it:

   % Only crash the calling process if Key doesn't exist
   {ok, Setting} = conf_man:config(Key),
   % ...

etc.

I just noticed this in retrospect. Small detail, but can have enough of
an impact on calling code that it is worth mentioning.

-Craig
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic Configuration Database in Erlang

Roger Lipscombe-2
In reply to this post by zxq9-2
On Wed, 4 Dec 2019 at 15:10, zxq9 <[hidden email]> wrote:

>
> On 2019/12/04 22:03, Stefano Bertuola wrote:
> > I am looking for understanding how to implement a configuration database
> > in Erlang (which allows dynamic configuration).
> >
> > Does anyone have more details about this or any other implementation?
>
> Hi, Stefano.
>
> I make a configuration manager/DB in just about any program I write that
> needs configuration. Usually this takes the form of a single gen_server
> that is started first at the beginning of execution (you can have it
> start later, but usually your system configuration is pretty close to
> the very top of your supervision tree).

Counterpoint: if you're dealing with a larger, mixed, system that has
Erlang and non-Erlang components, you probably want to look at
external configuration databases, such as etcd or consul. They'll deal
with service discovery, health-checking, and all that other
complicated stuff.

Or you might want a mixture of the two approaches.