Global lock is unfair?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Global lock is unfair?

Ryan Stewart
My team has used global:set_lock() to implement a sort of simple mutex system on a set of resources. It turns out that this function seems to have no concept of fairness. One process might request a lock and not get it until 20 other processes have requested and gotten said lock, out of request order. Is there any easy way to do fair locking?

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Global lock is unfair?

Michael Truog
Use a single Erlang process.  It will only be able to handle a single message at a time (other messages will get queued in-order) and the behavior will be fair.


On 05/14/2018 12:55 PM, Ryan Stewart wrote:
My team has used global:set_lock() to implement a sort of simple mutex system on a set of resources. It turns out that this function seems to have no concept of fairness. One process might request a lock and not get it until 20 other processes have requested and gotten said lock, out of request order. Is there any easy way to do fair locking?


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Global lock is unfair?

Ryan Stewart
Funneling all the calls through a single process would bring the system to a grinding halt. Sure, I could write a system of processes, each of which acts as a mutex to one, particular resource, but why would I want to? All I need is locking, just like global locks, but fair. It seems like something that should already exist in a proven library.

On Mon, May 14, 2018 at 3:41 PM Michael Truog <[hidden email]> wrote:
Use a single Erlang process.  It will only be able to handle a single message at a time (other messages will get queued in-order) and the behavior will be fair.

On 05/14/2018 12:55 PM, Ryan Stewart wrote:
My team has used global:set_lock() to implement a sort of simple mutex system on a set of resources. It turns out that this function seems to have no concept of fairness. One process might request a lock and not get it until 20 other processes have requested and gotten said lock, out of request order. Is there any easy way to do fair locking?

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Global lock is unfair?

Dan Gudmundsson-2
If you are already using mnesia you can use mnesia:lock({global, GlobalKey, Nodes}, write|read) but you will have to run it in a
transaction on nodes where mnesia is running.

/Dan


On Mon, May 14, 2018 at 11:53 PM Ryan Stewart <[hidden email]> wrote:
Funneling all the calls through a single process would bring the system to a grinding halt. Sure, I could write a system of processes, each of which acts as a mutex to one, particular resource, but why would I want to? All I need is locking, just like global locks, but fair. It seems like something that should already exist in a proven library.


On Mon, May 14, 2018 at 3:41 PM Michael Truog <[hidden email]> wrote:
Use a single Erlang process.  It will only be able to handle a single message at a time (other messages will get queued in-order) and the behavior will be fair.

On 05/14/2018 12:55 PM, Ryan Stewart wrote:
My team has used global:set_lock() to implement a sort of simple mutex system on a set of resources. It turns out that this function seems to have no concept of fairness. One process might request a lock and not get it until 20 other processes have requested and gotten said lock, out of request order. Is there any easy way to do fair locking?
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Global lock is unfair?

Andrew Thompson-5
In reply to this post by Ryan Stewart
On Mon, May 14, 2018 at 04:53:08PM -0500, Ryan Stewart wrote:
> Funneling all the calls through a single process would bring the system to
> a grinding halt. Sure, I could write a system of processes, each of which
> acts as a mutex to one, particular resource, but why would I want to? All I
> need is locking, just like global locks, but fair. It seems like something
> that should already exist in a proven library.
>

For simple cases, wrapping a process around the individual resource and
using it as a lock is actually fine and should scale well. If you want
something fancier, there's Ulf's library:
https://github.com/uwiger/locks

Andrew
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Global lock is unfair?

Ryan Stewart
Thanks for the pointer, Andrew. The process per resource would work, but how would you ensure that only one process is started per resource? The only way I know would be to write a central "manager" process that keeps track of the resource processes and directs traffic according to the resource "key", ensuring a new process is started when a new resource comes in demand. On top of that, the manager has to be aware of other nodes in the cluster and resource processes that might already exist elsewhere That's still a lot more work than seems necessary.

On Tue, May 15, 2018 at 10:53 AM Andrew Thompson <[hidden email]> wrote:
On Mon, May 14, 2018 at 04:53:08PM -0500, Ryan Stewart wrote:
> Funneling all the calls through a single process would bring the system to
> a grinding halt. Sure, I could write a system of processes, each of which
> acts as a mutex to one, particular resource, but why would I want to? All I
> need is locking, just like global locks, but fair. It seems like something
> that should already exist in a proven library.
>

For simple cases, wrapping a process around the individual resource and
using it as a lock is actually fine and should scale well. If you want
something fancier, there's Ulf's library:
https://github.com/uwiger/locks

Andrew
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Global lock is unfair?

Ryan Stewart
Dan, thanks for the mnesia hint. It looks like a murky pool to dive into, but I'll read into it more.

On Tue, May 15, 2018 at 12:52 PM Ryan Stewart <[hidden email]> wrote:
Thanks for the pointer, Andrew. The process per resource would work, but how would you ensure that only one process is started per resource? The only way I know would be to write a central "manager" process that keeps track of the resource processes and directs traffic according to the resource "key", ensuring a new process is started when a new resource comes in demand. On top of that, the manager has to be aware of other nodes in the cluster and resource processes that might already exist elsewhere That's still a lot more work than seems necessary.

On Tue, May 15, 2018 at 10:53 AM Andrew Thompson <[hidden email]> wrote:
On Mon, May 14, 2018 at 04:53:08PM -0500, Ryan Stewart wrote:
> Funneling all the calls through a single process would bring the system to
> a grinding halt. Sure, I could write a system of processes, each of which
> acts as a mutex to one, particular resource, but why would I want to? All I
> need is locking, just like global locks, but fair. It seems like something
> that should already exist in a proven library.
>

For simple cases, wrapping a process around the individual resource and
using it as a lock is actually fine and should scale well. If you want
something fancier, there's Ulf's library:
https://github.com/uwiger/locks

Andrew
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Global lock is unfair?

Jesper Louis Andersen-2
In reply to this post by Ryan Stewart
On Tue, May 15, 2018 at 7:53 PM Ryan Stewart <[hidden email]> wrote:
Thanks for the pointer, Andrew. The process per resource would work, but how would you ensure that only one process is started per resource? The only way I know would be to write a central "manager" process that keeps track of the resource processes and directs traffic according to the resource "key", ensuring a new process is started when a new resource comes in demand. On top of that, the manager has to be aware of other nodes in the cluster and resource processes that might already exist elsewhere That's still a lot more work than seems necessary.


What are your failure semantics?

One "solution" is to shard the key to a node, and register the process locally on that node. Since only one can be registered at a time, this work somewhat and provides a lock on the resource. I.e., just steal riak_core and use it :P

Another solution, which you shouldn't dismiss, is to put all locks on one node only. This gives you a single-point-of-failure, but unless you need several 9's of uptime, the solution is easy to implement, easy to reason about and the 9's it gives you are very nice to have. Beware the advanced algorithm which has a bug because it can easily make your reliability measure in 9's _worse_ than having a deliberate S.P.O.F you understand.

However,

* Failure semantics matter. If one node goes down, all the locks on that node will be lost, etc.
* Ulf Wiger's locks library should work in a distributed setting and solve this problem, but with a different set of failure semantics.

Usually, a distributed lock that is truly resistant to node failure and so on is a hard problem. Theoretically, because just coming up with an algorithm is hard (Raft, PAXOS (multi-PAXOS, Fast-PAXOS, etc). Practically because implementing said algorithms in an error-free way is equally hard.

If you haven't, the SRE Handbook by Google has a chapter[0] on this. It is a decent survey which gives you an overview of the problem space without delving too much into Erlang. Once you know what you are looking for and what tooling you have, it is usually easier to pick a solution.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions