Selecting records from Mnesia using partial key

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Selecting records from Mnesia using partial key

Frans Schneider-2
Dear list,

Before trying to reinvent the wheel, maybe somebody can point me in the
right direction.

I have a request with a variable set of attributes [A, B, C...] and a
Mnesia table with policies, each applicable for a also variable set of
attributes [X, Y, Z...]. Policies may be applicable for the same set of
attributes, in which case all these policies have to be evaluated.

A policy should be selected if its attributes is the set _or a subset_
of attributes from the request: i.e. if the request has the set of
attributes [A, B] than all policies with attributes [A, B], [A] and [B]
should be selected.

Attributes currently are records holding two strings but could be turned
into atoms.

Does anybody know a nice way to implement this in Mnesia?

Thanks,

Frans
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Selecting records from Mnesia using partial key

Mikael Pettersson-5
On Fri, Jul 6, 2018 at 1:34 PM, Frans Schneider <[hidden email]> wrote:

> Dear list,
>
> Before trying to reinvent the wheel, maybe somebody can point me in the
> right direction.
>
> I have a request with a variable set of attributes [A, B, C...] and a Mnesia
> table with policies, each applicable for a also variable set of attributes
> [X, Y, Z...]. Policies may be applicable for the same set of attributes, in
> which case all these policies have to be evaluated.
>
> A policy should be selected if its attributes is the set _or a subset_ of
> attributes from the request: i.e. if the request has the set of attributes
> [A, B] than all policies with attributes [A, B], [A] and [B] should be
> selected.
>
> Attributes currently are records holding two strings but could be turned
> into atoms.
>
> Does anybody know a nice way to implement this in Mnesia?

Without further information about or restrictions to your problem, I guess the
simplest solution is to use a qlc to query the policies table, and in the qlc
filter by available attributes.

This is effectively going to traverse the entire table, so it won't be
very efficient,
but it may be enough for your case (we don't know your table size, number of
attributes, etc).

If you want something more efficient, you'll need one or more indices, i.e.
mappings from attributes (or sets of attributes) to applicable policies.  Since
your input may contain a superset of a policy's attributes, you'll have to
create it/them yourself rather than relying on mnesia's built-in
indexing support.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Selecting records from Mnesia using partial key

Frans Schneider-2
Yes, I know I had to be more specific, but this was one of those Friday
afternoon posts looking for an easy answer.

The thing is, looking for applicable policies all combinations of
attributes from the request have to be tried: i.e. the number of
combinations becomes an issue. Fetching policies from Mnesia based on
their attributes is trivial using hashing.

The number of policies in the Mnesia table may be up to 10K, maybe 100K.
The number of attributes per request, as far as I have seen, varies
mostly between 4 and 8. The repository has a primary key for the policy
id and a index key which is the hash of the attribute ids used by that
policy. The number of requests is high, as high as possible, and come
from applications making authorization calls. For horizontal scaling,
the repository should be distributed.

Looking up policies probably should be implemented using some digraph
type of structure on top of some storage back-end, maybe a graph
database. I have seen some implementations of that but all lack the
possibility to remove or make changes to policies after the graph has
been created without rebuilding the whole graph. With the regular
changes made to authorizations, making changes to the graph is mandatory.

Maybe somebody did something before like this and I would like some input.

Frans

[1]

On 07/08/2018 04:45 PM, Mikael Pettersson wrote:

> On Fri, Jul 6, 2018 at 1:34 PM, Frans Schneider <[hidden email]> wrote:
>> Dear list,
>>
>> Before trying to reinvent the wheel, maybe somebody can point me in the
>> right direction.
>>
>> I have a request with a variable set of attributes [A, B, C...] and a Mnesia
>> table with policies, each applicable for a also variable set of attributes
>> [X, Y, Z...]. Policies may be applicable for the same set of attributes, in
>> which case all these policies have to be evaluated.
>>
>> A policy should be selected if its attributes is the set _or a subset_ of
>> attributes from the request: i.e. if the request has the set of attributes
>> [A, B] than all policies with attributes [A, B], [A] and [B] should be
>> selected.
>>
>> Attributes currently are records holding two strings but could be turned
>> into atoms.
>>
>> Does anybody know a nice way to implement this in Mnesia?
>
> Without further information about or restrictions to your problem, I guess the
> simplest solution is to use a qlc to query the policies table, and in the qlc
> filter by available attributes.
>
> This is effectively going to traverse the entire table, so it won't be
> very efficient,
> but it may be enough for your case (we don't know your table size, number of
> attributes, etc).
>
> If you want something more efficient, you'll need one or more indices, i.e.
> mappings from attributes (or sets of attributes) to applicable policies.  Since
> your input may contain a superset of a policy's attributes, you'll have to
> create it/them yourself rather than relying on mnesia's built-in
> indexing support.
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Selecting records from Mnesia using partial key

Frans Schneider-2
Nice, thanks Richard,

I am trying to implement a XACML policy decision point [1, 2] to see if
it can do authorization in a electronic health record system. I can not
find much data and examples of its use and it is hard to say anything
about things like the number of policies, distinct attributes etc. in a
typical installation.

Attributes are user defined and not defined by the standard, but it may
be quite a few (hundreds?) and growing during the life-span of an
application. In (academic) literature there is nothing mentioned on this
point. It seems reasonable to assume attributes may be added but not
removed from the system (they may become unused however).

I like your idea a lot and, at least for a PoC, a nice option to get
something working. Thanks,

Frans

[1]
https://www.oasis-open.org/committees/download.php/2713/Brief_Introduction_to_XACML.html
[2]
http://docs.oasis-open.org/xacml/3.0/errata01/os/xacml-3.0-core-spec-errata01-os-complete.html


On 07/09/2018 02:57 PM, Richard O'Keefe wrote:

> You said that the number of attributes per *request* is 4 to 8, but not
> how many
> distinct attributes there may be in total.  If that number is not
> unreasonably
> high, say a couple of hundred, you could
> (1) map distinct attributes to distinct non-negative integers
> (2) map sets of attributes to bit strings (integers where bit k is 1 if
> and only attribute k is in the set)
> (3) test whether Policy_Attrs is a subset of Request_Attrs by asking
>      Policy_Attrs band Request_Attrs =:= Policy_Attrs
>
>
> If a query has N attributes, there are 2**N - 1 non-empty subsets.
> You could take a "subset" query and turn it into 2**N-1 "exact match"
> queries
> and form the union of the results.  Depending on the maximum number of
> all attributes, various kinds of slicing and dicing might pay off, but
> with respect, something about this bothers me.
>
> I think it would be helpful to describe the actual application in a little
> more detail.
>
> On 9 July 2018 at 19:44, Frans Schneider <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Yes, I know I had to be more specific, but this was one of those
>     Friday afternoon posts looking for an easy answer.
>
>     The thing is, looking for applicable policies all combinations of
>     attributes from the request have to be tried: i.e. the number of
>     combinations becomes an issue. Fetching policies from Mnesia based
>     on their attributes is trivial using hashing.
>
>     The number of policies in the Mnesia table may be up to 10K, maybe
>     100K. The number of attributes per request, as far as I have seen,
>     varies mostly between 4 and 8. The repository has a primary key for
>     the policy id and a index key which is the hash of the attribute ids
>     used by that policy. The number of requests is high, as high as
>     possible, and come from applications making authorization calls. For
>     horizontal scaling, the repository should be distributed.
>
>     Looking up policies probably should be implemented using some
>     digraph type of structure on top of some storage back-end, maybe a
>     graph database. I have seen some implementations of that but all
>     lack the possibility to remove or make changes to policies after the
>     graph has been created without rebuilding the whole graph. With the
>     regular changes made to authorizations, making changes to the graph
>     is mandatory.
>
>     Maybe somebody did something before like this and I would like some
>     input.
>
>     Frans
>
>     [1]
>
>
>     On 07/08/2018 04:45 PM, Mikael Pettersson wrote:
>
>         On Fri, Jul 6, 2018 at 1:34 PM, Frans Schneider
>         <[hidden email] <mailto:[hidden email]>> wrote:
>
>             Dear list,
>
>             Before trying to reinvent the wheel, maybe somebody can
>             point me in the
>             right direction.
>
>             I have a request with a variable set of attributes [A, B,
>             C...] and a Mnesia
>             table with policies, each applicable for a also variable set
>             of attributes
>             [X, Y, Z...]. Policies may be applicable for the same set of
>             attributes, in
>             which case all these policies have to be evaluated.
>
>             A policy should be selected if its attributes is the set _or
>             a subset_ of
>             attributes from the request: i.e. if the request has the set
>             of attributes
>             [A, B] than all policies with attributes [A, B], [A] and [B]
>             should be
>             selected.
>
>             Attributes currently are records holding two strings but
>             could be turned
>             into atoms.
>
>             Does anybody know a nice way to implement this in Mnesia?
>
>
>         Without further information about or restrictions to your
>         problem, I guess the
>         simplest solution is to use a qlc to query the policies table,
>         and in the qlc
>         filter by available attributes.
>
>         This is effectively going to traverse the entire table, so it
>         won't be
>         very efficient,
>         but it may be enough for your case (we don't know your table
>         size, number of
>         attributes, etc).
>
>         If you want something more efficient, you'll need one or more
>         indices, i.e.
>         mappings from attributes (or sets of attributes) to applicable
>         policies.  Since
>         your input may contain a superset of a policy's attributes,
>         you'll have to
>         create it/them yourself rather than relying on mnesia's built-in
>         indexing support.
>
>     _______________________________________________
>     erlang-questions mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://erlang.org/mailman/listinfo/erlang-questions
>     <http://erlang.org/mailman/listinfo/erlang-questions>
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions