Storing Map into DB

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Storing Map into DB

Danniel Condez
Hi,

Any recommendation for persisting a Map into an unstructured db/filesystem. The goal is to store and fetch it as is, so that both erlang backend and FrontEnd (GraphQL) can easily fetch it.

Thanks!
Danniel

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Leo
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Leo


2017-03-23 22:39 GMT+01:00 Danniel Condez <[hidden email]>:
Hi,

Any recommendation for persisting a Map into an unstructured db/filesystem. The goal is to store and fetch it as is, so that both erlang backend and FrontEnd (GraphQL) can easily fetch it.

I would say a document-oriented nosql DB: https://en.wikipedia.org/wiki/Document-oriented_database

cheers,
Leo
 

Thanks!
Danniel



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Steve Davis
In reply to this post by Danniel Condez
Simplest (and most portable) would be:

for the put/write: term_to_binary(Map), for the get/read: binary_to_term(Blob)
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Vans S
A mnesia table with the key being the uuid of the object.

{db_mytable, uuid4, #{}}

You have full fold/match/select available to you now. 

[Valid] = :mnesia.match_object({:db_mytable, _, #{auth=>#{token=> "353442323232"}})


On Friday, March 24, 2017 10:13 AM, Steve Davis <[hidden email]> wrote:


Simplest (and most portable) would be:

for the put/write: term_to_binary(Map), for the get/read: binary_to_term(Blob)
_______________________________________________
erlang-questions mailing list



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Jesper Louis Andersen-2
In reply to this post by Danniel Condez
On Thu, Mar 23, 2017 at 10:51 PM Danniel Condez <[hidden email]> wrote:
Hi,

Any recommendation for persisting a Map into an unstructured db/filesystem. The goal is to store and fetch it as is, so that both erlang backend and FrontEnd (GraphQL) can easily fetch it.


Our GraphQL system uses Postgres as a database and utilizes jsonb columns to store map data inside the database. The tradeoff is that your map keys are binary(), but the advantage is that you have the ability to add indexes over your jsonb columsn for quicker access to report-style queries in your data set.

We essentially took Facebook's TAO-paper (Types, Associations & Objects) and implemented it on top of Postgres rather than MySQL. We can then use the TAO-API in order to satisfy queries in the GraphQL system. Preliminary analysis and benchmarking shows that this works exceptionally well. Most queries are satisfied by index lookups from memory and is in the sub 3ms range, even for larger blobs of data.

Our current primary gotcha is that you can't define

-spec m() :: #{ <<"foo">> := integer(), <<"bar">> := integer() }.

for the dialyzer. So we can't make the dialyzer efficient at debugging right now. However, the keys are static keys, so we could probably convert them into atom()'s without too much hassle. They are limited in what they can be. This requires some future rework of our GraphQL layer as well though.

(For those interested in the GraphQL stuff: talk and release is currently slated for EUC 2017).

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Éric Pailleau

Hi,
Sharing a nice article on postgresql and jsonb...
http://obartunov.livejournal.com/193870.html

"Envoyé depuis mon mobile " Eric



---- Jesper Louis Andersen a écrit ----

On Thu, Mar 23, 2017 at 10:51 PM Danniel Condez <[hidden email]> wrote:
Hi,

Any recommendation for persisting a Map into an unstructured db/filesystem. The goal is to store and fetch it as is, so that both erlang backend and FrontEnd (GraphQL) can easily fetch it.


Our GraphQL system uses Postgres as a database and utilizes jsonb columns to store map data inside the database. The tradeoff is that your map keys are binary(), but the advantage is that you have the ability to add indexes over your jsonb columsn for quicker access to report-style queries in your data set.

We essentially took Facebook's TAO-paper (Types, Associations & Objects) and implemented it on top of Postgres rather than MySQL. We can then use the TAO-API in order to satisfy queries in the GraphQL system. Preliminary analysis and benchmarking shows that this works exceptionally well. Most queries are satisfied by index lookups from memory and is in the sub 3ms range, even for larger blobs of data.

Our current primary gotcha is that you can't define

-spec m() :: #{ <<"foo">> := integer(), <<"bar">> := integer() }.

for the dialyzer. So we can't make the dialyzer efficient at debugging right now. However, the keys are static keys, so we could probably convert them into atom()'s without too much hassle. They are limited in what they can be. This requires some future rework of our GraphQL layer as well though.

(For those interested in the GraphQL stuff: talk and release is currently slated for EUC 2017).

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Danniel Condez
In reply to this post by Jesper Louis Andersen-2
Thanks Jesper! The TAO-paper is very helpful. One of the data structures I'm looking to work with are RDF triples. Right now, the end goal is rapid prototyping with no overhead on db administration and normalization. So as much as possible, I would want to keep almost a 1-1 mapping from Erlang maps to GraphQL data in the frontend.

And it sounds like you built your own graphQL layer as well. What did you think about https://github.com/graphql-erlang/graphql ?

Thanks!
Danniel

On Fri, Mar 24, 2017 at 8:01 AM, Jesper Louis Andersen <[hidden email]> wrote:
On Thu, Mar 23, 2017 at 10:51 PM Danniel Condez <[hidden email]> wrote:
Hi,

Any recommendation for persisting a Map into an unstructured db/filesystem. The goal is to store and fetch it as is, so that both erlang backend and FrontEnd (GraphQL) can easily fetch it.


Our GraphQL system uses Postgres as a database and utilizes jsonb columns to store map data inside the database. The tradeoff is that your map keys are binary(), but the advantage is that you have the ability to add indexes over your jsonb columsn for quicker access to report-style queries in your data set.

We essentially took Facebook's TAO-paper (Types, Associations & Objects) and implemented it on top of Postgres rather than MySQL. We can then use the TAO-API in order to satisfy queries in the GraphQL system. Preliminary analysis and benchmarking shows that this works exceptionally well. Most queries are satisfied by index lookups from memory and is in the sub 3ms range, even for larger blobs of data.

Our current primary gotcha is that you can't define

-spec m() :: #{ <<"foo">> := integer(), <<"bar">> := integer() }.

for the dialyzer. So we can't make the dialyzer efficient at debugging right now. However, the keys are static keys, so we could probably convert them into atom()'s without too much hassle. They are limited in what they can be. This requires some future rework of our GraphQL layer as well though.

(For those interested in the GraphQL stuff: talk and release is currently slated for EUC 2017).


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Danniel Condez
In reply to this post by Vans S
Vans, I think the second approach you mentioned is more in-lined to what I'm looking for cause I'm trying to stay away from the data being relational. Instead I would like to work more with RDF triples. Thank you so much for your help!

-Danniel

On Fri, Mar 24, 2017 at 10:25 AM, Vans S <[hidden email]> wrote:
>  I try to stay away from normalized data so have more flexibility in terms of storing and retrieving data.

In an ideal world I would like mnesia to support maps. Then you can drop the record and mnesia becomes a document db, you give it the primary/secondary indexes as you create the table,
it will find those in the maps that are the content.


> However, making relationships or pointers to other records with different types is still kinda vague. I'm prett ynew to erlang but I think I can start with this and learn further.
This is tough for me too. If anyone has good ideas I'm all ears.

The simplest approach is writing a module with the same name as the tables record name and putting the logic into there.  The associations would be %{address_ref=> uuid4, wallet_ref=> uuid4}, then to find the wallet associated with a particular user you would just do :mnesia.read({:db_wallet, wallet_ref}).


Another approach is a seperate sql style reference table,  db_r_user_wallet to map the uuid references.  This has the advantage of keeping the data cleaner as you don't have refs in there.

Personally I use the first approach, and only considered the second.

Mnesia's transactions either succeed or fail, so you can do complex things in a single transaction like touching multiple tables, and if something goes wrong you wont have a half committed transaction.

One of my recent projects was a compute cloud and attaching disks to virtual machines looks like this:

-module(db_disk).


attach(UuidOwner, UuidVm, UuidDisk) ->
    ?ENSURE_OWNER(OwnerUuid, UuidVm, UuidDisk),
    ?ENSURE_ATTACHABLE(UuidVm, UuidDisk),

    ?T(
        ok = mnesia:write({db_disk, UuidDisk, %{attached_to=> %{UuidVm=> UuidVm}}})
    ).


-module(db_vm).

disks(OwnerUuid, UuidVm) ->
    ?ENSURE_OWNER(OwnerUuid, UuidVm),
    mnesia:activity(transcation, fun()->
        mnesia:match_object({db_disk, UuidVm, %{attached_to=> %{UuidVm=>_}}})
    end).


The reason 'attached_to' == %{} is because of a hack for a missing data type / maps functionality in erlang.
Maps are new and selects/matches are not able to work on lists, unless someone can tell me how.

Ideally %{UuidVm=> UuidVm} could be simplified to %{UuidVm} if erlang were to allow sets like usage with
map syntax. And %{UuidVm=> _} would also become %{UuidVm}.

Im not aware of a way in a match/select to do a lists:member/2.

Each disk can be attached to multiple VMs in the case of network shared block storage.



On Friday, March 24, 2017 12:28 PM, Danniel Condez <[hidden email]> wrote:



Hi Vans,

Looks like this is a pretty simple and flexible solution. I'm looking deeper into mnesia and found `Object based programming in mnesia`(http://erlang.org/doc/apps/mnesia/Mnesia_chap5.html#id82246) to be close to what I'm trying to achieve. I try to stay away from normalized data so have more flexibility in terms of storing and retrieving data. However, making relationships or pointers to other records with different types is still kinda vague. I'm prett ynew to erlang but I think I can start with this and learn further.


On Fri, Mar 24, 2017 at 7:45 AM, Vans S <[hidden email]> wrote:

A mnesia table with the key being the uuid of the object.
>
>{db_mytable, uuid4, #{}}
>
>You have full fold/match/select available to you now.
>
>[Valid] = :mnesia.match_object({:db_ mytable, _, #{auth=>#{token=> "353442323232"}})
>
>
>
>On Friday, March 24, 2017 10:13 AM, Steve Davis <steven.charles.davis@gmail. com> wrote:
>
>
>
>Simplest (and most portable) would be:
>
>
>
>for the put/write: term_to_binary(Map), for the get/read: binary_to_term(Blob)
>
>______________________________ _________________
>
>erlang-questions mailing list
>
>[hidden email]
>
>http://erlang.org/mailman/ listinfo/erlang-questions
>
>
>


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Storing Map into DB

Jesper Louis Andersen-2
In reply to this post by Danniel Condez
On Sat, Mar 25, 2017 at 3:43 AM Danniel Condez <[hidden email]> wrote:
 
And it sounds like you built your own graphQL layer as well. What did you think about https://github.com/graphql-erlang/graphql ?


 Looks nice. Our implementation is a bit ahead of that though:

* We can lint and type check schemas as well as queries
* We have validations in place
* We have directives in place
* We parse the graphql.schema and bind it functions of the form

execute(Context, CurObj, Field, Args) -> {ok, Res} | {error, Reason} | no_return()

Our implementation is also closer to a typical compiler toolchain: compile queries into query plans. Execute those. I've started a bit on the documentation, but I'll make sure to open up our repository before that is totally done.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...