PosegreSQL driver

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

PosegreSQL driver

Paul Guyot-2
Hello,

As there was yet another discussion on PostgreSQL drivers, we decided to eventually publish our own.

It can be found on GitHub:
https://github.com/semiocast/pgsql

Compared to existing drivers which served as an inspiration (and which we used initially), our driver features:

* OTP-supervision;
* transparently handling many PostgreSQL types, including arrays and numerics;
* cancellation of running queries using out-of-band protocol;
* SSL support;
* timeout for queries;
* iteration on results using protocol-level implicit portals and cursors, with fold, map and foreach;
* mapping of types to Erlang using a mapping of known types, handling new types that may arise through the life of the connection (this feature is an improvement of oidmap handling in Christian Sunesson's driver);
* handling both floating point and integer datetimes (this feature is an improvement of timestamp handling in Will Glozer's driver).

All these features and more (e.g. the driver passes the erlang's node name as the application_name) were developed for our needs and we hope that others would enjoy them.
This driver is currently in use with PostgreSQL 9.2 servers.

We don't use rebar internally but I made sure ./rebar compile and ./rebar eunit works.

Paul
--
Semiocast            http://semiocast.com/
+33.183627948 - 20 rue Lacaze, 75014 Paris



Reply | Threaded
Open this post in threaded view
|

PosegreSQL driver

Max Lapshin
What should I do this weekend? Take a beer, write postgresql erlang driver
or just write my own nosql database?


It is really interesting: why are there so many forks? Is it possible to
consolidate all efforts? Maybe it is impossible at all because different
drivers implement different approaches of querying async database from
async erlang?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130307/a11e2189/attachment.html>

Reply | Threaded
Open this post in threaded view
|

PostgreSQL driver

Paul Guyot-2
Le 7 mars 2013 ? 17:00, Max Lapshin a ?crit :

> What should I do this weekend? Take a beer, write postgresql erlang driver
> or just write my own nosql database?
>
> It is really interesting: why are there so many forks? Is it possible to
> consolidate all efforts? Maybe it is impossible at all because different
> drivers implement different approaches of querying async database from
> async erlang?

:)

Likewise, there are many HTTP servers, several XML and JSON parsers, etc. For example, we mostly use mochiweb here, but we built our own HTTP client and after having used mochiweb for JSON parsing, we transitioned to jiffy.
There even are several UUID modules written in Erlang!

The reasons are probably diverse, including:
- the ease to implement a working network tool, as Erlang is particularly good at protocols. Indeed, you can get a simple working PostgreSQL driver in a weekend;
- the existence of a built-in, but unsatisfying implementation (inets for httpd, odbc for PostgreSQL);
- the lack of support from first authors and the inactivity of their projects;
- the fact that some features may eventually require a rewrite.

To illustrate this point, I did some digging before open-sourcing our driver. Initially, we used ODBC, as we try to favor OTP built-in solutions. Then, because ODBC didn't satisfy our needs, in 2009, we went with ejabberd's version of Christian Sunesson's driver. I remember doing a diff with jungerl's version to find out bug fixes from ProcessOne. This driver was the most popular at the time, but it lacked OTP-supervision as we do releases and code_changes. So we implemented that, as well a thin-layer resembling ODBC API and the API that we had with other database drivers we used. At that point, we could still call our implementation a fork, with a couple of modules untouched.

I believe we fixed several bugs in this driver and I might have filed a request at ProcessOne (I surely did submit patches to ProcessOne for eunit). It didn't handle latest PostgreSQL protocol if I remember properly. It also didn't handle timestamp well, so we imported some timestamp code from Will Grozer's driver. Entirely switching to Will Grozer's driver was not so much an option as it didn't feature OTP compliancy either. We eventually had to rewrite all the code to support fold/map/foreach, asynchronous cancellation and timeouts. This was quite deep as these features are based on a different sequencing of PostgreSQL backend protocol.

At every conversations on postgresql drivers, I wondered if we shall open source our own (which cannot really be considered a fork as of today) and decided so after Anton's comments that timeouts are hard. I don't mean we solved the issue, but we actually use query timeouts in our code using our driver, and it required parting from Will Grozer's and Christian Sunesson's code, which Anton didn't do with his fork.

Paul
--
Semiocast            http://semiocast.com/
+33.183627948 - 20 rue Lacaze, 75014 Paris



Reply | Threaded
Open this post in threaded view
|

PosegreSQL driver

Tristan Sloughter-2
In reply to this post by Max Lapshin
That is why I sent my original email. There are some patches that forks
have done are design/feature decisions that can't necessarily be merged
together, but a lot can.

I didn't want to simply make tsloughter/epgsql and merge them together
without reaching out to those with the forks since then it would just end
up being another fork from wg/egpsql like all the rest.

Not sure how to move forward with this. Though I'm trying out semiocast's
client now as well.

I think it also shows an issue that I was talking about in #erlounge
yesterday. It would be great to have a page on erlang.org that breaks apps
into categories (json parser, http client, postgres/mysql drivers, ...) and
links to their source on github. But apps that are known to work and be
still in use, not just anything that shows up in a google or github search.

Tristan

On Thu, Mar 7, 2013 at 10:00 AM, Max Lapshin <max.lapshin> wrote:

> What should I do this weekend? Take a beer, write postgresql erlang driver
> or just write my own nosql database?
>
>
> It is really interesting: why are there so many forks? Is it possible to
> consolidate all efforts? Maybe it is impossible at all because different
> drivers implement different approaches of querying async database from
> async erlang?
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130311/3c147dae/attachment.html>

Reply | Threaded
Open this post in threaded view
|

PosegreSQL driver

Helge Rausch-2
On 03/11/2013 09:02 PM, Tristan Sloughter wrote:
> I think it also shows an issue that I was talking about in #erlounge
> yesterday. It would be great to have a page on erlang.org
> <http://erlang.org> that breaks apps into categories (json parser, http
> client, postgres/mysql drivers, ...) and links to their source on
> github. But apps that are known to work and be still in use, not just
> anything that shows up in a google or github search.

I'd like that, too. Something like https://www.ruby-toolbox.com/ , which
categorizes libraries and ranks them according to a metric, considering
the number of forks, stars (GitHub), activity, etc..


--
Helge Rausch

https://github.com/tsujigiri
https://twitter.com/helgerausch

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130314/7687594a/attachment.bin>

Reply | Threaded
Open this post in threaded view
|

PosegreSQL driver

Wes James-2
In reply to this post by Tristan Sloughter-2
trapexit.org has this, I noticed:

Hi there!


As you can see our forums are not available anymore. They're in a read only
mode which makes you able to access the content, but not to edit or add to
it.
Sorry for the inconvenience.

But don't worry, we're working on a new Erlang community site, which is
weeks if not days away from official launching!

-------------------

Maybe trapexit would be a good place to do this, but this will take effort
from every one to keep it up to date.

Anyone have an idea when the new site will be ready?

wes



On Mon, Mar 11, 2013 at 2:02 PM, Tristan Sloughter <
tristan.sloughter> wrote:

> That is why I sent my original email. There are some patches that forks
> have done are design/feature decisions that can't necessarily be merged
> together, but a lot can.
>
> I didn't want to simply make tsloughter/epgsql and merge them together
> without reaching out to those with the forks since then it would just end
> up being another fork from wg/egpsql like all the rest.
>
> Not sure how to move forward with this. Though I'm trying out semiocast's
> client now as well.
>
> I think it also shows an issue that I was talking about in #erlounge
> yesterday. It would be great to have a page on erlang.org that breaks
> apps into categories (json parser, http client, postgres/mysql drivers,
> ...) and links to their source on github. But apps that are known to work
> and be still in use, not just anything that shows up in a google or github
> search.
>
> Tristan
>
> On Thu, Mar 7, 2013 at 10:00 AM, Max Lapshin <max.lapshin>wrote:
>
>> What should I do this weekend? Take a beer, write postgresql erlang
>> driver or just write my own nosql database?
>>
>>
>> It is really interesting: why are there so many forks? Is it possible to
>> consolidate all efforts? Maybe it is impossible at all because different
>> drivers implement different approaches of querying async database from
>> async erlang?
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130314/91ccd819/attachment.html>

Reply | Threaded
Open this post in threaded view
|

PosegreSQL driver

Francesco Mazzoli-2
A very toned down version of Erlang Central (erlangcentral.org) will be launched on Monday. A hackage style repo is on the road map, but probably not until June or July. We are happy over all the help we can get.

F

Wes James <comptekki> wrote:

>trapexit.org has this, I noticed:
>
>Hi there!
>
>
>As you can see our forums are not available anymore. They're in a read
>only
>mode which makes you able to access the content, but not to edit or add
>to
>it.
>Sorry for the inconvenience.
>
>But don't worry, we're working on a new Erlang community site, which is
>weeks if not days away from official launching!
>
>-------------------
>
>Maybe trapexit would be a good place to do this, but this will take
>effort
>from every one to keep it up to date.
>
>Anyone have an idea when the new site will be ready?
>
>wes
>
>
>
>On Mon, Mar 11, 2013 at 2:02 PM, Tristan Sloughter <
>tristan.sloughter> wrote:
>
>> That is why I sent my original email. There are some patches that
>forks
>> have done are design/feature decisions that can't necessarily be
>merged
>> together, but a lot can.
>>
>> I didn't want to simply make tsloughter/epgsql and merge them
>together
>> without reaching out to those with the forks since then it would just
>end
>> up being another fork from wg/egpsql like all the rest.
>>
>> Not sure how to move forward with this. Though I'm trying out
>semiocast's
>> client now as well.
>>
>> I think it also shows an issue that I was talking about in #erlounge
>> yesterday. It would be great to have a page on erlang.org that breaks
>> apps into categories (json parser, http client, postgres/mysql
>drivers,
>> ...) and links to their source on github. But apps that are known to
>work
>> and be still in use, not just anything that shows up in a google or
>github
>> search.
>>
>> Tristan
>>
>> On Thu, Mar 7, 2013 at 10:00 AM, Max Lapshin
><max.lapshin>wrote:
>>
>>> What should I do this weekend? Take a beer, write postgresql erlang
>>> driver or just write my own nosql database?
>>>
>>>
>>> It is really interesting: why are there so many forks? Is it
>possible to
>>> consolidate all efforts? Maybe it is impossible at all because
>different
>>> drivers implement different approaches of querying async database
>from
>>> async erlang?
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>erlang-questions mailing list
>erlang-questions
>http://erlang.org/mailman/listinfo/erlang-questions

--
Erlang Solutions Ltd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130314/51bbeb13/attachment.html>