erl_scan, erl_parse, macros and records

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

erl_scan, erl_parse, macros and records

Dan Kelley

Hi,

I'm working on a router of sorts.  I'd like to have a config file which specifies how to classify messages, what channels are available, and routing rules based on classifications and channels.

When my application starts, I'd like to transform the configuration into a module (let's call it 'router') so I can do someting like 'router:route(Message)'.  Message will be a record that encapsulates a bunch of things about the message, and the returned value will be the channel to which the message should be delivered.  The Message record is reasonably complicated - the top level thing contains several layers of different records.  I also have a large number of macros that I use to extract different fields/subrecords of a top-level Message.

When I started looking at how to do this, I quickly found erl_scan, erl_parse, and friends.  I can easily create a module on the fly with these, but everything falls apart when I need to get at the Message record and macro definitions.  Despite the access provided to compiler-ish things by erl_scan, erl_parse and friends, it looks like the only way to get at the preprocessor is to use epp on files.  This is a problem because the code I want to dynamically create needs to use the record definitions and macros from the Message header file.

Trying to figure a way around this led me to aleppo, which neatly solved the problem of including the header file, but aleppo doesn't process records.  For a brief and exciting moment, it looked like erl_expand_records might save the day, but it expects forms, and I can't get anything with records through erl_parse:parse_form.

Am I missing an easier way to dynamically create code that references macros and records?

Thanks,

Dan



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Mihai Balea

On Jun 16, 2011, at 3:48 PM, Dan Kelley wrote:

>
> Hi,
>
> I'm working on a router of sorts.  I'd like to have a config file which specifies how to classify messages, what channels are available, and routing rules based on classifications and channels.
>
> When my application starts, I'd like to transform the configuration into a module (let's call it 'router') so I can do someting like 'router:route(Message)'.  Message will be a record that encapsulates a bunch of things about the message, and the returned value will be the channel to which the message should be delivered.  The Message record is reasonably complicated - the top level thing contains several layers of different records.  I also have a large number of macros that I use to extract different fields/subrecords of a top-level Message.
>
> When I started looking at how to do this, I quickly found erl_scan, erl_parse, and friends.  I can easily create a module on the fly with these, but everything falls apart when I need to get at the Message record and macro definitions.  Despite the access provided to compiler-ish things by erl_scan, erl_parse and friends, it looks like the only way to get at the preprocessor is to use epp on files.  This is a problem because the code I want to dynamically create needs to use the record definitions and macros from the Message header file.
>
> Trying to figure a way around this led me to aleppo, which neatly solved the problem of including the header file, but aleppo doesn't process records.  For a brief and exciting moment, it looked like erl_expand_records might save the day, but it expects forms, and I can't get anything with records through erl_parse:parse_form.
>
> Am I missing an easier way to dynamically create code that references macros and records?
>
> Thanks,

Why not generate a temporary file and apply compile:file/2 on it?

Mihai

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Dan Kelley


On Thu, Jun 16, 2011 at 4:12 PM, Mihai Balea <[hidden email]> wrote:

On Jun 16, 2011, at 3:48 PM, Dan Kelley wrote:

>
> Hi,
>
> I'm working on a router of sorts.  I'd like to have a config file which specifies how to classify messages, what channels are available, and routing rules based on classifications and channels.
>
> When my application starts, I'd like to transform the configuration into a module (let's call it 'router') so I can do someting like 'router:route(Message)'.  Message will be a record that encapsulates a bunch of things about the message, and the returned value will be the channel to which the message should be delivered.  The Message record is reasonably complicated - the top level thing contains several layers of different records.  I also have a large number of macros that I use to extract different fields/subrecords of a top-level Message.
>
> When I started looking at how to do this, I quickly found erl_scan, erl_parse, and friends.  I can easily create a module on the fly with these, but everything falls apart when I need to get at the Message record and macro definitions.  Despite the access provided to compiler-ish things by erl_scan, erl_parse and friends, it looks like the only way to get at the preprocessor is to use epp on files.  This is a problem because the code I want to dynamically create needs to use the record definitions and macros from the Message header file.
>
> Trying to figure a way around this led me to aleppo, which neatly solved the problem of including the header file, but aleppo doesn't process records.  For a brief and exciting moment, it looked like erl_expand_records might save the day, but it expects forms, and I can't get anything with records through erl_parse:parse_form.
>
> Am I missing an easier way to dynamically create code that references macros and records?
>
> Thanks,

Why not generate a temporary file and apply compile:file/2 on it?

Two reasons:

First, I can provide better error messages, in the sense that I'll have smaller compile units that I'll trace back to specific parts of the config.

Second,  I'm not sure that I can make that non-ugly.  I had hoped to write my dynamic module to disk using erl_pp so it'd be human-readable.  

But yeah, that's my fallback position.

Dan


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Anders Nygren-2
On Thu, Jun 16, 2011 at 3:19 PM, Dan Kelley <[hidden email]> wrote:

>
>
> On Thu, Jun 16, 2011 at 4:12 PM, Mihai Balea <[hidden email]> wrote:
>>
>> On Jun 16, 2011, at 3:48 PM, Dan Kelley wrote:
>>
>> >
>> > Hi,
>> >
>> > I'm working on a router of sorts.  I'd like to have a config file which
>> > specifies how to classify messages, what channels are available, and routing
>> > rules based on classifications and channels.
>> >
>> > When my application starts, I'd like to transform the configuration into
>> > a module (let's call it 'router') so I can do someting like
>> > 'router:route(Message)'.  Message will be a record that encapsulates a bunch
>> > of things about the message, and the returned value will be the channel to
>> > which the message should be delivered.  The Message record is reasonably
>> > complicated - the top level thing contains several layers of different
>> > records.  I also have a large number of macros that I use to extract
>> > different fields/subrecords of a top-level Message.
>> >
>> > When I started looking at how to do this, I quickly found erl_scan,
>> > erl_parse, and friends.  I can easily create a module on the fly with these,
>> > but everything falls apart when I need to get at the Message record and
>> > macro definitions.  Despite the access provided to compiler-ish things by
>> > erl_scan, erl_parse and friends, it looks like the only way to get at the
>> > preprocessor is to use epp on files.  This is a problem because the code I
>> > want to dynamically create needs to use the record definitions and macros
>> > from the Message header file.
>> >
>> > Trying to figure a way around this led me to aleppo, which neatly solved
>> > the problem of including the header file, but aleppo doesn't process
>> > records.  For a brief and exciting moment, it looked like erl_expand_records
>> > might save the day, but it expects forms, and I can't get anything with
>> > records through erl_parse:parse_form.
>> >
>> > Am I missing an easier way to dynamically create code that references
>> > macros and records?
>> >
>> > Thanks,
>>
>> Why not generate a temporary file and apply compile:file/2 on it?
>
> Two reasons:
> First, I can provide better error messages, in the sense that I'll have
> smaller compile units that I'll trace back to specific parts of the config.
> Second,  I'm not sure that I can make that non-ugly.  I had hoped to write
> my dynamic module to disk using erl_pp so it'd be human-readable.
> But yeah, that's my fallback position.
> Dan
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>

Hi
Take a look at erl_syntax.

/Anders
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Zsolt Keszthelyi
In reply to this post by Dan Kelley
Have you tried smerl? http://code.google.com/p/smerl/
I used it for something very similar (dynamically generating email sorting functions from a text file containing rules)

Zsolt

On Thu, Jun 16, 2011 at 4:19 PM, Dan Kelley <[hidden email]> wrote:


On Thu, Jun 16, 2011 at 4:12 PM, Mihai Balea <[hidden email]> wrote:

On Jun 16, 2011, at 3:48 PM, Dan Kelley wrote:

>
> Hi,
>
> I'm working on a router of sorts.  I'd like to have a config file which specifies how to classify messages, what channels are available, and routing rules based on classifications and channels.
>
> When my application starts, I'd like to transform the configuration into a module (let's call it 'router') so I can do someting like 'router:route(Message)'.  Message will be a record that encapsulates a bunch of things about the message, and the returned value will be the channel to which the message should be delivered.  The Message record is reasonably complicated - the top level thing contains several layers of different records.  I also have a large number of macros that I use to extract different fields/subrecords of a top-level Message.
>
> When I started looking at how to do this, I quickly found erl_scan, erl_parse, and friends.  I can easily create a module on the fly with these, but everything falls apart when I need to get at the Message record and macro definitions.  Despite the access provided to compiler-ish things by erl_scan, erl_parse and friends, it looks like the only way to get at the preprocessor is to use epp on files.  This is a problem because the code I want to dynamically create needs to use the record definitions and macros from the Message header file.
>
> Trying to figure a way around this led me to aleppo, which neatly solved the problem of including the header file, but aleppo doesn't process records.  For a brief and exciting moment, it looked like erl_expand_records might save the day, but it expects forms, and I can't get anything with records through erl_parse:parse_form.
>
> Am I missing an easier way to dynamically create code that references macros and records?
>
> Thanks,

Why not generate a temporary file and apply compile:file/2 on it?

Two reasons:

First, I can provide better error messages, in the sense that I'll have smaller compile units that I'll trace back to specific parts of the config.

Second,  I'm not sure that I can make that non-ugly.  I had hoped to write my dynamic module to disk using erl_pp so it'd be human-readable.  

But yeah, that's my fallback position.

Dan


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Ulf Wiger
In reply to this post by Dan Kelley

This sounds in part similar to what the Diameter application (part of OTP R14B03) does.

The Diameter application uses a parse_transform called diameter_exprecs, which is essentially the same as the exprecs module in http://github.com/esl/parse_trans

For one thing, this allows users to work with record definitions generated at compile-time, without having to deal with include files. It also allows the diameter_service agent to check ubiquitous attributes like 'Destination-Host', 'Destination-Realm' in any record, as long as that record is known by the encode-decode ("dictionary") module:

get_avp_value(Dict, Name, Rec)
  when is_tuple(Rec) ->
    try
        Dict:'#get-'(Name, Rec)
    catch
        error:_ ->
            undefined
    end.

(from diameter/src/app/diameter_service.erl)

Another thing that might help is parse_trans_codegen.erl, which provides a (IMHO) more readable way to generate code.
The idea is that you simply wrap the code you want to generate in a fun(), possibly using some simple expansion techniques to import values, etc. If records are known at generation time, they can simply be included and used as-is, with normal record syntax.


See also https://github.com/esl/parse_trans/blob/master/examples/ex_codegen.erl for some (admittedly pretty corny) examples.

BR,
Ulf W

On 16 Jun 2011, at 21:48, Dan Kelley wrote:


Hi,

I'm working on a router of sorts.  I'd like to have a config file which specifies how to classify messages, what channels are available, and routing rules based on classifications and channels.

When my application starts, I'd like to transform the configuration into a module (let's call it 'router') so I can do someting like 'router:route(Message)'.  Message will be a record that encapsulates a bunch of things about the message, and the returned value will be the channel to which the message should be delivered.  The Message record is reasonably complicated - the top level thing contains several layers of different records.  I also have a large number of macros that I use to extract different fields/subrecords of a top-level Message.

When I started looking at how to do this, I quickly found erl_scan, erl_parse, and friends.  I can easily create a module on the fly with these, but everything falls apart when I need to get at the Message record and macro definitions.  Despite the access provided to compiler-ish things by erl_scan, erl_parse and friends, it looks like the only way to get at the preprocessor is to use epp on files.  This is a problem because the code I want to dynamically create needs to use the record definitions and macros from the Message header file.

Trying to figure a way around this led me to aleppo, which neatly solved the problem of including the header file, but aleppo doesn't process records.  For a brief and exciting moment, it looked like erl_expand_records might save the day, but it expects forms, and I can't get anything with records through erl_parse:parse_form.

Am I missing an easier way to dynamically create code that references macros and records?

Thanks,

Dan


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Ulf Wiger, CTO, Erlang Solutions, Ltd.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Tim Watson-5
I can also whole-heartedly recommend parse_trans, and in particular
the exprecs module. You can see another usage of this in
https://github.com/hyperthunk/nodewatch/blob/master/dxcommon/src/dx_subscription.erl
and of the API for accessing record info in
https://github.com/hyperthunk/nodewatch/blob/master/dxcommon/src/dxcommon.erl
- look at the function record_to_proplist/1, which is completely
generic in the type of the record and works with recursive record
structures.

Very useful stuff!

On 17 June 2011 07:42, Ulf Wiger <[hidden email]> wrote:

>
> This sounds in part similar to what the Diameter application (part of OTP
> R14B03) does.
> The Diameter application uses a parse_transform called diameter_exprecs,
> which is essentially the same as the exprecs module in
> http://github.com/esl/parse_trans
> For one thing, this allows users to work with record definitions generated
> at compile-time, without having to deal with include files. It also allows
> the diameter_service agent to check ubiquitous attributes like
> 'Destination-Host', 'Destination-Realm' in any record, as long as that
> record is known by the encode-decode ("dictionary") module:
> get_avp_value(Dict, Name, Rec)
>   when is_tuple(Rec) ->
>     try
>         Dict:'#get-'(Name, Rec)
>     catch
>         error:_ ->
>             undefined
>     end.
> (from diameter/src/app/diameter_service.erl)
> Another thing that might help is parse_trans_codegen.erl, which provides a
> (IMHO) more readable way to generate code.
> The idea is that you simply wrap the code you want to generate in a fun(),
> possibly using some simple expansion techniques to import values, etc. If
> records are known at generation time, they can simply be included and used
> as-is, with normal record syntax.
> https://github.com/esl/parse_trans/blob/master/doc/parse_trans_codegen.md
> See
> also https://github.com/esl/parse_trans/blob/master/examples/ex_codegen.erl for
> some (admittedly pretty corny) examples.
> BR,
> Ulf W
> On 16 Jun 2011, at 21:48, Dan Kelley wrote:
>
> Hi,
> I'm working on a router of sorts.  I'd like to have a config file which
> specifies how to classify messages, what channels are available, and routing
> rules based on classifications and channels.
> When my application starts, I'd like to transform the configuration into a
> module (let's call it 'router') so I can do someting like
> 'router:route(Message)'.  Message will be a record that encapsulates a bunch
> of things about the message, and the returned value will be the channel to
> which the message should be delivered.  The Message record is reasonably
> complicated - the top level thing contains several layers of different
> records.  I also have a large number of macros that I use to extract
> different fields/subrecords of a top-level Message.
> When I started looking at how to do this, I quickly found erl_scan,
> erl_parse, and friends.  I can easily create a module on the fly with these,
> but everything falls apart when I need to get at the Message record and
> macro definitions.  Despite the access provided to compiler-ish things by
> erl_scan, erl_parse and friends, it looks like the only way to get at the
> preprocessor is to use epp on files.  This is a problem because the code I
> want to dynamically create needs to use the record definitions and macros
> from the Message header file.
> Trying to figure a way around this led me to aleppo, which neatly solved the
> problem of including the header file, but aleppo doesn't process records.
>  For a brief and exciting moment, it looked like erl_expand_records might
> save the day, but it expects forms, and I can't get anything with records
> through erl_parse:parse_form.
> Am I missing an easier way to dynamically create code that references macros
> and records?
> Thanks,
> Dan
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
>
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Jon Watte
In reply to this post by Dan Kelley
First, why is it not sufficient to provide your router rules as configuration information for a static module?
If you do that, then you will have no problems with in-place code update etc. It will also be easier to debug!

Second, if you need to parameterize routing, then you can use parameterized modules. There are some gotchas, such as the "real" arity of functions is actually higher than the "apparent" arity, but they work well for certain cases.

If you really need to go code generation, then I suggest you consider the case of changing rules carefully, and plan out how your system will perform in that case. (Also, HiPE vs interpreter, etc)

Sincerely,

jw


--
Americans might object: there is no way we would sacrifice our living standards for the benefit of people in the rest of the world. Nevertheless, whether we get there willingly or not, we shall soon have lower consumption rates, because our present rates are unsustainable.



On Thu, Jun 16, 2011 at 12:48 PM, Dan Kelley <[hidden email]> wrote:

Hi,

I'm working on a router of sorts.  I'd like to have a config file which specifies how to classify messages, what channels are available, and routing rules based on classifications and channels.

When my application starts, I'd like to transform the configuration into a module (let's call it 'router') so I can do someting like 'router:route(Message)'.  Message will be a record that encapsulates a bunch of things about the message, and the returned value will be the channel to which the message should be delivered.  The Message record is reasonably complicated - the top level thing contains several layers of different records.  I also have a large number of macros that I use to extract different fields/subrecords of a top-level Message.

When I started looking at how to do this, I quickly found erl_scan, erl_parse, and friends.  I can easily create a module on the fly with these, but everything falls apart when I need to get at the Message record and macro definitions.  Despite the access provided to compiler-ish things by erl_scan, erl_parse and friends, it looks like the only way to get at the preprocessor is to use epp on files.  This is a problem because the code I want to dynamically create needs to use the record definitions and macros from the Message header file.

Trying to figure a way around this led me to aleppo, which neatly solved the problem of including the header file, but aleppo doesn't process records.  For a brief and exciting moment, it looked like erl_expand_records might save the day, but it expects forms, and I can't get anything with records through erl_parse:parse_form.

Am I missing an easier way to dynamically create code that references macros and records?

Thanks,

Dan



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_scan, erl_parse, macros and records

Robert Virding-2
In reply to this post by Dan Kelley
Sorry, I have not really been able to understand exactly what you want to do and what you are having problems with. A short review of what is done where:

erl_scan - generates tokens from characters
epp (the preprocessor) - handles macros, both definition and expansion, and include files. That's all it does not, no more. It works on tokens and generates tokens.
erl_parse - parses everything except macros which have been done in epp. So it parses records, both the definition and use of.
compile (the compiler) - takes the abstract syntax from erl_parse and generates code. All record processing is done here (in a very early pass).

Erl_scan, erl_parse and the compiler are easy to run directly but unfortunately epp, the preprocessor, only works on files. There have been numerous suggestions to modify it to work directly on tokens but this has never been done.

So if you want to generate code from within a program without writing to temporary files you can basically do everything, including records, except for macros. Include files can easily handled explicitly by parsing them and including the abstract forms directly in with your own forms. If you are generating your own code it should not be too difficult to get around the lack of macros.

An alternative to using the standard libraries to generate code is to use the syntax_tools application with the erl_syntax module to build an erlang abstract syntax. These tools can handle macros. It does not internally use the same abstract syntax as the standard tools but it has functions for generating the standard abstract syntax from its representation, which can then be used by the compiler.

An example of generating a record definition from a string would be:

{ok,Tokens,Line} = erl_scan:string("-record(foo, {a,b=42,c}). "),
{ok,FooDef} = erl_parse:parse_form(Tokens)

where FooDef is the abstract syntax form of the record definition. The abstract syntax of a module is just a list of the abstract syntax of al the forms (attributes and functions) in the module. This is why include files are basically simple to handle.

At what level do you want to start generating input? From strings, or what?

Robert


----- "Dan Kelley" <[hidden email]> wrote:

>
>
Hi,

>
I'm working on a router of sorts.  I'd like to have a config file which specifies how to classify messages, what channels are available, and routing rules based on classifications and channels.

>
When my application starts, I'd like to transform the configuration into a module (let's call it 'router') so I can do someting like 'router:route(Message)'.  Message will be a record that encapsulates a bunch of things about the message, and the returned value will be the channel to which the message should be delivered.  The Message record is reasonably complicated - the top level thing contains several layers of different records.  I also have a large number of macros that I use to extract different fields/subrecords of a top-level Message.

>
When I started looking at how to do this, I quickly found erl_scan, erl_parse, and friends.  I can easily create a module on the fly with these, but everything falls apart when I need to get at the Message record and macro definitions.  Despite the access provided to compiler-ish things by erl_scan, erl_parse and friends, it looks like the only way to get at the preprocessor is to use epp on files.  This is a problem because the code I want to dynamically create needs to use the record definitions and macros from the Message header file.

>
Trying to figure a way around this led me to aleppo, which neatly solved the problem of including the header file, but aleppo doesn't process records.  For a brief and exciting moment, it looked like erl_expand_records might save the day, but it expects forms, and I can't get anything with records through erl_parse:parse_form.

>
Am I missing an easier way to dynamically create code that references macros and records?

>
Thanks,

>
Dan

>

>

> _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions