Request for opinions on using process dictionary for meta information

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Request for opinions on using process dictionary for meta information

Jay Doane-2
Our team is currently using the process dictionary to store meta information for long running processes, and would like to hear from folks familiar with beam internals whether this is a recommended practice.

Essentially we add meta information about the current request in a process dictionary of every process related to handling of this request. This information could be:
- current file
- user name of initiator of the request
- http query parameters
- type of a process (indexer | compactor | http_handler)

For example, when we detect performance problems, we run a function that returns the Pids of (say) top memory using processes. We use the meta info stored in process dictionaries to determine which documents are using excessive memory so we can take corrective action.

In practice, we are unable to use normal OTP messaging because in many cases, the high memory using processes have also grown huge message queues, and so would be unresponsive to any additional messages.

Given our existing constraints, is this a reasonable use of the process dictionary?

Thanks,
Jay

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Request for opinions on using process dictionary for meta information

Dmytro Lytovchenko
If you store some data that doesn't change much later — I see no problem here, perfectly fine example of PD use.
But on the other hand, if you store mutable data in PD — this is considered an antipattern and it makes your code have side effects which are hard to track. Also when your process dies, all data in PD will be lost, which is OK if data is set once, but not OK if the data was important.

2017-04-17 22:43 GMT+02:00 Jay Doane <[hidden email]>:
Our team is currently using the process dictionary to store meta information for long running processes, and would like to hear from folks familiar with beam internals whether this is a recommended practice.

Essentially we add meta information about the current request in a process dictionary of every process related to handling of this request. This information could be:
- current file
- user name of initiator of the request
- http query parameters
- type of a process (indexer | compactor | http_handler)

For example, when we detect performance problems, we run a function that returns the Pids of (say) top memory using processes. We use the meta info stored in process dictionaries to determine which documents are using excessive memory so we can take corrective action.

In practice, we are unable to use normal OTP messaging because in many cases, the high memory using processes have also grown huge message queues, and so would be unresponsive to any additional messages.

Given our existing constraints, is this a reasonable use of the process dictionary?

Thanks,
Jay

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Request for opinions on using process dictionary for meta information

Max Lapshin-2
Yes, this is a good way.

You can inspect process in such non-blocking way, but use this information as a hint, not as a proper and reliable data.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Request for opinions on using process dictionary for meta information

Mikael Pettersson-5
In reply to this post by Jay Doane-2
Jay Doane writes:
 > Our team is currently using the process dictionary to store meta
 > information for long running processes, and would like to hear from folks
 > familiar with beam internals whether this is a recommended practice.
 >
 > Essentially we add meta information about the current request in a process
 > dictionary of every process related to handling of this request. This
 > information could be:
 > - current file
 > - user name of initiator of the request
 > - http query parameters
 > - type of a process (indexer | compactor | http_handler)
 >
 > For example, when we detect performance problems, we run a function that
 > returns the Pids of (say) top memory using processes. We use the meta info
 > stored in process dictionaries to determine which documents are using
 > excessive memory so we can take corrective action.
 >
 > In practice, we are unable to use normal OTP messaging because in many
 > cases, the high memory using processes have also grown huge message queues,
 > and so would be unresponsive to any additional messages.
 >
 > Given our existing constraints, is this a reasonable use of the process
 > dictionary?

Yes, I think this is perfectly reasonable.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Request for opinions on using process dictionary for meta information

Gianfranco Alongi-2
Hi!

It kind of sounds similar to the idea behind gproc, which is a widely used library,
so as everyone has said already - this is perfectly fine.


Use case: System inspection

Gproc was designed to work as a central index for "process metadata", i.e. properties that describe the role and characteristics of each process. Having a single registry that is flexible enough to hold important types of property makes it easier to (a) find processes of a certain type, and (b) query and browse key data in a running system.


On Tue, Apr 18, 2017 at 11:33 AM, Mikael Pettersson <[hidden email]> wrote:
Jay Doane writes:
 > Our team is currently using the process dictionary to store meta
 > information for long running processes, and would like to hear from folks
 > familiar with beam internals whether this is a recommended practice.
 >
 > Essentially we add meta information about the current request in a process
 > dictionary of every process related to handling of this request. This
 > information could be:
 > - current file
 > - user name of initiator of the request
 > - http query parameters
 > - type of a process (indexer | compactor | http_handler)
 >
 > For example, when we detect performance problems, we run a function that
 > returns the Pids of (say) top memory using processes. We use the meta info
 > stored in process dictionaries to determine which documents are using
 > excessive memory so we can take corrective action.
 >
 > In practice, we are unable to use normal OTP messaging because in many
 > cases, the high memory using processes have also grown huge message queues,
 > and so would be unresponsive to any additional messages.
 >
 > Given our existing constraints, is this a reasonable use of the process
 > dictionary?

Yes, I think this is perfectly reasonable.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Request for opinions on using process dictionary for meta information

Roger Lipscombe-2
On 18 April 2017 at 10:29, Gianfranco Alongi <[hidden email]> wrote:
Hi!

It kind of sounds similar to the idea behind gproc, which is a widely used library,
so as everyone has said already - this is perfectly fine.


Doesn't gproc store the process metadata in an ETS table? That's a valid approach, too, but you need a monitor process to clean up if the process dies.
As another pro-example of storing information in the process dictionary; this is what lager does with metadata -- see lager:md.

Both approaches are valid: we use all of the following:

1. Process dictionary (usually for lager, but sometimes separately).
2. Separate process registry using one or more ETS tables (we don't use gproc; we wanted slightly different semantics).
3. Process responds to gen_server:call(Pid, info)

They're useful for different information, for different scenarios.

I'd be reluctant to use the process dictionary for anything other than process-internal-use or debugging information, though.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...