Solving the right problem

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Solving the right problem

Marthin Laubscher

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

I reckon the problem I’m looking to address is intrinsically concurrent and if I can design the solution just right I just might be able to avoid introducing sequential code and choke points to create an dynamic (learning, responsive to conditions) distributed server capable of using all or most of its resources for any mixture of trivial, simple, complex and massive service requests whether it’s coming from a few clients or billions of them. Essentially as illustrated in the diagram below:

 

 

I’d like to ask your advice and check some assumptions if I may impose.

 

  1. Is my conviction that Erlang (and OTP) is ideally if not best suited to addressing this type of problem objectively justified or likely based on loyalty and familiarity?
  2. Is my aspiration to scale the same code from one server to potentially hundreds of thousands of cores spread across the globe hopelessly romantic or somewhere within the realm of possibility?
  3. Assuming the network remains HTTP/HTTPS based, would Erlang’s inets module allow the code accepting new requests to interact with and control load balancing hardware to ensure each such request is served on the best available server, or will I need custom load balancers of my own for that?
  4. Still assuming HTTP/HTTPS will inets allow me to break up request processing across multiple processes and threads based on incremental scanning of the requests themselves?
  5. Are there lessons from previous (or current) successes and/or failures to achieve similar results to learn from available in the public domain like maybe from ejabberd or Yaws?  (I’m not attempting to reinvent any wheels or address a general purpose need like Yaws et al. Internet and web protocols may be involved but I have a singular focus on delivering user-specific perspectives of a single large dataset to a custom client app.)  

 

Given my two additional objectives of eventually:-

  1. ending up with a boringly simple system which elegantly encapsulates all these complex requirements, and
  2. open-sourcing the entire system once it’s beyond reach of those with nefarious intentions,

would anybody like to get involved in helping design and implement this project or even take the lead on it?

 

Thank you in advance for your kind consideration.

 

Warm regards,

 

Marthin Laubscher

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

I Gusti Ngurah Oka Prinarjaya
I recommended you to use Erlang.

Erlang can save you from nowadays complex microservice architecture. If you not use Erlang, it will become so so so fckn complex architecture, too many external tools to wired and manage.
Erlang is all in one toolbox. 
You need Redis? Erlang have ETS and or Mnesia, 
You need Consul (services discovery) in order all of your micro-services able to communicate to each other? Oh come on.. Erlang is distributed by default. you can use rpc:call() or spawn() process in other server, or send message RemoteNodePid ! {Your, Message} 

ALL tools nowadays are mimicking Erlang's features



Pada tanggal Sen, 4 Nov 2019 pukul 19.10 Marthin Laubscher <[hidden email]> menulis:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

I reckon the problem I’m looking to address is intrinsically concurrent and if I can design the solution just right I just might be able to avoid introducing sequential code and choke points to create an dynamic (learning, responsive to conditions) distributed server capable of using all or most of its resources for any mixture of trivial, simple, complex and massive service requests whether it’s coming from a few clients or billions of them. Essentially as illustrated in the diagram below:

 

 

I’d like to ask your advice and check some assumptions if I may impose.

 

  1. Is my conviction that Erlang (and OTP) is ideally if not best suited to addressing this type of problem objectively justified or likely based on loyalty and familiarity?
  2. Is my aspiration to scale the same code from one server to potentially hundreds of thousands of cores spread across the globe hopelessly romantic or somewhere within the realm of possibility?
  3. Assuming the network remains HTTP/HTTPS based, would Erlang’s inets module allow the code accepting new requests to interact with and control load balancing hardware to ensure each such request is served on the best available server, or will I need custom load balancers of my own for that?
  4. Still assuming HTTP/HTTPS will inets allow me to break up request processing across multiple processes and threads based on incremental scanning of the requests themselves?
  5. Are there lessons from previous (or current) successes and/or failures to achieve similar results to learn from available in the public domain like maybe from ejabberd or Yaws?  (I’m not attempting to reinvent any wheels or address a general purpose need like Yaws et al. Internet and web protocols may be involved but I have a singular focus on delivering user-specific perspectives of a single large dataset to a custom client app.)  

 

Given my two additional objectives of eventually:-

  1. ending up with a boringly simple system which elegantly encapsulates all these complex requirements, and
  2. open-sourcing the entire system once it’s beyond reach of those with nefarious intentions,

would anybody like to get involved in helping design and implement this project or even take the lead on it?

 

Thank you in advance for your kind consideration.

 

Warm regards,

 

Marthin Laubscher

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

I Gusti Ngurah Oka Prinarjaya
And, if your team feels so desperate to code in Erlang, because of its weird syntax, 
you can use Elixir to utilise ALL of Erlang's power in a more friendly and easy-to-understand syntax


Pada tanggal Sel, 5 Nov 2019 pukul 11.59 I Gusti Ngurah Oka Prinarjaya <[hidden email]> menulis:
I recommended you to use Erlang.

Erlang can save you from nowadays complex microservice architecture. If you not use Erlang, it will become so so so fckn complex architecture, too many external tools to wired and manage.
Erlang is all in one toolbox. 
You need Redis? Erlang have ETS and or Mnesia, 
You need Consul (services discovery) in order all of your micro-services able to communicate to each other? Oh come on.. Erlang is distributed by default. you can use rpc:call() or spawn() process in other server, or send message RemoteNodePid ! {Your, Message} 

ALL tools nowadays are mimicking Erlang's features



Pada tanggal Sen, 4 Nov 2019 pukul 19.10 Marthin Laubscher <[hidden email]> menulis:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

I reckon the problem I’m looking to address is intrinsically concurrent and if I can design the solution just right I just might be able to avoid introducing sequential code and choke points to create an dynamic (learning, responsive to conditions) distributed server capable of using all or most of its resources for any mixture of trivial, simple, complex and massive service requests whether it’s coming from a few clients or billions of them. Essentially as illustrated in the diagram below:

 

 

I’d like to ask your advice and check some assumptions if I may impose.

 

  1. Is my conviction that Erlang (and OTP) is ideally if not best suited to addressing this type of problem objectively justified or likely based on loyalty and familiarity?
  2. Is my aspiration to scale the same code from one server to potentially hundreds of thousands of cores spread across the globe hopelessly romantic or somewhere within the realm of possibility?
  3. Assuming the network remains HTTP/HTTPS based, would Erlang’s inets module allow the code accepting new requests to interact with and control load balancing hardware to ensure each such request is served on the best available server, or will I need custom load balancers of my own for that?
  4. Still assuming HTTP/HTTPS will inets allow me to break up request processing across multiple processes and threads based on incremental scanning of the requests themselves?
  5. Are there lessons from previous (or current) successes and/or failures to achieve similar results to learn from available in the public domain like maybe from ejabberd or Yaws?  (I’m not attempting to reinvent any wheels or address a general purpose need like Yaws et al. Internet and web protocols may be involved but I have a singular focus on delivering user-specific perspectives of a single large dataset to a custom client app.)  

 

Given my two additional objectives of eventually:-

  1. ending up with a boringly simple system which elegantly encapsulates all these complex requirements, and
  2. open-sourcing the entire system once it’s beyond reach of those with nefarious intentions,

would anybody like to get involved in helping design and implement this project or even take the lead on it?

 

Thank you in advance for your kind consideration.

 

Warm regards,

 

Marthin Laubscher

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

Marthin Laubscher
In reply to this post by I Gusti Ngurah Oka Prinarjaya

Thank you. Yes, Erlang has brilliant building blocks, as far as I can tell, but that is the problem – it’s only as far as I can tell. I’ve never had a chance to test Erlang’s limits. I cannot expect people who built their business’ core competencies around having learnt about those limits the hard (expensive way) to give that away for free, so I’d be grateful for any pointers and insights I can get to inform my design decisions.

 

From: I Gusti Ngurah Oka Prinarjaya <[hidden email]>
Date: Tuesday, 5 November, 2019 at 07:00
To: "[hidden email]" <[hidden email]>
Cc: Erlang <[hidden email]>
Subject: Re: Solving the right problem

 

I recommended you to use Erlang.

 

Erlang can save you from nowadays complex microservice architecture. If you not use Erlang, it will become so so so fckn complex architecture, too many external tools to wired and manage.

Erlang is all in one toolbox. 

You need Redis? Erlang have ETS and or Mnesia, 

You need Consul (services discovery) in order all of your micro-services able to communicate to each other? Oh come on.. Erlang is distributed by default. you can use rpc:call() or spawn() process in other server, or send message RemoteNodePid ! {Your, Message} 

 

ALL tools nowadays are mimicking Erlang's features

 

 

 

Pada tanggal Sen, 4 Nov 2019 pukul 19.10 Marthin Laubscher <[hidden email]> menulis:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

I reckon the problem I’m looking to address is intrinsically concurrent and if I can design the solution just right I just might be able to avoid introducing sequential code and choke points to create an dynamic (learning, responsive to conditions) distributed server capable of using all or most of its resources for any mixture of trivial, simple, complex and massive service requests whether it’s coming from a few clients or billions of them. Essentially as illustrated in the diagram below:

 

cid:16e39dee43d5b16b21

 

I’d like to ask your advice and check some assumptions if I may impose.

 

  1. Is my conviction that Erlang (and OTP) is ideally if not best suited to addressing this type of problem objectively justified or likely based on loyalty and familiarity?
  2. Is my aspiration to scale the same code from one server to potentially hundreds of thousands of cores spread across the globe hopelessly romantic or somewhere within the realm of possibility?
  3. Assuming the network remains HTTP/HTTPS based, would Erlang’s inets module allow the code accepting new requests to interact with and control load balancing hardware to ensure each such request is served on the best available server, or will I need custom load balancers of my own for that?
  4. Still assuming HTTP/HTTPS will inets allow me to break up request processing across multiple processes and threads based on incremental scanning of the requests themselves?
  5. Are there lessons from previous (or current) successes and/or failures to achieve similar results to learn from available in the public domain like maybe from ejabberd or Yaws?  (I’m not attempting to reinvent any wheels or address a general purpose need like Yaws et al. Internet and web protocols may be involved but I have a singular focus on delivering user-specific perspectives of a single large dataset to a custom client app.)  

 

Given my two additional objectives of eventually:-

  1. ending up with a boringly simple system which elegantly encapsulates all these complex requirements, and
  2. open-sourcing the entire system once it’s beyond reach of those with nefarious intentions,

would anybody like to get involved in helping design and implement this project or even take the lead on it?

 

Thank you in advance for your kind consideration.

 

Warm regards,

 

Marthin Laubscher

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

Ulf Wiger-2
In reply to this post by Marthin Laubscher
Re. 3, you should definitely look into using existing solutions for HTTP/HTTPS load-balancing. This will work every bit as well with Erlang/inets as with any other technology.

Re. 4, yes, and you're not limited to inets. Take a look e.g. at Cowboy [1]

Re. 5, well, your description is vague enough that it's hard to answer, but you seem to be aiming for some form of SOA architecture. If you want to proceed quickly with prototyping and MVPs, you could implement a component architecture inside a single Erlang node and make some minimal preparations for being able to later break them apart into a larger network of services. A single Erlang node running on a decent cloud instance is likely to handle a fairly large number of devices without breaking a sweat, unless your applications are expected to be very computationally heavy.

This way, you can defer the many messy issues of going full SOA from the beginning, and benefit from Erlang's outstanding "micro-SOA" capabilities.


Den mån 4 nov. 2019 kl 13:10 skrev Marthin Laubscher <[hidden email]>:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

I reckon the problem I’m looking to address is intrinsically concurrent and if I can design the solution just right I just might be able to avoid introducing sequential code and choke points to create an dynamic (learning, responsive to conditions) distributed server capable of using all or most of its resources for any mixture of trivial, simple, complex and massive service requests whether it’s coming from a few clients or billions of them. Essentially as illustrated in the diagram below:

 

 

I’d like to ask your advice and check some assumptions if I may impose.

 

  1. Is my conviction that Erlang (and OTP) is ideally if not best suited to addressing this type of problem objectively justified or likely based on loyalty and familiarity?
  2. Is my aspiration to scale the same code from one server to potentially hundreds of thousands of cores spread across the globe hopelessly romantic or somewhere within the realm of possibility?
  3. Assuming the network remains HTTP/HTTPS based, would Erlang’s inets module allow the code accepting new requests to interact with and control load balancing hardware to ensure each such request is served on the best available server, or will I need custom load balancers of my own for that?
  4. Still assuming HTTP/HTTPS will inets allow me to break up request processing across multiple processes and threads based on incremental scanning of the requests themselves?
  5. Are there lessons from previous (or current) successes and/or failures to achieve similar results to learn from available in the public domain like maybe from ejabberd or Yaws?  (I’m not attempting to reinvent any wheels or address a general purpose need like Yaws et al. Internet and web protocols may be involved but I have a singular focus on delivering user-specific perspectives of a single large dataset to a custom client app.)  

 

Given my two additional objectives of eventually:-

  1. ending up with a boringly simple system which elegantly encapsulates all these complex requirements, and
  2. open-sourcing the entire system once it’s beyond reach of those with nefarious intentions,

would anybody like to get involved in helping design and implement this project or even take the lead on it?

 

Thank you in advance for your kind consideration.

 

Warm regards,

 

Marthin Laubscher

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

james
In reply to this post by Marthin Laubscher
My question would be - why HTTP/HTTPS?

I know 'everybody loves stateless' but actually that's not always a good
model.

If you were to design based on gRPC and drop the connection when the
user is not actively working, how many concurrent connections will you
have, and how much latency can you wear?


Reply | Threaded
Open this post in threaded view
|

Re: [Original] Re: Solving the right problem

Marthin Laubscher
Hi James,

Nice questions. Thank you.

HTTP(S) was assumed for historical reasons which might no longer be valid. Frankly, it was probably more about the ports which were the first to be opened for outgoing traffic by panic-struck workplaces, and the level of packet inspection that accompanied that than the specific properties of the protocol. Neither the problem nor the solutions I foresee are actually stateless by nature, that much is true. Tracking cursor states in the server allows for much more intelligent choices and proactive actions to ensure that each next bit of data is delivered to the client via the most opportune path from where it is mastered, cached or calculated.

Having said that, I'd think maintaining sessions too low down the network stack might at some point start working against the type of cursor mobility I wish to have. It remains to be seen. Ideally the time it takes to set-up and tear down network level connections between individual servers and individual clients can also be factored into how the work is distributed across nodes with available computing and network capacity.

Network latency is critically important but only from the perspective of how it impacts on user experience.  The project's design approach aims to address this by progressively rolling out and using servers that are physically close (in network terms) to the clients they serve and then using smart pre-fetch logic to get the data there before the user needs it. It's impossible to make up for network latency using faster processing, but people take several orders of magnitude longer to consume a chunk of data than computers and that create the opportunity to "catch-up" on latency.

I'd love to discuss alternatives if you'd like to offer some? Making the most of ready-made components is crucial, but not if they have made conflicting design decisions which end up keeping the project overall from attaining its goals.

Marthin

On 2019/11/06, 01:16, "james" <[hidden email]> wrote:

    My question would be - why HTTP/HTTPS?
   
    I know 'everybody loves stateless' but actually that's not always a good
    model.
   
    If you were to design based on gRPC and drop the connection when the
    user is not actively working, how many concurrent connections will you
    have, and how much latency can you wear?
   
   
   



Reply | Threaded
Open this post in threaded view
|

Re: [Original] Re: Solving the right problem

t ty
I would suggest you ignore the Erlang/Elixir mode of thought as a starting point.

Your architecture is closely related to either XMPP or AMQP protocol and suggest you base it off these instead of heading straight down do a coding language. Of course I would be bias towards MongooseIM or RabbitMQ. This abstracts the entire communication question away and you can focus on the business logic.

In XMPP, each client device and server would be XMPP clients and both would communicate using simple text based coded messages. You automatically get federation and presence notification. Store and forward is automatically built in. XMPP is also moving towards addressing IoT devices which might further simplify your architecture.

In RabbitMQ each client/server would have a two queue (in/out) and all communications goes through these. Multiple clients can share a single queue pair if they are grouped together e.g. all clients within the same geo-location.

Cheers
Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

Marthin Laubscher
In reply to this post by Ulf Wiger-2

Thank you Ulf,

 

I’m hesitant to label what I’m going for here with loaded terms such as SOA architecture or “micro-SOA”. I’ve found in the many years I’ve worked in telco’s as a Systems Architect or Design Authority that we do better when we don’t overplay architectural choices as if they are goals in themselves or even as reusable solutions. Yes, for trivial problems with trivial solutions where the cost of a bespoke architecture would never be warranted it makes sense to squeeze the solution into an architecture that there or there about and deal with the little things that gets in the way as they come up. Been there, done that too.

 

When, by contrast, the project’s value proposition is such that the implications on cost, TTM or otherwise, of a custom architecture is warranted by the benefits to viability, sustainability and profitability, then using a “best fit” ready-made architecture tends to lead not to minor issues to overcome, but impedance mismatch level issues to contend with when one can least afford those – while scaling and rolling out.

 

I’m hoping to leverage my extremely narrow problem domain to arrive at the simplest possible design rules and principles which will achieve the time to market, performance and scalability I’m aiming for. If the result turns out to be applicable to more people’s problem domains than my own that would be cool, but I’m obliged to prioritise my own project above all else.

 

I’m sorry about the vagueness in my question. I will endeavor to clarify more as questions like yours come up. The project I’m working aims to have massive impact from a simple design. At this juncture I should describe it as a giant global key-value store where the keys have many dimensions including space and time and payload which, if actually stored in a key-value store, would usually be a fraction of the size of the composite key. Therefore, and because the data is intrinsically relational (i.e. everything is related to everything either directly or indirectly, but yes, Relational as in the 5th Normal Form too), these intrinsic properties makes true key-value store often sub-optimal.

 

The idea is to have the client request data based on a cursor or user/session specific context. Each client would maintain a logical connection with its closest (network wise) server where it makes the requests. The request is load-balanced at that server address to a host with available capacity and knowledge of the context.  At that host, the request gets interpreted/processed by combining a) the context in which the request was made, such as 3D viewpoint and zoom level, b) the request itself, i.e. what data is being requested, c) how and where the raw or derived data can best be obtained from.  This I’d hope to determine both recursively and without sequential code bottlenecks by splitting up the request and context matching across as many nodes as are required and available. The same then goes to how the request is fulfilled. By the client including data in the request about the user’s apparent momentum the matching process is also capable of anticipating what data and/or results might be requested next. Up to 100% of available resources may be spent on such pre-fetching, but it may not hold back the current request nor may it demand resources required for other priority requests. Simply put, every core in every data center and every provisioned network link should run at 100% all the time doing the most opportunistic work possible to provide each user fair and maximal use of the resources.

 

To implement that simply and in a manner that can evolve naturally, I was hoping to find the ideal place in code terms for a function which takes a “parsed” request and cursor data structure as parameters and returns a result structure which will be filled in in due course (not a completed result). The returned result would be handed back to an appointed client handler which would compile and send results back to the client as and when it comes in from wherever it had to be fetched. The client itself would be aware of the fact that not all data is present yet when it gets results, and would set up to be “subscribed” to data arriving later.

 

I know this is still very vague and I am sorry about that. But maybe there is enough in there to kick off the next level discussion about what I asked in the first place, which is to learn about past and/or present experiences using erlang to solve similar problems and what worked out well vs what didn’t.

 

Thanks,

 

Marthin

 

 

 

From: Ulf Wiger <[hidden email]>
Date: Tuesday, 5 November, 2019 at 17:03
To: "[hidden email]" <[hidden email]>
Cc: erlang-questions <[hidden email]>
Subject: Re: Solving the right problem

 

Re. 3, you should definitely look into using existing solutions for HTTP/HTTPS load-balancing. This will work every bit as well with Erlang/inets as with any other technology.

 

Re. 4, yes, and you're not limited to inets. Take a look e.g. at Cowboy [1]

 

Re. 5, well, your description is vague enough that it's hard to answer, but you seem to be aiming for some form of SOA architecture. If you want to proceed quickly with prototyping and MVPs, you could implement a component architecture inside a single Erlang node and make some minimal preparations for being able to later break them apart into a larger network of services. A single Erlang node running on a decent cloud instance is likely to handle a fairly large number of devices without breaking a sweat, unless your applications are expected to be very computationally heavy.

 

This way, you can defer the many messy issues of going full SOA from the beginning, and benefit from Erlang's outstanding "micro-SOA" capabilities.

 

 

Den mån 4 nov. 2019 kl 13:10 skrev Marthin Laubscher <[hidden email]>:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

I reckon the problem I’m looking to address is intrinsically concurrent and if I can design the solution just right I just might be able to avoid introducing sequential code and choke points to create an dynamic (learning, responsive to conditions) distributed server capable of using all or most of its resources for any mixture of trivial, simple, complex and massive service requests whether it’s coming from a few clients or billions of them. Essentially as illustrated in the diagram below:

 

cid:16e3c0ba2df5b16b21

 

I’d like to ask your advice and check some assumptions if I may impose.

 

  1. Is my conviction that Erlang (and OTP) is ideally if not best suited to addressing this type of problem objectively justified or likely based on loyalty and familiarity?
  2. Is my aspiration to scale the same code from one server to potentially hundreds of thousands of cores spread across the globe hopelessly romantic or somewhere within the realm of possibility?
  3. Assuming the network remains HTTP/HTTPS based, would Erlang’s inets module allow the code accepting new requests to interact with and control load balancing hardware to ensure each such request is served on the best available server, or will I need custom load balancers of my own for that?
  4. Still assuming HTTP/HTTPS will inets allow me to break up request processing across multiple processes and threads based on incremental scanning of the requests themselves?
  5. Are there lessons from previous (or current) successes and/or failures to achieve similar results to learn from available in the public domain like maybe from ejabberd or Yaws?  (I’m not attempting to reinvent any wheels or address a general purpose need like Yaws et al. Internet and web protocols may be involved but I have a singular focus on delivering user-specific perspectives of a single large dataset to a custom client app.)  

 

Given my two additional objectives of eventually:-

  1. ending up with a boringly simple system which elegantly encapsulates all these complex requirements, and
  2. open-sourcing the entire system once it’s beyond reach of those with nefarious intentions,

would anybody like to get involved in helping design and implement this project or even take the lead on it?

 

Thank you in advance for your kind consideration.

 

Warm regards,

 

Marthin Laubscher

 

 

Reply | Threaded
Open this post in threaded view
|

Re: [Original] Re: Solving the right problem

Marthin Laubscher
In reply to this post by t ty

Thanks. I’m sorry for not making it clear that in my case the communication question is integral to my business logic. I certainly don’t want to abstract it away behind a general purpose tool which involves sequential fundamentals such as single threaded message queues. XMPP would inevitably come into play higher up in the application stack but I’d be hard-pressed to latch my design paradigm to the general messaging mindset. From my project’s perspective messaging would be of its many derivatives.

 

I’m interested in learning as much as possible from those who designed other more general purpose implementations such a you mentioned. I particularly found the RabbitMQ best practices article very useful. The general gist of those boils down to using your best understanding of your clients, data and resources to design your messages and message queues so bottlenecks are avoided as a result. That’s fine, when you know those things in advance which I simply don’t. Everything about my clients, data and resources would be constantly changing. I’d be a fool to try and predict any of it at design time and an idiot to then try impose my predictions on reality. But I am confident that I can abstract the nature of changes my clients, data and resources would changes would be subjected to well enough to design and build the code to measure and respond to those changes as and when they happen. In terms of the RabbitMQ best practices I mentioned, that would translate into adapting the design of my queues, nodes and messages back and forth based on transient, slow-changing and fast-acting changes – not what I’d call simple nor practical.

 

I’ve managed to simplify the solution down to a single core datatype in which all data is stored, exchanged and referenced, ranging from a few bytes to as big as needed but with a large composite key. The obvious storage problem this introduces is overcome by applying the optimization techniques which Edgar Codd taught and many more dynamically at run-time based on measured meta-data as opposed to design time based on forecast-based modelling. While that may seem impossibly complex to pull off, it actually becomes brilliantly simple and easy once you’ve set up the environment at the appropriate level of abstraction.

 

 

From: T Ty <[hidden email]>
Date: Wednesday, 6 November, 2019 at 11:04
To: "[hidden email]" <[hidden email]>
Cc: Erlang Questions <[hidden email]>
Subject: Re: [Original] Re: Solving the right problem

 

I would suggest you ignore the Erlang/Elixir mode of thought as a starting point.

 

Your architecture is closely related to either XMPP or AMQP protocol and suggest you base it off these instead of heading straight down do a coding language. Of course I would be bias towards MongooseIM or RabbitMQ. This abstracts the entire communication question away and you can focus on the business logic.

 

In XMPP, each client device and server would be XMPP clients and both would communicate using simple text based coded messages. You automatically get federation and presence notification. Store and forward is automatically built in. XMPP is also moving towards addressing IoT devices which might further simplify your architecture.

 

In RabbitMQ each client/server would have a two queue (in/out) and all communications goes through these. Multiple clients can share a single queue pair if they are grouped together e.g. all clients within the same geo-location.

 

Cheers

Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

Jesper Louis Andersen-2
In reply to this post by Marthin Laubscher
On Mon, Nov 4, 2019 at 1:10 PM Marthin Laubscher <[hidden email]> wrote:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.


If you have been working on this design since 1991, I would recommend you start some experiments into parts of the design. You can often simulate a large amount of devices, and you can also simulate the server side. If there are multiple servers involved, solving the problem in the small scale is often useful: you get to learn about some of the small-scale problems of the design right away. As you scale such a system up, you will run into trouble that has to be solved. But often such solutions stem from the smaller scale ideas you tackled first.

At scale, I claim any system breaks down. The real work starts when you reach that level. Almost any web server technology has a breaking point, and you will hit it sooner or later. Erlang has a breaking point and you will hit it sooner or later. It also depends on the nature of the problem you are trying to solve. If you are looking at number crunching, I'd definitely design a hybrid where that is outsourced to some other language suited for fast execution. On the other hand, if you are looking at a problem where you have massive concurrency, you might get away with Erlang or Go. If you add the requirement of robustness on top, at a fine granular level, then I know no other thing than the Erlang stack which can do it. Other languages achieve robustness by containerization of computation and then restarting faulty containers. Yet, the disadvantage of such a method are that errors in such systems can "bleed" to sibling requests. This might not be suitable, depending on the problem at hand.

But you really need to start experiments. It is almost impossible to design a software system without the duality of design and implementation helping each other along. That is, design leads to partial implementation. Partial implementation leads to insights. And insights leads to a better design. Over time, it is often the case the design shrinks to a fundamental core. You want that as small as possible as it makes it understandable.


image002.png (85K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Solving the right problem

Marthin Laubscher

Spot on. I really should allow the rubber to hit the road now, which I am doing.

 

For the record, what happened in 1991 wasn’t to that I started designing the server I talk about, but that is when I accepted it as my mission to solve a problem I could no longer ignore like others still do. Erlang/OTP was one of many things I filled my toolbox with for use as and when the time comes.

 

Some of the hurdles and mental blocks I am looking to break through include:-

  • the apparent manual labour involved in setting up and managing distributed Erlang nodes including setting up physical machines, configuring virtual machines, installing OS, installing Erlang, installing the app, controlling automatic startup and shutdown, plumbing the network and hooking up release management so changes in the experimental code gets rolled out orderly and automatically,
  • the incredible performance of modern hardware in a single processor node,
  • the cost and complexity of monitoring virtual machines’ activity on the underlying hardware,
  • that generating suitable traffic to stress test candidate code might be more troublesome to simulate than it’s worth and
  • programming isn’t what gives meaning to my life but I love it and tends to get too deeply involved and thus distracted by it from other responsibilities.

 

So by all means, come and join me and my project. Help me overcome and/or sidestep my issues so we can collaborate at the appropriate level on that setting up this adaptive hyper-scaling platform for matching requests and resources to deliver near-real-time responses. We might even call it NoQ, NoMQ or OPR (see, no Q 😊, or Optimal Processing Resources).

 

From: Jesper Louis Andersen <[hidden email]>
Date: Thursday, 7 November, 2019 at 16:48
To: "[hidden email]" <[hidden email]>
Cc: "Erlang (E-mail)" <[hidden email]>
Subject: Re: Solving the right problem

 

On Mon, Nov 4, 2019 at 1:10 PM Marthin Laubscher <[hidden email]> wrote:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

If you have been working on this design since 1991, I would recommend you start some experiments into parts of the design. You can often simulate a large amount of devices, and you can also simulate the server side. If there are multiple servers involved, solving the problem in the small scale is often useful: you get to learn about some of the small-scale problems of the design right away. As you scale such a system up, you will run into trouble that has to be solved. But often such solutions stem from the smaller scale ideas you tackled first.

 

At scale, I claim any system breaks down. The real work starts when you reach that level. Almost any web server technology has a breaking point, and you will hit it sooner or later. Erlang has a breaking point and you will hit it sooner or later. It also depends on the nature of the problem you are trying to solve. If you are looking at number crunching, I'd definitely design a hybrid where that is outsourced to some other language suited for fast execution. On the other hand, if you are looking at a problem where you have massive concurrency, you might get away with Erlang or Go. If you add the requirement of robustness on top, at a fine granular level, then I know no other thing than the Erlang stack which can do it. Other languages achieve robustness by containerization of computation and then restarting faulty containers. Yet, the disadvantage of such a method are that errors in such systems can "bleed" to sibling requests. This might not be suitable, depending on the problem at hand.

 

But you really need to start experiments. It is almost impossible to design a software system without the duality of design and implementation helping each other along. That is, design leads to partial implementation. Partial implementation leads to insights. And insights leads to a better design. Over time, it is often the case the design shrinks to a fundamental core. You want that as small as possible as it makes it understandable.