Static files served through Webmachine
Locked
Static files served through Webmachine
 
|
The Webmachine list seems pretty low traffic, so I thought I send this here to see if anyone had some interesting ideas.
I've been mulling around in my head serving up static content in webapps I write with Webmachine. The main pattern I'm reaching is heavily using frontend tools like JQuery templates, Knockout.js, Spine, Backbone, etc and having the Webmachine backend essentially serve only as a RESTful JSON interface. No templating on the backend and things like that.
Webmachine fits that part perfectly! It has made web development for me more than bearable but enjoyable for once. And I hope to release a sort of Erlang Webframework (a set of applications) in case others also find this method to be fitting to them as well.
But the problem lies in the inefficient serving of static files like html, js and css. The call comes in, matches the last dispatch rule I have of '*' and goes to a resource to server static content like this:
maybe_fetch_object(Ctx, Path) -> % if returns {true, NewCtx} then NewCtx has response_body case Ctx#ctx.response_body of undefined ->
case file_exists(Ctx, Path) of {true, FullPath} -> {ok, Value} = file:read_file(FullPath), {true, Ctx#ctx{response_body=Value}};
false -> {false, Ctx} end; _Body -> {true, Ctx} end. An obvious optimization here is to cache files in memory and check that before doing a read_file.
But a real solution to me seems to be not using Webmachine for static content at all somehow. The problem for an example url: /user/new I server up the file under /priv/user/new.html for /user/user_id I serve up /priv/user/show.html (show.html then references javascript that finds the user_id from the url and populates the content, so no server-side templating is needed.
So the problem there is it relies on Webmachine dispatch rules and resource logic to know [user, new] -> /priv/user/new.html That means my two thoughts of nginx match on subdomain and sending like api.domain.com to Webmachine and anything else it would serve itself.
Can anyone think of a way I can keep the nice URLs and serve the static html files through nginx or another webserver. Or am I just wasting time? :) Thanks,
Tristan
_______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
> Can anyone think of a way I can keep the nice URLs and serve the static html
> files through nginx or another webserver. The typical solution to this is to reverse proxy webmachine under nginx. You can set pattern matches in Nginx which will forward dynamic traffic to webmachine running on a different port (invisibly to the client) and serve static content in all its optimized nginx glory. Here's something from my own setup which forward /reset and /forgot to webmachine: location /forgot { proxy_pass http://127.0.0.1:8191 } location /reset { proxy_pass <a href="http://127.0.0.1:8191;">http://127.0.0.1:8191; } This will cause /reset to go to http://127.0.0.1:8191/reset. Webmachine runs on port 8191. jack. _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
Yeah, I do things like that already with an nginx proxy in front of Webmachine.
But my problem is wanting to be able have nice URLs like /user/user_id that if its a request for text/html returns a file from /priv/user/show.html while if its a request for application/json it returns the json representation of the user. Obviously I do that simply in Webmachine. But I'm not even sure if thats possible to filter out and proxy with nginx this way or f it is just a stupid thing to do or not.
Thanks, Tristan On Mon, Jul 18, 2011 at 9:39 PM, Jack Moffitt <[hidden email]> wrote:
_______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
> But my problem is wanting to be able have nice URLs like /user/user_id that
> if its a request for text/html returns a file from /priv/user/show.html > while if its a request for application/json it returns the json Ah, I see the issue. I assume show.html is mostly empty and loads data from the json file? I'm not sure if nginx's matchers are sophisticated enough to route by content requested. You could return a 302 from webmachine, but this is probably going to make it less efficient than just dumping the file from memory. If you're willing to use the Rails-type .json/.html format stuff on the ends of your URLs, this becomes easy. Now you've made me curious whether web caching servers are smart enough to distinguish by content type as well. jack. _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
Yeah, I agree a 302 would probably be less efficient.
I'm not familiar with Rails doing .json/.html. My understand was Rails used urls like I was wanting to use: domain.com/user/<user-id> -- no .html. Do you mean something else where they use .json/.html?
Good question about caching servers as well.
On Mon, Jul 18, 2011 at 10:06 PM, Jack Moffitt <[hidden email]> wrote:
_______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
> I'm not familiar with Rails doing .json/.html. My understand was Rails used
> urls like I was wanting to use: domain.com/user/<user-id> -- no .html. Do > you mean something else where they use .json/.html? Rails will parse the extension as the format, and defaults to HTML I believe. So calling /users/1.xml for example would do an XML dump of the dump. Similarly, you can do this for .json as well (but by default I think it does html and xml). This is redundant information due to this information being in HTTP headers preferably, but I think they did it as a practical solution for XMLHttpRequest, which has a limited ability to set headers. But if you did add such a thing, then nginx can easily proxy one and not the other or vice versa. jack. _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
Rails does check the Accept: header as well, so quite a lot more
processing goes into choosing the correct content-type for a request. On the XMLHttpRequest (xhr) instance, you can use xhr.setRequestHeader("Accept", <accept options>) and you can get to the xhr object via jQuery's beforeSend function to set this. Have you looked at using sendfile? This is essentially a way of letting erlang choose which file to send, but getting nginx to actually send it. You'll need to be proxying through either nginx or apache, but apart from that it's quite simple. Here's a rundown of how it works in nginx: http://wiki.nginx.org/XSendfile Sam On Tue, Jul 19, 2011 at 4:49 AM, Jack Moffitt <[hidden email]> wrote: > > > I'm not familiar with Rails doing .json/.html. My understand was Rails used > > urls like I was wanting to use: domain.com/user/<user-id> -- no .html. Do > > you mean something else where they use .json/.html? > > Rails will parse the extension as the format, and defaults to HTML I > believe. So calling /users/1.xml for example would do an XML dump of > the dump. Similarly, you can do this for .json as well (but by default > I think it does html and xml). > > This is redundant information due to this information being in HTTP > headers preferably, but I think they did it as a practical solution > for XMLHttpRequest, which has a limited ability to set headers. > > But if you did add such a thing, then nginx can easily proxy one and > not the other or vice versa. > > jack. > _______________________________________________ > erlang-questions mailing list > [hidden email] > http://erlang.org/mailman/listinfo/erlang-questions erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
In reply to this post by Tristan Sloughter
On Tue, Jul 19, 2011 at 03:27, Tristan Sloughter
<[hidden email]> wrote: > Can anyone think of a way I can keep the nice URLs and serve the static html > files through nginx or another webserver. Put a Varnish accelerator in front of your system (https://www.varnish-cache.org/). That way, it doesn't matter if your backend is slow at serving files as the accelerator will just cache static stuff for you. In addition, you avoid the trouble of going through another system as a proxy for static content. Also, the solution is quite modular. On the development system, you don't need more than a single system running Erlang. In my opinion, there is little reason not to plug into the whole industry there is where the main point is to make serving HTTP go faster. Trying to beat that with Erlang is probably possible, but I don't think it is beneficial. Varnish is really really hard to beat. It is built specifically for being insanely fast and it serves its data from a shared mmap()'ing, scales to multiple CPUs and is a big blob of nasty C code. I'd rather stand on the shoulders here than trying to mess with it myself. -- J. _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
Sam, Jesper both these sound great, thanks! I'm also going to look into what Jack was saying about how Rails handles some stuff.
But I'm leaning towards Jesper's idea with Varnish being the best... I'm one of those people who scoff at most benchmarks so not sure I'll bother to do one for this, but maybe, if someone here can A) suggest the best setup for it B) if it makes sense at all or would just be another worthless benchmark that really gives no information about reality.
Thanks! Tristan On Tue, Jul 19, 2011 at 7:00 AM, Jesper Louis Andersen <[hidden email]> wrote:
_______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
If you're already using Nginx and just want control over URLs, you
have access to the standard rewrite module: http://wiki.nginx.org/HttpRewriteModule This is not 30x redirection btw, unless of course you want that. On Tue, Jul 19, 2011 at 9:24 AM, Tristan Sloughter <[hidden email]> wrote: > Sam, Jesper both these sound great, thanks! I'm also going to look into what > Jack was saying about how Rails handles some stuff. > But I'm leaning towards Jesper's idea with Varnish being the best... I'm one > of those people who scoff at most benchmarks so not sure I'll bother to do > one for this, but maybe, if someone here can A) suggest the best setup for > it B) if it makes sense at all or would just be another worthless benchmark > that really gives no information about reality. > Thanks! > Tristan > On Tue, Jul 19, 2011 at 7:00 AM, Jesper Louis Andersen > <[hidden email]> wrote: >> >> On Tue, Jul 19, 2011 at 03:27, Tristan Sloughter >> <[hidden email]> wrote: >> >> > Can anyone think of a way I can keep the nice URLs and serve the static >> > html >> > files through nginx or another webserver. >> >> Put a Varnish accelerator in front of your system >> (https://www.varnish-cache.org/). That way, it doesn't matter if your >> backend is slow at serving files as the accelerator will just cache >> static stuff for you. In addition, you avoid the trouble of going >> through another system as a proxy for static content. Also, the >> solution is quite modular. On the development system, you don't need >> more than a single system running Erlang. >> >> In my opinion, there is little reason not to plug into the whole >> industry there is where the main point is to make serving HTTP go >> faster. Trying to beat that with Erlang is probably possible, but I >> don't think it is beneficial. Varnish is really really hard to beat. >> It is built specifically for being insanely fast and it serves its >> data from a shared mmap()'ing, scales to multiple CPUs and is a big >> blob of nasty C code. I'd rather stand on the shoulders here than >> trying to mess with it myself. >> >> >> -- >> J. > > > _______________________________________________ > erlang-questions mailing list > [hidden email] > http://erlang.org/mailman/listinfo/erlang-questions > > erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
I wonder if an ets solution wouldn't be as good, as these solutions are just going to find some way of holding the data in memory anyways. WIth ets, you can do things like hold compiled erlydtl/mustache templates inside of it...
Kenny
On Tue, Jul 19, 2011 at 11:47 AM, Garrett Smith <[hidden email]> wrote: If you're already using Nginx and just want control over URLs, you _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
Kenny, yeah, thats what I was thinking of doing as a cache method if I couldn't use something easily in front of Webmachine like Varnish (didn't actually think of Varnish until Jesper brought it up. Maybe a bit of both... Since while I don't want to use any templating on the backend, I want to end up with a general web framework from all this.
Tristan
On Wed, Jul 20, 2011 at 3:47 PM, Kenny Stone <[hidden email]> wrote: I wonder if an ets solution wouldn't be as good, as these solutions are just going to find some way of holding the data in memory anyways. WIth ets, you can do things like hold compiled erlydtl/mustache templates inside of it... _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
One way to look at it is that Erlang has (more powerful) versions of redis/memcached built in with ETS and Mnesia. These are pretty nice features for web development, and I'm always a fan of cutting down the external dependencies for deployment and dev. Just rebar your repo and you get this powerful, self-contained web server executable (one of the selling points of couchdb, actually).
Kenny
On Wed, Jul 20, 2011 at 6:38 PM, Tristan Sloughter <[hidden email]> wrote: Kenny, yeah, thats what I was thinking of doing as a cache method if I couldn't use something easily in front of Webmachine like Varnish (didn't actually think of Varnish until Jesper brought it up. Maybe a bit of both... Since while I don't want to use any templating on the backend, I want to end up with a general web framework from all this. _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
So I'm messing around with using nginx's xsendfile. I have the proxy setup and its working fine. And I have the following in the config to serve the static files:
location /files {
root /var/www; internal; } The proxying works fine and when I make a request to the nginx server it properly ends up going to the Webmachine server, which dispatch rule sends it to my resource for static files. Here I return:
NewReqData = wrq:set_resp_header("X-Accel-Redirect", "/files/test.html", ReqData), {"", NewReqData, Ctx}. But this does not work to end up having nginx serve up test.html from /var/www. Should it? Am I doing it completely wrong? Any suggestions?
Thanks, Tristan On Wed, Jul 20, 2011 at 6:51 PM, Kenny Stone <[hidden email]> wrote: One way to look at it is that Erlang has (more powerful) versions of redis/memcached built in with ETS and Mnesia. These are pretty nice features for web development, and I'm always a fan of cutting down the external dependencies for deployment and dev. Just rebar your repo and you get this powerful, self-contained web server executable (one of the selling points of couchdb, actually). _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
Add a trailing slash to your root delcaration. It's currently looking for '/var/www/files/test.html' - with the trailing slash it will look for '/var/www/test.html' (Yes, this seems like odd behaviour, but that's what they say to do on the page about it)
Sam
-- Sam Elliott [hidden email] -- On Fri, Jul 22, 2011 at 2:17 AM, Tristan Sloughter <[hidden email]> wrote: So I'm messing around with using nginx's xsendfile. I have the proxy setup and its working fine. And I have the following in the config to serve the static files: _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Static files served through Webmachine
|
|
So like:
root /var/www/ Didn't seem to work. I need to find where OSX stores the damn nginx logs... That will probably explain whats going on.
On Fri, Jul 22, 2011 at 5:15 AM, Sam Elliott <[hidden email]> wrote: Add a trailing slash to your root delcaration. It's currently looking for '/var/www/files/test.html' - with the trailing slash it will look for '/var/www/test.html' (Yes, this seems like odd behaviour, but that's what they say to do on the page about it) _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
«
Return to Erlang Questions
|
1 view|%1 views
Loading...
| Powered by Nabble | Edit this page |