Unfortunately Nick doesn’t go into detail as to what the specific bottleneck was for CrazyEgg but after speaking with a few friends and reading DHH’s entry on scaling Rails, I’m thinking its an issue of an architect that doesn’t know how to scale his app or has coded the system poorly. David Hansson’s point is that if you design your system to not maintain state at the app level and keep in line with shared nothing at the app level, then you can easily scale infinitely by adding more app nodes much like LiveJournal, eBay and even Google do. David concedes that Rails can only handle half as many requests per second as PHP and therefore you are roughly going to have to spend $500 a month for 2.6 million requests/day on Rails versus $250 on PHP. If you’ve built a Rails app you know this is absolutely an inconsequential cost compared to the additional cost of man hours to build, maintain, and enhance a PHP app.

There aren’t a lot of mega traffic sites running on Rails yet but CrazyEgg certainly isn’t touching the traffic 43things brings in daily. Also, a friend of mine runs SoulCast and while I don’t know his daily stats I imagine he does as many pages views a day as CrazyEgg and I’ve never seen the site lag.

Update

Holy crap what a small internet world it is! Ok, after blogging this I was just sending a few emails reaching out to some folks I hung out with at Pubcon Vegas last week and I didn’t get a card from Neil Patel so in the process of hunting down his email I discovered that he is CTO for the company that built CrazyEgg and now I feel like an ass. Sorry Neil. I’m absolutely sure you are a skillfull architect and look forward to hearing more details.

Update 01/26/2007

I didn’t want to create a new post so I decided to append to the bottom of this post. David on scalability in the middle of a very long thread I just discovered. Great synopis and worth your time if you are still on the fence:

 I've said it before, but it bears repeating: There's nothing interesting about how Ruby on Rails scales. We've gone the easy route and merely followed what makes Yahoo!, LiveJournal, and other high-profile LAMP stacks scale high and mighty.

Take state out of the application servers and push it to database/memcached/shared network drive (that's the whole Shared Nothing thang). Use load balancers between your tiers, so you have load balancers -> web servers -> load balancers -> app servers -> load balancers -> database/memcached/shared network drive servers. (Past the entry point, load balancers can just be software, like haproxy).

In a setup like that, you can add almost any number of web and app servers without changing a thing.

Scaling the database is the "hard part", but still a solved problem. Once you get beyond what can be easily managed by a descent master-slave setup (and that'll probably take millions and millions of pageviews per day), you start doing partitioning.

Users 1-100K on cluster A, 100K-200K on cluster B, and so on. But again, this is nothing new. LiveJournal scales like that. I hear eBay too. And probably everyone else that has to deal with huge numbers.

So the scaling part is solved. What's left is judging whether the economics of it are sensible to you. And that's really a performance issue, not a scalability one.

If your app server costs $500 per month (like our dual xeons does) and can drive 30 requests/second on Rails and 60 requests/second on Java/PHP/.NET/whatever (these are totally arbitrary numbers pulled out of my...), then you're faced with the cost of $500 for 2.6 million requests/day on the Rails setup and $250 for the same on the other one.

Now. How much is productivity worth to you? Let's just take a $60K/year programmer. That's $5K/month. If you need to handle 5 million requests/day, your programmer needs to be 10% more productive on Rails to make it even. If he's 15% more productive, you're up $250. And this is not even considering the joy and happiness programmers derive from working with more productive tools (nor that people have claimed to be many times more productive).

Of course, the silly math above hinges on the assumption that the whatever stack is twice as fast as Rails. That's a very big if. And totally dependent on the application, the people, and so on. Some have found Rails to be as fast or faster than comparable "best-of-breed J2EE stacks" -- see http://weblog.rubyonrails.com/archives/2005/04/04/justingehtland-is-back-with-numbers-to-back-it-up/

The point is that the cost per request is plummeting, but the cost of programming is not. Thus, we have to find ways to trade efficiency in the runtime for efficiency in the "thought time" in order to make the development of applications cheaper. I believed we've long since entered an age where simplicity of development and maintenance is where the real value lies.
David Heinemeier Hansson
Tuesday, July 12, 2005

IDIOT!

Heres the idea:

• During the pledge drive you promote the URL krcw.com/keeplistening.
• When I, the listener from across the ether goes to that URL I can easily register an account and donate money
• As soon as I donate the minimum amount I am given a login and password that allows me access to a private streaming feed that brings me the usual awesome streaming indie rock that I love. I don’t have to wait for 3 week long pledge drive to end.

Pay to Play!

I’m sure your bottom line will more than double and I can keep listening to commercial free radio.

Once I switched to Mac I really never had any desire to waste any of my precious disk space on Windows. I use Remote Desktop to take care of my few Windows needs. However RDP often fails to work properly and is slow. So when Jared showed me how freaking fast and reliable and how little disk space a Windows XP install took on Parallels I was convinced. Its crazy. Boot XP in 30 seconds, in FULL SCREEN mode and it feels like you are on the fastest PC available. Not only that but you can boot just about any OS such as SusE or Fedora in Parallels.

Important one: It functions way better if you follow this tip from Jared (reply to my question about two mouse pointers and slow response):

Regarding the double cursor..

Load the Windows VM, then make it a partial window (via [alt]+[Return]).

Then, from the menu bar, go to VM/Install Parallels tools. I think that takes care of it. They enhance various OS level integrations.

Danah let it slip that she had collected info on nearly all Friendster and MySpace users which I interprete to mean that she wrote code to scrape all profiles. Cool! I can relate! So it was you that was the cause for Friendster to be so slow back in the JSP days. (P.S. to both Danah and Fred: the 100 million figure on MySpace is bogus. I know people generating MANY believeable profiles on MySpace with bots and I think they account for a pretty large chunk. MySpace is pretty inept to dealing with bots. Open up a thousand threads and let her rip!). But most of the talk was non-geeky and focused on how teens and adults interact in the virtual world. Here are some of my notes:

As a result of the 60’s backlash society has sought to constrain teens
and keep in a constant state of activity. Myspace is an outlet in a world in which teens are mostly occupied with school related activities.

These are just rough notes. I’ll clean them up this weekend.

Properties of MySpace
persistance – conversation continues when you aren’t around
searchability -
replicability – publish your conversation you had with your friend
invisible audiences -

Diliemna for teens – be cool but not get in trouble with adult peers

battlecry – social network by jerry fallwell

privacy problem with FB feeds
yes you can get that data by refreshing someones profile daily or through a script but not likely

exposure – don’t tell the world when i join a threesome group. friends with that geeky person
but don’t want all your friends to be notified

invasive – information that people are not ready deal with. cobot collected data in rooms.

cognitive caps to how much info you can deal with. following all that info is not reciprocal.
just b/c i follow someone carefully doesn’t mean they are there for you. what are the social
costs of having this info?

We’re taking on quite a few new projects here that involve heavy development so I decided it was time to take the plunge. Immediately I started looking at the few available quick immersion courses offered around the country to see if we could get up to speed in a hurry. Bummer! Everything was booked.

So then I got the hairbrained idea that if I was going to spend the time locating someone to come do private session in house I may as well start a Ruby on Rails training series here in Raleigh with open registration. After a lot of calling around (thanks Nathaniel!) I was fortunate enough to establish a relationship with David A. Black who is highly regarded in the industry as an excellent trainer with 13 years experience as a professor, an acclaimed author of a new book Ruby for Rails which received a 10/10 rating on Slashdot, as well as a dedicated founding co-director of Ruby Central. So registration is officially open for the Introduction to Ruby on Rails course being offered by Raleigh on Rails October 2nd – 5th!

Pay particular attention to the subdomains and what they spell out. Very slick. I’m sure your phone will be ringing off the hook over at Virante now Russ.