Defensio Defends Against Spam With Merb

Case Study

The Opportunity

Over the last few years, blogs, online forums, and wikis have grown increasingly in popularity—and as a target for spam. Carl Mercier realized that traditional spam blockers were not effective and saw an opportunity to create Defensio, a spam filtering web service to weed out spam with 99.8 percent accuracy. He also determined that Ruby and Rails were the best tools for the task.

“It’s a really elegant language that is easy to learn, easy to use, and easy to hack if you need to do something special,” he said. Getting the Defensio application built and up and running was a fast, painless process with Rails.

As the company matured and larger websites began integrating Defensio, the development team faced challenges with Rails that they hadn’t anticipated. Rails didn’t offer the speed Defensio needed to quickly and effectively filter spam for its larger clients, some of which hosted upwards of 40,000 blogs; scaling became an issue. In addition, while Rails offered many features that were superfluous with Defensio’s simple API, it lacked the ability to easily customize those features as needed.

Most importantly, though, was that Rails wasn’t threadsafe. Thread-safety was a major issue, given Defensio’s need to simultaneously handle numerous processing requests to keep its clients’ sites up and running smoothly. Processing each request and analyzing the content for spam was an involved process. Bounced or queued requests as a result of server overload meant a greater chance that spam would make it through.

“When you need to maintain 10x more servers to handle a high load of requests because the framework you are using isn’t threadsafe, your operation becomes costly and cumbersome,” said Mercier.

If Mercier wanted to turn Defensio into the best spam filtering web service around and keep his clients, he needed to fix these Rails-related problems. His choices were to port the application into another language, or find a way to work with Ruby without relying solely on Rails.

The Solution

Fortunately for Mercier, a new open source Ruby framework was emerging. The Merb project, sponsored by Engine Yard, had identified the same issues with Rails that Mercier experienced, so their goal of building a fast, lightweight, threadsafe and customizable alternative framework resonated strongly with him. After hearing buzz on Merb from his peers in the “Montreal on Rails” community, he made the switch.

Rails’ monolithic code base was both too restrictive and too extensive; with everything offered in one bundle, it was hard work to tweak things to meet Defensio’s needs. On the other hand, Merb’s simple microkernel architecture gave Mercier the flexibility he needed to customize Defensio without having to work around Rails’ unnecessary features and code. Moreover, a vibrant community had grown around Merb to create an ever-expanding pool of plugins that would only enhance choice for users.

Where Rails fell short for Defensio’s API on speed, flexibility and modularity, Merb succeeded, allowing Defensio to effectively process requests and filter spam before it created problems for its customers’ sites. But Mercier is quick to point out that this doesn’t mean Rails is becoming obsolete.

“Rails will serve 80 percent of people pretty well, but for the 20 percent that requires more flexibility than Rails can offer, it is great to know that Merb is out there,” said Mercier.

Results

Now that the Merb-based Defensio application has been running for five months, Mercier is seeing amazing results in performance and efficiency.

Unlike Rails, Merb is threadsafe and can handle simultaneous multiple requests easily. For customers, this ability to manage more queries per Mongrel at the same time—up to 10x what Rails could have achieved—has meant more reliable protection, as spam won’t slip through the cracks to the website because Defensio couldn’t properly process the request. For Mercier, this has resulted in easier and cheaper scaling on half of the hardware required by Rails and better manageability of Defensio’s typical traffic patterns—long periods of steady or low traffic punctuated by shorter spurts of very high request volumes.

Mercier also appreciates the peace of mind that comes from having an active community standing behind the framework. Not only does Merb have a strong and involved team of developers and contributors who are well known in the Ruby community, but the project is also backed by Engine Yard, a solid company with a strong reputation in the Ruby community. “At the end of the day, if there is a problem or a bug, I know that there are Merb experts who can help us take care of it,” he said.

Whether a developer uses Merb or Rails, Mercier stresses the importance of assessing their project goals and needs first.

“It’s good for the Ruby community to have a choice of frameworks. Merb and Rails really complement each other depending on the needs of a particular development team. For some websites, including ours, Rails is great. For our API, though, Rails just didn’t cut it, and I am grateful that Merb gave us an alternative.”