Archive for November, 2013

Version 5.2 and the New Doghouse

Thursday, November 28th, 2013

doghouseSmallWe are very excited to announce the latest version, 5.2, of all the Pedias. Although this version includes quite a good number of fixes and enhancements the real new features are in the brand-new Doghouse. The original Doghouse was very popular and served us well for over a year. However with its impressive growth we needed to expand and not only modernize the core language but also take the opportunity to make it easier to add expansions in the future. Launching today Doghouse is faster, 100% redundant, has more relevant results, matches more UPCs and ISBNs, contains improved moderator tools and has a new web page.

Instead of two complete dedicated servers at Site5 and Hostgator we now moved to Heroku, a platform that lets us deploy any number of virtual servers within seconds to handle peak loads. We can now also deploy exclusive worker servers to do background maintenance without affecting the API response time.

We are now using Ruby instead of PHP for the API language. It was a language we started using to develop the online admin as it allowed for a modern interface and less development time with the use of Ruby gems. Early on we realized it would make a good fit for the API and that we could migrate the entire platform to Ruby and maintain a single language. A language more similar to Objective-C which we use everyday when working on The Pedias and Pocketpedia.

Flying Sphinx Heroku Puma




The database is now hosted by ClearDB, with redundant servers and guaranteed uptime as well as automatic backups. This was previously work that we had to execute and monitor ourselves: synchronizing Site5 and Hostgator and doing daily local backups. Now we have less code to maintain and more time to develop the features we really want to see in Doghouse. We also re-structured the database for efficiency and speed. The next step is to evaluate adopting PostgreSQL that integrates better with Heroku. ClearDB was not built for a dynamic system such as Heroku that can scale, the limits are too low to easily add to a Heroku Ruby application.

Images are now hosted at Google Storage and Rackspace. Rackspace’s content delivery network delivers incredible speeds when it comes to downloading the cover, the slowest part when downloading details for any item. We also have automatic generation of a mini size thumbnail for Pocketpedia that makes the results faster and more responsive.

We have been using Sphinx since the beginning, an amazing search engine ┬ádelivering accurate search results in single digit milliseconds. But like MySQL we were also managing Sphinx and keeping it upgraded and maintained. In the spirit of Heroku we moved to a dedicated server operated by Flying Sphinx. Built by Pat Allan, an Australian who also built the Ruby gem “Thinking Sphinx” for integrating with Sphinx. Pat has been incredibly helpful responding to issues and updating his code within days to fix bugs. Going way beyond run of the mill good support and even pushing code fixes into our own repository. We now update the index live, meaning contributions are available to all within a minute or two, instead of the nightly indexing that was done by the previous version.

SphinxRackspaceRuby Sidekiq




Moderating entries and viewing them online happens all under a single domain now: Logged in users will get access to an edit button as well as special admin pages. Version 5.2 also makes it easier to sign up as a moderator, simply include an email in the “Doghouse -> Settings” window. This will allow you to fix any small errors you might run into with one of the Doghouse items. There is no commitment for becoming a moderator. We want to encourage all users to contribute and correct entries, as many or as few as they like.

In future posts, we will share details of a more technical nature when we have the time to write them up. They include: running Puma on Heroku to maintain a fast response time; Sidekiq and Redis to handle background jobs; using Heroku’s API to scale dynamically and automatically; integrating automatic error reporting; automated testing with Rspec and New Relic monitoring.

We would like to thank all our beta testers who made sure that the API was running smoothly in the new version before the launch of 5.2. Also all the moderators who fixed countless small issues in tens of thousands of books, movies, albums and games. But most all the users who contributed entries and helped us grow Doghouse with every submission.

Regardless of whether you’re a veteran user or just bought the programs, do test the new Doghouse and let us know if there are any rough edges that need smoothing. We’ll be rolling out more new Doghouse features slowly once we are satisfied with the transition so stay tuned for updates.