MerbOverheard

Better late than never eh? Three weeks after the end of RailsConf, I thought I’d write a wrap-up post. Me and my wife turned the trip over to Vegas into a much needed holiday - we were out there four days before the conference began, and had a few more days left in Vegas after it ended too. Overall we had a great time, although I can’t say that it was a particularly relaxing break - Vegas tends to leave you more tired than when you arrived! In this post though I’ll focus on my thoughts of the conference itself.

Day 0: The Tutorials

I only attended the morning tutorial on the Monday sessions, as I was meeting with Mike in the afternoon. However, the jQuery on Rails session I attended in the morning was very good. It was a nice mix of things I knew (but good to run through again to reaffirm that knowledge), as well as some cool bits I hadn’t yet come across. It was very technical which was great, and while it was more of a jQuery focused session, rather than specific jQuery on Rails, that was fine by me as I was hoping for a real focus on the framework and it’s great features. This session didn’t disappoint, and Yehuda and Andy did very well given it was the first session, and there were a few technical challenges to overcome (namely, lack of power strips and issues with the wifi).

Day 1

Keynote: I enjoyed the keynote, but was also left a little disappointed by it in the end. It was well delivered, and had a good message, however some of the Rails 3 concepts touched on didn’t have very much detail with them (understandable, given the state of a lot of the Rails 3 features) - but in a few cases it may have been better to have left them out, rather than bringing them up and creating more questions than the keynote answered. Specifically, how some of the newer features would deal with legacy compatibility, and things like supporting HTML5 data attributes and how older browsers like IE6 might deal with that, or what could be done to support those browsers when they don’t support more modern features. I get that I could just stick with earlier versions of Rails, but it’d be a shame if the fancy new version (with plenty of other new features that would be useful regardless of which browsers you’re targeting) doesn’t work at all because of a few things that just won’t work on browsers older than three or four years old.

On the whole though, it was at least a thought provoking opening to the conference.

GitHub Panel: this session was excellent. The GitHub guys gave off a great vibe the entire time - it was fun, informative, and enjoyable. It’s obvious they are really doing something they enjoy, and they’ve built it all themselves from the ground up. Was great to see them answer a lot of questions about GitHub, and about starting up a business like GitHub; it was really very interesting.

Behind Call of Duty: World at War: this talk was a little disappointing. I expected a lot more technical detail, and was actually hoping for a focus on how they handled stuff like stats tracking, aggregating stats from millions of consoles into their community portal. Instead the talk seemed a little bit underwhelming.

Writing Modular Applications: I know this talk was a stand-in replacement at the last minute for something else, but it was an inspired choice. Fairly high level, but extremely thought provoking, it was a great 40 minutes. Jim himself said the title was a bit inaccurate, but it was interesting to see the topics he spoke about presented in the specific way he did - an almost out of context look at decoupling of code. Hard to explain, but it’s well worth trying to find video or slides for this one.

Smacking Git Around: I think on reflection, this was my favourite talk of the entire conference. Scott’s energy and enthusiasm for Git is incredible, and his talk was well delivered, had a decent dose of humour, but above all, was incredibly useful. I think it was actually about three or four hours worth of Git tips and tricks delivered in just 40 minutes, and it was brilliant. The slides are online, and for someone that’s got the Git basics down, but wants a run down of some of the more interesting tricks for Git, it’s very useful.

Scaling Rails: lastly on the first day, this talk from the guys at Phusion was a funny end to the day. A bit light on technical details, it was still entertaining, and it was a must see if only for the great demo of Rubystein 3D, a basic game they wrote in Ruby, borrowing heavily from Wolfenstein, and with a lot of “cameos” from members of the community. Funny stuff.

Day 2

Rails Metal, Rack and Sinatra: really interesting session from Adam Wiggins of Heroku, touching on a few of the more interesting aspects of Rack, and specifically what Rack allows us to do - from Rails Metal, through to the embedding of Sinatra apps within Rails apps. Very cool stuff, interesting from start to finish.

Rails 3: Step Off of the Golden Path: another interesting one, delving into some of the Rails 3 stuff in more detail than the keynote did. It’s still a little tricky to get into too much detail simply because so much stuff within Rails 3 is open to change, or not yet finished, but Matt did a good job here of bringing together some of the more interesting bits, specifically in and around the options you will have for alternatives to the norm when building a Rails app (testing, JavaScript libraries etc) and how Rails 3 makes using these alternatives easier.

What Makes Ruby Go: An Implementation Primer: this I think was my second favourite talk after the Git one - two guys who really know the topic (Charles Nutter and Evan Phoenix, who work on JRuby and Rubinius respectively) chatting over some technical Ruby stuff, including specific sneaky performance issues. Very interesting, shame it was only 50 minutes really.

Heroku Q&A: I was interested in Heroku already before this talk, so was pleased to get the opportunity to watch the guys behind it demo the service, and take questions. They started with the basics, getting started with Heroku and hosting basic apps, before answering some questions on more advanced stuff and demoing a few other bits. Was good to watch to get a better idea of how Heroku works, and how to use it.

Day 3

Building A Video Portal In Rails: this was a solid talk, if a little disappointing. I expected a talk on the more interesting technical challenges facing a video portal, specifically transcoding. Unfortunately, the particular solution being presented outsourced the video storage, transcoding and distribution to a third party platform, meaning that the talk was relegated to simply speaking about the integration. I was definitely hoping for more information on the technical side of running a site like that.

Skipped the next session and instead me and Mike went to LarkConf. To quote the always awesome Jon Larkowski who arranged it, LarkConf is “the premiere ad hoc coffee shop networking un-conference opportunity”. And it was awesome. Was cool to meet Jon, as well as a couple of the guys behind Exceptional, and the owner/founder of CafeCourses. Great stuff, and the most fun I had networking all week.

Russian Doll Pattern: Mountable Apps in Rails 3: this talk was a little underwhelming. Like the opening keynote, it left me with more questions than it answered, and while it went into a lot more detail than the keynote did, there was definitely some mixed messages coming across. It was unfortunate, but Drupal was mentioned rather a lot, and I think that confused a lot of the participants - it wasn’t entirely clear on the purpose of mountable apps, and it probably would have been better to make clear the relation this work obviously has to merb-slices. However, there were still some interesting tidbits of information in the talk, and it was obvious a lot of the confusion was down to most of these things still being up in the air, as Rails 3 is still a long way away.

Closing Keynote: This was a Q&A session with some of the Rails Core guys, and ranged from being interesting, to a little bit boring. It was a good way to end the conference, but obviously the content relied entirely on the quality of the questions. There was some great questions (things like the process for having patches dealt with by the Core team, with Koz pointing out that they are humans too, and sometimes make mistakes), to more boring questions. There was also the odd funny one (such as Obie asking “are you guys letting Yehuda turn Rails 3 into Drupal?”, which coming right after the Russian Doll talk, was forefront on a lot of peoples minds still). Overall, there were some interesting things to take away from the session, and it was a solid way to finish up.

Overall

On the whole, I immensely enjoyed RailsConf 2009. It was my first conference, and it was a good mixture of some fun networking, good sessions, and thanks to the location, made for a great break away too. My only real complaints about the conference itself is that there were a few weak sessions in there, and I definitely would have liked to have seen more technical content. While I understand that given the timeslots, there isn’t always time to do too much low level content, I do wonder whether a separate track on a slightly different schedule (one session to every two on other tracks, for example) could have held some more engaging talks on lower level ideas and concepts. Kind of a mix between the 40 minute talks, and the tutorial sessions held on the Monday. However, the Git talk and the Ruby implementation session were absolutely brilliant, and contained a great deal of useful information and content, packed into one session.

I’ll be interested to see where RailsConf takes place next year, but either way I definitely enjoyed the conference atmosphere, and meeting people, and will be thinking of attending a few other upcoming conferences, especially ones that have a more technical focus.

What’s MacRuby?

MacRuby is an Apple-sponsored, open source, full Ruby implementation on top of Objective-C runtime. In other words, whatever code runs on Ruby 1.9, should/will run on MacRuby. Yes, you read correctly, MacRuby can/will be able to run all your Ruby code. That means that eventually you will even be able to run your Rails/Sinatra/new-sexy-ruby-framework app on MacRuby.

Unlike RubyCocoa, MacRuby is not a bridge, it is a full implementation of the Ruby language on top of Apple’s Objective-C runtime. Taking a huge shortcut, MacRuby implements the Ruby syntax by aliasing it to the Obj-C language counterpart. A Ruby string instance is really in fact, an instance of NSMutableString. This is obviously transparent for you as a developer since you have the same Ruby API, but it also means that MacRuby can make use of the various Objective-C’s goodies such as native threads,  garbage collector in the background as well as the runtime performance.

On top of that, you have full access to the Obj-C API from your Ruby code. (tip: in macirb, try “my string”.methods(true, true).sort to see the available Ruby + Objective-C methods on your String instance)  The reason why having access to Objective-C is important is because it gives you the possibility to write native Cocoaapps using Ruby. For those who don’t know Cocoa, is a set of APIs for MacOSX development.

However, note that even though, the Cocoa support is almost complete and stable, MacRuby is still in development, especially on the Ruby side of things.

What is it not?

• MacRuby is not a fork of Ruby. Full rubyspec compliance is expected! It’s true that MacRuby supports smalltalk/Obj-C method selectors so it might be considered a language superset.
• MacRuby is not limited to the OSX platform. All its dependencies are open source and could possibly be compiled for other POSIX-based systems such as Linux.. (not done yet)
• Even if MacRuby’s primary goal is to allow you to write efficient Cocoa apps, it does not mean that MacRuby is limited to that.
• MacRuby doesn’t require you to learn Objective-C in order to develop Cocoa apps. (you just need to understand Obj-C selectors)

What’s coming up?

The current version of MacRuby (today being the 27th of May 2009) is version 0.4. You might have heard of things about MacRuby crazy performance, LLVM, AOT compilationAOT compilation etc… This is all happening in the ‘experimental’ branch.

What’s going on is that up to MacRuby 0.5, MR was  using YARV (Ruby 1.9 interpreter) on top of Obj-C and which obviously limited MacRuby’s  to YARV’s performance. After RubyConf 2008, Laurent Sansonetti, MacRuby’s lead developer, decided to try removing YARV to only keep its AST and experiment with using LLVM instead of the 1.9 interpreter.

This switch turned out to be very promising and was noticed right away by influential people such as IBM’s Antonio Cangiano. Since then, performance and compatibility have increased. Laurent even started working on an Ahead Of Time (AOT) compiler: http://pastie.org/485095 What’s really impressive is that in this specific example (Fibonacci sequence), MacRuby’s compiled code is faster than Objective-C! But let’s not jump the gun. First this is a very very early prototype and most of your apps won’t be using Fib. sequences In this case, MacRuby’s recursive method dispatch is faster but again, this is just a proof of concept and even though MacRuby is getting close to Obj-C speed, it’s still far from matching Obj-C’s impressive performance.

What this basically means is that you will be able to compile your Ruby code down to binary code. Imagine, taking your Rails app and compiling it down to a binary file that you can just push to your server But really, what’s almost more important is that Ruby will get closer to Objective-C’s speed.

Why will MacRuby change the Ruby ecosystem?

As a web developer, getting better performance is great. But Ruby is already fast enough. Rails is faster than any PHP framework out there and when doing Merb’s benchmarks we proved that Ruby for the web can definitely be fast.

Now if MacRuby ends up running 3-7X faster than Ruby 1.9 (no one can tell for sure), existing Ruby developers will certainly be really pleased but it will probably affect more people outside of our community than within. Let’s face it, our community isn’t that big but it’s growing fast and people are mainly coming to Ruby for its web frameworks. But Ruby has much more than that to offer. Desktop applications, video games, scientific computation and even embedded apps. Apple is betting on Ruby probably because of the fact that they see the potential in the language itself.

Would people still use Java if Ruby is as fast/faster than Java? Probably! Would they think about using Ruby for their next project? I would certainly hope so!

Ruby is viral, it’s such a great language, people who have started using it are having a hard time going back to what they used before. But to spread the ‘love’, we need to give people the opportunity to discover why Ruby is so great and to do that, we need to make sure Ruby is relevant to them. By making Ruby a realistic option to write desktop/mobile applications, we are targeting a new audience. An experienced audience which will be able to bring a new perspective to our ecosystem and help it grow.

Of course, MacRuby isn’t the only implementation out there trying to do that. JRuby and IronRuby are also very interesting projects. My take on it, is that MacRuby will be able to change things because of its new approach and potential community. It will more than likely be the first Ruby implementation compiling down to binary code, it will more than likely be the fastest implementation and it will more than likely draw a different type of developer.

Does it mean that JRuby, IronRuby, BlueRuby will be useless? Absolutely not! Matz is the first one to encourage diversity and I agree that this is a great way to move forward. These projects solve different problems and all have pros and cons, but they also all share a similar goal: making Ruby a better and more popular language outside of its original community. JRuby, IronRuby and MacRuby bring Ruby to the respective Java, .net and Cocoa communities, and indirectly bring fresh developers to Ruby. These implementations are critical in the way they actually bridge existing communities and in the end, all Rubyists will benefit from it. Also, even though MacRuby and IronRuby are in active development, JRuby is the only mature alternative to MRI/YARV at this point and it proved that Ruby can coexist in a different community as well as contribute a lot of interesting things back to the Ruby community.

To summarize, I see tremendous potential in MacRuby. There is the obvious technical aspect of the implementation, but also the indirect affect MacRuby could have on our community. I believe that MacRuby is an agent of change and will help bringing more diversity to our community. It will change mentalities and push Ruby to places where it’s being struggling to make a mark. I can see the so-called “Enterprise” people looking at Ruby differently. I also think that MacRuby has the potential to be at the origin of a new type of hybrid application, mixing desktop/mobile/distributed applications with centralized web applications. Why not dream of p2p applications/games using a subset of Rails to communicate between each other and with a central server?

If you are interested in learning more about MacRuby, check the list of resources available on the MacRuby site.

Rich Kilmer and I are also working on a documentation application for Cocoa and HotCocoa as well as a MVC framework for writing Cocoa apps using HotCocoa (HotCocoa is a thin, idiomatic Ruby layer that sits above Cocoa and other frameworks).

Make sure to keep an eye on @macruby and the MacRuby website if you want to keep track of the latest news.

The other night, during our monthly SDRuby meetup, lots of people were very interested in learning more about CouchDB and Ruby. I tried to show what Couch was all about but I didn’t have time to show how to use CouchDB with Ruby.
Here is me trying to do that in 10 minutes or less. I’ll assume you don’t have CouchDB installed.

Install CouchDB, if you are on MacOSX, you are in luck, download and unzip the standalone package called CouchDBX.
That’s it you have couch ready to go, press play and play with the web interface.

Next, let’s write a quick script. Let’s say we want to write a script that manages your contacts.

First, let’s install CouchRest:


$ sudo gem install couchrest


Now, let’s open a new file and write our script.

In line 4 and 5 we are just setting up the server(by default, localhost is being used). If the database doesn’t exist, it will get created.

SERVER = CouchRest.new

DB = SERVER.database!('contact-manager')

Then, we define your ‘model’, we set the default database to use and define a list of properties. Properties are not required, but they generate getters and setters for you. They are also used to set default values and validate your model. Line 11 shows how to use an alias that will provider a getter and a setter for the property name and the alias name:

property :last_name, :alias => :family_name

Line 14 does something that might seem strange at first. We are casting the address property as an instance of the Address class. Here is what the implementation of the Address class could look like:

Address is just an instance of Hash with some extra methods provided by the CouchRest::CastedModel module. (If you wonder why it’s called CastedModel instead of the more grammatically correct CastModel, the answer is simple: I suck at English grammar :p )

So here is a quick example of how to use a ‘CastedModel’:

That’s part of what’s great with CouchDB, you don’t need to worry too much about storage. Just define your properties, cast to models if needed and save everything as a document.

For more examples checkout the CouchRest spec fixtures and the examples.

To learn more about couchdb, read the (free) online draft of the CouchDB book and of course you probably should read the CouchRest source on GitHub.

Justin Smestad invited me up to factory labs to talk about sinatra recently for one of their brown bag lunches. He recorded the talk from his laptop and I’m amazed you can see or hear me.

My slides aren’t sync’d up but they’re here.

Sinatra 0.9 – Corey Donohoe

So Tim and I were talking about how we needed hoptoad for our rack apps the other day. This rack hoptoad notifier should get you going for rack and sinatra. Raise issues on github if you run into problems.

If all you want to do is install the gem you can install it like this:
% sudo gem install rack_hoptoad

RailsConf 2009 has now finished.  This time last year, no one would have ever guessed that the Merb and Rails teams would join forces and focus on what will hopefully be known as one of the best Web Frameworks.

It was encouraging to see so many people excited about what’s being ported over from Merb and the new options available to people who are currently limited by the existing stack. For those interested in pushing Rails further and doing stuff out of the norm, here are my slides. Arthur Zapparoli from Brazilian Rails squad recorded most of the talk and told me he will upload the video ASAP. You can also read Yehuda Katz’ blog which covers what he talked about.

It was really great to meet a lot of new people as well as people I only knew via IRC/IM/twitter.

It was a great honor to finally meet Dan Kubb (DataMapper), Ninh Hernandez-Búi & Hongli McLovin Lai (Phusion), Peter Cooper (RubyInside), Raimonds Simanovskis (Oracle adapter for AR), Arun Gupta (Sun/Glassfish),  Jeremy Hinegardner (crate), Michael Maxilien (IBM), Dana Jones (railsbridge), Zach Zolton & Geoff Buesing (CouchRest) and of course the Brazilian crew (lots of awesome .br guys came this year, I’m looking forward to RailsSummit) and last but not least, the French speaking crew (I’m glad to see Ruby is picking up back home). (I know I’m forgetting people… sorry about that)

It was also really nice to talk with some experts like Dave Astels, Aslak Hellesøy, Rich Kilmer, David Chelimsky, Ryan Brown, Derek Neighbors etc.. to get their feedback on various projects I’m working on.

Leaving Vegas, I feel like the Rails community is expanding quickly (it was the first RailsConf for 1/4 to 1/5 of the attendees) and that the community is organizing itself to welcome a new audience (better documentation, great initiatives like railsbridge.org, willingness to help), as well as trying to be more available to the ‘Enterprise’ world.

These feelings were enforced during our Rails Activism BOF and after talking with 3rd party developers and sponsors really trying to solve problems that newcomers to Rails are now facing. This is an exciting time.

Here are the slides from my keynote this morning at the SF Erlang Factory conference:

To start with, I would like to make it clear to everyone that I do sincerely care about the larger gender issues that my presentation touched off.  I have also replied and otherwise corresponded with everyone who has contacted me about my presentation, just as I have tried to reply to all of the blog posts that have been brought to my attention.  At this point, however, it is clear that this issue has grown too large to be resolved through one-on-one contact, hence this public statement.

I have made a specific point of exchanging emails with most of the bloggers who addressed the gender issue, and I did so because I care about minority involvement in our community.  I cared about this issue well before GoGaRuCo or this particular presentation, despite what anyone might think to the contrary, running the presentation past my wife/business partner and other colleagues well before I gave my talk.  They, at least, understood the goals of my presentation, which were to leaven an otherwise dry topic with humor.  It genuinely was not my intention to cause offense.  People may be driven by personal choice or cultural background to take offense at any number of things, of course, but I think there is always a clear difference between trying to offend people vs people choosing to take offense.

My view is that offending someone is walking up to them and saying: “You suck, your code sucks and your partner’s code sucks!”.
That is not what I did in my talk.  In the case of my talk, people knew what to expect, they *picked* the talk, and were warned by the organizers before I started that I would be using imagery potentially offensive to some.  The topic of my talk was obvious, and I would have hoped that people who were likely to be offended would have simply chosen not to attend my talk or read my slides on the internet.  It’s like complaining that television has too much material unsuitable for children, yet not taking steps to limit their viewing of it.  You can’t have it both ways.

In the same way, people have been rather quick to judge the entire Ruby/Rails community based on my presentation and the comments of a few people. I have noticed, for example, a lot of comments making sweeping generalizations about Americans, Republicans, women, Ruby, Rails, men etc.. leading to an exchange of insults, things getting personal, and everybody feeling the need to fuel the flames by sharing their own opinion in public.  While there is nothing wrong with sharing your approval or disapproval, I think it has also contributed to blowing this entire thing completely out of proportion.  I would like to ask people to step back and reflect upon whether we are contributing anything useful to the discussion.

We can argue forever about morals, professionalism, ethics, respect, etc., though this is all a distraction from the real problem that was raised by Sarah, namely that we have very few minorities in the Ruby community, especially women.  Minorities do need to be more represented!

So, instead of continuing an increasingly pointless debate about my slides, I would like to encourage the community to look further and see how we can change things.
Education, outreach, and self-censorship are certainly some of the options available.   Please read Aaron’s blog post and see how you might make a difference.

For those who still want to talk with me, I will be at the RailsConf devchix’s discussion panel and more importantly, if you care about Rails Activism and you want to see things evolve, be sure to come to our Birds of Feather discussion of Rails Activism.

Update: Some people thought I wasn’t clear enough about my position and I therefore tweeted the following: “I obviously made a mistake. I didn’t mean to offend anyone but since I did, I failed.”

We made the decision to slightly delay the release of Merb 1.1 as we ended up changing the scope of what we wanted to make available in the 1.1 release. If you have been following our releases, you know that this is not something we usually do, but we strongly believe that this is actually something that will save us time for the next release.

The big themes for 1.1 are full Ruby 1.9 support and Rails3 compatibilty: action-orm(previously called active-orm) and the new router.

While on one hand, Ruby 1.9 work is 99% done (we still have a couple of failing specs with action-args) and action-orm just needs to be merged in, on the other hand, the new router currently does more than what we initially planned for. It actually covers stuff we scheduled for 1.2.

Here is a quick preview of what Carl has been working on:

Merb’s router is now extracted into a rack middleware library and a bunch of features to try to get “mountable apps” working in Merb 1.1 have been added.

The proof of concept has been submitted to the Rack development mailing list and the draft is available at:
http://github.com/carllerche/rack-router

Merb, CloudKit, Sinatra and more than likely Rails3 should be using this new rack based router. This is a huge step for the Ruby community!

Here is the abstract explained by carl:

Conceptually, rack-router allows you to create a two way map between HTTP requests and Rack applications. It is built as a piece of middleware that takes in a set of routes.
When a request comes in, the router will compare that request against the set of routes until it finds one that matches. It then calls the associated rack app.
It can also generate URL’s that you can use to link to other mountable apps.
It also goes quite a bit further and attempts to make reusing rack applications completely painless (what we are tentatively calling “mountable apps”).

For more information, check the mailing list thread.

Now that the proof of concept has been accepted, the new implementation needs to be optimized to match the speed of the previous router. Currently the new router is pretty slow compared to 1.x router.

I meant to write this article four weeks ago, as I had just finished in my full time position with Touch Local, and was about to start working freelance full-time. Since then, my freelance work has kept me plenty busy, and I never got chance to write this post. So today I figured I’d take some time out to write it, with the added bonus that I now have a month worth of working for myself to reflect upon.

The last month has been incredibly exciting. I’ve updated this site to make it more professional, and to give potential clients more information about me and what it is I do. I’ve had a lot of work on which has been great, but I’ve also had a lot of business admin to do, which has taken some getting used to. Working on business accounts, raising invoices, writing proposals and contracts has all taken some time, but I’ve now got everything setup in a way which makes admin tasks a lot quicker. I’m going to write in more detail soon about the tools that can help freelancers with a lot of day to day tasks.

Not having to commute has definitely had a positive impact, and I get that time back now to put back into working, or into relaxing. Being more flexible is also great - I work in the evenings and the weekends, but can take time out in the week, or have the occasional late rise. Sometimes it can be hard to switch off from the work that needs doing, but at least in that regard I really enjoy what I do, so it isn’t so bad.

And that’s really what makes working for myself as a freelancer and running my own business so enjoyable. I’m working on things that I enjoy doing, from home, working hours that suit me. I also have the time to work on some of my own ideas now too, which is very exciting indeed.

So if anyone reading this needs Ruby on Rails development doing, please get in touch. You can also browse the rest of this site to find out more about me, and what it is I offer. And if anyone else reading this is considering becoming a freelancer, then don’t hesitate to drop me a line if you’d like to hear more about my experiences so far!

I’m looking for a few good folks to join my team. You need to be able to work on site in San Francisco, we can assist with moving expenses for the right candidate.

You would work with my team(currently 5 devs). We do no bs agile style development with pairing where appropriate and test infected mentality . generally we have a fun, tight team and get to work on some of the hardest and most rewarding areas of software development relevant today.

You would be on my team working on http://engineyard.com/solo and it’s successors as well as Engine Yard’s on premises infrastructure.

You would be working on/with the following ruby technologies:

Ruby
Merb
Rails
Sinatra
DataMapper
Nanite
Chef
Event Machine
Many others..

You would be working on the following problem domains:

Scalable Ruby Deployment Architectures
Scalable Cloud Computing Architectures
Scalable Database Architecture
Distributed Computing
Key Value Data Stores
Infrastructure automation
Cloud provider API's
Multi-Cloud portability
Monitoring and alerting systems
Security in the Cloud
Horizontally Scalable Architectures
Many other interesting areas of computing

Familiarity with the following tech is nice but not necessarily required if you can learn fast:

Gentoo Linux
Nginx
Apache
Monit
Daemontools
Mysql
Postgresql
Memcached
Passenger
Mongrel
Thin
SSL Certs
Iptables/Networking
Erlang

If you think you have the “right stuff” and you have the kick ass take names attitude then send a resume with a short intro about who you are and why you are the person we should hire.

ez@engineyard.com

Following the DataMapper 0.9.11 release, we just pushed a new minor Merb release.

This release is mainly targeting new developers and Windows users wanting to install the full Merb stack. Others can simply update their dependencies if they use the dependencies.rb file or install the new gems if nothing is bundled and no hard dependencies are set.

Merb is a metagem which installs a bunch of other gems (merb-core, DataMapper and a lot of small gems). The problem was that Merb was trying to install DM and dm-types, unfortunately, dm-types had a dependency on a gem which couldn’t be installed on Windows. All of that is now fixed and Windows users can install Merb 1.0.11 without having to manually pick the gems they need.

This release also includes a fix for people using CouchRest, a CouchDB Document Mapping DSL.

Merb 1.1 is still planned to be released in April. A majority of the work has been done, but since Yehuda and myself are going to be traveling, the release will be slightly delayed.

The great news regarding Merb 1.1 is that, on top of being fully Ruby 1.9 compatible, and using action-orm, and being closer to Rack, Yehuda and Carl have been working on the router to make it awesomer and ready for mountable apps

Stay tuned for more news.

update: People using CouchRest or another CouchDB ORM/DSL make sure you define your resources route with an identifier:

resources :articles,        :identify => :id

Otherwise, resource(@article) won’t work. (This is usually done by the merb orm plugin and I might add it to CouchRest in the future)

Introduction

The adapter API for DataMapper has been in a bit of flux recently. When I submitted my proposal for a talk at MountainWest, adapters were irritatingly complex to write. You just needed to know too much about DataMapper’s internals to be able to write one. A week before the conference began, I started a significant effort to re-write the API to make it easier. I succeeded, a little too well; my 30 minute talk only took 15. Since then, I’ve written a couple more adapters from scratch, and refined the API further. This post will serve as notes on the changes that I’ve made, and a tutorial on writing adapters.

The API changes are currently only in my branch, but they will be merged into the DataMapper/next branch. For now, you’ll need to use my adapters_1.0 branch.

This tutorial will follow my process as I make a DataMapper adapter for TokyoTyrant. You can grab the code from my github repo, paul/dm-tokyotyrant-adapter.

Setup

I’ll assume you know how to build a gem, and get it all set up using your favorite gem builder, so I’m going to skip all that. To begin, we only need a couple files. First (of course!), the spec:

spec/dm-tokyotyrant-adapter_spec.rb

require File.dirname(__FILE__) + '/spec_helper'

require 'dm-core/spec/adapter_shared_spec'

describe DataMapper::Adapters::TokyoTyrantAdapter do
  before :all do
    @adapter = DataMapper.setup(:default, :adapter   => 'tokyo_tyrant',
                                          :hostname  => 'localhost',
                                          :port      => 1978)
  end

  it_should_behave_like 'An Adapter'

end

And thats all there is to it. We make an @adapter instance var, which gets returned from DataMapper.setup, and then run the adapter shared spec. As of now, the shared spec is fairly thorough, but its far from comprehensive. If we run this now, we’ll get some errors about not finding the TokyoTyrantAdapter. So, lets go make it.

Initialization

lib/dm-tokyotyrant-adapter.rb

require 'dm-core'
require 'dm-core/adapters/abstract_adapter'       # 1

require 'tokyotyrant'

module DataMapper::Adapters

  class TokyoTyrantAdapter < AbstractAdapter      # 2
    include TokyoTyrant

    def initialize(name, options)
      super                                       # 3

      @options[:hostname] ||= 'localhost'         # 4
      @options[:port]     ||= 1978

      @db = RDB::new                              
    end
  end

end

Some of this is pretty TokyoTyrant-specific. Since the Ruby API isn’t very Rubyish, I’m going to skip over a lot of it, and just talk about the DataMapper/adapter specific stuff. Referencing the comments in the code above:

1. require the abstract adapter explicitly, since its not require‘d as part of requiring dm-core.
2. Make a class that follows the naming convention #{AdapterName}Adapter so that DataMapper can find it when we use the :adapter => 'adapter_name' option. Inherit from AbstractAdapter as well, as it will provide us with many helpers we’ll be using.
3. Make an initialize method, and call super. This will turn any provided options into a Mash (a Hash that can use a string and a symbol as the same key. It handles a little other setup for you, as well.
4. The rest is Tyrant-specific, but useful to know. We set some default connection options, and initialze a @db object.

If we run the spec now, it connects, and we get a bunch of pending specs, saying we need to implment #read, #create, etc…

dm-tokyotyrant-adapter/master % rake spec
(in /home/rando/dev/dm-tokyotyrant-adapter)
*****

Pending:

DataMapper::Adapters::TokyoTyrantAdapter needs to support #create (Not Yet Implemented)
/usr/lib/ruby/gems/1.8/gems/dm-core-0.10.0/lib/dm-core/spec/adapter_shared_spec.rb:52

DataMapper::Adapters::TokyoTyrantAdapter needs to support #read (Not Yet Implemented)
/usr/lib/ruby/gems/1.8/gems/dm-core-0.10.0/lib/dm-core/spec/adapter_shared_spec.rb:75

DataMapper::Adapters::TokyoTyrantAdapter needs to support #update (Not Yet Implemented)
/usr/lib/ruby/gems/1.8/gems/dm-core-0.10.0/lib/dm-core/spec/adapter_shared_spec.rb:107

DataMapper::Adapters::TokyoTyrantAdapter needs to support #delete (Not Yet Implemented)
/usr/lib/ruby/gems/1.8/gems/dm-core-0.10.0/lib/dm-core/spec/adapter_shared_spec.rb:129

DataMapper::Adapters::TokyoTyrantAdapter needs to support #read and #create to test query matching (Not Yet Implemented)
/usr/lib/ruby/gems/1.8/gems/dm-core-0.10.0/lib/dm-core/spec/adapter_shared_spec.rb:289

Finished in 0.005982 seconds

5 examples, 0 failures, 5 pending

Create

def create(resources)                                     # 1
  db do |db|                                              # 2
    resources.each do |resource|                          # 3
      initialize_identity_field(resource, rand(2**32))    # 4
      save(db, key(resource), serialize(resource))        # 5
    end
  end
end

1. resources is an Array of DataMapper Resource objects.
2. #db is a helper to make TokyoTyrant’s api a little more friendly. It handles connecting to the ttserver, and yields the connection to the block. When finished, it closes the connetion.
3. Some adapters might be able to support bulk creates, like SQL INSERT. This one doesn’t, so we’ll loop over every resource.
4. We’ll need to set the identity field. More on this later.
5. Put the resource into the database. #key and #serialize are helpers, I’ll explain them in a bit.

Something useful to note here: The resources being passed in to this method are the actual resources in use by DataMapper. That means that any modifications you make to them will also be automatically availble to anything using DataMapper. This is extremely useful for any data store that can provide a representation of the created object. If the data store set some fields as a result of creation, eg, a created_at timestamp, or an href linking to the location of the resource, you can update the resource right here, and not have to have DataMapper perform a #read to update the resource object.

If you’re coming from an RDBMS world, you’ll be familiar with sequences. Since you’re here, learning how to write adapters, I’m going to assume you’re not going to be talking to a relational database. If thats the case, and you don’t need to support these kinds of sequences, you should probably use UUIDs or something similar for your identity fields. Sequences are not scalable or distributable, they’re a relic of the big RDBMSs. I only have this #initialize_identity_field line in there to show how its done. As you can see, I’m not even picking it sequentially, but choosing a random number, instead, because I don’t have a resonable way to keep track of sequences. The method won’t try to overwrite a value if one is already set, so take the opportunity to use a UUID instead, and save everyone involved a bunch of trouble.%lt;/soapbox>

Because TokyoCabinet & Tyrant are key-value stores, I’ve written a couple helpers to try and coerce resources into a single key and value. First, I choose a key from the model name, and keys in the model, like so:

def key(resource)
  model = resource.model
  key = resource.key.join('/')
  "#{model}/#{key}"
end

We get the model, and the keys from the resource. One thing to keep in mind, is that DataMapper assumes composite keys for every model, so even if a model has only a single key, Resource#keys will always return an array. We use that to build a string, like Article/1234. I chose a slash as the delimiter, because TokyoTyrant has a ReSTful interface, and it will make for pretty urls.

We also need to serialze the resource. I chose to serialize it as JSON, because its cross-platform, and lightweight. YAML or even XML would also be ok choices, depending on what you may be interoperating with.

def serialize(resource)
  resource.attributes(:field).to_json
end

resource#attributes normally returns a Hash of {:property_name => value} pairs. DataMapper properties also can take an option, :field, which is used to indicate the name of the field used by the data store. Because we’re writing an adapter to a data-store, thats what we want. #attributes can take an optional argument to indicate what we want to use as keys. Here, I used :field, meaning I want the field attribute of the property. It will then return a Hash of the form {"field_name" => value} There usually won’t be a difference, but its important that adapters use the field instead of the name, so that someone writing a model can use the :field option to property correctly.

Let’s run the spec again, and see how we did:

dm-tokyotyrant-adapter/master % rake spec
(in /home/rando/dev/dm-tokyotyrant-adapter)
/usr/lib/ruby/gems/1.8/gems/rake-0.8.3/lib/rake/gempackagetask.rb:13:Warning: Gem::manage_gems is deprecated and will be removed on or after March 2009.
****..

Finished in 0.009957 seconds

6 examples, 0 failures, 4 pending

Read

def read(query)
  model = query.model

  db do |db|
    keys = db.fwmkeys(model.to_s)
    records = []
    keys.each do |key|
      value = db.get(key)
      records << deserialize(value) if value
    end
    filter_records(records, query)
  end
end

#read takes a DataMapper::Query object, which has everything needed to filter, sort, and limit records. For simple adapters, that don’t have a native query language, you don’t need to care. The #filter_records helper in AbstractAdapter will take care of everything for you. All you need to do it provide it an Array of Hashes, using the field name of the property as the key. Since we use json to serialize the value, here we deserialize it back into a hash. We used field names as the keys, so no further translation is needed. TokyoTyrant provides the #fwmkeys method as a way to search for a key prefix, so we pass the model name in, because the model name is the first part of the key we used. We pass all the records we found in to #filter_records, which performs the filtering, and we then return the result.

Update

def update(attributes, collection)                                 # 1
  attributes = attributes_as_fields(attributes)                    # 2
  db do |db|
    collection.each do |resource|                                  # 3
      attributes = resource.attributes(:field).merge(attributes)   # 4
      save(db, key(resource), serialize(resource))                 # 5
    end
  end
end

1. We take an attributes hash and a DataMapper::Collection. The attributes are in the form of {Property => value}, using the actual property object. A Collection is a set of resources.
2. We need to convert the keys in the attributes has from Property objects into :field name. Luckily, AbstractAdapter provides #attributes_as_fields, which does exactly that.
3. Iterate over every resource in the collection
4. Update the attributes hash with the combination of the existing attributes, merged with the attributes we wish to update.
5. Write the whole thing back to the database.

You may also want to take a look at how the InMemoryAdapter in dm-core accomplishes the same task. It extracts the query used to build the collection, and looks for those records in its data store, using #filter_records. It then updates each record in-place. Either way works fine, and the ease of which may depend upon the adapter. In TokyoTyrant, finding the records is harder than retrieving them, so I opted to just re-save the ones I already had in the collection. An SQL adapter is able to update the records without loading them, so using the query is faster. ( “UPDATE {attributes} WHERE {query}” ).

Delete

def delete(collection)
  db do |db|
    collection.each do |resource|
      db.delete(key(resource))
    end
  end
end

At this point, it should all be self-explainatory. Just iterate over every resource in the colleciton, and delete its key from the db. Yay.

Conclusion

And thats all there is to it. 3 hours, 2 beers, and ~100 LOC later, and we have a fully-capable adapter that can be used with DataMapper. I was running the specs at every stage, but left them out for brevity. Here’s the final run:

dm-tokyotyrant-adapter/master % rake spec
(in /home/rando/dev/dm-tokyotyrant-adapter)
......................................

Finished in 0.175668 seconds

38 examples, 0 failures

As I said before, the specs aren’t exactly comprehensive, but they will be added to over the next few weeks. For now, they’re good enough that you can be pretty confident your adapter will work for most things.

Thanks for tuning in, leave a comment, or come visit me in #datamapper on freenode if you have any adapter questions.

Confreaks posted my talk. Everyone go make fun of that huge nerd up there!

I pushed some of the changes I talked about to my github branch. This covers the Conditions objects.

Next on my personal roadmap for adapters one-point-oh edition are for Repository to handle turning the responses from adapters into Resource objects, if they aren’t already.