I started writing this last night, but just got time to publish it.

The Ripple Hackathon has finished! Today felt even more on-focus than yesterday, but also less frantic/stressed. We were in “the zone” for sure. Here’s what we accomplished today:

• Duff finished off the “stored key” associations, including incorporating conflict resolution on them! (I’ll be writing up a discussion of the new conflict resolution API later.) He also fixed a small inconsistency in an internal API around associations.
• Nathaniel teased apart some specs that were invoking save or create on Document models but weren’t in the integration suite, and he also lent me a lot of help on improving the build on Travis.
• Myron implemented an awesome decoupled serialization API for Riak::RObject. You can now register your own serializers for Ruby objects that you assign to the data on a RObject. So if you want to implement a custom serialization routine for application/x-my-content-type, you can do it now.
• Kyle continued work on refactoring the Riak::Client code so that you can specify multiple hosts among which to multiplex requests at runtime. This will include configurable load-balancing strategies (starting with a default round-robin strategy).
• I reviewed a number of works-in-progress from the other developers but spent most of my time bringing our Travis build toward green. There are still some outstanding runtime/build environment/worker-related issues on Travis, but in the last few builds we’ve had at least one Ruby version passing in the build.

Retrospective

This was a phenomenal week; I can tell because we kept a kanban on one of the whiteboards and the list in the “Done” column grew quite long in the three days (or you can just look at the commit list and changes). More than just the things completed, I believe we developed a rapport and understanding that can only be done in-person. We also have a clearer way forward, and some momentum behind key features that have been waiting to be started.

Thanks to all the committers who attended, and a very special thanks to the people at Basho who helped make this happen:

• Mark wrote the proposal and finances for the event, handled ordering lunch each day, and took notes and pictures of the event.
• Maureen, our office manager, handled booking hotels and other necessary logistics on very short notice.
• Marisa, our VP Finance, ran the numbers faster than should be humanly possible to meet accelerated timeframe.
• John and Justin let me take 3 days off the normal schedule to come to San Francisco and run this.
• Tony made everyone feel welcome and arranged an awesome dinner on Tuesday.

It’s the end of day 2 of the Ripple Hackathon. Today was shaped a lot differently than day 1 — we did a lot more heads-down hacking and a lot less discussion and planning. Luckily, this means there were a lot of bugs fixed and features added.

Today’s progress

Myron hacked through a lot of smaller issues:

• Undefined (“phantom” or “ghost”) attributes are now better accessible and saved along with the Ripple::Document instance. This might occur if you’ve removed attributes from the document class that still exist inside the Riak object and you want to still get to them.
• You can now use the guard tool to automatically run the specs in riak-client or ripple. It works much better than autotest so we’re pretty happy with it.
• There was an error stemming from beefcake that exists on the latest release version but not an earlier version. It was locked to the proper version until we can resolve the issue in the beefcake library itself.
• There were a few incompatibilities with Ruby 1.8.7 that were resolved, specifically in the way super was called, and neither singleton_method nor Symbol#intern exist in 1.8.
• Spec files now require 'spec_helper' instead of using a relative path to the spec_helper or a call to File.expand_path to get an absolute path. We didn’t need to have the relative requires because RSpec puts the spec directory on the load path automatically.
• There’s now a rake task that will clean trailing whitespace from all Ruby source files.
• There was an incompatibility with the way Set instances were encoded to JSON when called from inside ActiveSupport.

Nathaniel continued working on associations where the target is found using Riak Search. In the course of his investigation, we discovered bug #867, namely that sorting is applied at the wrong time. This put a halt on moving forward with the pattern.

Duff made headway on the “many” association of a similar type to the “one” using a stored key that was completed yesterday.

Kyle added some awesome conflict resolution logic to Risky and started on a refactoring of the Riak::Client class to support multiple hosts.

I completed some logic for encapsulating the retrying of failed requests automatically and helped Myron with some of the JSON issues. I also discussed with Kyle the plan for splitting the host configuration from the Client object, and how we might implement various strategies for deciding how to multiplex and failover connections.

The larger picture

The items we worked on today contribute toward two of the major goals for Ripple 1.0, stability and usability. Naturally, fixing existing bugs is important work, but we also want to improve the internal semantics of the client so it is more resilient to temporary failures and doesn’t hammer a single node with all of its requests. We want to make the Document modeling have stronger and more diverse patterns so that you can express relationships that have higher cardinalities and less surprising results in production. I believe we made headway on those goals today, and tomorrow has lots of promise as well.

So you may have heard about this event going on around Ripple this week. Basically, we have committers and interested parties gathering in the offices of Basho’s SF office to hack on Ripple and move it toward 1.0. Today was the first day of the hackathon/sprint/thing. Here’s a quick rundown on what happened:

Planning & Priorities

Because our committers are volunteers and are taking time out of their regular work to be here and to contribute, I wanted to make it worth their while – to let them work on what would make the most impact for them. So we started with a discussion that consumed most of the morning, dropping our personal priorities into either the “Gripes” (actual problems encountered) or “Wants” (desired features that would make things easier) columns. Mark took a picture of this discussion in-progress:

Some of these were things that can be directly addressed by the library, some will need to be done in Riak itself. The list, roughly as it went up on the board, is below:

Wants

• Count things (e.g. keys in bucket)
• Indexes (Risky does this)
• Separate Repos for subprojects
• Migrations framework
• Rake tasks - db:*
• Riak as a gem (node generation, install)
• Instrumentation

Gripes

• “Many” should be “few” (too many links)

  • Principle of Least Surprise
  • double-storage (backlinks)
  • Assigning inverses

• list-keys (list a small bucket should not be expensive) * key-filters suck too

• Connection resiliency

  • retries
  • PBC problems
  • multiple hosts

• Associations

  • embedded docs by key
  • module scope for class names
  • SBI is problematic

• JSON problems

• Test Server

We all then took some things that we wanted to work on and sat down to work on them. I also fielded a lot of questions about certain aspects of the things others were working on.

Wrap-up & Demos

One thing I wanted to do with each day was to prevent it from fizzling out, so we all discussed one thing we had accomplished (or decided on) and talked about it or showed off code.

• Nathaniel and Duff worked on a new kind of association for documents where the key of the other document(s) is stored in the body rather than with links (in a header). This starts to address the first bullet point under the “Gripes” above (aka issue #136) has a neat related feature - the “many” side can look up related objects using Riak Search. Their work-in-progress is on the “stored_key” branch.

• Myron worked tirelessly with the guys from travis-ci to get a continuous integration server running. This involved getting Riak 0.14.2 installed on their worker boxes and then figuring out the idiosyncracies of why our tests fail! All in all, an awesome effort that is already bearing fruit.

• Kyle discussed lots of strategies for both teasing out HTTP response validation logic from the individual client library adapters, as well has how to do request retries efficiently. The syntax we’re going to go with will be a simple block that wraps the lower-level request invocation so we implement the retry logic separately. This should land early tomorrown now that we have a solid plan.

  Kyle also created an interesting conflict resolution DSL for Risky which encapsulates various resolution strategies at a fine granularity (at the property level, actually). We discussed borrowing that syntax to build on Myron’s existing conflict-resolution code.

It was a very exhilirating day (if a bit distracted at times), and I trust tomorrow will be really productive as well.

Come on by

As the original announcement mentions, if you’re in the San Francisco Bay area and want to talk about Ripple, Riak, or contribute to the library, feel free to drop by the Basho office (929 Market, Suite 500) anytime during business hours tomorrow or Wednesday. We also have formal “Riak Office Hours” on Wednesday too, if your questions are more general.