Ruby Findings

I was really impressed by ActiveRecord’s ability to abstract an underlying database schema into a full-featured set of class based access methods. Defining relationships, decorating them with speficic methods, etc and so on. There is one aspect of ActiveRecord, however, that I’m really learning to dislike: database concurrancy.

So, first of all, if you already know that ActiveRecord is not thread safe by default, then you can probably move on to reading something else. You’ve been through the frustration I’ve been going through and you’ve either coded your way around ActiveRecord’s blasie attitude towards threads or you gave up trying to use it in a threaded application. If, however, you haven’t see the pain and carnage that ActiveRecord can introduce into your nice, thread safe projects then read on. I may be able to save you some pain.

ActiveRecord is not, by default, thread safe. I haven’t hacked into the code to figure out why it isn’t but through trial and error and reading pages on the Intertubes, it’s clear that ActiveRecord is thread challenged. There is a very cheesy way to attempt to ensure that ActiveRecord won’t immediately barf in a threaded environment: ActiveRecord::Base.allow_concurrency.

Oh yes, you think when you first see this, this will solve my problems. Well, sort of. If you put

ActiveRecord::Base.allow_concurrency = true

in your code ActiveRecord will change the way it talks to its backend database. In the Mysql world, this means that ActiveRecord will open a new database connection thread each time it needs to query the database - I think. I have to be honest I’m not sure this is exactly accurate: it may only open a new thread when it needs to execute a query that it hasn’t already executed. Either way, even a trivial script that just imports data into models can easily open 100+ threads to the database. Which will make your database quite unhappy and result in anger on your part.

My understanding is that in addition to creating a mega amount of database connections, ActiveRecord will also not release its database resources until the process in which ActiveRecord is running dies. If your script will only create a few database calls, ActiveRecord’s concurrency option is probably satisfactory for the job (although, if your script is that light-weight, why would you need threads anyway) however, for long term processes like daemons, this really a show stopper.

Even better - the Mysql driver has a lock that engages when the database connection is waiting for a statement result set to return from the database. This results in a lock on the Ruby interpreter which results in your entire script grinding to a halt while waiting for Mysql to strut its stuff.

If you are part of the Ruby 1.9 crowd, you can take a look at the brand-spank’in-new git project espace/mysqlplus that offers “An enhanced mysql driver with an async interface and threaded access support”.

But I’m stuck on 1.8.7 so I’m going to spend some time over the next few days looking for a reusable and sustainable solution that will permit my scripts to use threads to divide work and be able to use ActiveRecord at the same time.

Time to upgrade your ruby installs. This advisory came out yesterday regarding 5 CVE’s impacting every current ruby release.

Officially impacted versions: 1.8 series

• 1.8.4 and all prior versions
• 1.8.5-p230 and all prior versions
• 1.8.6-p229 and all prior versions
• 1.8.7-p21 and all prior versions

1.9 series

• 1.9.0-1 and all prior versions

Read the advisory for remediation information. Matasano also has a great writeup on the advisory here. Wonder what it would take to blow out the rails params array with this (I’ll leave that to the professionals, I’d rather just patch it and move on right now). Scary stuff!

Update Per comments on the Ruby on Rails blog post you will break your rails install if you upgrade to 1.8.6-p230. And 1.8.7 is only compatible with Rails 2.1. You might want to test out the latest ruby with your app on a dev site before blindly upgrading your production ruby install.

If you want to put a little laugh back into the day, read Zed Shaw’s rant. And yes, there is actually some good info there, it’s worth a read.

I work on a number of projects, some in ruby, some in “other” languages. Time and time again I find myself working on a piece of code in those “other” projects and thinking “man I wish this was in ruby”. The most recent one was with arrays in PHP, so I figured I’d write a little piece on why it made me sad. Sometimes, the simple things in a language are what make it pleasing to work in.

$emotions = array('happy', 'sad', 'angry');
if (is_array($emotions))
  foreach($emotions as $e) {
    print $e . "\n";
  }

$emotions = array('happy', 'sad', 'angry');
foreach((array)$emotions as $e) {
  print $e . "\n";
}

$emotions = array('happy', 'sad', 'angry');
foreach((array)$emotions as $k => $v) {
  print "[$k] => $v\n";
}

$emotions = array('happy', 'sad', 'angry');
for ($i = 0; isset($emotions[$i]); $i++) {
  print $emotions[$i] . "\n";
}

emotions = ['happy', 'sad', 'angry']
emotions.each {|e| puts e}

['happy', 'sad', 'angry'].each {|e| puts e}

emotions = ['happy', 'sad', 'angry']
emotions.each_index {|k| puts "[#{k}] => #{emotions[k]}"}

emotions = ['happy', 'sad', 'angry']
emotions.each {|e| puts e} if emotions.is_a? Array

happy = ['ecstatic', 'joyous']
sad = ['depressed', 'suicidal']
angry = ['raging', 'mad']
emotions = [happy, sad, angry]
emotions.flatten.each {|emotion| puts emotion}

$happy = array('ecstatic', 'joyous');
$sad = array('depressed', 'suicidal');
$angry = array('raging', 'mad');
$emotions = array_merge_recursive($happy, $sad, $angry);
foreach((array)$emotions as $k => $v) {
  print "[$k] => $v\n";
}

I’m feeling kind of dumb tonight. I must have read the little (and I do mean little) section on the Regexp class in my Ruby in a Nutshell book a billion times now (if that doesn’t say stupid, I dunno what does) and somehow every time I’ve missed this little beauty.

@words=["important","monkey","dancing"]

def highlight(input)
  input.gsub(/#{@words.join('|')}/, '*\0*')
end

puts highlight("I think it's important to have a dancing monkey in your bedroom.")
# => I think it's *important* to have a *dancing* *monkey* in your bedroom.

@words=["important","monkey","dancing"]

def highlight(input)
  input.gsub(Regexp.union(*@words), '*\0*')
end

puts highlight("I think it's important to have a dancing monkey in your bedroom.")
# => I think it's *important* to have a *dancing* *monkey* in your bedroom.

I needed to write a reporting script the other day that dealt with time durations. The script basically processed a series of log files determining the ‘start’ and ‘end’ times of certain events. Part of the report output needed to be the duration of each event. Calculating this using DateTime objects should have been as easy as this:

def duration
  @end_datetime - @start_datetime
end

Ack, Instead of providing me with something intuitive, I got an object back of class Rational. A fraction of all things:

start = DateTime.parse('2/1/2008 10:35:00')
finish = DateTime.parse('2/1/2008 21:15:00')
duration = finish - start   # => Rational(4, 9)

Well, what the heck do I do with a fraction?

It turns out that Date has a class method called day_fraction_to_time() that takes a Rational and returns the number of hours, minutes, seconds and factions of a section. The beast in action:

start = DateTime.parse('2/1/2008 10:35:00')
finish = DateTime.parse('2/1/2008 21:15:00')
duration = finish - start
Date.day_fraction_to_time(duration)   => [ 10, 40, 0, Rational(0, 1) ]

Joy, this works, I now have my duration in hours minutes and seconds. The only conclusion I can draw regarding using Rational is that it can provide more accuracy than a Double or Float. In other words, it’s a more accurate representation of the percentage of a day since Rational( 1, 3 ) will not lose precision whereas ‘.3333333333’ is only as precise as the number of bytes a floating point number is stored as.

I was a little bored tonight and thought I’d respond to a post over Happy Python with some code. Thanks for the inspiration Matt!

ruby>> require 'ieeemac'
# => true  
ruby>> macs = Mac.find_macs `ifconfig -a`
# => [#<Mac:0x1357f40 @mac=["00", "16", "cb", "95", "13", "50"]>, #<Mac:0x1357f2c @mac=["00", "16", "cb", "ff", "fe", "5d"]>, #<Mac:0x1357d74 @mac=["00", "17", "f2", "41", "ec", "ec"]>, #<Mac:0x1357c20 @mac=["00", "50", "56", "c0", "00", "08"]>, #<Mac:0x1357ae0 @mac=["00", "50", "56", "c0", "00", "01"]>, #<Mac:0x135798c @mac=["00", "1c", "42", "00", "00", "00"]>, #<Mac:0x13577e8 @mac=["00", "1c", "42", "00", "00", "01"]>]  
ruby>> macs[0].to_format :cisco
# => "0016.cb95.1350"  
ruby>> macs[0].to_cisco
# => "0016.cb95.1350"  

Read the rest of this post for the full class.

# We got your MAC manipulation here
class Mac
  #bare:    001122334455
  #windows: 00-11-22-33-44-55
  #unix?:   00:11:22:33:44:55
  #cisco:   0011.2233.4455  
  VALID_MACS=/[0-9a-f]{12}|[0-9a-f]{2}[:-][0-9a-f]{2}[:-][0-9a-f]{2}[:-][0-9a-f]{2}[:-][0-9a-f]{2}[:-][0-9a-f]{2}|[0-9a-f]{4}\.[0-9a-f]{4}\.[0-9a-f]{4}/

  def initialize(str)
    if ismac?(str)
      # Convert to internal octet representation
      @mac = str.gsub(/[.:-]/,'').scan(/../)
    else
      raise ArgumentError, "Invalid MAC", caller
    end
  end

  # return all the valid MAC's in a given string as an array
  def self.find_macs(str)
    macs = str.scan VALID_MACS
    ret = []
    macs.each { |mac| ret << Mac.new(mac) }
    ret
  end

  def to_format(fmt)
    # Convert to symbol for ease of use
    fmt = fmt.to_sym
    case fmt
    when :bare
      to_bare
    when :windows
      to_windows
    when :unix
      to_unix
    when :cisco
      to_cisco
    end
  end

  def ismac?(str)
    VALID_MACS =~ str
  end

  def to_bare
    @mac.join('')
  end
  alias :mac :to_bare

  def to_windows
    @mac.join('-')
  end
  def to_unix
    @mac.join(':')
  end
  def to_cisco
    sprintf("%s%s.%s%s.%s%s", @mac[0],@mac[1],@mac[2],@mac[3],@mac[4],@mac[5])
  end
end

# macs = Mac.find_macs(`ifconfig -a`)
# macs.each do |mac|
#   puts "BARE:    #{mac.to_format(:bare)}"
#   puts "WINDOWS: #{mac.to_format(:windows)}"
#   puts "UNIX:    #{mac.to_format(:unix)}"
#   puts "CISCO:   #{mac.to_format(:cisco)}"
# end

Sometimes you run across neat and interesting libraries when you are reading up on something completely different. Case in point, via another blog, I stumbled on Reliable Messaging with Ruby. This looks to solve a problem I haven’t yet run into, but I know I’ll find a use for it in the future (maybe for a game I’m slowly working on, or a work project). As of today, there haven’t been any commits to the project for two years, but it’s entirely possible there are no bugs (except for the one I found below) and the author considers it feature complete.

Current feature list:

• Simple API.
• Transction processing.
• Disk-based and MySQL message stores.
• Best effort, repeated and once-only delivery semantics.
• Priority queues, message expiration, dead-letter queue.
• Message selectors.
• Local and remote queue managers using DRb.

$ sudo gem install reliable-msg -y

$ sudo queues install disk foo
Created queues configuration file: /Library/Ruby/Gems/1.8/gems/reliable-msg-1.1.0/queues.cfg

$ sudo queues manager start
Loaded queues configuration from: /Library/Ruby/Gems/1.8/gems/reliable-msg-1.1.0/queues.cfg
Using message store: disk
Accepting requests at: druby://localhost:6438

@drb_server = DRb::DRbServer.new drb_uri, self, :tcp_acl=>ACL.new(drb["acl"].split(" "), ACL::ALLOW_DENY)

@drb_server = DRb::DRbServer.new drb_uri, self, :tcp_acl=>ACL.new(drb["acl"].split(" "), ACL::DENY_ALLOW)

$ ruby q.rb
Local: Hello queue master
Queue: Hello queue master

require 'rubygems'
require 'reliable-msg'

world = String.new "Hello queue master"
queue = ReliableMsg::Queue.new 'my-queue'
queue.put world
msg = queue.get
puts "Local: #{world}"
puts "Queue: #{msg.object}"

I’ve read a bunch of conflicting advice on how to use active record models defined in a Rails application from outside Rails.

The following is the result of experimenting with others’ advice plus my own digging around:

require 'rubygems'
require 'active_record'
require '/path/to/rails/config/environment'

This should allow you to use your Rails models just like you would in a Rails controller:

#!/usr/local/bin/ruby

require 'rubygems'
require 'active_record'
require '/var/www/railsapp/config/environment'

m = MyModel.new
m.find_all.each { |row|
  puts row.some_column
}

def dqtocidr( mask )
  bitcount = 0
  mask.split('.').each { |octet|
    o = octet.to_i
    bitcount += (o & 1) and (o >>= 1) until o == 0
  }
  return bitcount
end

irb(main):016:0> dqtocidr(‘128.0.0.0’) => 1

irb(main):017:0> dqtocidr(‘192.0.0.0’) => 2

irb(main):018:0> dqtocidr(‘224.0.0.0’) => 3

irb(main):019:0> dqtocidr(‘240.0.0.0’) => 4

irb(main):020:0> dqtocidr(‘248.0.0.0’) => 5

irb(main):021:0> dqtocidr(‘252.0.0.0’) => 6

irb(main):022:0> dqtocidr(‘254.0.0.0’) => 7

irb(main):023:0> dqtocidr(‘255.0.0.0’) => 8

Ruby does not provide programmer defined destruction methods. In C++ you can define a method that is messaged when the object instance is destroyed. This method can perform clean-up operations such as deleting temporary files.

Ruby, however, destroys objects dynamically during a scheduled garbage collection. This means that, during a run, it is impossible to know when an object instance will be destroyed. This fact combined with the lack of a destruction method can create headaches when there is a need to clean-up during object destruction.

Since there is no direct support for a destructor method, you must call a custom function, or more specific a proc object, when the garbage collector is about to destroy the object. As stated before it is unpredictable when this occurs.

Also if such a finalizer object has a reference to the orignal object, this may prevent the original object to get garbage collected.

Because of this problem, the finalize method below is a class method and not a instance method. So if you need to free resources for an object, like closing a socket or killing a spawned subprocess, you should do it explicitly with the ObjectSpace class.

class MyClass
  def initialize
    ObjectSpace.define_finalizer(self,
     self.class.method(:finalize).to_proc)
  end

  def MyClass.finalize(id)
    puts "Object #{id} dying at #{Time.new}"
  end
end

# test code
3.times {
    MyClass.new
}
ObjectSpace.garbage_collect