cowboyrushforth.com

consistent hash ring in node.js

Fri, 03 Jun 2011 11:00:00 -0700

If you ever need to create a consistent hash ring in node.js, this may help you. This is compatible with redis-rb's dht algorithm. This is not well tested yet, but in preliminary tests it is working good. Thanks to this stackoverflow post for the crc32 algo. Future improvements may include parts of this re-done as a c module for added speed. Currently I am not using this per inbound request, only once per server start, so speed is less of an immediate priority. Enjoy!

this.ring = {};
this.sorted_keys = [];

this.generateRing = function(servers) {
  for(var j = 0; j < servers.length; j++) {
    this.addServerToRing(servers[j]);
  }
}

this.addServerToRing = function(server) {
  for(r = 0; r < 160; r++) {
      key = this.crc32("redis://"+server+"/0:"+r);
      this.ring[key] = server;
      this.sorted_keys.push(key);
  }

  this.sorted_keys.sort(function (a, b) {
    return a-b;
  });
}

this.getServer = function(key_query) {
  key = this.crc32(key_query);
  index = this.ringSearch(key);
  return this.ring[this.sorted_keys[index]];
}

this.ringSearch = function(val) {

  var upper = this.sorted_keys.length - 1;
  var begin = upper;
  var lower = 0;
  var k = 0;

  while(lower <= upper) {
    k = parseInt(((lower + upper) / 2),10);
    idx = this.sorted_keys[k];

    if(idx == val) {
      return k;
    } else if ( idx > val ) {
      upper = k - 1;
    } else {
      lower = k + 1;
    }
  }

  if(upper < 0) {
    upper = begin;
  }
  return upper;
}


this.crc32 = (function() {
    function utf8encode(str) {
        var utf8CharCodes = [];

        for (var i = 0, len = str.length, c; i < len; ++i) {
            c = str.charCodeAt(i);
            if (c < 128) {
                utf8CharCodes.push(c);
            } else if (c < 2048) {
                utf8CharCodes.push((c >> 6) | 192, (c & 63) | 128);
            } else {
                utf8CharCodes.push((c >> 12) | 224, ((c >> 6) & 63) | 128, (c & 63) | 128);
            }
        }
        return utf8CharCodes;
    }

    var cachedCrcTable = null;

    function buildCRCTable() {
        var table = [];
        for (var i = 0, j, crc; i < 256; ++i) {
            crc = i;
            j = 8;
            while (j--) {
                if ((crc & 1) == 1) {
                    crc = (crc >>> 1) ^ 0xEDB88320;
                } else {
                    crc >>>= 1;
                }
            }
            table[i] = crc >>> 0;
        }
        return table;
    }

    function getCrcTable() {
        if (!cachedCrcTable) {
            cachedCrcTable = buildCRCTable();
        }
        return cachedCrcTable;
    }

    return function(str) {
        var utf8CharCodes = utf8encode(str), crc = -1, crcTable = getCrcTable();
        for (var i = 0, len = utf8CharCodes.length, y; i < len; ++i) {
            y = (crc ^ utf8CharCodes[i]) & 0xFF;
            crc = (crc >>> 8) ^ crcTable[y];
        }
        return (crc ^ -1) >>> 0;
    };
})();

Altly Begins

Thu, 19 May 2011 21:01:00 -0700

Since the last update, I have quit my job, moved to the beach, and started a company with a small team of awesome people. For the first time in quite some time, I am getting a lot of fresh air and sun light regularly, and it is an awesome thing.

Reserve your username today! More details soon :)

zeromq adventures and linkfinder

Thu, 07 Apr 2011 20:49:00 -0700

I have been hearing about ZeroMQ for a bit now, but had never really dug in until 72 hours ago. I must say, I am impressed. Maybe I am easily amused (likely), or maybe I have just been having a good run with technology lately, but nevertheless, I will share a bit of my experience.

What is ZeroMQ (aka '0mq') you ask? At first glance it comes across as a networking library, but that is just the tip of the iceberg. Once cracked open it is a concurrency library, networking library, and socket library all wrapped into one really simple and concise api.

Sounds like it is chalk full of features and bloatware, but right on the front of their site they have 'Less is more' in big bold letters. I like this concept in engineering most of the time. Actually I like this concept applied to life itself most of the time. Unless I am at a vegas buffet anyways. Hah!

So I figured for a quick test trial of the library I would write a service in C that returns links in a given webpage. One or more clients will connect to the service and send a url that it would like scanned. This url is then downloaded, parsed, and the links in it, if any, are returned in JSON format.

Without further ado, check it out on github: https://github.com/cowboyrushforth/linkfinder. Turned out to be about 245 sloc and only took a few spare hours to write. How it is setup is that the main thread has one big event loop that 0mq runs and accepts requests. These requests go into a queue that one of 4 worker threads pick up. The worker picks up a request, does all the downloading, parsing, and then sends the request back to the mainthread which then delivers it to the client. It could use a bit of additional error handling thought, but is incredibly simple.

If your feeling like a technologically inspirational read, check out the ZeroMQ guide. It is very comprehensive and really shows the full potential of the library.

Cheers!

on node.js, trendy technology, and the development community

Sat, 19 Mar 2011 16:30:00 -0700

Ok, so I have dived into node.js, as seen on my recent entry on building a realtime, web based, minesweeper clone. I dove into this mostly to just see what all the fuss is about, but I would be lying if I said I was not pleasantly surprised.

This seems to be a topic that one must tread lightly about as node is getting quite a bit of flack as well as quite a few praises and success stories. Seems like a love/hate relationship with no counselor! Why the great divide?

My goal in writing this is to just outline both sides of the story and hopefully give a fairly neutral opinion on the matter but shining some light on all angles. So here goes: (continued after break)

MineSpotter v0.0.1

Sat, 19 Mar 2011 09:57:00 -0700

So with all the excitement in the node.js community, I thought I would see what all the fuss is about, so I went to work on my first node.js application, MineSpotter.

MineSpotter is essentially a minesweeper clone, but heavily inspired by a website called WordSquared. MineSpotter is to MineSweeper as WordSquared is to Scrabble. Ultimately the game is designed to have an infinitely expanding board, and can be played in realtime, with multiple players.

MineSpotter was built using node.js, along with a node library called DNode on top of the geospatial features of MongoDB.

Still have a lot of things to finish, such as a leader board, facebook connect, and more work on flagging and explosions, but the core gameplay is pretty much completed.

Source is up at github here for your reading and hacking pleasure, and thanks to DuoStack I have been able to get some free hosting for it.

Check it out at http://play.minespotter.com!

new adventures

Sat, 23 Oct 2010 15:22:00 -0700

In Seattle for my new gig for a couple weeks, and looks like I will be coming up here regularly for a few more weeks. Definitely living life in the fast lane, but trying to balance it between the moist northwest, the dry high desert, and the busy metropolis of LA turns out to be pretty fun. Here is a few snaps from some recent adventures:

Crazy wall covered in gum at Pike Place, Seattle. Very awesome and very nasty:

high desert Lightning storm:

red vortex sunrise:

phenomenal

Wed, 06 Oct 2010 22:43:00 -0700

Burning Man was epic once again. What can I say. From the insane flaming bar that our camp built, to being able to play to hundreds of people during one of the most epic sunsets of my life, words really just do not do it justice. Coming back into the real world was once again very intense in many different ways from logistically to emotionally. But alas I have finally returned, and while there is pieces of my mind and soul that are still far from home, overall I have re-assimilated into society as much as I am probably going to, and onward ho!

On on that note, shortly after my return this year I was given the exciting opportunity to come on board MySpace.com as a full time senior engineer, so life will be shifting around a bit for me here very shortly. I have been primarily working from home for the past 4+ years, so going back to working with people and not being such a caveman I am looking forward too, as well as working on some really big projects.

mid summer stretch

Thu, 29 Jul 2010 18:46:00 -0700

Sometimes, well a lot of the times, life is weirdly pleasant. Just as I had started to dig in with my spare time and find other ways to up my game and find interesting things to do, I have gotten hit with quite a freight train of work you could say.

So lately I have been laying low, trying to get all my ducks in a row, and coincidentally, I have not been learning how to sew. I did manage to however have enough free time to make this mix last weekend, and finally it is on sound cloud for your downloading pleasure.

724 by spotman

Other than that, I am not sure how long this flood of work will last, but I am going to milk it up while I can. Only 30 days or so until I leave for burning man which is insane and awesome at the same time. This year I will definitely be gone a whole week, but not sure how much more than that I will have time for. Very excited about the camp this year though :)

Thats all I got for now. Enjoy!

oil spills suck but math does not

Wed, 02 Jun 2010 18:22:00 -0700

Life has been weird lately to say the least. The oil spill really is very sad and unfortunate, and it seems like there has been quite a bit of natural disasters recently too which is a bummer. On top of that, I have for better or for worse, come to the realization that (imo) the economy is probably never going to go back to the good ol' days. I hypothesize it will probably make a very slow recovery as some of the other glooming problems in our country are slowly resolved. When people are all employed and things start picking up, I suspect people will be spending their money in different ways.

Taking that all into consideration, I was sort of feeling bummed for a bit. Business has been slow and tedious as we as a small company try to pick up new clientele. It seems as though everyone is trying to cut corners and just try to stay afloat right now. In previous years, our company has thrived from the constant desire of any business to turn their ideas into a reality. We specialize in providing custom software and we will turn anyone's idea into a good program. These days, people do not want to innovate as much, and the ones that do, have a large pool of people like us to pick from currently. It is a hard game out there.

Recently though, I have decided to not just going to sit around and pout about it. As things evolve around me, I must evolve too. So I have decided to arm myself with a significant amount more math. Math is one thing that when you are a kid you just have no clue how this will apply to your life later. You also probably don't (or maybe you do) think sitting around all day writing code is fun, but turns out it can be extremely rewarding at times. I feel like math is one area that I did not take seriously enough as a kid, and now in my career choice I am finally starting to pay for it as I want to expand my programming into different realms.

I am not going back to school or anything extreme like that, but believe it or not I have hired a personal tutor, and plan to be really hone my skills in linear algebra, trigonometry and calculus into hopefully a sharp toolset that will help me take my coding to new places.

Besides all that jibber jabber of work/career/etc, I am also still trying to have some fun. I will be going to element11 here in just another week where Bat Country will be representin' and I will get another chance to DJ, which is very exciting for me. Hopefully I will remember to record my set this time!

That's all for now!

time flies fast

Tue, 04 May 2010 11:37:00 -0700

Wow I can't believe we are almost 1/2 way through the year! Life has been so busy it has been hard to keep up lately! Lucky for me it's a pretty even split between work and play so I can't complain much!

Work wise we just launched our new website to give us a bit fresher of a look, and we also just attended a tradeshow in hopes of drumming up some more business. Other than that we have had an ebb and flow of business lately, but we have built some really cool things this year!

In other news, Bat Country's final plan was submitted to Burning Man a couple weeks ago! I am very excited about this year, and we have a ton of momentum so far.

Thats it for now!
-s