- Posted: 11 months ago
- Tags:consistent, hashring, node.js
If you ever need to create a consistent hash ring in node.js, this may help you. This is compatible with redis-rb's dht algorithm. This is not well tested yet, but in preliminary tests it is working good. Thanks to this stackoverflow post for the crc32 algo. Future improvements may include parts of this re-done as a c module for added speed. Currently I am not using this per inbound request, only once per server start, so speed is less of an immediate priority. Enjoy!
this.ring = {};
this.sorted_keys = [];
this.generateRing = function(servers) {
for(var j = 0; j < servers.length; j++) {
this.addServerToRing(servers[j]);
}
}
this.addServerToRing = function(server) {
for(r = 0; r < 160; r++) {
key = this.crc32("redis://"+server+"/0:"+r);
this.ring[key] = server;
this.sorted_keys.push(key);
}
this.sorted_keys.sort(function (a, b) {
return a-b;
});
}
this.getServer = function(key_query) {
key = this.crc32(key_query);
index = this.ringSearch(key);
return this.ring[this.sorted_keys[index]];
}
this.ringSearch = function(val) {
var upper = this.sorted_keys.length - 1;
var begin = upper;
var lower = 0;
var k = 0;
while(lower <= upper) {
k = parseInt(((lower + upper) / 2),10);
idx = this.sorted_keys[k];
if(idx == val) {
return k;
} else if ( idx > val ) {
upper = k - 1;
} else {
lower = k + 1;
}
}
if(upper < 0) {
upper = begin;
}
return upper;
}
this.crc32 = (function() {
function utf8encode(str) {
var utf8CharCodes = [];
for (var i = 0, len = str.length, c; i < len; ++i) {
c = str.charCodeAt(i);
if (c < 128) {
utf8CharCodes.push(c);
} else if (c < 2048) {
utf8CharCodes.push((c >> 6) | 192, (c & 63) | 128);
} else {
utf8CharCodes.push((c >> 12) | 224, ((c >> 6) & 63) | 128, (c & 63) | 128);
}
}
return utf8CharCodes;
}
var cachedCrcTable = null;
function buildCRCTable() {
var table = [];
for (var i = 0, j, crc; i < 256; ++i) {
crc = i;
j = 8;
while (j--) {
if ((crc & 1) == 1) {
crc = (crc >>> 1) ^ 0xEDB88320;
} else {
crc >>>= 1;
}
}
table[i] = crc >>> 0;
}
return table;
}
function getCrcTable() {
if (!cachedCrcTable) {
cachedCrcTable = buildCRCTable();
}
return cachedCrcTable;
}
return function(str) {
var utf8CharCodes = utf8encode(str), crc = -1, crcTable = getCrcTable();
for (var i = 0, len = utf8CharCodes.length, y; i < len; ++i) {
y = (crc ^ utf8CharCodes[i]) & 0xFF;
crc = (crc >>> 8) ^ crcTable[y];
}
return (crc ^ -1) >>> 0;
};
})();
- Posted: 1 year ago
- Tags:altly
Since the last update, I have quit my job, moved to the beach, and started a company with a small team of awesome people. For the first time in quite some time, I am getting a lot of fresh air and sun light regularly, and it is an awesome thing.
Reserve your username today! More details soon :)
- Posted: 1 year ago
- Tags:c, linkfinder, zeromq
I have been hearing about ZeroMQ for a bit now, but had never really dug in until 72 hours ago. I must say, I am impressed. Maybe I am easily amused (likely), or maybe I have just been having a good run with technology lately, but nevertheless, I will share a bit of my experience.
What is ZeroMQ (aka '0mq') you ask? At first glance it comes across as a networking library, but that is just the tip of the iceberg. Once cracked open it is a concurrency library, networking library, and socket library all wrapped into one really simple and concise api.
Sounds like it is chalk full of features and bloatware, but right on the front of their site they have 'Less is more' in big bold letters. I like this concept in engineering most of the time. Actually I like this concept applied to life itself most of the time. Unless I am at a vegas buffet anyways. Hah!
So I figured for a quick test trial of the library I would write a service in C that returns links in a given webpage. One or more clients will connect to the service and send a url that it would like scanned. This url is then downloaded, parsed, and the links in it, if any, are returned in JSON format.
Without further ado, check it out on github: https://github.com/cowboyrushforth/linkfinder. Turned out to be about 245 sloc and only took a few spare hours to write. How it is setup is that the main thread has one big event loop that 0mq runs and accepts requests. These requests go into a queue that one of 4 worker threads pick up. The worker picks up a request, does all the downloading, parsing, and then sends the request back to the mainthread which then delivers it to the client. It could use a bit of additional error handling thought, but is incredibly simple.
If your feeling like a technologically inspirational read, check out the ZeroMQ guide. It is very comprehensive and really shows the full potential of the library.
Cheers!
- Posted: 1 year ago
- Tags:node.js
Ok, so I have dived into node.js, as seen on my recent entry on building a realtime, web based, minesweeper clone. I dove into this mostly to just see what all the fuss is about, but I would be lying if I said I was not pleasantly surprised.
This seems to be a topic that one must tread lightly about as node is getting quite a bit of flack as well as quite a few
praises and success
stories. Seems like a love/hate relationship with no counselor! Why the great divide?
My goal in writing this is to just outline both sides of the story and hopefully give a fairly neutral opinion on the matter but shining some light on all angles. So here goes: (continued after break)
Continue Reading -->
So with all the excitement in the node.js community, I thought I would see what all the fuss is about, so I went to work on my first node.js application, MineSpotter.
MineSpotter is essentially a minesweeper clone, but heavily inspired by a website called WordSquared. MineSpotter is to MineSweeper as WordSquared is to Scrabble. Ultimately the game is designed to have an infinitely expanding board, and can be played in realtime, with multiple players.
MineSpotter was built using node.js, along with a node library called DNode on top of the geospatial features of MongoDB.
Still have a lot of things to finish, such as a leader board, facebook connect, and more work on flagging and explosions, but the core gameplay is pretty much completed.
Source is up at github here for your reading and hacking pleasure, and thanks to DuoStack I have been able to get some free hosting for it.
Check it out at http://play.minespotter.com!
