Keyspace = {

  Column Family : {

    Row Key : {
      Column Name : 'Column Value'
      Column Name : 'Column Value'
      Column Name : 'Column Value'
      ...
    }

  }

}

Think you understand it? Pretend that you don’t know the meaning of Row, Key or Column. I wish someone told me that before I got into Cassandra.

Pop your stack: the relational model (mostly) does not apply

A paradigmatic dichotomy of data modeling is the notion of serial vs. parallel. Whether in databases or programming languages etc., a lot of effort has been made to express data in series, parallel, or some composition of both.

Series

a number of things, events, or people of a similar kind or related nature coming one after another …

In programming languages you have arrays, lists, vectors, etc.; in databases you have tables of rows. In a database you might find a table named user, probably stores one or more user records; any record in the user table does not necessarily relate to any other; any limits to the number of records in the user table are implementation dependent and for the most part be considered arbitrary.

Parallel

occurring or existing at the same time or in a similar …

or

The opposite of series.

Databases represent these as row of columns; Programming languages have tuples, maps, hashes, structs, classes, etc. Continuing the database example, a record in the user table probably stores exactly one fixed size set of facts about a user as fields; all fields are assumed to be true for that user in some greater disposition; altering the set of fields fundamentally changes the model.

Does Cassandra do this? Yes & No, at the same time.

In Cassandra you can have many columns in a row, at the time of writing about 2 Billion. Major relational databases have limits on the order of Thousands. If unintuitive, take a moment to grok the difference between Thousands and Billions, we are not comparing apples to apples.

In this example, rows model entities and columns model keys. Depending on the replication strategy, bic and pilot might not exist on the same node. This is something to consider when defining the Keyspace.

Manufacturers = {

  bic : {
    origin : 'france'
    year : 1945
  }

  pilot : {
    origin : 'japan'
    year : 1918
  }

}

In this example rows are series of model numbers and their respective description.

ProductDescriptionsByMfgr = {

  bic : {
    FRM41 : 'BIC 4-Color Ballpoint Pen Refill, Fine Point'
    MRM41 : 'BIC 4-Color Ballpoint Pen Refill, Medium Point'
  }

  pilot : {
    77227 : 'Dr. Glip, Better & EasyTouch Retractable Ballpoint Pen Refill, Medium, Black'
    77228 : 'Dr. Glip, Better & EasyTouch Retractable Ballpoint Pen Refill, Medium, Blue'
    77210 : 'Dr. Glip, Better & EasyTouch Retractable Ballpoint Pen Refill, Fine Point, Black'
    77221 : 'Ballpoint Pen Refill, Medium Point, Black'
    77222 : 'Ballpoint Pen Refill, Medium Point, Blue'
    77215 : 'Ballpoint Pen Refill, Fine Point, Black'
  }

}

You should be experiencing some cognitive dissonance. It’s OK, Cassandra conflates series and parallel within a row. So add columns all day long, but still add rows all day long too.

It is important to know that all columns in a row are sorted. If I add another column to this row it will be inserted at the correct position. Even in the previous example, the column origin will precede year. How columns are sorted within a row is configurable per Column Family.

Also, no distribution or replication takes places within a row: each copy of a row will contain all columns; conversely if your replication doesn’t take this into consideration integrity or performance may suffer.

If you want something a more complex than a single datum as a column’s value check out Super Columns

The norm is to Denorm: model for queries.

If the intent of a query is to retrieve a single item then model that single item:

Procucts = {

    77227 : {
      description : 'Dr. Glip, Better & EasyTouch Retractable

Ballpoint Pen Refill, Medium, Black'

manufacturer : 'pilot'
      price : '$1.59'
      quantity : '2'
    }
}

If you need a listing of products with descriptions per manufacturer, store exactly that (as in the ProductDescriptionsByMfgr example). Don’t feel the need to normalize.

Is this a waste of disk? disk is cheap, querying a big dataset is not.

Mixing implementation and domain modeling …

It may seem as if there is no clean separation of domain modeling and actual implementation. The structures available in Cassandra come with many critical implementation-specific strings attached. Very true, but before holding this against Cassandra, consider if other database systems are free from this. Things like replication, sharding, data warehousing, denormalized data, etc. are common in decent sized implementations and will definitely leak back into the domain modeling.

Terminology killed the cat

Not only is Cassandra’s terminology confusing it’s downright misleading. Row, Column & Key all have existing semantics in the land of databases. To make matters worse, Cassandra’s definitions are not even orthogonal to the existing ones — they exist in a difficult state of quasi-synonymity.

Despite this disservice, the set of small unused words apropos of database is probably depleting as fast four-letter English profanity. I’ll take key over distributed ordered set descriptor any day.

I was thrown into the Cassandra pool without knowing how to swim, I hope this helps anyone in the same situation. Expert swimmers out there please correct me where wrong.

Permalink | Leave a comment  »

Why FFI?

Interfacing Ruby with a C library is a pain. If you are like me, before you dig in, you make damn sure that: a gem doesn’t already exist; parsing (read scraping) the output of back-ticking is too dirty; you really have to extend Ruby to solve your problem; you really have to solve your problem. Extending Ruby for me usually is a last resort.

Your goal is most likely to expose an interface of the desired behavior in a C library and move data between ruby-space and c-space. Normal Ruby extensions are all C code. The gap between the C library and Ruby must be bridged with C. You are literally extending the Ruby interpreter, a C program.

With FFI you do it all in Ruby. In essence you implement the constructs of a C header in a Ruby DSL. If that seems simple … yes, it is.

If you can read .h files then you can FFI

Step 1: get ffi

% gem install ffi

Step 2: make module

require 'ffi'

module KQueue
  extend FFI::Library
  ffi_lib FFI::Library::LIBC
end

Make this module interface to a library and that library is libc.

Step 3: put some stuff inside

Functions

C from sys/event.h

int kevent(int kq, const struct kevent *changelist, int nchanges,
           struct kevent *eventlist, int nevents,
           const struct timespec *timeout);

Ruby-FFI: the first param is the name of the function; second is an array of types; third is the return type.

attach_function :kevent, 
                [ :int, :pointer, :int, :pointer, :int, :pointer ],
                :int

Call them like a normal module function. No compiling nor linking, just use it as you would Ruby code.

number_of_events = KQueue.kevent kq, change, 1, event, 1, timeout

Structs

C from sys/event.h

struct kevent {
    uintptr_t   ident;      /* identifier for this event */
    int16_t     filter;     /* filter for event */
    uint16_t    flags;      /* general flags */
    uint32_t    fflags;     /* filter-specific flags */
    intptr_t    data;       /* filter-specific data */
    void        *udata;     /* opaque user data identifier */
};

Ruby-FFI: not quite as clean as functions but almost as intuitive.

class KEvent < FFI::Struct

  layout(:ident,  :uintptr_t, # identifier for this event
         :filter, :int16,     # filter for event
         :flags,  :uint16,    # general flags
         :fflags, :uint32,    # filter-specific flags
         :data,   :intptr_t,  # filter-specific data
         :udata,  :pointer)   # opaque user data identifier

end

Use FFI:Struct intances like a Hash.

e = KEvent.new
e[:flags] = EVFILT_VNODE

Caveats: Constants & Macros

Originally I wanted the previous section to read If you can read man pages then you can FFI, but man pages don’t have everything, luckily headers do.

Constants & macros are not C code per se, they are replaced with C code by the preprocessor. Long story short, they don’t end up as symbols in the shared libraries and therefore FFI can’t relate to them.

# taken from sys/event.h
EV_ADD = 0x0001  # add event to kq (implies enable)

# derived from macro EV_SET in sys/event.h
def ev_set event, ident, filter, flags, fflags, data, udata
  event[:ident] = ident
  event[:filter] = filter
  event[:flags] = flags
  event[:fflags] = fflags
  event[:data] = data
  event[:udata] = udata
end

Technically you don’t need them but practically you do. Just make C constants into Ruby constants and implement C macros as Ruby methods.

That’s it, just use it.

Long Term …

FFI not only lowers the barrier to interfacing with existing libraries, it makes interfacing cheap. Cheap could mean more effort towards crafting good Ruby instead of just thin layers above the library.

All those libraries out there just got moved from a an ivory tower into Walmart, i.e. accessible to anyone. Ruby being such a community driven platform, one can imagine the impacts of FFI for Ruby are more than just technical.

I got started on this subject by being subliminally exposed to ffi in the docs of other platforms I was tinkering with (haskell, node.js, racket). Then I wanted to make a tool that would perform a task when a file changed, like autotest but more general. Ultimately I googled “ruby ffi”, the rest is history.

Check out FFI’s wiki for more details. If I’m wrong about any of this, please have at it.

Permalink | Leave a comment  »

Do you wish you could write ruby like this …

1
2
3
4
5
6
7
8
9

if r == P
  #do_something ...
elsif r * 2 < Q
  #do_something ...
elsif some_test(r)
  #do_something ...
else
  raise "unpossible"
end

… but using a case statement? You can in ruby 1.9.

Why? Case statements are sooooo much nicer

Multi-condition if/elsif statements allow entirely unrelated conditions. Meaning they need not share any terms, even though they commonly do. Unfortunately the grok-ability tends to be inverse to the number of conditions. These can get pretty smelly.

In a case statement the common term is factored to one place at the top of the statement. This sets up a semantic subject and many predicates, a very familiar and intuitive idiom for users of natural languages (like you).

Why can’t I use a case statement?

While ruby (1.8.x and earlier) allows many useful expressions in when clauses, they all revolve around the implementation of the === method. Meaning you must have an object that has an instance method named === that will evaluate to a truth value. This is very good for a single canonical implementation of === per class, but does not help with more abstract and ad-hoc assertions.

So how can It be done in ruby 1.9?

In ruby 1.9 you can. Revisiting the term predicate, a definition:

something that is affirmed or denied concerning an argument of a proposition.

This sounds a lot like a function that takes a parameter returns a truth value; that sounds a lot like each test in a long if/elsif statement; and that sounds like a whole lot of fun, here we go:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

threeven = -> n { n % 3 == 0 }
fourven = -> n { n % 4 == 0 }
fiven = -> n { n % 5 == 0 }

rs = (0..10000).to_a.sample(30)

rs.each do |r|

  case r

  when -> n { n.zero? }
    puts "#{r} is zero"

  when fiven
    puts "#{r} is fiven"

  when fourven
    puts "#{r} is fourven"

  when threeven
    puts "#{r} is threeven"

  when -> n { n.even? }
    puts "#{r} is even"

  when -> n { n.odd? }
    puts "#{r} is odd"

  else raise "unpossible"
  end

end

two nice idioms:

when -> x { x.something? }
  ...

will apply the subject of the case statement to the lambda. This seems very handy for the simple expressions that are easy to read on their own.

approaching_critical_levels = -> n { n > 10e9 }
...
when approaching_critical_levels
  ...

will apply the subject of the case statement to the lambda bound to the name. This is very readable but requires a definition somewhere.

Both forms do the same thing, YMMV. But your mileage will be better than using an equivalent if/elsif statement.

Why does this work?

Ruby 1.9 doesn’t really have this feature per se. There are modifications to improve lambdas but that really isn’t the point. The point is that the semantics of case/when didn’t change, it still calls === as each clause’s predicate. Ruby 1.9 implemented Proc#=== such that the RHS operand is applied to the lambda. When used in the context of a case statement we get this gem of an idiom.

You don’t get more abstract or ad-hoc as the humble lambda and case statements sure are swell. What a nice fit, polymorphism FTW!

Permalink | Leave a comment  »

TL;DR Redis the AK-47 of databases

Easy

Installation was super easy, on a mac:

brew install redis

Starting it up was just as easy:

redis-server /usr/local/etc/redis.conf

Using it?

% redis-cli
redis> set this.is.a.key "this is a value"
OK
redis> get this.is.a.key
"this is a value"

Redis also supports more complex data stuctures, but the idea is the same: you set some state to a name. No tables, no schema, no JSON, no map nor reduce. If you don’t get the juxt of this, then please take out your safety pencil and a circle of paper.

Simple

The protocol is machine & human readable. Don’t believe me?

% nc localhost 6379
get this.is.a.key # i typed this
$15               # redis says 15 characters are coming
this is a value   # redis sent 15 characters

The stock config file has on the order of 30 options, 25 uncommented active ones:

% grep -v '^#\|^$' /usr/local/etc/redis.conf
daemonize no
pidfile /usr/local/var/run/redis.pid
port 6379
timeout 300
loglevel verbose
logfile stdout
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump.rdb
dir /usr/local/var/db/redis/
appendonly no
appendfsync everysec
vm-enabled no
vm-swap-file /tmp/redis.swap
vm-max-memory 0
vm-page-size 32
vm-pages 134217728
vm-max-threads 4
glueoutputbuf yes
hash-max-zipmap-entries 64
hash-max-zipmap-value 512
activerehashing yes

It’s not that hard to guess what most of these are, but why guess when stock config is so well documented? It seems like the useful 80% of what you can know about the server side is documented there.

There is one authentication method: a password, off by default. If you specify a password in the config file then its simple:

redis> AUTH thepasswordintheconfigfile
OK

Now you can use the database, just use it, seriously. But be warned:

Note: because of the high performance nature of Redis, it is possible to try a lot of passwords in parallel in very short time, so make sure to generate a strong and very long password so that this attack is infeasible.

Backup? Use cp, as in /bin/cp …

% cp /usr/local/var/db/redis/dump.rdb /some/place/to/backup.rdb

Need some more dials, knobs, domain specific configuration languages, obfuscated enterprise grade protocols, ldap authentication, Photoshop integration or some sort of bean? Call up Larry Ellison, he’ll be glad to provide you with an aspect certified enterprise-oriented solution.

Powerful

Does it do transactions? Yes

redis> MULTI
redis> ...
redis> ...
redis> ...
redis> EXEC

The above code will get exclusive rights to the dataset. All other clients' commands will queue up. EDIT: this explanation is wrong, commands are queued, clients are not locked, please see the the docs

But Redis also has optimistic locks that don’t exclusively lock the entire dataset.

redis> WATCH mykey
redis> ...
redis> ...
redis> ...
redis> EXEC

if mykey is modified by another connection any time between the WATCH and EXEC commands, a rollback will result. The important part is that if no one is stepping on anyone else’s toes (data) then there is no queueing. If by chance you do step on someone’s toe, just back off for second and try again.

Pub/Sub

Pub/Sub is a unique and useful feature. A client publishes messages on a channel and zero or more subscribers receive that message. Its kinda like IRC.

# client 1
redis> SUBSCRIBE the.chan
Reading messages... (press Ctrl-c to quit)
1. "subscribe"
2. "the.chan"
3. (integer) 1

# client 2
redis> PUBLISH the.chan "hey guise whats cookin"

# client 1
redis> SUBSCRIBE the.chan
1. "message"
2. "the.chan"
3. "hey guise whats cookin"

Not enough XML for you? Go fsync yourself.

No Surprises

Does it scale? Yep, via replication. One master to many slaves. The slaves ask the master for new data, then everyone has the same data.

The entire dataset is in RAM. This makes things as fast as well, RAM, always. What about when the power goes out? It’s periodically saved to disk.

No virtual memory by default. Everything really is in RAM. If you don’t have enough RAM then turn virtual memory support on.

What about between saves? A log (since the last save) is kept on disk. It can be replayed to recover unsaved data.

Is that log fsynced? Every second by default, but you can configure it.

There are potential windows of time when data can be lost. The important thing is that those windows are known and can be considered instead of surprising you.

AK-47 Appeal

Why Redis appeal to me? Pareto’s principle could be applied: When using software, one could argue that 80% of the time is spent using 20% of features. Redis seems to implement the vital few features very well.

According to AK-47 legend, assault rifles were not popular because of their tendency to consume large amounts of ammo. The Soviets embraced the idea and simply supplied their troops with more ammo. Kinda like Redis and RAM.

There are more and precise guns out there, but you can’t pack them with wet dirt and expect them to fire unconditionally. There are more sophisticated designs, but factories cannot crank them out anywhere in the world, 2nd world or 3rd world. This is because the design of the AK-47 is, for lack of a better term, simple. There is inherent robustness in simplicity. You can shoot an AK-47 if you have two things: an AK-47 and ammo. You can host or replicate a Redis dataset on any machine that has two things: Redis and enough RAM.

In the spirit of full disclosure, I’m a newb to Redis. My knowledge is basically the contents of this post (at the time of writing). I don’t use it in production (yet). Likewise I’m no expert in AK-47s (I’ve never even fired one) or guns in general. I’m aware via notoriety alone.

See also: Unix – The Hole Hawg, by Neal Stephenson

Permalink | Leave a comment  »

Can certain tactics give a players an edge at Uno? The following simulations play five players, one of which uses a tactic that presumably gives an edge, in 10,000 independent games of Uno.

Control: all random

This is important so we can compare how much of an edge plain old randomness can have.

edge is calculated by games won - ( number of games / number of players) as a percentage of all the games.

run-out is when there are no more cards to draw, the came cannot continue. I'm displaying them as positive because when you add all the numbers up they should total 0%.

So in this case, out of 10,000 games, player 2 came out ahead 0.04% of the time while player 5 fell behind 0.11% of the time, and nobody won 0.03% of the time (3 games) due to run-out. But really every player was within a small window of breaking even.

Choose the most abundant color

When playing a wild card you have a choice of which color to set for the next player to play. Reason dictates if you choose the most abundant color in your hand you are more likely to decrease the size of your hand. What happened when facing four random players against one who picks the most abundant color in their hand?

Wow, if player 1 just picked a color at random it probably wouldn't have made a difference. Keep in mind the random players don't even consider their own hand and might choose a color they don't have.

Play draw cards as soon as possible

If you have draw cards, play them soon right? Hit 'em hard!

hmm, I don't see anything noteworthy here. If you must infer something, infer that out of 10,000 games this tactic slightly loses.

Play draw cards as late as possible.

Don't play the draw cards soon, play them late. If draws are stacked you will be safe and it makes others more likely to have more cards when you have less.

We broke 1%, but player 5 was close and is only random. I don't consider it that significant.

In the spirit of full disclosure ...

These simulations are far from complete. The human interaction aspects like calling uno when a player has 1 card left, seeing the faces of other's cards accidentally, drawing more cards than needed to play, cheating, etc. are not simulated and probably effect the game.

I didn't consider the ability for a player to look at the size of another player's hand. These are key factors when I play skip or reverse cards. So i really didn't test any tactics that specifically use skips or reverses. Does anyone know a better way to play skips or reverses?

All my simulations test a strategy against 4 random players. The combinatorics of all possible tactics with possible games with 2 to 7 players is beyond my motivation for this casual exercise.

Externalities: does a strategy effect other players without giving the user a significant edge or handicap? I have no idea but if you are qualified to answer these questions please let me know.

Also, I could have made errors, I'm only human.

Is the case closed?

Will these tactics will give you an edge? Nope, I think we are fooled by randomness when applying them.

Have I made it less fun? A bit. Then again maybe one can focus on the harder to simulate, human aspects of the game with the peace of mind that the above tactics are safe to not consider.

Will I ever play Uno again? Sure, its fun foiling other people's tactics and seeing the looks on their faces. It is also a great way to kill time, the average number of rounds of these 40,000 games was about 24.

If you have a tactic you think has an edge, or have found a flaw in my reasoning let me know I'll check it out, or grab the code!

Why did I do this? To get an edge in Uno.

Permalink | Leave a comment  »

Permalink | Leave a comment  »

It sends out a range of signals that commonly turn off TVs. Take this thing to a sportsbar, bestbuy, or an old-folks home and have some fun.

My friends Marly & Christian at hackerspace let/helped me put this together. It wasn't that hard to wire up, but it was hard to wire up in a size that fits in a dental floss box. It was like making a ship in a bottle.

For the real scoop: Marly's post & Christian's post

Permalink | Leave a comment  »

Kombucha is a carbonated, slightly alcoholic, probiotic beverage that tastes like cider mixed with beer and champaign. it has numerous claims of health benefits and tastes great. One problem, it's pretty expensive, like $4 for a single serving bottle. Can I make it? so far so good.

My recipe: the dregs of a bottle of store-bought kombucha + room temperature sweet-tea. So far it looks as if the culture is growing healthy. It has the kombucha smell.

Permalink | Leave a comment  »

Bad data occurs when updating a Digest::SHA1 object with a string of size 512 MiB or larger.

code that shows

output

Digest::SHA1#file:        e0c0e4aafa03db337897c27eb2cc531efac674e8
updated < 512 MiB:        e0c0e4aafa03db337897c27eb2cc531efac674e8
updated = 512 MiB:        93174e243c120d7bca6851e3a2e014c415ab6605
updated = file size:        93174e243c120d7bca6851e3a2e014c415ab6605

this was discovered in ruby 1.8.7 (2009-06-12 patchlevel 174) [universal-darwin10.0] running on Mac OS X Version 10.6.5.

Linux checks out fine, I ran it on a few linux machines, no worries there.

patch

next step: debug ruby to see what is going on

Permalink | Leave a comment  »

Look at this thing!

Weighing in at 1.13lbs

It tasted pretty good, not the best, but hey, they don't let me taste apples before I buy them.

Permalink | Leave a comment  »

While in the checkout aisle I was pleasantly surprised by Stride's Mega Mystery gum.

It tastes pretty good, but that's not why it's noteworthy. I liked that I knew nothing about it and therefore couldn't rationally compare it to other gums.

My gum shopping experience went from:

pick the a gum from 10 or so options, weighing in flavors, price, trying new experiences, brands I've had good experience with, etc.

to:

this mystery gum or pick the a gum from 10 or so options, weighing in ...

no choosing, no weighing, I didn't even look at the price.

I suffer the Paradox of Choice, and I recommend this gum.

Permalink | Leave a comment  »

Best soy sauce ever (so far).

A new asian market, Eastern Market, opened up in Gainesville and has a very large selection of Kim Lan sauces. I went with the Piau Shiang variety.

Google translated 飄香醬油 into Fragrance soy sauce, and the description below:

金蘭飄香醬油加入特有的二次發酵醬汁，使風味、成份更加濃厚香醇，讓每道菜均色、香、味俱全，滿足全家的口慾。

translated into:

Jin Lan Jia Rute some secondary fermentation of soy sauce fragrant sauce to make the flavor, alcohol content is more intense, so that each dish are color, smell, and taste, to meet the whole family's appetite.

You know the non-salty flavor in soy sauce? Make it very pleasant, very big, and very aromatic. It's seems very close to Kim Lan's excellent regular variety I've bought before.

Permalink | Leave a comment  »

This map has UF, the airport, I-75 the major roads between them all, and nothing else. Its purpose is to get an intuitive feel for Gainesville. If someone gives you verbal directions and you understand this map my hope is that the majority of the route can be expressed by this map.

I-75 & the airport are the big ways to enter and leave the city, if you arrive/depart any other way you probably know a bit about Gainesville.

UF is a big deal. You are probably going here or some place relative to it.

The other items are major the roads, simple as that. 39th Avenue was a tough decision and it is only included because of the airport.

I didn't choose the items on the map per se, as much as I chose to omit everything non-essential.

Permalink | Leave a comment  »