Alice, Bob and Mallory

JavaScript hash table keys

Jonas Elfström — Fri, 05 Mar 2010 17:42:00 +0100

In JavaScript you can add properties to objects dynamically. You can access those properties both by object.foo and object['foo']. The later is commonly used to use JavaScript objects as hash tables (associative arrays).

While implementing a simplistic unique random number generator I happened to use keys(obj). Unfortunately keys(obj) is part of ECMAScript 5. See chapter 15.2.3.14 in ECMA-262. The web browsers of today mostly implements ECMAScript 3.

Here's an implementation of keys(obj) for ECMAScript 3 browsers (tested in Google Chrome, IE8 and Firefox 3.5). If the browser already has a keys function then nothing will be done.

if (typeof keys == "undefined") 
{ 
  var keys = function(obj) 
  {
    props=[];
    for (k in ht) if (ht.hasOwnProperty(k)) props.push(k);
    return props;
  }
}

The simplistic unique random number generator looks like this

function uniqueRndNumbers(min, max, quantity) {
  var ht={}, i=quantity;
  while ( i>0 || keys(ht).length<quantity) 
    ht[Math.floor(Math.random()*(max-min+1))+min]=i--;
  return keys(ht);
}

This function has not undergone any serious testing. Also if the quantity is more than a fraction of (max-min) then another algorithm like the Fisher–Yates shuffle might be a better choice.

Ira Glass on Storytelling

Jonas Elfström — Wed, 03 Feb 2010 23:48:00 +0100

I don't live on that same continent as Ira Glass and because of that I can't listen to the wonderful This American Life on the radio so instead I subscribe to the podcast. Every time a new episode shows up that is the first thing I listen to. There are a lot of great podcasts out there but if I had to choose only one, I think I would go for This American Life (sorry RadioLab, I love you). Recently I learned that the host and producer of the show, Ira Glass, can be found on YouTube talking about storytelling. It's amazing how he covers how he thinks you should tell a story for a radio/TV show. I recognize how the show executes that narrative but it also doesn't take anything away from the fact the show is excellent and that there are a lot of workmanship and talent put into it.

Ira Glass on Storytelling #1

Ira Glass on Storytelling #2

Ira Glass on Storytelling #3

Ira Glass on Storytelling #4

C# implicit string conversion

Jonas Elfström — Wed, 03 Feb 2010 17:32:00 +0100

I know how it works and I think I can see why but I'm still not very fond of how eager C# is to perform implicit string conversion.

Contrived example:

12	string s = -42 + '+' + "+" + -0.1 / -0.1 + "=" + (7 ^ 5) + " is " + true + " and not " + AddressFamily.Unknown;

s will be set to "1+1=2 is True and not Unknown"

The answer is in white text above, select the text to see it.

A more real problem is something like this

string str = 1 + 2 + "!=" + 1 + 2;

str will be set to "3!=12".

Edit 2010-02-08
This wouldn't be much of a problem if all objects in .NET always returned a decent string representation of their current state/value with ToString() but that's not the case. Instead "The default implementation returns the fully qualified name of the type of the Object.".
I don't like the inconsistency. It's way too late now but I think it would have been much better if only objects that really produces a human readable output of the data in the object should implement ToString(). If you want the name of the type of the Object there should be another way.

String concatenation in Ruby

Jonas Elfström — Tue, 02 Feb 2010 00:04:00 +0100

There's no StringBuilder class in Ruby because the String class has the << for appending. The problem is that not every Ruby programmer seems to be aware of it. Recently I've seen += being used to append to strings where << would have been a much better choice.

The problem with using += is that it creates a new String instance and if you do that in a loop you can get really horrible performance.

If you are dealing with an array you don't even have to use << because Array#join is even faster and shows intent in a nice way.

require 'benchmark'

array_of_random_strings=(0...262144).map{65.+(rand(25)).chr}.join.scan(/.{1,8}/m)

Benchmark.bm do |benchmark|
  benchmark.report do
    str=array_of_random_strings.join
  end
  benchmark.report do
    str2=""
    array_of_random_strings.each do |s|
      str2<<s
    end
  end
  benchmark.report do
    str3=""
    array_of_random_strings.each do |s|
      str3+=s
    end
  end
end

The array_of_random_strings is an array of 32768 8 characters long random strings.

user	system	total	real
0.030000	0.000000	0.030000	( 0.027184)
0.160000	0.010000	0.170000	( 0.190277)
106.020000	0.300000	106.320000	(113.457793)

The performance of += was even worse than I imagined!

Finding primes in parallel

Jonas Elfström — Thu, 14 Jan 2010 22:55:00 +0100

Justin Etheredge has been blogging about his challenge to find prime numbers with LINQ. He later used AsParallel() (coming in .NET 4) to speed things up and then followed that up with a post about using The Sieve Of Eratosthenes.

As you can see in the comments of those posts I tried to speed the Sieve of Eratosthenes up by using Parallel.For in the inner loop. I also tried AsParallel() in the LINQ expression but it made no difference in either case. At most it got 5% faster. I'm not sure but it could be that because SoE is very memory intense we could have a scaling issue and maybe also memory bandwidth exhaustion. This is mere speculation.

I then searched for other algorithms and found The Sieve of Atkin. It uses less memory than SoE so I thought I'd give it a try.

I set the limit to 20,000,000 and then benchmarked it. It timed in on 2.48s so actually worse than the 2.2s that SoE took. Not good! Then I added Parallel.For in the loop that did most of the work and lo and behold, it scaled! I have two cores in my machine (T7200@2.0GHz) and the average runtime went down to 1.26s. That's almost linear and surprisingly good! If you happen have a quad core (or more) and feel like trying it out then please contact me. It would be interesting to see if it scales further.

static List<int> FindPrimesBySieveOfAtkins(int max)
{
    var isPrime = new BitArray((int)max+1, false);
    var sqrt = (int)Math.Sqrt(max);

    Parallel.For(1, sqrt, x =>
    {
        var xx = x * x;
        for (int y = 1; y <= sqrt; y++)
        {
            var yy = y * y;
            var n = 4 * xx + yy;
            if (n <= max && (n % 12 == 1 || n % 12 == 5))
                isPrime[n] = !isPrime[n];

            n = 3 * xx + yy;
            if (n <= max && n % 12 == 7)
                isPrime[n] = !isPrime[n];

            n = 3 * xx - yy;
            if (x > y && n <= max && n % 12 == 11)
                isPrime[n] = !isPrime[n];
        }
    });

    var primes = new List<int>() { 2, 3 };
    for (int n = 5; n <= sqrt; n++)
    {
        if (isPrime[n])
        {
            primes.Add(n);
            int nn = n * n;
            for (int k = nn; k <= max; k += nn)
                isPrime[k] = false;
        }
    }

    for (int n = sqrt + 1; n <= max; n++)
        if (isPrime[n])
            primes.Add(n);

    return primes;
}

This is C# 4.0 code, compiled in Visual C# 2010 Express Beta 2.

Edit 2010-01-20

Indications are that this does in fact not scale very good on a quad core. It's even worse, it seems it scales good on my old T7200 but not on a dual core E6320. I don't know why but of course the shared state of the isPrime BitArray is a huge problem and maybe it could be that differences in CPU architecture (FSB speed, caches and so on) in the E6320 is an explanation. Average execution time on the E6320 was 1290ms in a single thread and 1064ms in two.

If you want to try this in an older version of C# than 4.0 then check out this post.

A reader asked how I timed the executions. Here's how.

var steps = new List<long>();
var watch = new Stopwatch();

for (int i = 0; i < 10; i++) 
{
    watch.Reset();
    watch.Start();
    var primes = FindPrimesBySieveOfAtkins(20000000);
    watch.Stop();
    Console.WriteLine(watch.ElapsedMilliseconds.ToString());
    steps.Add(watch.ElapsedMilliseconds);
}
Console.WriteLine("Average: " + steps.Average().ToString());

Comparing instance variables in Ruby

Jonas Elfström — Mon, 02 Nov 2009 22:50:00 +0100

Say you have two objects of the same class and you want to know what differs between them. Well actually you just want to know the instance variables in object b that differs from the ones in object a.

To begin with, we need a class. I like cheese.

class Cheese
  attr_accessor :name, :weight, :expire_date
  def initialize(name, weight, expire_date)
    @name, @weight, @expire_date = name, weight, expire_date
  end
end

Then we need some ~~cheese~~ objects.

12	stilton=Cheese.new('Stilton', 250, Date.parse("2009-11-02"))gorgonzola=Cheese.new('Gorgonzola', 250, Date.parse("2009-11-17"))

With only name, weight and an expiration date it would be easy to compare those but imagine that these two objects has 42 properties. It does not stop there, you are being asked to compare 24 different classes in this way. Are you cringing yet?

Object#instance_variables to the rescue! Well, that and a small hack by me. Below I add a new method called instance_variables_compare to Object. The long method name is because I wanted to follow the naming already in place. Usually I prefer to do these kind of things as a module and then include them where appropriate but in this case I find that a monkey patch will do.

class Object
  def instance_variables_compare(o)
    Hash[*self.instance_variables.map {|v|
      self.instance_variable_get(v)!=o.instance_variable_get(v) ? 
      [v,o.instance_variable_get(v)] : []}.flatten]
  end
end

It returns the instance variables that differs as a hash because it's handy and because I like it that way.

>> stilton.instance_variables_compare(gorgonzola)
=> {"@name"=>"Gorgonzola", "@expire_date"=>#<Date: 4910305/2,0,2299161>}
>> gorgonzola.instance_variables_compare(stilton)
=> {"@name"=>"Stilton", "@expire_date"=>#<Date: 4910275/2,0,2299161>}
>> stilton.expire_date=gorgonzola.expire_date
=> #<Date: 4910305/2,0,2299161>
>> stilton.instance_variables_compare(gorgonzola)
=> {"@name"=>"Gorgonzola"}
>> stilton.instance_variables_compare(stilton)
=> {}

If you ever think of using this code you should be aware of two things.

This code is very untested and comes with no guarantees.
Since instance variables spring into life the first time they are assigned to you either have to work with objects that always initialize everything or you have to change instance_variables_compare to handle this.

Infinite ranges in C#

Jonas Elfström — Tue, 20 Oct 2009 20:41:00 +0200

I recently learned that C# is compliant with the IEEE 754-1985 for floating point arithmetics. That wasn't a big surprise but that division by zero is defined as Infinity in it was! It actually kind of bothers me that I didn't know this.

In mathematics division by zero is undefined for real numbers but I guess Infinity is a more pragmatic result. Or as a friend put it "IEEE stands for Institute of Electrical and Electronics Engineers not Institute of Mathematics"

1234	double n = 1.0;n = n / 0;if (n > 636413622384679305) System.Console.WriteLine("Yes it certainly is!");

This C# code does not throw an exception, it simply leaves n defined as Infinity and a line written to the console.

Ruby is also IEEE 754-1985 compliant. It even lets you define infinite ranges.

Infinity=1.0/0
=>Infinity
(1..Infinity).include?(162259276829213363391578010288127)
=> true
(7..Infinity).step(7).take(3).inject(&:+) # 7+14+21
=> 42

I can't say I see very much use of this but it brings a kind of completeness to the handling of infinities. Unfortunately it seems we don't get that in C# out of the box because Enumerable.Range takes <int>,<int> as parameters and there's no Infinity definition for int. That's unless someone wrote a generic Range class. Turns out none other than Jon Skeet did in his MiscUtil. Dowload MiscUtil and then by using MiscUtil.Collections; you can:

double n = 1.0;
var infinity = n / 0;
var r = new Range<double>(0, infinity);
if (r.Contains(4711))
  System.Console.WriteLine("Yes it certainly does!");
var sum = r.Step(7.0).Take(3).Sum();

And guess what, it works like a charm! 4711 is part of positive infinity and sum is 42.0 and all is good.

Edit

There's also a couple of predefined constants. Thanks to Eric for pointing that out.

12	var r = new Range<double>(7, System.Double.PositiveInfinity);var sum = r.Step(7.0).Take(3).Sum();

Counting the number of Google Readers

Jonas Elfström — Mon, 19 Oct 2009 21:55:00 +0200

I run this blog on a 9 year old laptop hidden in a cabinet in the living room. It's not a powerful machine but it has been up to the job since I turned it into a web server 7 years ago. This could maybe be one of the last HP Omnibook 4150b still in use, at least it has to be in a very exclusive club of laptops being switched on for the past 7.5 years. Recently I've seen an increase in traffic and especially from Feedfetcher-Google. It so happens that Feedfetcher also shows the number of subscribers.

[19/Oct/2009:22:01:19 +0200] "GET /xml/rss20/feed.xml HTTP/1.1" 304 0 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 4 subscribers; feed-id=7686756599804593322)"

The above is only one out of five different feed-ids because I have both atom and rss and for a short while this blog was at another address. The fifth feed is actually myself subscribing to the comments.

I'm not using FeedBurner so I can't get my statistics from there but I still wanted to be able to see the number of Google Readers of my blog (as far as I can see I only have one other type of subscriber).

Usually I script anything more advanced than a grep in Ruby but this time I made an exception and stayed in Bash.

tail -1000 /www/logs/access.log |
grep Feedfetcher |
cut -d ";" -f 4 | sort -u |
while IFS= read -r line
do
   tac /www/logs/access.log | grep -m 1 $line
done |
sed 's/^.*html; \([0-9]*\) subscribers.*/\1/' |
awk '{tot=tot+$1} END {print tot}'

Most certainly this can be optimized in a number of ways. Don't be shy, just tell me!

So what's going on there? Well, first I get the last 1000 rows from my access log and right now my traffic is so low that that is way more than I really would have to. Then I get all unique feeed-ids from the rows containing Feedfetcher. I pipe those to a loop that gets the very last access for each one of them. Then I parse out the number of subscribers with a regexp in sed and count them with awk .

It turns out that I have a whopping number of 14 15 subscribers and I am one of them.

The Thrush combinator in C#

Jonas Elfström — Tue, 06 Oct 2009 20:35:00 +0200

Last year I read Reg "Raganwald'" Braithwaite's excellent post The Thrush and he explains it as

The thrush is written Txy = yx. It reverses evaluation.

Back then I didn't even consider trying to implement it in C#. That was before I digged deeper into lambda expressions and extension methods in C# 3.0 and way before last night when I read Debasish Ghosh's post on how to implement the Thrush in Scala. After reading that my first thought was if it was possible to do the same in C#. Here's my attempt.

At first I struggled with the static typing and headed for an easy way out using Object in the extension method of Object:

12	public static object Into(this Object obj, Func<object, object> f){ return f.Invoke(obj); }

My goal was to translate the Ruby example

(1..100).select(&:odd?).inject(&:+).into { |x| x * x }

in Raganwald's post to C#.

Which reads "Take the numbers from 1 to 100, keep the odd ones, take the sum of those, and then answer the square of that number."

But with the Object based extension method I had to do some ugly casts.

var r = Enumerable.Range(1, 100).Where(x => Odd(x)).Sum().Into(x => (int)x * (int)x);

With som added typing I could do:

var result = Enumerable.Range(1, 100).Where(x => Odd(x)).Sum().Into(x => x * x);

But that merely moved the cast to the ext. method and also made it work for integers only.

12	public static int Into(this Object obj, Func<int, int> f){ return f.Invoke((int)obj); }

Then I remembered generics and method type inference.

12	public static T Into<T>(this T obj, Func<T, T> f){ return f(obj); }

And now not only the casts were gone but I also got a thrush combinator almost as flexible as the one in Ruby.

Contrived example follows:

12	var test = "ball";var ball = test.Into(s => "Are we having a " + s + " yet?");

The odd part

The Odd(x) method call in the calculation above is a plain static method.

12	private static bool Odd(int n){ return (n % 2 != 0); }

If you want an even more terse syntax you could try an ext. method on IEnumerable like this:

12	public static IEnumerable<int> Odd(this IEnumerable<int> en){ return en.Where(n => n % 2 != 0); }

Gives:

var result = Enumerable.Range(1, 100).Odd().Sum().Into(x => x * x);

Also as a general alternative to .Sum() I could have used .Aggregate((x, y) => x + y)) but I found it a bit verbose.

In C# I don't think it's possible to pull off the Symbol#to_proc stuff that Ruby does. That's the &: in the select(&:odd?) and the inject(&:+) in the Ruby example. Raganwald has a great post on that too.

Edit

Check out Jon Skeet's nice answer on StackOverflow to my question on how to make this even more Ruby-like. I have to try out that Operator class later though.

Edit 2009-10-07

One thing I found a bit surprising is that by implementing the Into ext. method in this way it not only works for all objects based on System.Object but it also works for value types.

1234	int n=4711;int oddOrZero = n.Into(x => x % 2 !=0 ? x : 0); // 4711n = 4712;oddOrZero = n.Into(x => x % 2 != 0 ? x : 0); // 0

Edit 2009-10-12

My confusion did stem from my lack of understanding of extension methods. Ex. methods are in fact not extending System.Object or any other type, they are "nothing more than a pleasant syntax for calling a static method" in case no instance method with the same name can be found.

A case for using only three different digits in keypad codes

Jonas Elfström — Sun, 27 Sep 2009 21:02:00 +0200

Keypads have obvious security problems and keypads accepting a stream of digits with no # or enter in between, while checking for the four digit long code, are even worse.

The important part is to not leak the digits in the code by wear or intentional markings because if they leak it's suddenly very far from 10000 combinations.

If the "lock picker" only knows that the code contains four digits there are 10000 combinations. Keypads accepting a stream of digits can then be opened in a maximum of 10003 keystrokes using the De Bruijn sequence. That is still quite a lot.

Below is a Ruby implementation of the De Bruijn sequence.

# De Bruijn sequence
# Original implementation by Frank Ruskey (1994)
# translated to C by Joe Sawada
# and further translated to Ruby by Jonas Elfström (2009)

@n=4
@k=10
@a=[0]
@sequence=[]

def debruijn(t, p, alike)
  if t>@n
    if @n%p==0
      1.upto(p) {|j| @sequence<<@a[j]}
    end
  else
    @a[t]=@a[t-p]
    if @a[t]>0
      debruijn(t+1,p,alike+1)
    else
      debruijn(t+1,p,alike)
    end
    (@a[t-p]+1).upto(@k-1) {|j|
      @a[t]=j
      debruijn(t+1,t,alike+1)
    }
  end
end

debruijn(1,1,0)
print @sequence.join

It's not uncommon to find keypads with 4 of the 10 keys worn down and if you do you can be pretty sure that the code contains those four different digits. The number of possible combinations are 4! = 4x3x2x1 = 24. I got curious to see if there's a kind of De Bruijn sequence for this that brings down the 4*24=96 keystrokes. By scribbling in a text editor I quickly realized there's not a clean sequence. Not clean in the way that a sequence following the rules can be created. Also it's probably even quite daunting to present it as mathematically dense and beautiful as the De Bruijn but that could be my less than great combinatorics speaking.

I made a quick and dirty brute force hack to try to find a shorter sequence.

seq=[]
1.upto(4) {|a| 1.upto(4) {|b| 1.upto(4) {|c| 1.upto(4) {|d|
  seq << "%d%d%d%d" % [a,b,c,d] if !(a==b || a==c || a==d || b==c || b==d || c==d)
}}}}

s=seq[0]
seq.delete_at(0)
while (seq.length>0)
  next_code=(seq.select {|c| c[0..2]==s[-3..-1]})
  if next_code.empty?
    next_code=(seq.select {|c| c[0..1]==s[-2..-1]})
    if next_code.empty? 
      next_code=(seq.select {|c| c[0]==s[-1]})
      s+=next_code[0][1..3]
      seq.delete_at(seq.index(next_code[0]))
    else
      s+=next_code[0][2..3]
      seq.delete_at(seq.index(next_code[0]))
    end
  else
    next_code=(seq.select {|c| c[0..2]==s[-3..-1]})
    s+=next_code[0][3].chr
    seq.delete_at(seq.index(next_code[0]))
  end
end

The above code takes the first code "1234" of the 24 and then searches the rest of the array for a code beginning with "234". It finds "2341" and adds "1" to the end of s and continues to look for "341" and so on. Relatively soon there is no three digit match and then it tries two digits and eventually even that fails and then it gets the first one digit match. The resulting sequence is:

123412314231243121342132413214321

From 96 to 33 keystrokes. Not as effective as De Bruijn but still significant. Unlike De Bruijn I have absolutely no proof that this is the shortest one possible but it seems likely. Also notice that in the middle of the sequence we find "3121" and "1213". Those break the criteria of four different digits but they seem to be necessary to be able to enter the reversed mode. Try reading the sequence forward and backwards to see what I mean.

If the code only contains two digits it's gets even more trivial to try them all. There are 14 possible codes and by compressing those to one sequence you get down to 20 keystrokes.

Things get a little more interresting if three buttons are worn. It turns out that the repeated digits can be placed in the code in six different ways.

0012,1002,1200,0102,0120,1020

That's 6x2x3=36 combinations and, maybe a little unintuitive, 12 more than if you are using four different digits. I compressed it down to 49 key strokes. Unlike the sequence for four different digits I can't find it with google and I know it's kind of security by obscurity but that could give a tiny amount of extra trouble to an attacker. I will not be the first one publishing it.

Be aware that if an attacker knows you are using a 0012-like code he gets a smaller space to search. 6x8x9x10=4320 instead of 10000. You have to weight the risk of button leaks against a code protocol leak.