Disable the ←’s in Rails on Windows

If you run the demo Rails server on Windows you’ll soon see something like this in the console:

Started GET "/rails/info/properties" for 127.0.0.1 at 2011-03-05 19:45:31 +0200
Processing by Rails::InfoController#properties as HTML
←[1m←[36mSQL (0.0ms)←[0m ←[1m SELECT name
FROM sqlite_master
WHERE type = 'table' AND NOT name = 'sqlite_sequence'
←[0m
Rendered inline template (0.0ms)
Completed 200 OK in 80ms (Views: 1.0ms | ActiveRecord: 0.0ms)


The weird sequences with the arrows are ANSI escape codes which are supposed to make the output colorful (the arrows represent the ASCII “escape” character). Unfortunately, cmd.exe (the Windows console) doesn’t support these sequences and displays them in their raw ugliness.

On Rails 3, you can tell Rails not to use these sequences by setting the colorize_logging variable to false. Add these lines to the Application class in application.rb:

    # Windows cmd.exe doesn't support ANSI colors, so disable them
    config.colorize_logging = false

If you’re wondering how some Windows programs such as PowerShell have color output in the console, the answer is that they probably use the Windows API function SetConsoleTextAttribute for this.

win32console

An alternative solution is to to use win32console. This is a ruby module which overrides ruby’s output methods so they emulate ANSI color support. To use it, add this line to your application’s Gemfile (located at the application root directory):

    gem 'win32console', :platforms => :mingw

Why does the other lane always seem to go faster?

Because it does go faster. Well, not always, but more often than your own lane is faster.

Before I tell you why this is, let’s consider a simpler scenario for a moment. You’re driving in a congested single lane road. For the sake of this thought experiment, assume that the road is made of two equal parts. Through half of the length of the road the traffic moves relatively fast, but through the rest of the length the traffic moves slower. Let’s call the fast part A and the slow part B. Then you’ll spend more time at part B than at part A. If for example the traffic moves twice as fast in part A than in part B, you’ll spend 1/3 of the driving time in part A and 2/3 of the time at part B.

So even though the slow and the fast parts are equal in length, they are not equal in time. And this is true no matter how the slow and fast parts are distributed throughout the road. The fast and slow parts could be interleaved for example, so that for the first km the traffic goes fast, the second km the traffic is slow, and so on; in the end the result is the same: you spend more time in the slow parts.

Now back to the original question. The road has two lanes (let’s call them X and Y) and they’re both made of “fast” segments and “slow” segments. It’s safe to assume that the lanes are symmetric, in the sense that the total length of segments where X is faster than Y is equal to the total length of segments where Y is faster than X. (Even if the lanes aren’t identical, the probability of you choosing to enter each one of them is 50:50, so statistically speaking the assumption holds).

Let’s say that you only stay in one lane throughout the whole road. Then by the assumption above, the length of the road where your own lane moves faster than the other lane is equal to the length of the road where your lane moves slower. But remember that if you move through a road composed of a slow part and a fast part of equal lengths, you’ll spend more time at the slower part. Consequently, you spend more time at the segments where the other lane is faster! Just like in the first example, even though the two types of segments are equal in length, they’re different in time.

So next time you’re in a traffic jam and wondering why the cars in the other lane go faster, remember that Murphy doesn’t hate you; it’s just that you drove fast through the fast parts.

What happens if you move between lanes? It depends on your timing and on the distribution of the fast and slow segments along the road. For example, it seems to be the case that if a segment along the lane is slow, it says nothing about the probability that the next segments is slow (the events are statistically independent). If the segments are short, then by the time you change the lane the traffic could already move in a different pattern and you gain nothing. In that case, it would seem to you that the lane became slower when you moved to it, when in fact you just experience the same effect as staying in the original lane.

A demo HTTPS server with OpenSSL

In this post I’ll show how to set up a simple HTTPS server with OpenSSL. I needed to learn this in order to research a problem regarding client certificate authentication on .NET. However, the instructions here contain some sub-steps that can be useful to know in other situations. The instructions are for Windows but, again, they may be useful on other platforms.

We’ll be using the openssl utility and ca.pl, supplied with the OpenSSL package. Before you begin, make sure the directory containing OpenSSL binaries is in the system path.

The openssl utility can already serve as a web server. We just need certificates for the server and for the client. We’ll begin by generating a sample Certificate Authority (CA). The certificates will be signed by this CA.

Generating the CA

To generate the CA, type

ca.pl -newca

You can press Enter to use the defaults for all fields except common name (type some identifier you’ll want to use for the CA) and password (must be at least 4 chars). You’ll be asked to enter a password three times; enter the same password at each time.

You now have a demo CA in the folder demoCA. Create an X509 .cer file that can be imported in the Windows trusted root certificates store:

openssl x509 -outform der < demoCA\cacert.pem > cacert.cer

Now import the certificate into the trusted root certificates store. Remember to remove the certificate from the store when you’re done, to reduce security risks.

Creating the server certificate

ca.pl -newreq

Use default values for everything except the common-name. For the common name, type the server name (e.g. http://www.example.com). Make sure that the name resolves to the local machine. You can simply use the local machine’s name, or add an appropriate entry to the hosts file.

ca.pl -sign

Type in the password for the CA key, and accept the certificate. Now you have the certificate in newcert.pem and the private key in newkey.pem. Make a unified PEM file for use with OpenSSL:

copy newcert.pem+newkey.pem server.pem

Creating a user certificate

The first two steps are more or less what the same as when you created the server certificate.

ca.pl -newreq

Use default values for everything except the common-name. For the common name, type a display name for the user. Sign the certificate:

ca.pl -sign

To be able to import the certificate and the associated private key into the Windows personal certificate store, we’ll need to create a pfx file. (The file is PKCS #12 format. Apparently, Firefox can also use this format.)

openssl pkcs12 -export -in newcert.pem -inkey newkey.pem > user.pfx

Import the certificate into the browser, and you’re ready for the big moment.

Running the server

openssl s_server -www -Verify 1 -CAfile demoCA\cacert.pem

You’ll need to enter the password for the server’s private key.

This will start a server on port 4433. Start up IE, and navigate to https://hostname:4433, where hostname is the name you typed earlier into the server certificate’s common name field. If everything was set up correctly, you’ll be prompted for a client certificate. After selecting the client certificate you’ll see a status page sent by OpenSSL.

Troubleshooting

On Windows, you may see an unable to write ‘random state’ error with every command.The commands work nonetheless. I believe it can be safely ignored in this context; it just means the private keys will be less than 100% random, which is OK for our purposes (but probably not for production use).

If you get a TXT_DB error number 2 error message when signing the certificate, it could mean that a certificate with the requested serial number is already registered in the CA database. You can edit the file demoCA/serial to change the next assigned serial number to something else.

How to resize a VMware disk

These instructions worked for me for resizing guest Windows partitions hosted with VMware Workstation 6.5.1.

1. Back up your data. The steps below include messing with the virtual machine files and partitions and could conceivably destroy all your data (although they worked reliably for me so far).
2. Power off the virtual machine. Note: in order to resize Windows NTFS partitions you must shut down Windows properly.
3. The next step won’t work if you have snapshots. If you don’t need older snapshots, delete them using the Snapshot Manager.
4. Resize the disk:

   vmware-vdiskmanager.exe -x 36GB myDisk.vmdk

   The disk is now bigger, but the partitions inside it stay the same. You’ll need to edit the partition table. See next steps…

5. Dowload the GParted live CD. GParted is an open-source partition editor.
6. In the VM Settings dialog box, set the CD drive to use the GParted ISO image file you’ve downloaded.
7. Make sure the CD drive is before the hard drive in the virtual machine’s BIOS boot order. To do this, add the following entry to the VM’s .vmx file:

   bios.forceSetupOnce = "TRUE"

8. Power on the virtual machine. Because of the previous step, you’ll enter the BIOS setup utility.
9. Reorder the boot sequence if necessary so that CD drive will be before the hard drive. Save the settings and exit the setup utility.
10. GParted Live should load now. The main window should appear after pressing Enter at a few prompts. When the main window appears, edit the partitions as necessary.
    (For some reason I have strange problems with the mouse pointer in GParted, so I use the keyboard). When you’re done, choose Edit -> Apply All Operations.
11. Disconnect the ISO image, and restart the VM.

What can the eSlick do?

Before buying my eSlick e-reader, I looked around and saw many reviews that described its functions, but none of them answered the question: what can you actually do with it? Can you read academic papers? How about ePub books?

Of course, the specs says it supports PDF and other formats, but it says nothing about the reading experience of these formats.

Now, after using an eSlick for a few weeks I can attempt to answer this question.

There’s only one thing that the eSlick does well: display PDF documents that are sized for its display. Most of the stuff you’ll want to read will be in another format (or created for a bigger page size) but that’s not a problem if you can convert it to the required format.

Here’s how I read different kinds of texts using the eSlick:

PDF’s created for a regular (about A4) page size

Since the eSlick’s display is smaller than that of a regular page, you can’t view these files directly. There are 3 ways that you can handle them.

(1) Zoom out: Not useful for prolonged reading since the text looks tiny.

(2) Normal size, but move around with the arrow keys: Very annoying, unusable.

(3) Reflow: (In reflow mode the eSlick rearranges the text and adapts it for its display). Doesn’t always work. Sometimes the text is arranged in the wrong order. When it does work it works pretty nice. It’s good enough for me that I’m now reading an entire book in reflow mode. There’s a bug that if a paragraph is split between two pages, the second part isn’t shown; I have to turn off reflow mode, finish the paragraph, and then turn on reflow again. However, this procedure is easy enough that it doesn’t hurt the reading experience very much.

Two-column academic articles

Reflow doesn’t work for the articles I’ve tested; the text comes out in the wrong order. However, due to a wonderful coincidence, each column in this type of text is in about the same width as the eSlick display. A program called PaperCrop renders the PDF and then automatically segments the resulting image into columns. The result can be saved as a PDF file that can be read on the eSlick. I’ve tested PaperCrop on one article and it worked perfectly, detecting columns and diagrams correctly.

EDIT Papercrop also knows how to reflow one-column articles! Just choose the “reflow” preset.

ePub files

This is a popular eBook format. You can download free books from the Project Gutenberg site in this format. eSlick supports it natively (in newer firmware versions) but not very well. It doesn’t even support text styles such as bold and italics. Also, it doesn’t display Hebrew text (and probably other non-Latin languages). The solution is to convert the ePub to PDF. I’ve used an online converter successfully on Hebrew books. (In that site, set the target ebook reader to Kindle as it has the same display size). However, it failed to convert some of the books. I’m looking for other ways to do it.

An open source command line tool called epub2pdf looks promising for this end but I couldn’t make it to work.

Web pages

This is the procedure I’m using to create eSlick-compatible PDFs from interesting web pages. First, use readability to remove all the supporting extras — sidebars, footers and so on — leaving only the content. (In the readability configuration, set the font to small and margins to extra-narrow). Next, print it to PDF using the Foxit PDF Creator. I’m doing it on Explorer since the PDF Creator stopped working for me from Firefox after the first time I’ve used it. (It doesn’t look a very high quality software).

Even though the PDF Creator is bundled with the eSlick, it doesn’t have the eSlick listed as a recognized target in its settings, so you have to set a custom page size. I’m using 13.2cm x 9.9cm. (Derived from the display aspect ratio and an estimate of the display width, as I couldn’t find the height and width specified anywhere).

String literals with embedded nulls in Boost

Boost’s string library makes global string replace in C++ easy:

std::string str = "$greeting, world!";
boost::replace_all(str, "$greeting", "Hello");
std::cout << str << "\n"; // print "Hello, world!"

Suppose however that you want the search string to contain a null character. Why would you want such a thing? Consider escaping strings for safe inclusion in some contexts:

std::string str2 = get_string_from_remote_source();
boost::replace_all(str2, "\0", "(nul)");
// do something with str2 that depends on it not to contain null chars

(Remember that, unlike C strings, it’s perfectly valid for C++ standard library strings to contain NUL characters.)

Alas, the code above doesn’t work; the call to replace_all() doesn’t do anything. It turns out that when you given Boost a string literal, it uses strlen() to get the string’s length. Since strlen() works on C style null-terminated strings, it stops on the first null character it sees.

Why did I expect Boost to behave differently? In C++, string literals are array of chars. With the help of some template magic, the Boost library can know the string’s length at compile-time. It doesn’t need to rely on functions like strlen() to compute the string’s length, so it can handle arbitrary string literals, including ones with embedded nulls.

After some thinking and googling about it, it becomes clear why Boost doesn’t work this way, or at least why it isn’t the only reasonable way. The reason is that Boost cannot tell the difference between string literals and other character arrays. Consider this case:

char search[80];
strcpy(search, "foo");
boost::replace_all(str, search, "bar");

We probably wouldn’t want replace_all() to look for the whole 80 character long string which the input array happens to contain, but only for the part initialized with a null-terminated string. Actually, this seems to be some sort of gray area. When a zero character appears inside a string literal, it certainly means that the programmer intended the character to be a part of the string. But when it appears inside another character array, it may or may not mark the end of the string.

We need a way to tell Boost you want to treat a char array as an array instead of a null-terminated string literal. To do this, wrap the array in a call to boost::as_array. For example:

char nullchar[] = {'\0'};
boost::replace_all(str2, boost::as_array(nullchar), "(nul)");

In fact, you can also pass a string literal to as_array, but remember that the corresponding array contains an (additional) terminating null character. So, returning to the original problem, for a string containing solely one null character, use boost::as_array(“”). Don’t use boost::as_array(“\0”), as the latter will contain two characters.

CA2W found

Here’s a problem that was driving me crazy for a while. I got the following error trying to use ATL’s CA2W class in a C++ program:

error C3861: ‘CA2W’: identifier not found

Usually problems like this occur when you don’t include the correct header file. But in this case, I did include it (atlconv.h), exactly like the documentation says.

I made sure that CA2W is really defined in the header. I checked other things, but the result was always that the compiler should have seen the definition. Still the nefarious error message appeared. So why did the compiler pretend not to know this symbol? Finally I saw this line at the beginning of atlconv.h:

namespace ATL
{

D’oh!

So CA2W did exist, but only in the ATL namespace. I never noticed this before; it turns out that by default projects created with Visual C++ include atlbase.h which does “using namespace ATL”. Including this file solved the problem.

The documentation says nothing about this, of course. I couldn’t find any mention of the word “namespace” in there.

Incidentally, looks like we’ll be seeing this problem a lot in the future since the default has changed in Visual Studio 2010.

A TCP proxy in Ruby

A TCP proxy (or a tunnel, or a bridge) is a program that listens at a certain network address for connections. Whenever a connection is made to that address, the program connects to another predefined network address and starts transferring data between the two ends.

The reason I wanted a TCP proxy is this: I needed to run a program on a virtual machine. This program needs Internet access, but I couldn’t make the VM’s Internet access to work — it could only connect to programs on the host machine, i.e. my computer. I thus used a TCP proxy on the host machine to connect the VM to the outside world. (The fact that the program running on the VM needed to access only one predefined network address simplified things greatly).

Below is a Ruby script I used, made from bits of example code that I found on the Web. I tested it with Ruby 1.8.6 on Windows 7.

Several notes regarding the script:

Preventing threads from disappearing

The script is designed to exit with a stack trace on exception. More extensive error handling would be overkill for a quick script. The problem is that in Ruby, by default, threads silently exit on exception — it caused me quite a headache before figuring this out. This is fixed by setting Thread.abort_on_exception to true.

Exiting with Ctrl-C

It’s nice to be able to exit the script by pressing Ctrl-C. On Windows, Ruby doesn’t handle Ctrl-C keypresses inside socket.accept (and apparently during other blocking calls). To fix this, we need a special thread that spends most of its life sleeping, but wakes up once in a second. During that time Ruby will be able to process the keypress and exit.

The script

require 'socket'

if ARGV.length < 1
    $stderr.puts "Usage: #{$0} remoteHost:remotePort [ localPort [ localHost ] ]"
    exit 1
end

$remoteHost, $remotePort = ARGV.shift.split(":")
puts "target address: #{$remoteHost}:#{$remotePort}"
localPort = ARGV.shift || $remotePort
localHost = ARGV.shift

$blockSize = 1024

server = TCPServer.open(localHost, localPort)

port = server.addr[1]
addrs = server.addr[2..-1].uniq

puts "*** listening on #{addrs.collect{|a|"#{a}:#{port}"}.join(' ')}"

# abort on exceptions, otherwise threads will be silently killed in case
# of unhandled exceptions
Thread.abort_on_exception = true

# have a thread just to process Ctrl-C events on Windows
# (although Ctrl-Break always works)
Thread.new { loop { sleep 1 } }

def connThread(local)
    port, name = local.peeraddr[1..2]
    puts "*** receiving from #{name}:#{port}"

    # open connection to remote server
    remote = TCPSocket.new($remoteHost, $remotePort)
    
    # start reading from both ends
    loop do
        ready = select([local, remote], nil, nil)
        if ready[0].include? local
            # local -> remote
            data = local.recv($blockSize)
            if data.empty?
                puts "local end closed connection"
                break
            end
            remote.write(data)
        end
        if ready[0].include? remote
            # remote -&gt; local
            data = remote.recv($blockSize)
            if data.empty?
                puts "remote end closed connection"
                break
            end
            local.write(data)
        end
    end
    
    local.close
    remote.close
    
    puts "*** done with #{name}:#{port}"
end

loop do
    # whenever server.accept returns a new connection, start
    # a handler thread for that connection
    Thread.start(server.accept) { |local| connThread(local) }
end

PS

When I started writing this script I got a cryptic error message if a didn’t add a “require ‘rubygems'” line at the beginning. However I can’t reproduce the problem now. In fact the browser history doesn’t show all the googling I’ve done to find the solution and I’m beginning to think that I hallucinated it all.

Taking the swing out of songs

I recently read about a program called Swinger in a music blog. The post’s author, Paul Lamere, describes the program nicely:

The Swinger is a bit of python code that takes any song and makes it swing. It does this be taking each beat and time-stretching the first half of each beat while time-shrinking the second half. It has quite a magical effect.

The description is followed by several amusing audio samples of songs in “straight” rhythm turned into swing rhythm by the program.

For those unfamiliar with the musical terms, I’ll try to explain a little what the program does. Usually in music, each beat is divided into two notes of equal length. This is “straight” rhythm. You can hear it, for example, in the guitar intro for Every Breath You Take by The Police: all the notes are have uniform length. On the other, notes can be swung, which means that the second note of the beat comes a little bit later. In other words, the first note of the beat is longer than the other. You can hear it, say, in Personal Jesus by Depeche Mode (not the Johnny Cash version).

Naturally, if you can take a normal song and add a swing to it, you can also do the reverse. Instead of stretching the first half of each beat and shrinking the second half, you shrink the first half and stretch the other. Swinger can already do this. You just need to give it the ratio between the two notes of the beat, and it will stretch and shrink the notes so they will have uniform length.

The song I decided to try and de-swing was Revolution 1 by the Beatles. Here is the result:

Some parts of it are de-swinged pretty well (the first verse, the backing vocals), other parts (e.g. the intro) sound pretty much the same to me as the original.

What’s the “swing ratio” of Revolution? The way I hear the song, each beat is divided into three equal parts, the first note taking 2 parts and the second taking the last one, meaning a ratio of 2:1. Swinger actually takes as a parameter the number x-0.5, where x is the part of the beat taken by the first note. In other words, the parameter is the part of the second half of the beat taken by the first note. So I tried using 0.17. The result didn’t sound very different than the original. I finally settled on 0.25 — a ratio of 3:1.

You can notice that the effect is much less striking than the swinging effect shown in the blog. I also tried de-swinging another song (Terrapin by Syd Barrett) but it didn’t have much effect. It looks like de-swinging is harder than en-swinging. Why is that?

My theory is that in the beat wasn’t detected 100% correctly in these cases. The algorithm’s result greatly depends on the beat detection quality. In fact, if you think about it, en-swinging and de-swinging are actually the same process applied with slightly different time shifts: in both cases, the algorithm alternately stretches and shrinks small chunks of the song. The only difference is which part of the beat the stretching happens. If the beat isn’t detected correctly, the de-swinging process may even make the song swing even more! I suspect that the algorithm has a harder time detecting the beats for songs with swing.

Another explanation is that when the song is swinging, the rhythm is more loose. The different musical parts (vocals, percussion) are less in sync with each other. Simple time-stretching therefore can’t “fix” the rhythm.

std::string is contiguous

You can safely assume that the memory buffer used by std::string is contiguous. Specifically, the address of the string’s first character can be used as the address for the whole string, just like a C-style char array:

std::string str = "foo";
strncpy(&str[0], "bar", 3); // str now contains "bar".

Why is this safe? The current C++ standard apparently doesn’t guarantee that the string is stored contiguously, but it is in all known implementations. Additionally, the next C++ standard (C++0x) will make this guarantee. So the above usage is valid on all present and future C++ implementations.

Why is this important? It’s common for functions, especially in the Windows API, to “return” strings by copying them into a buffer passed to the function. Since the memory buffer used in std::string is contiguous you can safely pass it to the function, after resizing the string to the correct size.

A typical usage for Windows API functions:

// get required buffer size
DWORD bufSize = 0;
GetComputerNameA(NULL, &bufSize);
if (!bufSize && GetLastError() != ERROR_BUFFER_OVERFLOW) {
  throw std::runtime_error("GetComputerNameA failed");
}
// bufSize now contains required size of buffer, including null terminator
std::string buf(bufSize, '\0');
if (!GetComputerNameA(&buf[0], &bufSize)) {
  throw std::runtime_error("GetComputerNameA failed");
}
// bufSize now contains actual size of data
buf.resize(bufSize);
// now use buf as a regular std::string

This is cumbersome but actually easier than plain C code, since you don’t have to manage the memory yourself.

Note that the expression &str[0] is valid only if str isn’t empty. Also, everything I’ve said also applies to std::wstring, the wide-character version of std::string.

References:

• Contiguousness of STL containers and string
• Cringe not: Vectors are guaranteed to be contiguous