Baptiste Fontaine’s Blog  (back to the website)

Syntax Quiz #1: Ruby’s mysterious percent suite

This is the first post of a serie I’m starting about syntax quirks in various languages. I’ll divide each post in two parts: the first one states the question (mainly “Is this valid? What does it do? How does it work?”); the second one gives an answer.

In Ruby, any sequence of 3+4n (where n≥0) percent signs (%) is valid; can you guess why?

Here are the first members of this suite:

%%%             # n=0
%%%%%%%         # n=1
%%%%%%%%%%%     # n=2
%%%%%%%%%%%%%%% # n=3

I’m using Ruby 2.3.0 but this I was able to test this behavior on the almost-10-years-old Ruby 1.8.5, so you should be fine with any version.

You can stop here and try to solve this problem or skip below for an answer. I’m including an unrelated image below so that people with large screen are not spoiled.

The answer to the problem lies in two things: string literals and string formatting.

You might know you can use %q() to create a string; which can be handy if you have both single and double quotes and don’t want to escape them:

my_str = %q(it's a "valid" string)

This method doesn’t support interpolation with #{} but its uppercased friend does:

my_str = %Q(i's still #{42 - 41} "valid" string)

The equivalent also exists to create arrays of strings with %w() and %W(), regular expressions with %r, as well as %i to create arrays of symbols starting in Ruby 2.0.0:

names = %w(alice bob charlie) # => ["alice", "bob", "charlie"]
names.each { |name| puts "Hello #{name}" }

my_syms = %i(my symbols) # => [:my, :symbols]

puts "yep" if %q(my string) =~ %r(my.+regexp)

You can also use [],{} or <> instead of parentheses:

%w{foo bar}             # => ["foo", "bar"]
%q[my string goes here] # => "my string goes here"
%i<a b c>               # => [:a, :b, :c]

Ruby lets you use a percent sign alone as an alias of %Q:

%(foo bar) == %<foo bar> # => true
%{foo bar} == "foo bar"  # => true

But wait; there’s more! You can also use most non-alphanumeric characters like | (%|my string|), ^ (%w^x y z^), or… %:

%w%my array%        # => ["my", "array"]
%q%my string%       # => "my string"
%%my other string%  # => "my other string"

This means that %||, %^^ or %%% can be used to denote an empty string (don’t do that in real programs, please). It answers the problem for the case n=0: %%% is an empty string; the first percent sign indicates it’s a literal string, and the following two are respectively the beginning and end delimiters.

The second part of our answer is string formatting.

If you have ever written a Python program you know it supports string formatting à la sprintf with %:

print "this is %s, I'm %d years-old" % ("Python", 25)

Well, Ruby supports the same method, called String#%:

puts "this is %s, I'm %d years-old" % ["Ruby", 21]

In both languages you can drop the array/tuple if you have only one argument:

print "I'm %s" % "Python"
print "I'm %d" % 25
puts "I'm %s" % "Ruby"
puts "I'm %d" % 21

Both will raise an exception if you have not enough arguments but only Python will do it if you have too many of them:

# Python
print "I'm %s" % ["Python", "Ruby"]
# => TypeError: not all arguments converted during string formatting
# Ruby
puts "I'm %s" % ["Ruby", "Python"]
# prints "I'm Ruby"

This means that while "" % "" is syntaxically valid in both languages, only Ruby runs it without error, because Python raises an exception telling that the argument (the string on the right) is not used.

If we combine this knowledge with what we have above with literal strings we now know we can write the following in Ruby:

%%% % %%% # equivalent to "" % ""

The last key is that it works without spaces and can be chained:

""  % ""  % ""  # => ""
%%% % %%% % %%% # => ""
%%%%%%%%%%%     # => ""

The 3+4n refers to the way the expression is constructed: the first three percent signs are an empty string, and the next four ones are the formatting operator followed by another empty string.


Want more of these? Here are a few other valid Ruby expressions using strings and percent signs (one per line); guess how they’re parsed and evaluated:

%*%%*%%*%%*
%_/\|/\_/\|/\__
%_.-''-._%%_.-''-._%%_.-''-_%%.etc.
%/\/\/\/\/\______/
%# <- is it really valid? :-#

A JavaScript Modules Manager That Fits in a Tweet

ES6 does provide modules; but unless you’re using Babel you’ll have to rely on third-party libraries such as RequireJS until all major browsers support them.

I use D3 everyday to visualize data about ego networks and have a small (400-500 SLOC) JavaScript codebase I need to keep organized. In the context I work in I must keep things simple as I won’t always be there to maintain the code I’m writing today.

How simple a modules implementation could possibly be? It should at least be able to register modules and require a module inside another; much like Python’s import. It should also handle issues like circular dependencies (e.g. foo requires bar which requires foo) and undeclared modules. Modules should be lazily loaded, i.e. only when they are required; and requiring twice the same module shouldn’t execute it twice.

Well, here is one:

p={f:{},m:{},r:function(a,b){p.f[a]=b},g:function(a){if(!p.m[a]){if(p.m[a]<1|!p.f[a])throw"p:"+a;p.m[a]=0;p.m[a]=p.f[a](p)}return p.m[a]}};

It’s 136-bytes long. 139 if you count the variable definition. At this level you can’t expect long function names but here is an usage example:

// register a "main" module. A module consists of a name and a
// function that takes an object used to require other modules.
p.r("main", function(r) {
    // get the "num" module and store it in a `num` variable
    var num = r.g("num");

    // use it to print something
    console.log(num.add(20, 22));
});

// register a "num" module
p.r("num", function(r) {
    // a module can export bindings by returning an object
    return {
        add: function(a, b) { return a+b; },
    };
});

// call the "main" module
p.g("main");

This code will print 42 in the console. It only uses two modules but the implementation works with an arbitrary number of modules. A module can depend on any number of other modules that can be declared in an arbitrary order.

Consider this example:

p.r("m1", function(r) { r.g("m2"); });
p.r("m2", function(r) { r.g("m3"); });
p.r("m3", function(r) { r.g("m1"); });

p.g("m1");

m1 depends on m2 which depends on m3 which itself depends on m1. The implementation won’t die in an endless loop leading to a stack overflow but will fail as soon as it detects the loop:

p:m1

Admittedly this error message doesn’t give us too information but we have to be thrifty in order to fit under 140 characters. The prefix p: tells you the error comes from p, and the part after is the faulty module. It can either be a wrong name (the module doesn’t exist) or a circular dependency.

Walk-through

Note: don’t use this at home. This is just an experiment; I eventually used Browserify for my project.

We need an object to map modules to their functions; we’ll populate it on calls to register. We need another object to store the result of their function call; i.e. what they export. I added a third object to “lock” a module while it’s executed in order to detect circular dependencies.

We’ll have something like that:

var p = {
    _fn: {},   // the functions
    _m: {},    // the modules’ exported values
    _lock: {}, // the locks

    register: function(name, callback) {
        // add the function in the object
        p._fn[name] = callback;
    },

    get: function(name) {
        // if we have a value for this module let’s return it.
        // Note that we should use `.hasOwnProperty` here
        // because this’ll fail if the module returns a falsy
        // value. This is not really important for this problem.
        if (p._m[name]) {
            return p._m[name];
        }

        // if it’s locked that’s because we’re already getting
        // it; so there’s a recursive requirement
        if (p._lock[name]) {
            throw "Recursive requirement: '" + name + "'";
        }

        // if we don’t have any function for this we can’t
        // execute it and get its value. See also the
        // remark about `.hasOwnProperty` above.
        if (!p._fn[name]) {
            throw "Unknown module '" + name + "'";
        }

        // we lock the module so we can detect circular
        // requirements.
        p._lock[name] = true;

        try {
            // execute the module's function and pass
            // ourselves to it so it can require other
            // modules with p.get.
            p._m[name] = p._fn[name](p);
        } finally {
            // ensure we *always* remove the lock.
            delete p._lock[name];
        }

        // return the result
        return p._m[name];
    },
};

This works and is pretty short; but that won’t fit in a Tweet ;)

Let’s compact the exceptions into one because those strings take a lot of place:

if (p._lock[name] || !p._fn[name]) {
    throw "Module error: " + name;
}

The error is less explicit but we’ll accept that here.

We try to get as little code as possible then use YUI Compressor to remove the spaces and rename the variables. This means we can still work with (mostly) readable code and let YUI Compressor do the rest for us.

I measure the final code size with the following command:

yuicompressor p.js | wc -c

Right now we have 240 bytes. We need a way to remove 100 bytes. Let’s rename the attributes. _fn becomes f; _m becomes m, _lock becomes l and the public methods are reduced to their first letter. We can also remove the var since p will be global anyway. Let’s also reduce the error message prefix to "p:".

p = {
    f: {}, m: {}, l: {},

    r: function(name, callback) { p.f[name] = callback; },

    g: function(name) {
        if (p.m[name]) {
            return p.m[name];
        }

        if (p.l[name] || !p.f[name]) {
            throw "p:" + name;
        }

        p.l[name] = true;

        try {
            p.m[name] = p.f[name](p);
        } finally {
            delete p.l[name];
        }

        return p.m[name];
    },
};

That’s 186 bytes once compressed. Not bad! Note that we have twice the same line in the g function (previously known as “get”):

return p.m[name];

We can invert the first if condition and fit the whole code in it; combining both returns into one. This is equivalent to transforming this code:

function () {
    if (A) {
        return B;
    }

    // ...
    return B;
}

Into this one:

function () {
    if (!A) {
        // ...
    }

    return B;
}

The first form is preferable because it removes one indentation level for the function body. But here return is a keyword we can’t compress.

Speaking of keyword we can’t compress; how could we remove the delete? All we care about is to know if there’s a lock or not, so we can set the value to false instead, at the expense of more memory. This saves us only one byte but since we only care about the boolean values we can replace true with 1 and false with 0.

We’re now at 166 bytes and the g function looks like this:

function(name) {
    if (!p.m[name]) {
        if (p.l[name] || !p.f[name]) {
            throw "p:" + name;
        }

        p.l[name] = 1;

        try {
            p.m[name] = p.f[name](p);
        } finally {
            p.l[name] = 0;
        }
    }

    return p.m[name];
}

Now, what if we tried to remove one of the three objects we’re using? We need to keep the functions and the results in separate objects but we might be able to remove the locks object without losing the functionality.

Assuming that modules only return objects let’s merge m and l. We’ll set p.m[A] to 0 if it’s locked and will then override the lock with the result. p.m[A] then have the following possible values:

  • undefined: the key doesn’t exist; the module hasn’t been required yet
  • 0: the module is currently being executed
  • something else: the module has already been executed; we have its return value

We need to modify our code a little bit for this:

function(name) {
    if (!p.m[name]) {
        if (p.m[name] === 0 || !p.f[name]) {
            throw "p:" + name;
        }

        p.m[name] = 0;
        p.m[name] = p.f[name](p);
    }

    return p.m[name];
}

Note that this allowed us to get ride of the try/finally which let us go down to 143 bytes. We can already save two bytes by using < 1 instead of === 0.

Replacing || (boolean OR) with | (binary OR) saves one more byte and allows us to fit in 140 bytes! We can go further and remove the brackets for the inner if since it only has one instruction. We need to do that after the compression because YUI Compressor adds brackets if they’re missing.

The final code looks like this:

p = {
    f: {}, m: {},

    r: function(name, callback) { p.f[name] = callback; },

    g: function(name) {
        if (!p.m[name]) {
            if (p.m[name] < 1 | !p.f[name])
                throw "p:" + name;

            p.m[name] = 0;
            p.m[name] = p.f[name](p);
        }

        return p.m[name];
    },
};

That’s 139 bytes once compressed! You can see the result at the top of this blog post.
Please add a comment below if you think of any way to reduce this further while preserving all existing features.

Thank you for reading!

Preventing Bash Pranks

The easiest and most popular Bash pranks involve someone messing up with your ~/.bashrc. For example, here is a real-life example:

#! /bin/bash
b=~/.bashrc
echo >>$b
echo "echo sleep 1 >>$b" >>$b

If you execute this script, it’ll add a newline in your ~/.bashrc just in case it doesn’t end with a newline, then add this line:

echo sleep 1 >>~/.bashrc

The effect of this isn’t immediately visible to the pranked user. When they’ll start a new Bash session, e.g. by opening a new terminal window, the code in ~/.bashrc will be executed, and the previous line will add sleep 1 at the end of it, which means it’ll be executed and the user will have to wait one more second before having their prompt. The next time they’ll open a session, it’ll add one more line and thus will wait 2 seconds, and so forth.

In this post, I’ll give you an overview of the existing solutions to prevent these pranks.

Note that I’m referring to ~/.bashrc as your startup Bash file because it’s commonly used, but some people directly use ~/.bash_profile instead, or another one. When you start a session, Bash reads /etc/profile, then tries ~/.bash_profile, ~/.bash_login, and ~/.profile, (in that order). In most environments the default ~/.bash_profile file sources ~/.bashrc.

User Rights

The first solution is to protect your ~/.bashrc by restraining the access. Nobody should be able to edit your file except you (and root). It should be the default, but if you messed up with user rights, here is how to reset the file to a safe state (read and write for you, and that’s all):

$ chmod 600 ~/.bashrc

Most attacks thus involve you executing a script, which allows them to bypass the rights because the script is executed by you with your editing rights.

One solution would be to remove your own writing right and adding it only when you need it:

# add this in your ~/.bashrc
secure-edit() { chmod u+w $*; ${EDITOR:-vi} $*; chmod u-w $*; }

Then remove your writing right:

$ chmod 400 ~/.bashrc

You can’t edit your file anymore, but you can use your new secure-edit command:

$ secure-edit ~/.bashrc

It temporarily allows you to modify the file, open your editor, then put the restricted rights back.

The “last line protection”

This one is easy to use but easy to circumvent. The goal is to prevent one-line insertions, such as:

echo "alias ls=cd" >> ~/.bashrc

and the solution is as simple as:

#

Yes, that’s just an hash symbol. If you ends your ~/.bashrc with it, the first inserted line will be commented out:

#alias ls=cd

It doesn’t work if the prankster adds multiple lines, or adds a newline before the prank.

return

You can exit from a Bash script with exit. Your ~/.bashrc is not executed like a script, it’s sourced. This means Bash doesn’t start a subshell for it and execute all its content in the current shell. This also means if you write exit it’ll exit your current shell.

The solution here is to use return at the end of your file:

return

Any line added after this one won’t be executed because Bash will stop the evaluation. Note that while it’s better than the previous solution, it can be nullified by a sed call (e.g. sed 's/return//').

The disguised return

This one is the same as the previous one, but prevents pranksters from removing it with calls to sed or similar search & replace techniques. It uses the fact than in Bash you can execute a command contained in a variable by using it at the proper place:

print_something=echo
$print_something hello

These lines are equivalent to echo hello. We use the same thing here with return. The idea is to execute an obfuscated version of return, e.g.:

b=tu
a=re
c=rn

$a$b$c

And voilà! It’s now nearly impossible to detect the return execution without manually editing the ~/.bashrc file.

This is still vulnerable to file replacement, e.g.:

rm ~/.bashrc
echo 'echo "sorry, no."' > ~/.bashrc

This wipes the existing ~/.bashrc file and replace it with another one.