A Quick Link Previewer for Mattermost

Mattermost is a Slack-like self-hosted and open-source alternative. We use it at work but for some reason link previews don’t work. Before diving into Mattermost’s internals I wanted to see if I could write a quick workaround using the fact that Mattermost does show an image if you post a link ending with .png or .jpg.

The Current Situation

When you post an image link, Mattermost makes a request to show it in the application. It detects those images using a regexp; not by e.g. sending a HEAD request to get the content type. If you have an image URL that doesn’t end with common extentions Mattermost won’t show it.

Mattermost doesn’t serve you a preview of the image; it rather gives you an img with the original URL. That means every single person reading the channel will request the image from its original location. Slack, on the other hand, fetch images, cache them, and serves them from its own domain, https://slack-imgs.com. Slack uses a custom user-agent for its request so you know where it comes from.

User-Agent: Slackbot-LinkExpanding 1.0 (…)

Mattermost, on the other hand, can’t use a custom user-agent because the request is done by your browser. The only thing distinguishing Mattermost’s request for a preview and any other request is it asks for an image:

Accept: image/webp,image/*,*/*;q=0.8

The header above is Chrome telling the Web server it can deal with WebP images, then images in any format, then anything; in that order. Note it explicitly says it accepts WebP images because some browsers don’t support the format.

Unfortunately not all browsers are explicit. Firefox sends Accept: */* since Firefox 47 and so did IE8 and earlier versions. In those cases we can’t really do anything beside complicated rules based on the user-agent and other headers.

Proxying Requests

If we know how to tell if a request comes from Mattermost asking for an image preview rather than a “normal” user we can serve different contents to them: a link preview as an image to Mattermost, and the normal content to the user.

All we have to do is to make some sort of intelligent proxy. Using Flask we can make something like this:

from flask import Flask

app = Flask(__name__)

@app.route("/<path:path>/p.png")
def main(path):
    if not request.headers.get("Accept", "").startswith("image/"):
        return redirect(path)

    return "Hello, I'm an image"

This is a small web application that takes any route in the form of /<url>/p.png and either redirects you to that URL if your Accept header doesn’t start with image/; either serves you a Hello, I'm an image page.

All we have to do now is to return an actual image in lieu of that placeholder text. I used Requests and Beautiful Soup to fetch and parse webpages, and Pillow to generate images.

When one requests http://example.com/http://...some...url.../p.png, the app fetches that URL; parses it to extract its title and some excerpt; write that on a blank image; and serves it.

Extracting the title is as easy as grabbing that title HTML tag. If available, I also try the og:title and twitter:title meta tags. If none of those are available, I fallback on the first h1 or h2 tag.

Getting an usable excerpt is not too hard; here again I search for common meta tags: description, og:description, twitter:description. If I can’t get any of them I take the first p that looks long enough.

You can check that code on GitHub.

Generating Images

The Pillow library makes it easy to write text on an image. I replaced the (ugly) default font with Alegreya. The tricky part is mostly to fit the text on the image. I used a combination of Draw#textsize to get the dimensions of some text as if it were written on the image and Python’s textwrap module to cut the title and excerpt so that they wouldn’t cross the right side of the image.

I used fixed dimensions for all images (400×70) and kept a small padding along their sides. Previews with small or missing excerpts get some unused white space at the bottom; this could be fixed by pre-computing the final size of the image before creating it.

On most websites the preview works fine. We could tweak the text size as well as add a favicon or an extracted image.

Conclusion

In the end it took a couple hours to have a working prototype. Most of that time was spent dealing with encoding issues and trying various webpages to find edge cases.

The result is acceptable; it has the issue of not being accessible at all but that’s still better than nothing.

The source code is on GitHub.

Add This Directory to Your PATH

You’ve probably read this instruction quite a few times in online tutorials about installing command-line tools. What is this PATH for? Why should we add directories “to” it? This is the subject of this post.

How the Shell Finds Executables

When you type something like ls, your shell has to find what is this ls program you’re trying to run. Typical shells only have a handful predefined commands like cd; most of the commands you use everyday are standalone programs.

In order to find that ls program; your shell looks in a few directories. Those directories are stored in an environment variable called PATH; you can look up its value in most shells using the following command:

echo $PATH

You can see it contains a list of paths separated by colons. Finding a program in these is just a matter of checking each directory to see if it contains an executable with the given name. This lookup is implemented by the which program. The algorithm is pretty simple and goes like this:

cmd = "ls"
for directory in $PATH.split(":") {
    candidate = $directory/$cmd ;
    if candidate exists and is executable {
        return candidate ;
    }
}

This is pseudo-code, but there are implementations in various languages: C, Go, Ruby, Dart, etc.

As you can see, the order of these directories matters because the first matching candidate is used. That means if you have a custom ls program in /my/path, putting /my/path at the beginning of the PATH variable will cause ls to always refer to your version instead of e.g. /bin/ls because /bin appears after /my/path.

You can perform this lookup using the which command. Add -a to get all matching executables in the order in which they’re found. This is what which python gives on my machine:

$ which python
/usr/local/bin/python

$ which -a python
/usr/local/bin/python
/usr/bin/python

You can see I have two python’s but the shell picks the one in /usr/local/bin instead of /usr/bin because it appears first in my PATH. You can bypass this lookup by giving a path to your executable. This is why you run executables in the current directory by prefixing them with ./:

./my-program

This tells the shell you want to run the program called my-program in the current directory. It won’t search in the PATH for it. It also works with absolute paths. The following command runs my python in /usr/bin regardless of what’s in my PATH variable:

/usr/bin/python

For performance reasons a shell like Bash won’t look executables up all the time. It’ll cache the information for the current session and will hence do this lookup only once per command. This is why you must reload your shell to have your PATH modifications taken into account. You can force Bash to clear its current cache with the hash builtin:

hash -r

Now that we know how our shell find executables; let’s see how this PATH variable is populated.

Where Does That PATH Come From?

This part depends on both your shell and your operating system. Bash reads /etc/profile when it starts. It contains some setup instructions, including initial values for the PATH variable. On macOS, it executes /usr/libexec/path_helper which in turns looks in /etc/paths for the initial paths.

The file looks like this on my machine:

$ cat /etc/paths
/usr/bin
/bin
/usr/sbin
/sbin

The actual code to set the PATH variable (or any variable for that matter) in Bash is below:

PATH="/one/directory:/another/directory:/another/one"

By default, Bash doesn’t pass its variables to the child processes. That is, if you set a variable in Bash then try to use it in a program it’ll fail:

$ myvar=foo
$ ruby -e 'puts ENV["myvar"] || "nope :("'
nope :(

Bash allows one to mark a variable as exported to subprocesses with the export builtin command:

$ myvar=foo
$ export myvar
$ ruby -e 'puts ENV["myvar"] || "nope :("'
foo

This is usually done when setting the variable:

$ export myvar=foo
$ ruby -e 'puts ENV["myvar"] || "nope :("'
foo

Technically Bash doesn’t need you to export the PATH variable to use it but it’s better if for example a program you use executes another program; in this case the former must be able to find the latter using the correct PATH.

How Do We Modify It?

Each shell has its own file in the user directory to allow per-user setup scripts. For Bash, it’s ~/.bash_profile, which often sources ~/.bashrc. You can use this file to override the default PATH. It’ll be loaded when starting a session; meaning you have to either reload your shell either re-source this file after modifying it.

We saw in the previous section how to set the PATH variable; but most of the time we don’t want to manually set the whole directories list; we only want to modify its value to add our own directories.

We won’t dive into the details in that post, but Bash has a syntax to get the value of a variable by prefixing it with a dollar symbol:

echo myvar  # prints "myvar"
echo $myvar # prints "foo", i.e. myvar's value

Bash also supports string interpolation using double quotes: You can include the value of a variable in a double-quotes string by just writing $ followed by its name:

echo "hello I'm $myvar"  # prints "hello I'm foo"

We use this feature to append or prepend directories to the PATH variable: prepending means setting the PATH’s value to that directory followed by a colon followed by the previous PATH’s value:

PATH="/my/directory:$PATH"

You usually don’t need to re-mark this variable as exported but using export at the beginning of the command doesn’t hurt.

Wrapping Things Up

Modifying the PATH is not something we do very often because most tools are installed in standard locations—already in our PATH. Most package managers install executables in their own location and need the user to modify their PATH. Homebrew, for example, installs them under /usr/local/bin and /usr/local/sbin by default. If those are not already in the PATH, one needs to add them:

# In e.g. ~/.bash_profile
export PATH="/usr/local/bin:/usr/local/sbin:$PATH"

This means the shell will first look in these directories for executables. It allows one to “override” existing tools with more up-to-date ones.

Where to Buy (Cheap) Shirts Online

This post title sounds like a spam email subject but I was asked twice in the past three weeks where I buy my t-shirts, so here’s an answer that everybody can benefit from.

Living in Europe means we can get quite high shipping costs on US-based websites. The highest shipping cost per t-shirt I ever got was $23 for two t-shirts and on some websites up to 40% of the total cost is due to shipping.

The main criteria I have when buying t-shirts online are:

1. Their price. I very rarely buy t-shirts that are more expensive than $20-25. The reason is I like to buy multiple t-shirts at once and having to pay $100 for four t-shirts plus shipping isn’t something I’d consider.
2. The quality of their design. Of course if all you want is a unicolor t-shirt you can go to virtually any clothing store in the street to get one.
3. Their shipping costs and how much time I’ll have to wait (spoiler: at the very least a couple weeks if it comes from the US).

A bonus point is if they accept Paypal. This is because Paypal does the Euro/Dollar conversion for me; otherwise if I pay with my credit card in Dollars my bank steals takes me ~1% of the price, or €1 if it’s lower than €100.

Not-that-fun story: I once had to pay a $0.02 AWS bill. It cost me €1.02; €1 for my bank and ~€0.02 for AWS.

Note that I’m not affiliated with any of the websites listed in this post and I bought one or more times on each one.

Threadless

Threadless is the first website I ordered shirts on; back in 2009. They have a large choice of t-shirts; you have to really dig to find geeky ones. A lot of them are as low as $10-12 and they have promos where shirts can go as low as $5. All the ones I bought in 2009 are still in perfect condition. The shipping costs are acceptable at ~30% of the total cost (e.g. 5 t-shirts for $45 + $27 for shipping = $72) but I need at least a month to receive my package.

Pluses: Large choice, good prices.
Minuses: YMMV but I need to dig a lot to find t-shirts I like.

6 Dollar Shirts

I don’t remember how I found 6 Dollar Shirts but it was quite a pleasant surprise to see a website that sells shirts at $6 each. They even have a special price at $50 for ten of them. They have a lot more geeky shirts and ones with references to popular culture. There are also a lot of political t-shirts that I’m not interested in — but again you may like them.

The downside of the site are the high shipping costs: $35 for ten shirts. But given their shirt prices it stays very low: $50+$35 for ten t-shirts means only $8.5 per t-shirt. The first ones I bought in 2013 are still in pretty good shape. They’re not as good as Threadless’ but they’re really good for their price.

Pluses: Prices, and some designs are really great.
Minuses: After ~20 t-shirts I find it hard to find enough nice shirts to fill another 10-shirts bulk.

Teespring

Teespring is a time-limited t-shirts website. People submit their design and if there are enough buyers before a few days it goes to print. It’s not a site I often go on but it seems to be popular for when people want to print a special-occasion shirt.

I bought two shirts from them in 2014; each one cost $18 plus $12 of shipping cost. That’s quite high, which is why I don’t often visit their website.

Pluses: Some shirts are nice.
Minuses: Prices are high and there’s not a lot of choice.

Cotton Bureau

Cotton Bureau is similar to Teespring but with higher prices and goals appears to be higher than Teespring so less t-shirts go to print. They do have nice shirts, thought.

Last time I bought a t-shirt from them was on October 31 of this year. I picked a Mystery Tee as well, which is a cheaper shirt they choose for you. They should ship on November 28 so I should get them for Christmas — That’s a long time to wait. I paid $25 for the t-shirt I chose; $15 for the Mystery one; and $24 for the shipping. You have to really want that t-shirt to pay that much.

Pluses: Nice designs and if enough people subscribe to an already-printed t-shirt it goes to print again.
Minuses: Quite expensive and you have to wait until the end of the buying window to have it printed and shipped.

Redbubble

Redbubble is a marketplace that sells a lot of stuff with designs from a lot of people.

They have a lot of choice but in my experience the quality of their designs is lower than Threadless’ and 6 Dollar Shirts’. They even sometimes have duplicated or copied designs: two or three people got the same idea and submitted (almost) the same design.

The positive side of this is you’ll often find t-shirts that refer to current TV series that don’t have official stores (unlike CBS’ series), for example this Bojack-Horseman-themed shirt.

Their shipping costs are pretty low; I paid only €2.5 to get my €21 t-shirt.

Pluses: Low shipping costs and some nice finds if you search a lot.
Minuses: A lot of not-so-original designs that make it harder to find the good ones.

TeeFury

TeeFury is a daily t-shirt website. They have two shirts featured for a day at $11; otherwise you have to pay the full price ($20). I’m not a huge fan of their designs but they have very low shipping costs; only $3 for a $11 t-shirt.

Pluses: They’re the cheapest option of this list when you buy only one or two t-shirts at once.
Minuses: Only two t-shirts per day.

I hope this post may help you buy (cheap) t-shirts online. Note the shipping costs I gave were for me in France; they may vary depending on your location.

Please share your own recommendations in the comments!