Martin's blog

Who wants to be spammed?

2026-05-08T00:00:00+02:00

The German job market in 2026 is, roughly speaking, hell. So in addition to the usual venues I've been sending my profile to HackerNews' "Who wants to be hired" thread.

The rules of the thread are fairly simple:

Please only post if you are personally looking for work. Agencies, recruiters, job boards, and so on, are off topic here.

Readers: please only email these addresses to discuss work opportunities.

As expected, all but one (arguably two) of the replies I got are spam:

AI Cover letter writing service: let me be clear: cover letters are, with very few exceptions, bullshit. But it takes a certain type of personality to think that the solution to "every 'AI cover letter' tool churning slop" is... to make a new AI cover letter tool to churn slop.
AI Resume Tech: I don't understand what this letter wants to say or what they do. Apparently my CV is a "portfolio-only read", which is surprising because I don't have a portfolio in there.
Be a face for a foreign developer (received twice): I knew this was going on, but I was nonetheless surprised that it was offered to me. If you're not up to date, the trick is that you act as the face of a foreign ~~spy~~ developer -- you show up in meetings, they do the technical work, and you both split the salary 40/60. What happens to the company whose infrastructure is open to some unkown foreign developer is left as an exercise to the reader. Here's a hint.
Weird spam with no link
Buy a home spam
Join our API Hackathon and win $35k: apparently it is no longer enough to pass an interview - if you want to be hired you need to join a hackathon, build someone else's API, and only then you may be considered for a role.
Job recommendation service: few things say "here's your zero noise list" as an unrequested list of jobs that don't fit my profile.
Crypto-native talent network: the biggest surprise for me was to find out that, apparently, we are still doing blockchain in 2026.
Someone who isn't even trying
Someone who is trying really hard: from one guy who can't be bothered to learn about encodings to another one who wants to build an entire market. Perhaps I should put the previous guy in touch with this one.
Actual, non-spam job offer: they didn't reply to my follow-up e-mail but at least they were on topic.
Job recommendation service with recruiter referral code: I thought that job agencies weren't allowed to contact me at all but a deeper read of the rules reveals that this is technically not true -- they cannot post in the thread but there's nothing about them mailing me directly. And since the link is to a job (not one that this person is hiring for but a job nonetheless) it technically qualifies as "work opportunity". A previous version of this article had this e-mail classified as spam, but I've since bumped it to "not what I wanted but ok".

I know this sounds really dumb but part of me was hoping that the HN crowd would be different to that of literally every other forum on the internet. At least just because I hoped it would be better doesn't mean that I'd be naive enough to post my personal e-mail address. Despite all the noise I still find it mildly amusing to receive e-mails from bots that think my name is "Probably".

A eulogy for my Kindle DX

2026-01-29T00:00:00+01:00

More than a decade ago on my first trip to the US I bought a Kindle DX, an almost-A4-sized e-book reader that followed me through multiple countries, homes, and commutes. We were off to a rocky start when I slipped on the ice and fell breaking the screen in the process, but I got a free replacement that I've been using ever since¹.

I never used most of its functionalities - all I wanted was an e-book reader that I could load via USB with a big enough e-paper screen on which to read papers and comics in PDF format, and on that front my Kindle DX delivered like no other. I cannot speak to its audiobook-playing capacities nor 3G capabilities because, frankly, I never cared.

I kept the device in surprisingly good condition for a long time, with the only obvious problem being its micro USB port becoming progressively weaker as the years went by. So imagine my surprise when, about a month ago, my Kindle stopped working altogether regardless of a full battery and a still-working port. I tried all reset procedures I knew and I even exchanged its battery, but it is time for me to accept that the device is dead or, at least, beyond my skills to revive it. Whether its demise is related to me reading Careless people at the time remains an unanswered question.

I am currently deciding what to do with the remaining pieces. There seems to be exactly one project focused on controlling these screens and it's not precisely a friendly one, but I'm not in a rush to get it done. I will be needing a replacement, though, but whether it will be a second-hand Kindle DX or something else I cannot say. The smaller form factor of modern devices is mildly intriguing, but then I would also have to get a tablet for reading comics and that feels like a waste of money.

Goodbye, Martin's 2nd Kindle (as it was called). I wish you would have told me that you were close to the end so I could have saved, if not the books themselves, at least the bookmarks. And I would have had time to get replacement reading before my holidays, not to mention the 30€ I spent on a new battery. I guess now I'll never know whether it was MOBI or ePUB that I was supposed to use as ebook format, or at least not without consulting Wikipedia. I'm sure there's a lesson there somewhere.

Footnotes

While the Ship of Theseus school of thought would argue that this is not the same device, the Japanese school would argue the opposite - if you haven't seen it, this anecdote from Douglas Adams' "Last Chance to See" gives a good summary.

I need a senior computer

2026-01-29T00:00:00+01:00

After spending the holidays with my family I am once again reminded of the need for a computer with dedicated hardware for seniors. Whenever we talk about parents and tech we picture an old woman wearing a shawl trying to operate a computer that's turned off, and that's unfair: the problem is not that my mom doesn't understand how a mouse works (she does) but rather that her OS keeps moving things around, breaking the muscle memory on which she relies after losing her eyesight. And while a younger person would rely on accessibility software like the Orca screen reader, all my mom wants is to keep doing what she already knows.

My mom's computer needs are modest: listening to audiobooks, podcasts, voice notes, and getting reliable, repeatable answers to simple questions. Under her current setup only the first two are painless (relatively speaking) while the last two are impossible. Case in point: listening to an audiobook (that I download for her in a shared folder) requires turning on her computer, wait for the screen reader, press the Windows key, type "books", press Enter, Esc, Alt-Tab, scroll down to the right book, Enter, Ctrl+G, enter a page number, Enter, Esc, and finally Insert+Arrow Down, at which point the screen reader will finally read the book from the given page number.

One may think that a voice interface would be the solution, but our experiences have been mixed: while good for listening and replying to messages, asking Siri a question on her really-not-cheap iPhone ends up with Siri replying "I found this, take a look" (not the best phrasing, but whatever) and then performing an equally-complex series of swipes and clicks after which, more often than not, my mom gives up and asks whoever is around. Alexa isn't any better, alternating between misunderstanding our questions and trying to sell us premium apps.

The problem here is that my mom is expected to adapt to the technology instead of the other way around. What she needs is a simple device with roughly five customizable physical buttons (touchscreens are out for obvious reasons), two knobs, mic and speakers, and nothing else. One knob would select a function (list books, list audiobooks, list notes, etc), the other would control the volume, the buttons would depend on the current function (play and pause for podcasts, next page for audiobooks, and so on), and that's it. Connect the box to the WiFi (config provided via a file in the SD card) and we're good to go.

I know I can write the software for such a device, but the hardware is trickier. The input buttons are straightforward, the knobs less so, I have ideas for handling the mic, and the enclosure would be 3D printed. The OS would be Linux with remote access for troubleshooting, and all that's left is the two hundred unexpected tiny issues that always pop up. And then there's the question of whether a joystick wouldn't be a better choice...

I'm not saying I'll find the time to do it, but I'm definitely thinking about it.

Writer's block

2025-08-05T00:00:00+02:00

I wrote my last post in November 2024, which is a lot. Unlike other dry spells (seven months in 2017 and six months in 2019), this one was not because I forgot about the blog but because I have a severe writing block due to, you guessed it, AI.

I receive lots of bot traffic every day — way more, I suspect, than actual human readers. That's usually not a problem because I think of this blog as a message in a bottle, drifting lazily through the oceans until one day someone finds it by chance, reads it and thinks "yeah, VGA cables are indeed cool". But AI changes the equation — if for every human I reach there are one hundred bots, then all I'm effectively doing is creating AI training data for free.

I used to think that the terms under which I release this content (Creative Commons BY-SA 4.0) would help: the "ShareAlike" terms give you permission to share and adapt my ideas, but only as long as you give proper attribution and distribute the result under the same license. This is all well and good until you learn that AI companies have decided that they can use all content ever created under a "fair use" policy, licenses be damned and sell you the results without even attributting the original author (let's not even think about compensation).

Much more talented people than me have given their reasons for why they think this whole situation is a travesty — Dorothy Gambrell, the creator Cat and Girl has made an eloquent case for why AI feels so much like theft and Mike Krahulik of Penny Arcade has made comics like this one, this other one and this third one. None of those texts include the word "fucking assholes", though, but that's understandable. My version does, though.

Social problems cannot be solved with technology, but given the limited amount of power I have I will have to make do with the technology I have. I have decided to implement a three-step process:

Add all known AI crawlers to my robots.txt file and restrict them to the PR part of this website (who I am, what I do, publications). Everything else is off-limits.
Add some AI protection — I like the spirit behind Anubis but I may end up sticking to a combination of Apache's mod_evasive and Fail2Ban.
If all that fails, a Tarpit for particularly misbehaved bots. How to generate an infinite maze of URLs for cheap is an interesting problem to think about, and one that I've started working on.

One of my personal goals has always been to make this website as quick to load as possible, which rules out captchas of any kind. A Proof-of-Work solution like the above-mentioned Anubis can work, but that remains to be seen. Other measures are less drastic, with the simplest one being that I'll try and get at least one curse per post. The reasons for this are complex enough that they deserve their own post, so stay tuned.

Adventures in making

2024-11-27T00:00:00+01:00

There is a recurring joke about how the longer you work as a programmer, the stronger the urge to quit the industry and go live in a farm. I'm not at that point yet but I do have a soft spot for making physical things with my own two hands that, no matter how crooked, I can hold proudly and say "I did this". It is no surprise then that I'm a fan of the Maker Movement, the idea of figuring out how to do stuff on your own rather than purchasing it. I've been using my free time this year to build quite a lot of stuff, and yet 3D printing is the only one I wrote about.

Today's post is an attempt at remediating that. I don't expect you to like them all, but if you're looking for a new hobby then here's a list of hobbies a nerd can enjoy.

Drawing

I have a problem with drawing: I've been doing it for long enough that I'm objectively good at it (no just "good enough", but "good") and yet without a project at hand I just... don't. I had some success with small comics I drew during particularly depressing meetings and I've been chasing that high ever since.

A friend and I are currently looking into writing a comic together, so hopefully I'll have more to tell in the coming months. I would have liked to join a comic contest as our North star, but those are hard to come by when you're an adult.

I don't want to write much more about the topic because I already did it once.

Painting

I started taking painting classes about a year ago after realizing that you can learn from videos how to add purple shadows to portraits but you can't ask why you are adding purple shadows to pink skin. My drawing skills have helped with the first steps, but becoming the next Rembrandt will still take a while.

My main shock about painting was the stark difference between me just "going with it" versus having a specific goal: I can get good results whenever I paint for fun (or, more accurately, when following the tasks my teacher gives me) but getting a precise result is still a lost cause - the least I care, the better the result. And while I understand why this happens, I don't yet have a working solution.

My current main challenge with painting is the cleanup: I still haven't processed the trauma of that one night when paint splashed everywhere and I spent the next hour on my knees chasing tiny paint spots on the floor of my rented apartment. I know I can't get better without practice, but sometimes the apprehension is just too much.

Sewing

I have a general rule: if I face the same problem three times, and the problem is salient enough that I remember all three occasions at once, then I need to figure out how to solve it for good. October found me in this specific situation when I realized that I had

One nice shirt that's too big for me,
One summer shirt that's too big for me, and
One summer hoodie design that I've been wanting to have for almost twenty years.

My first idea was to learn sewing by hand. I wouldn't recommend it as a long-time strategy, but hand-making my own sweatshirt taught me a lot about why a sewing machine does what it does and when is hand stitching the faster, simpler alternative.

The next natural step was to get a sewing machine. The one in my local library is eternally on loan, but luckily second-hand sewing machines in my area are surprisingly cheap. The difficult part was finding a good first project: I checked every beginner book in every library I know, and yet all of them start with projects like a skirt, a top or a dress, none of which would be very useful for me. In the end my hand-sewn sweatshirt had to be redone when the material failed to shrink as promised and having to redo all of that gave me some good practice.

Having now resized one shirt, one t-shirt, one sweatshirt and having made a summer hoodie from scratch I believe I'm ready to tackle the nice shirt. I've been toying with some stupid ideas after that, but whether they end up being fashion or cosplay I cannot say.

Music

I never really stopped playing music, but my piano skills have not evolved much in the last ten years. Fate put recently an old guitar into my hands for repair, and this gave me a good chance to give the habit a bit of a refresh - I learned how to play guitar when I was 12, and I am glad to report that I still keep the muscle memory.

The weird part was singing - I stopped playing guitar because I was really bad at singing, and yet I am now at the point of my life in which I get to understand two things:

Singing is not that difficult once you understand that your voice and the music have to follow each other
As long as your singing isn't atrocious, no one really cares

I tried to write an app for showing a song and its corresponding tab, but the problem is harder than I thought: the whole song doesn't fit in a phone screen, you cannot click the screen to move on (both your hands are busy), and coordinating the text and music automatically is a tough problem. I considered real-time speech detection, but I'm not sure there really is interest for that as guitar players are notoriously uninterested in formalizing their art. I'm currently sticking to the tried-and-tested hand-written booklet until I find a better solution.

Penmanship

On the summer of 2002 I stopped writing cursive - I was taking my first Math courses and realized that formulas only make sense in block letters. My block handwriting declined steadily until 2014 when I decided to do something about it, but cursive remained forgotten until last year.

It was quite shocking to realize that I no longer remembered how to write some uppercase letters, particularly one that's part of my own name. I therefore made a conscious effort to start writing cursive more often, I looked into different types of scripts, and nowadays my cursive is nicer than it ever was - I'm still not as good as some teenage girls I know, but I'm getting there.

Make-up

I have no interest in wearing make-up on my day-to-day - in fact I wish less people would do it. But during Halloween, Karneval or similar? If you think of it as "applied painting" it turns out to be a lot of fun.

I believe my skull makeup is on point (I really need to find a new tube of cracked-skin white base) and my Picasso was also not that bad, but where to go from there is harder. I tried checking in social media, but the most interesting ones are meant for the camera and fall apart the second you look away. Latex prosthetic seem promising, so I'll probably be trying that for next February.

The way forward

I have a couple ideas I want to try next year. Sculpting in wax seems fun (messy, though) and woodworking (even messier) has definitely made it through my "three problems" threshold. And if I had a backyard you can bet I'd have a project boat underway - did you know you can just download instructions to make one?

I won't pretend that I'm particularly good at most of these hobbies, but I am fine and that's what matters. Mastery is a life-long process, but making your own t-shirt takes a couple days (about a week if you want welt pockets).

So go ahead and give any of them a try. Write a comic on post-its, draw a bad portrait, play some chords on a guitar and make a wax tiki lamp. Programming is a lot of fun, but remember that specialization is for insects.

Therac-25, LLMs and the Zen of Python

2024-09-02T00:00:00+02:00

If you are a professional software developer you probably heard at some point the cautionary tale of the Therac-25, a radiation therapy machine that gave massive overdoses of radiation to at least six patients in the 80s and killed at least three.

There are several layers of problems explaining how all of this came to be, but today I'd like to focus on a specific thread. The (in)famous race condition in the Therac-25 was caused by two simultaneous bugs, but these bugs were not new: the software for the Therac-25 was ported from a previous version of the machine (the Therac-20) and those bugs were already present there. The key difference being that, while the Therac-20 had hardware interlocks preventing those faults, the Therac-25 engineers decided to implement them exclusively in software.

I'm digging this story up because today I read yet again someone suggesting that using ChatGPT to rewrite code from one programming language to another is a great idea. I disagree: a programming language is an opinionated way on how to better achieve a certain task and switching between world views without understanding how and why they do things the way they do is a recipe for inefficient code at best and weird bugs at worse. Simple example: if you're a Python programmer who came from C then you probably wrote loops like this:

# C programmer version
for i in range(len(mylist)):
    item = mylist[i]
    f(item)

# Pythonic code
for item in mylist:
    f(item)

This is a very simple example that highlights a fundamental misunderstanding of the language. The first code will run just fine, but it's generally not the proper way of writing Python code. For a more "real-life" example you can read the introduction to Learn Rust With Entirely Too Many Linked Lists where the author points out that writing linked lists in Rust is, from a Rust point of view, overall a bad idea and something to avoid entirely.

Real-case LLM

I decided to test my theory with Google's Gemini - I've seen students using it in their actual coding (probably because it's free) making it a fair choice. I asked the following:

Convert the following code from Python to Elixir:
def change_eight(my_arr):
    my_arr[8] = None

And the answer was:

def change_eight(my_arr) do
  if length(my_arr) >= 9 do
    List.replace_at(my_arr, 8, nil)
  else
    my_arr
  end
end
# This Elixir code ensures that the index 8 exists in the list before
# attempting to replace the value with nil. If the index is out of bounds,
# the original list is returned unchanged.

In my original code passing a list with less than 9 elements would throw an IndexError exception, which is consistent with The Zen of Python principle "Errors should never pass silently". But if my code relies on the exception to detect that a precondition is not being met, this rewrite has now ensured that the error will not be detected because the exception won't be triggered.

I added more context to the prompt, passing this code instead:

def change_eight(my_arr):
    my_arr[8] = None

if __name__ == '__main__':
    mylist = list(range(10))
    try:
        change_eight(mylist)
    except IndexError:
        print("Precondition failed")

    mylist = list(range(20))
    try:
        change_eight(mylist)
    except IndexError:
        print("Second precondition failed")

which lead to this rewrite:

def change_eight(my_arr) do
  if length(my_arr) >= 9 do
    List.replace_at(my_arr, 8, nil)
  else
    raise RuntimeError, "Precondition failed"
  end
end

if __MODULE__ == __ENV__["__MODULE__"] do
  mylist = Enum.to_list(0..9)
  try do
    change_eight(mylist)
  rescue RuntimeError -> error
    IO.puts(error.message)
  end

  mylist = Enum.to_list(0..19)
  try do
    change_eight(mylist)
  rescue RuntimeError -> error
    IO.puts(error.message)
  end
end
# This Elixir code is similar to the previous response, but it includes two
# test cases to demonstrate how the RuntimeError is raised in different scenarios.

The new version of the function is marginally better: Gemini now throws exceptions where expected, but it has also hard-coded the first error message - if the code fails on the second call the user will be shown an error corresponding to the first. I later asked for a C version instead of Elixir, at which point the error messages where gone entirely.

This is how critical bugs are born: when your software depends on (reasonable!) assumptions about the world and you send it through a pipeline (be it human or automated) that has no idea about what those preconditions are. And if you send your code through a system that's been designed to understand code exclusively at a syntactic level then you don't get to be surprised when the resulting code is not semantically equivalent to your original one.

Final words

Software development is not just about writing code. Software development is about understanding the environment in which your code runs and the decisions that lead to it - some of them reasonable ("this condition can never happen, the hardware will catch it"), some of them arbitrary ("let's write it in Perl"). The Therac-25 incident was made possible because someone decided to use code on an unfamiliar environment without considering the repercussions, the same way that Gemini did not consider "The Zen of Python" nor my error reporting strategy while rewriting my code.

There is more to software development than "data comes in, data comes out". Thinking about systems in terms of the context in which they run (software, hardware and social) is the best way to avoid finding yourself one day unpleasantly surprised.

Or, perhaps more relevant, unpleasantly shocked.

Calculator texts

2024-07-22T00:00:00+02:00

Some weeks ago HN user wonger_ posted this list of words to Hacker News. This is a list of the words you can spell using only an upside-down calculator - the word "boobies" (5318008) is perhaps the most well-known, but it's far from the only one.

This comment by user chriscbr went a step further and annotated all words according to their Part-of-Speech while this one by user jprete raised the bar: can you do a long work of fiction only using calculator words?

I spent some time trying to make the longest possible text, and this post is a long, complicated way to say "probably not".

Randall Munroe of XKCD fame published some years ago this article on how to write texts using only a subset of letters - he was interested in phone keypads, but there's no reason why his code can't be adapted to our task. The basic idea is that of a trigram model: you train a model to predict "given these two words, this is an ordered list of the words that have the highest probability of coming next", you restrain your words to those that fit your constraint (in our case, letters that can be mapped to a calculator), train the model with some data, and you're done.

After downloading Randall's code, updating it for Python3 and making it slightly more efficient I trained it with as much Wikipedia text as I had patience for. The exercise gave some interesting word combinations, although nothing resembling a coherent long work of fiction:

Be less (would make a nice parody of "Be best")
Oh Ohio hills
High heel shoes
I'll go see his leg
His shoe size is big
He is obese she is his size
He is high, so is his boss, so is she

The next step was to use a more capable language model, and for that we move onto LLM territory. The idea is straightforward: during generation an LLM will look at the input words and make an ordered list of the most probable next words in the sequence. Usually we want to use one of the top words, but nothing stops us from using the most likely words that only uses a specific set of characters. We need to do this by hand because LLMs are incapable of backtracking - if they have generated "Hegel oozes ego" and then realize that they've written themselves into a corner, there's no way for them to backtrack and try something different. As a result sooner or later they all choose a word that doesn't fit the instructions, which is where we come into play.

Writing the prediction code was straight-forward, but punctuation was an issue: we do want to keep some punctuation for the text to look natural, but at the same time the phrase "Oil... Oil... Oil... Oil..." is more likely than something like "Hillbillies besiege his soil". And this meant that neither GPT-2 nor LLaMa 2 could generate anything long, although some short phrases ended up being interesting enough:

Bob is eligible.
He is 2 eggs high. See his gills.
He is 90. She... she is his lobolo.
Hello Bill, Bob is his big ol' goose.

At the end, the best approach for me was to go back to the beginning, use the annotated list of words and build phrases by hand. This led me to the short-story:

Bob sees Zoe boil his sole beige goose egg, sighs. She giggles.

HN user araes used ChatGPT and, after some sentences that didn't stick to the prompt (color me surprised) eventually generated the text:

Ellie sells loose shoes; Bill shills big goose oil.

Not one to be outdone, HN user bcherny used Claude 3.5 to obtain:

Ellie sees Bill. Bill sells bibles. Ellie lies, "Bill, sell bibles else." Bill sells bibles. Ellie gobbles bibles. Bill obsesses. Ellie sighs.

Which I now counteract with my own:

Leslie obliges. She'll go see Giles' bill "Oh, Ohio hills". She seizes Giles' high heel shoes, giggles. "Bozo, go sell blisses".

Or, if you want something even longer and feel like reading some really weird prose:

Giles sells big oil. Loose soil geologies is his hobbies, his Lego. He sees Shell be so big he is ill. He begs his boss Zoe, his gig is hell.

"Hell is high heel shoes. Hell is Boise, Ohio. Hell is hillbillies", his boss hisses. "Go be Bob's hell".

Giles obliges.

Bob oozes ego. Hegel is his Bible. Bob is so high he giggles. "Oh ho ho! Hello bozo".

Giles sighs. "Hello Bob. Zoe lobs blisses."

"Blisses? She lobs ills."

Can we do better? Probably. Could we generate "a long word of fiction"? I'm going to say "probably not": having 35 verbs and lacking both the letter "a" and the word "the" restricts the set of possibilities a lot. But given enough patience I bet you could easily write a page of more-or-less coherent text.

Let me close this post with some of my favorite generations that didn't make it into the longer stories because they are a bit inappropriate:

"Google is so big, ISIS less so": Google had ca. 180k full-time employees in 2023 while ISIS may have around 16k. So yes, ISIS is "less big" than Google.
"He is so high he sees Oz": That is, indeed, pretty high. L. Frank Baum would be proud.
"Hi, I'll go see hell": it's not the end of the World nor The restaurant at the End of the Universe, but maybe someone is just visiting the Museo del Prado in Madrid.
"Oslo is hellhole": Wow, rude!

Repairing a shoe with a 3D printer

2024-06-28T00:00:00+02:00

I got my 3D printing license last year, courtesy of the Stadtbibliothek Köln offering free printing to any card-carrying member who sits through their two-hours course "please don't burn down your fingers, the printer, or the library". The printer has been vital for replacing items that are too important to live without but too cheap to sell as spare parts, which brings me to today's topic: shoes.

If you haven't spent the last week obsessing over shoe spare parts you may not know that the heel cap is the part of the heel of a shoe that makes contact with the ground. A shoe without heel cap is useless because you can't tiptoe everywhere but there is no store around me that will sell a replacement for this specific model. A friend of mine lost one of those recently and you can imagine where this story goes from there.

The first step in making a replacement heel cap is to take measures of the part you want to replace. Since I didn't have my caliper at hand I took a picture of the heel where the replacement should go along with a reference of known size, a trick I picked from this scene of the 1999 movie "The bone collector". Here's the picture in all its grainy glory:

Next we need to trace the contour. A 20 cent Euro coin is 22.25mm wide so I imported the image in Inkscape, created a circle of exactly that size, and then resized the image until the coin in the picture matched the circle. I then traced the main forms on screen trying to take perspective deformation into account and ended up with a decent design.

While probably not the best tool for this specific use case, I used Blender to generate 3D shapes for these curves using extrusion, a process where you take (say) a circle, push it in a straight line, and end up with a 3D cylinder. I extruded every part individually, estimating 2mm for the bottom, 5mm for the guides, and 15mm for the main internal support. I finally saved the project as .stl for MakerBot Desktop, and we are ready to go.

One visit to the library and 30 minutes later, the first draft was ready for a test fit. I asked my friend for her shoe, took out my caliper for the necessary adjustments... and the draft piece fit perfectly on the first try. So all that remained was a second visit to make the piece tougher and deeper (just in case), and we are done.

This is not the first custom piece I print, but it's the one that made me realize how truly useful this technology has been to me. And you don't even need to know how to 3D model when sites like Thingyverse offer a wide array of ready-made designs for free. If you have the chance, you should definitely give it a try.

Building a Web App with nothing - Update

2024-03-15T00:00:00+01:00

In my last post I mentioned a small webapp that I had built in an hour. This app was okay for what it was, but using it revealed that it needed a bit more love before it could be useful. And that's fine! After all, that was the whole point of making a quick mockup: to put it into the hand of users and see what their usage reveals about what's working and what isn't.

I have now written an updated version that you can see following this link. This new version has a couple small but important improvements:

It adds a second tab where you can see all of your data. Useful for knowing which expenses you've already added and/or to make sure that your records are saved properly.
It adds a button for clearing all data. I use this button not really to clear data, but rather to query the current date and time.
It now shows messages showing that data was inserted properly. As it turns out, not even the developer trusts a program that stores data silently and gives no acknowledgement whatsoever.

I still have not fixed the CSV export error. That one will have to wait for a future version.

Building a Web App with nothing

2024-01-31T00:00:00+01:00

I recently had to solve a problem that I already faced before: I need an app to keep track of my daily expenses, it has to run on my phone (so I can enter expenses on the run before I forget), and pretty much nothing else.

Last time I scratched this itch I built an Android app that did exactly that, and it worked fine for what I wanted. But time has passed, I have forgotten almost everything I knew about the Android SDK, and I wanted to get this version of the app done from scratch in one hour - for a single CRUD application that doesn't do the UD part, this seemed more than reasonable. So this time I decided to go for good old HTML and plain JavaScript.

Building the interface

I decided to build the app in pure HTML - the requirements of my project are modest, and I didn't want to deal with setting up a server anyway (more on this later). I did want my app to look good or at least okay but, while I do have some art delusions, I was never particularly good at web design. Therefore, I am a fan of fast web design frameworks that take care of that for me, and this project is not the exception. I am weirdly fond of Pure.css but for this project I wanted something new and therefore I settled for good old Bootstrap.

If you have never used either of these frameworks, the concept is simple: you include their code in the header of your HTML file, you copy the template that better fits the look you're going for, and that's it. In my case that means that my empty template looked like this:

<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Spendings</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/css/bootstrap.min.css"
        rel="stylesheet" crossorigin="anonymous"
        integrity="sha384-T3c6CoIi6uLrA9TneNEoa7RxnatzjcDSCmG1MXxSR1GAsXEV/Dwwykc2MPK8M2HN">
</head>
<body>
    <main>
        <div class="my-4"></div>
        <div class="container">
            <h2>New expense</h2>
            <form id="main_form">
                <!-- Here comes the form fields -->
            </form>
        </div>
        <div class="my-4"></div>
        <div class="container">
            <h2>Export expenses</h1>
            <div class="row">
                <div class="col text-center">
                    <form>
                        <button class="btn btn-primary" onClick="export_csv(); return false;";>
                            Export data to .CSV
                        </button>
                    </form>
                </div>
            </div>
        </div>
    </main>
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/js/bootstrap.bundle.min.js"
        integrity="sha384-C6RzsynM9kWDrMNeT87bh95OGNyZPhcTNXj1NW7RuBCsyN/o0jlpcV8Qyq46cDfL"
        crossorigin="anonymous"></script>
</body>
</html>

Once this was up and running it was time to add form elements. In my case that meant fields for the following information:

Expense: what it is that I actually bought.
Amount: how much I paid for it.
Date and Time: when did I buy it.
Category: under which category it should be stored.
Currency: in which currency is the expense - useful when I'm traveling back home.
Recurrent: this is a checkbox that I use for expenses that I'll have to re-enter every other month such as rent. I have not yet implemented this functionality, but at least I can already enter this into the database.
Two buttons: "Add expense" which would save this data, and "Export to CSV" to download the data saved so far.

The first three fields (expense, amount, and currency) look like this. Note that most of this code is a slightly edited version from the examples I copy-pasted from the official Bootstrap documentation:

<form id="main_form">
    <div class="mb-3">
        <label for="inputDesc" class="form-label" required>Description</label>
        <input type="text" class="form-control" id="inputDesc" required>
        <div class="invalid-feedback">Enter a description</div>
    </div>
    <div class="mb-3 row">
        <div class="col-sm-8 input-group" style="width: 66%;">
            <span class="input-group-text" id="descAmount">Amount</span>
            <input type="number" step=".01" class="form-control" id="inputAmount" required>
            <div class="invalid-feedback">Amount missing</div>
        </div>
        <div class="col-sm-4 input-group" style="width: 33%;">  
            <select class="form-select" id="inputCurrency">
                <option selected value="eur">EUR</option>
                <option value="usd">USD</option>
                <option value="ars">ARS</option>
            </select>
        </div>
    </div>
    <!-- A lot of other fields -->
    <div class="col text-center">
        <button type="submit" class="btn btn-primary"
        onClick="add_expense(event); return false;">
            Add expense
        </button>
    </div>

And with that we are done with the interface.

Storing the data

Web browsers have become stupidly powerful in the last years, meaning that they not only include a full programming language (JavaScript) but also at least three mechanisms for saving data in your device: Cookies, Web Storage, and IndexedDB.

This mostly works for my project: having a database in my phone saves me the trouble of having to set a database in a server somewhere, but there's still a risk of data loss if and when I clean my device's cache. The solution, as you may have seen above, is an "Export to CSV" button that I can use to save my data manually once in a while - I could have chosen to e-mail myself the data instead, but that's the kind of feature I'll implement once I have a need for it.

We now need to delve into JavaScript. Most of the code I ended up using came from this tutorial, with the only exception that my code doesn't really need a primary index so instead of writing:

const objectStore = db.createObjectStore("name", { keyPath: "myKey" });

I went for:

const objectStore = db.createObjectStore("name", { autoincrement: true });

Once you have implemented most of that code, you are ready to insert data into your database. In my case that's what the function add_expense is for and it looks like this:

function add_expense(event)
{
    // Validate the expense using the mechanism provided by Bootstrap
    // See https://getbootstrap.com/docs/5.3/forms/validation/
    form = document.getElementById("main_form");
    form.classList.add('was-validated');
    // These are the only two fields that need proper validation
    expense = document.getElementById("inputDesc").value;
    amount = document.getElementById("inputAmount").value;
    if (expense.length == 0 || amount.length == 0)
    {
        event.preventDefault();
        event.stopPropagation();
    }

    // Open a connection to the database
    const transaction = db.transaction(["spendings"], "readwrite");
    const objectStore = transaction.objectStore("spendings");
    // Collect the data from the form into a single record
    var data = {spending: expense,
                date: document.getElementById("inputDate").value,
                time: document.getElementById("inputTime").value,
                category: document.getElementById("inputCateg").value,
                amount: Number(amount),
                currency: document.getElementById("inputCurrency").value,
                recurrent: document.getElementById("checkRecurrent").checked};
    // Store this record in the database
    const request = objectStore.add(data)

    transaction.onerror = (sub_event) => {
        // This function is called if the insertion fails
        console.log("Error in inserting record!");
        alert("Could not save record");
        // Displays a (currently hidden) error message
        document.getElementById("warning_sign").style.display = 'block';
        event.preventDefault();
        event.stopPropagation();
    }
    transaction.oncomplete = (sub_event) => {
        // This function is called if the insertion succeeds
        console.log("Record added correctly");
        // Ensures that the hidden error message remains hidden, or hides it
        // if an insertion had previously failed and now succeeded
        document.getElementById("warning_sign").style.display = 'none';
    }
}

The function for creating the output CSV file is a simple combination of a call to retrieve every record in the database and this StackOverflow answer on how to generate a CSV file on the fly:

function export_csv(event)
{
    // Open a connection to the database
    const transaction = db.transaction(["spendings"], "readonly");
    const objectStore = transaction.objectStore("spendings");
    // Request all records
    const request = objectStore.getAll();
    transaction.oncomplete = (event) => {
        // Header for the CSV file
        let csvContent = "data:text/csv;charset=utf-8,";
        csvContent += "date,time,expense,amount,category,currency,recurrent\r\n";
        // Add every record in a comma-separated format.
        // Can you spot how many bugs this code fragment has?
        for (const row of request.result)
        {
            fields = [row['date'], row['time'], row['spending'],
                        row['amount'], row['category'],
                        row['currency'], row['recurrent']];
            csvContent += fields.join(",") + "\r\n";
        }
        var encodedUri = encodeURI(csvContent);
        window.open(encodedUri);
    }
}

The above-mentioned code has a fatal bug: I am building a CSV file by simply concatenating fields together with a "," in between, and that's likely to break under many circumstances: if my description has a "," in it, if my locale uses "," as the decimal separator, if my description has a newline character for whatever reason, and many more. I could solve this by escaping/replacing every "," in the input fields with something else, but let's instead learn the proper lesson here: do not build CSVs by hand in production!

Conclusion

And just like that we are done. I set out to get it done in an hour, and I almost made it - I had some issues with the database not initializing properly, which I ended up solving by bumping up the database version and forcing the database to rebuild itself. The glorious final version can be found following this link. This app saves all of your data in your local device, meaning that you can start using it right now. But you don't have to take my word for it. If you don't trust some random dude with a blog you can always check the source code yourself and make sure that the version I'm linking to is the same I described above. Isn't open source code great?

I can imagine a couple reasons why one would want to develop an app like this. Perhaps you need a working prototype that you want to put in the hands of alpha testers right now. Perhaps you want an app that has to work in exactly one device. Or maybe you have to develop in an environment where all you have is a text editor, a web browser, and nothing more. This would be unacceptable for a professional environment, but sometimes the problems you're trying to solve are so simple that scaling up in resources and costs doesn't make sense. Or maybe you would like to go down to the basics once in a while and realize that not every app needs React, a web server, and a cloud deployment.

I didn't manage to do all I set up to do. Some upgrades that I'm planning to add gradually are:

A fix to the CSV bug mentioned above.
A message letting you know that you expense was inserted successfully.
Implement the "Recuring expense" feature - I'm thinking of checking whenever you open the app whether the current month has any recurring expenses. If not, then the app asks you whether you want to insert them, one at a time (in case one of the expenses is no longer valid).
A hint of what the conversion rate for that day is, in case I don't want to save "I bought this in US Dollars" but rather "I bought this in US Dollars and that converts to this many Euros".

And finally, I should point out that this project was originally conceived as a proof of concept for streaming and coding at the same time. I have video of the whole experience but, given that the camera died on me halfway, you will have to rely on my word when I say that you aren't missing much.

Memories

2024-01-30T00:00:00+01:00

As a software developer I constantly have to fight the urge to say "this is nonsense" whenever something new comes up. Or, to be more precise, as a software developer who would like to still find a job 10 years from now. I may still prefer virtual machines over clouds, but I also realize that keeping up to date with bad ideas is part of my job.

I've given around turns around the Sun by now to see the rise of technologies that we take for granted. Today's article is a short compilation of a couple of them.

Seat belts

When I was a child, 2-point seat belts were present in all cars but mostly as decoration. In fact, wearing your seat belt was seen as disrespectful because it signaled that you didn't trust the driver, or even worse, that you disapproved of their current driving. And they were not wrong: since most people didn't use their seat belts in general, the only reason anyone would reach out to use theirs was when they were actively worried about their safety. Drivers would typically respond with a half-joke about how they weren't that bad of a driver.

It took a lot of time and public safety campaigns to instill the idea that accidents are not always the driver's fault and that you should always use your seat belt. Note that plenty of taxi drivers in Argentina refuse to this day to use theirs, arguing that wearing a seat belt all day makes them feel strangled. Whenever they get close to the city center (where police checks are) they pull their seat belt across their chest but, crucially, they do not buckle it in. That way it looks as if they are wearing one, sparing them a fine.

Typewriters and printers

My dad decided to buy our first computer after visiting a lawyer's office. The lawyer showed my dad this new machine and explained how he no longer had to re-type every contract from scratch on a typewriter: he could now type it once, keep a copy of it (unheard of), modify two or three things, run a spell checker (even more unheard of!), and send it to the printer. No carbon paper and no corrections needed.

A couple years later my older brother went to high school at an institution oriented towards economics which included a class on shorthand and touch typing on a typewriter. Homework for the later consisted on typing five to ten lines of the same sentence over and over. As expected, the printer investment paid off.

... until one day my brother came home with bad news: he was no longer allowed to use the computer for his homework. As it turns out, schools were not super keen on the fact that you could correct your mistakes instantaneously, but electric typewriters could do that too so they let it slide. But once they found out that you could copy/paste text (something none of us knew at the time) and reduce those ten lines of exercise to a single one, that was the last straw. As a result printed pages were no longer accepted and, as reward for being early adopters, my parents had to go out and buy a typewriter for my brother to do his homework.

About two years later we changed schools, and the typewriter has been kept as a museum piece ever since.

Neighborhoods

My parents used to drink mate every day on our front yard, while my friends and I would do the same with a sweeter version (tereré). My neighbors used to do the same, meaning that we would all see each other in the afternoon. At that point kids would play with each other, adults would talk, and everyone got to know each other.

This is one of those things that got killed by modern life. In order to dedicate the evening to drink mate you need to come back from work early, something that's not a given. You also need free time meaning, on a way, that you needed a stay-at-home parent (this being the 90s that would disproportionately mean the mom). The streets were not also expected to be safe but also perceived to be safe, a feeling that went away with the advent of 24hs news cycles. Smartphones didn't help either, but this tradition was long extinct in my life by the time cellphones started playing anything more complex than snake.

Unlike other items in this list, I harbor hope that these traditions still live in smaller towns. But if you wonder why modern life is so isolated, I suggest you to make yourself a tea and go sit outside for a while.

Computers

Since this could be a book by itself, I'll limit myself to a couple anecdotes.

When Windows 3.11 came up, those who were using the command line had to learn how to use the new interface. Computer courses at the time would dedicate its first class to playing Solitaire, seeing as it was a good exercise to get your hand-eye coordination rolling. Every time I think about it I realize what a brilliant idea this was.

My first computer course consisted on me visiting a place full of computers, picking one of the floppy disks, and figure out how to run whatever that disk had. If it was a game (and I really hoped it would be a game) I then had to figure out how to run it, what the keys did, and what the purpose of the game was. There were only two I couldn't conquer at the time: a submarine game that was immune to every key I tried (it could have been 688 Attack Sub) and the first Monkey Island in English (which I didn't speak at the time). I had trouble with Maniac Mansion too until my teacher got me a copy of the codes for the locked door.

This course was the most educational experience of my life, and I've struggled to find a way to replicate it. Letting me explore at my own pace, do whatever I wanted and asking for help whenever I got stuck got me in an exploration path that I follow to this day. The hardest part is that this strategy only works with highly motivated students, which is not an easy requirement to fulfill.

Red Bull does not give you wings

2023-09-20T00:00:00+02:00

Today I'll take you on a small trip down the rabbit hole of verifying whether something reported on the media is actually true.

Let's start with this article from NPR about customers suing food companies for misleading companies. Let's take a look at this bit:

"Red Bull announced it would pay more than $13 million to settle a lawsuit brought by buyers who said the energy drink didn't — as the marketing materials promised — 'give you wings'".

What a stupid lawsuit, right? Of course drinking an energy drink won't give you actual wings!

Well, hold on. If we follow that link we reach an article on Business Insider with the click-bait title "Red Bull Will Pay $10 To Customers Disappointed The Drink Didn’t Actually Give Them 'Wings'". The text, however, explains that the wings are figurative, that Red Bull claimed that "the drink can improve concentration and reaction speeds", and that there was no scientific support for any of that.

However! Red Bull has caffeine, which is supposed to actually do all that. So what gives?

We need to go one level deeper. After plenty of searching through articles just copying each other we eventually reach this website with the text of the actual lawsuit. According to the complaint, Red Bull...

"During the class period, [Red Bull] have made various representations to consumers about the purported superior nature of Red Bull, over simpler and less expensive caffeine only products, such as caffeine tablets or a cup of coffee. To bolster those claims [Red Bull] post "scientific studies" on the Red Bull website which they say 'prove' Red Bull's superiority.

However, no competent, credible and reliable scientific evidence exists to support [Red Bull's] claims about the product".

Instead of fighting it in court, Red Bull settled for $13 million and retired the slogan.

So let's recap: we started with a perfectly reasonable complaint about false advertising, namely, that Red Bull's claims about being superior to plain coffee were false. This complaint ended up on a $13M settlement. It was then wrongly reported as "the drink does not improve concentration" (it might, just not any better than plain coffee), then was further reported as "customer angry that Red Bull doesn't give you (metaphorical) wings", and presented with click-bait titles that lead you to believe that the complaint was about actual, literal wings.

Remember, kids: when social media tells you something that sounds too good to be true, it usually is. Always check your sources! Otherwise you may end up backpacking through Europe convinced that touristic places are not full of tourists simply because Instagram told you so.

Bad AI (I): Meoweler

2023-08-09T00:00:00+02:00

Oh boy, where do I even start with this...?

Someone posted yesterday to Hacker News their new project: Meoweler, a cat-themed travel website that's almost entirely auto-generated.

I have written here before about the realistic dangers of AI, predicting that we would see "garbage in any medium that offers even the slightest of rewards, financial or not". And boy was I right...

Here's a short, non-exhaustive list of problems I found with this website:

Offensive stereotypes. About half the Argentinean cities I've checked are based on stereotypes about the country, regardless of whether they are even remotely applicable.
Offensive pictures. I grew up in not-so-popular cities across the country and I can tell you: I wasn't born in a war-torn city, I didn't go to school in Tatooine, I didn't study in a mirror-like desert, and I didn't go to University on a mountain city. I just lived in perfectly normal, standard, slightly boring cities. But you wouldn't know any of that by looking at the Midjourney-generated illustrations for each one of them.
Bad information. I don't care that much that the AI suggests that Potsdam's Brandenburg Gate is a "mini version" of the one with the same name in Berlin (which AFAIK is not true), nor that it talks about getting lost "in the maze-like streets of the Old Town" for the squarest blocks you'll find outside of Manhattan. But when your list of attractions includes sightseeing points in cities more than 200km away... and in a different Province... I hope you remember to bring your good walking shoes.
Dangerous information. The site includes an "LGBTQ safety" rating which is auto-generated and not checked against any official source. The guide will also gladly include locations "off the beaten path" that are plain unsafe for a tourist. One such advice suggests "lounging in sunbathed plazas" in a city that Travel Safe - Abroad rates as having a high risk of pickpockets and medium risk for muggings. And while I'm not here to tell you how to live your life, I think "don't" is a solid advice for this situation.

The source code for the website is open, and the license seems to be "do whatever you want with it, no restrictions". And you know what? People will do whatever they want with it: they will spin God-knows how many clones (dog themed, child themed, circus themed, ...), slap ads on it, and the internet will drown a little more in auto-generated, unchecked, ad-ridden SEO garbage.

But hey, at least the website has the tiny footer "Beware, most of the content was generated by AI". So if you end up being robbed at gunpoint because you went "off the beaten path" it will actually be your fault.

One final digression: I decided to check some of the cities listed as "Do not travel" in the US State website and I landed on Kabul, Afghanistan, which the US warns you against visiting due to armed conflict, civil unrest, crime, terrorism, and kidnapping. The AI description reads "Kabul is a friendly city: despite the violence, Kabul residents are extremely hospitable and friendly towards visitors", which to me as a (computational) linguist is fascinating: if the locals are hospitable and friendly, then who is perpetrating "the violence"? Is this sentence blaming foreigners for the situation in Kabul? Can someone kidnap you and be hospitable at the same time? And since we know that a guest who kills their hosts in their own house loses privileged status, does robbing, kidnapping, and maybe murdering your visitors mean that you lose your status as a friendly city?

I would say that I'm maybe overthinking it, but let's be honest: given that no human thought of any kind went into this entire website, literally any attempt at understanding it is, by definition, overthinking.

(I have changed the title of this article to add a "(I)" at the end. I get the feeling this is far from the last entry in this series...)

Virtual Conferences: a proposal

2023-08-02T00:00:00+02:00

Looks like I wrote this article and forgot to publish it for a whole month. I hope the extra length will make up for it.

Now that the 61st annual meeting of the ACL is over, and based on my experience as a member of the Virtual Infrastructure Committee, I would like to share some thoughts on what my ideal virtual conference infrastructure would look like.

A conference, for our purposes, is an event that takes place in some random venue around the world. Researchers submit papers with their latest research results and, if accepted, present those results in sessions where related papers are bundled together. Conference attendees typically hear a short talk about the paper (or read a poster) and then ask questions to the authors.

Given that we live in a post-pandemic world we are going to prepare for a hybrid conference. This is cheaper (less mouths to feed means lower fees) more accessible (researchers in countries with less resources will thank you for the plane ticket and hotel costs they'll save), greener (less people flying around), easier to archive, and a good idea overall.

The handbook

It is traditional in conferences to offer a handbook containing all information about the conference - schedule, paper abstracts, venue maps, social events, tourist info, sponsors, and so on. But given that this year's booklet reached an impressive 462 pages it may be time to think about a website or app to replace it.

I do not believe that the conference website and the handbook can be unified, for reasons I'll explain later. So let's assume we want to develop a website or app exclusively for the handbook. Let's call it AppBook.

The main use cases for AppBook are:

Checking the schedule to figure out either what's going on at some time (exploration) or to find out when is a specific event taking place (information retrieval).
Adding reminder for events you would like to visit, from entire events down to specific talks. It is not unusual to be there for Paper 1 in Room A, then go to Room B next door for Papers 2 and 3, and so on.
Get details about a specific paper: if the title catches your attention then you may want to read its abstract, and/or you may want to see what else the authors are presenting.
General information about the venue: maps, tourist info, vouchers, etc.

The schedule is, I believe, the hardest part. Sure, there are plenty of scheduling apps around that do the information retrieval part quite well, but how about the exploration part? Taking the EACL 2023 handbook as an example (because I have it here on my desk): I can see that Session 6 on Day 2 has two Tracks I'd like to visit, namely, "Information Extraction" and "Generation". With the paper booklet I can put one finger in the Schedule page, flip back and forth through the pages for each Track, decide which one is more interesting at which time, and plan my visit accordingly.

Compare that to a regular website calendar: clicking on a Track takes you to the abstracts, but clicking "back" takes you to the general calendar. In the specific case of the talk I heard today that means scrolling back down to the correct time slot, clicking on "+", finding the right Track (there are 13 parallel workshops), open this Track and, since the interface doesn't allow for right click, remember somehow what the previous one had to offer. Repeat this for all 3-4 Tracks that one could consider, and this quickly turns into a mess.

This leads me to the first three requirements:

AppBook has to be lightning fast, which in this case means offline. Sure, you can periodically download updates to the database, but there is no way to implement "lightning fast" when you have 2k+ attendees checking the website at the exact same time with shoddy WiFi. I can tell you that a full conference fits in ca. 10Mb of JSON.
The interface has to trivially allow back-and-forth between all events taking place in parallel. Swiping left/right to switch Tracks could work, but I'd have to play with a prototype first to ensure it's not confusing.
It has to be mobile first. If you want to get rid of the handbook you need to ensure that even the humblest of devices can provide an equal or better experience than the paper version. When in doubt, remember that computer interfaces could already do this type of stuff in real time 20 years ago.

Chat

Even though I insisted on AppBook being offline first, it would be foolish to ignore all it could do if we judiciously let it go online. At its most basic, you should be able to download the actual papers directly from the interface. The next step is allowing access to both pre-recorded and live videos, which is a lot of work on the backend but relatively straightforward on the frontend. And if your internet is down you'd still have the handbook information.

Chat, on the other hand, is where I think the app could shine. Having a dedicated chat for every paper is a great way to ensure that all attendees can contact you, which helps in-person attendees to talk to you even outside your presentation window and empowers virtual attendees to talk to authors they would otherwise miss. And even better: once you allow attendees to create their own channels you can sit back, relax, and watch communities of like-minded people come together, plan events, and have fun.

If you choose to go this way then you could consider adding a feature to AppBook: "log me in as author". If you register as the author of a specific paper then you would get some extra functionality such as administrator rights over channels for your papers, notifications whenever someone posts in there, and reminders of where and when you are supposed to go.

And finally, if you use something like Zoom then you may want to disable the chat function in videos. Sure, people may be used to it, but if you leave them on then now you have two sources of questions and you've fragmented your userbase (see the first paragraph of "Supporting Infrastructure").

Website

If you are a virtual attendee, spending a week looking at your phone would probably be torture. More so when there's almost certainly a big screen somewhere around that you could be using instead. So we need a website too.

Could AppBook and this website be merged together? I don't want to say "no", but I will however say "well... errmmm..." while making face gestures. See, the website and AppBook have different audiences and different use cases, to the point that unifying them might just be impossible:

Speed:
- AppBook: lightning fast.
- Website: regular website speed.
Data source:
- AppBook: offline first.
- Website: online first.
Device:
- AppBook: designed for your mobile phone.
- Website: designed for your laptop and/or desktop computer.
Interface:
- AppBook: one window at the time. Interaction is everything. You may need to go native, but maybe a web app would do.
- Website: multiple windows at the time. Most interaction issues can be solved opening a second tab.
What would you use it for:
- AppBook: figure out what's going on and/or where to go.
- Website: stream live events, read papers.

Poster sessions

Posters are tricky. The current best solution I've seen is Gather.Town, but now you are depending on an external platform. I imagine you could get rid of poster sessions altogether with a combination of pre-recorded videos and chat, which is coincidentally what the ACL Conference offers every other year. Then again, they also offer Gather.Town, so it's hard to say which one is better.

In-person notes

Organizing a conference is a lot of work, but luckily we've done it for long enough that there isn't much new that I could say. I should however point out some details to keep in mind:

All talks should be live-streamed and questions should be accepted both in-person and via chat.
Every room needs the proper equipment for remote presenters. That means being prepared for presenters joining via teleconference and, conversely, having a mic at hand for attendees to pose their questions.

Supporting infrastructure

In my ideal conference platform, there is exactly one source of truth and one way to get information. There are no two schedules needing coordination, nor two chat platforms, nor two video platforms. There is one way and one way only.

In other words, the infrastructure would be built as such:

A central "source of truth" database with all information about the conference. You change it there and it is reflected everywhere (which in the case of AppBook means "eventually").
A file hosting platform with every paper and poster.
Some kind of video platform for live-streaming events.
The website, which is your main portal to all those things above.
AppBook, which presents a subset of the above information but in an easy-to-query format.
A chat app: bundling this inside AppBook would probably be a bad idea, and a dedicated app may be better. AppBook could query the chat backend once in a while, but only as long as it doesn't make the app slower in any way. Bundling it with the website, on the other hand, would make perfect sense. And failing that, an "open in app" link should also work fine.

Should AppBook allow you to synchronize your reminders with the website (and vice-versa) to get notifications across devices? I get the feeling that "yes" would be the best solution (and again, as long as it doesn't make AppBook any slower), but this would significantly impact the complexity of the whole operation. So this remains an open question.

Conclusion

I don't know of any platform offering all of this at once. Most platforms I know offer one feature or the other with varying degrees of success but I haven't yet seen a good implementation of AppBook.

And please, I beg you: few things are as annoying as an app that abuses its dominant position. If you use my login information for crawling my LinkedIn information and "connect me" to other people in my (wrongly scraped!) company, I may not have a recourse right there and then (there's rarely an alternative conference app) but I will make it my goal to ensure that no one uses your app again.

10k running plan

2023-06-05T00:00:00+02:00

I am currently considering whether to run a 10k and decided to take a look at the available training plans. This being the internet I found a whole lot of reasonably good training calendars but, this being the internet, they are all detailing the distances in ~~freedom units~~ miles.

Seeing as I now spent an unreasonably long time translating one of those calendars, I have decided to put it here simply because I can. The original plan with much more detailed information can be found in this website.

Week	Mon	Tue	Wed	Thu	Fri	Sat	Sun
1	Rest	4.8k	CT	4.0k (race pace)	Rest	4.8k	30 min EZ
2	Rest	4.8k	CT	4.8k (race pace)	Rest	5.6k	35-40 min EZ
3	Rest	5.6k	CT	5.6k (race pace)	Rest	6.4k	35-40 min EZ
4	Rest	6.4k	CT	5.6k (race pace)	Rest	7.2k	40-45 min EZ
5	Rest	6.4k	CT	4.8k (race pace)	Rest	8.0k	40-45 min EZ
6	Rest	6.4k	CT	5.6k (race pace)	Rest	9.6k	40-45 min EZ
7	Rest	6.4k	CT	4.8k (race pace)	Rest	11.3k	40-45 min EZ
8	Rest	4.8k	CT or Rest	4.8k (race pace)	Rest	Rest	10k

Legends:

CT: Cross training. Swimming, biking, strength training, or even rest if you're feeling down. Anything but running.
EZ: Easy, comfortable pace. Walking is fine. Cross training is fine too.
Race pace: you are free to run at a comfortable pace the rest of the week, but on those days you should stick to your intended race speed (roughly 10km/h for the average runner).

Making your own Linux game cards

2023-05-28T00:00:00+02:00

As someone who's constantly fidgeting, I tend to play some Solitaire during online meetings. Note that I didn't say "boring online meetings" because, let's be honest, the meeting can be the most interesting thing ever and yet I still need something to keep my hands busy.

In Linux I use Aisleriot, and I'm really happy with it. The only (frankly, very minor) point I have to object is that I don't like the default card look. This being Linux, this is an issue that can be solved with a single click or two: Click in View, go to Card Style, and then choose a different style. And since I also installed gnome-cards-data, I have 30+ card styles to choose from.

One of those card styles is called "Slavic Costumes", a set based on this beautiful card deck from 1911 that was added to Gnome on May 1, 2022 and removed on August 16, 2022 for being on shaky ground regarding copyright. Lucky for me the change has not made it into Devuan yet, and therefore I get to use it without having to go to this commit and downloading the file myself.

But the problem (well, "problem") is that I'm not familiar with Cyrillic, and having cards with "В, Д, К, Т" instead of "J, Q, K, A" is breaking my brain. So I changed the design using a couple commands and Inkscape:

Find the source deck file, which in my case is stored in the directory /usr/share/aisleriot/cards/.
Copy the card file slavic_costumes.svgz to another directory and rename it to slavic_costumes_2.svg.gz. This is because, while Inkscape can open .svgz files without any issues, I want to edit the file as plain text in step 5.
Decompress the file with gzip: gzip -d slavic_costumes_2.svg.gz.
Edit the file with Inkscape. This is detailed a couple paragraphs down.
Edit the file with a text editor and change the sodipodi:docname property to match the new file name. I'm not sure whether this step is required.
Re-compress and re-rename the file: gzip slavic_costumes_2.svg and then mv slavic_costumes_2.svg.gz slavic_costumes_2.svgz
Moved the new file to the directory ~/.local/share/aisleriot/cards/

As for the edition with Inkscape, you need to do the following:

Select the card you want to edit and double-click on the element you want to change - in my case, that would be (say) the letter "Д" of one of the cards. In modern Inkscape this will allow you to edit a group of objects (which all cards are) without destroying the group itself.
Work on the card until it looks fine. In my case this meant replacing the two card letters with a boring "Q".
Click on another card, then right-click back on your original card and select "Object properties". Make sure that the "ID" and "Label" properties of the finished card have retained their original readable name, which is something like club_queen. Otherwise your card set may not work.
Repeat for all cards until you are happy.

And with that I have tweaked my own card set. I didn't find these steps detailed anywhere, so I thought it would be a good topic for a blog post. I can't promise that I'll look around for different deck designs to contribute (Heraclio Fournier, anyone?), but if you want to do it for your own entertainment, well, now you know how.

I miss VGA: On using computers when you don't want to

2023-05-16T00:00:00+02:00

I am constantly having issues with HDMI and I never really know what the actual issue is. A non exhaustive list of possible causes would include a broken cable, a screen that's on the wrong port, an unsupported resolution and/or refresh rate, a PC that's not sending a signal, a PC that's using a different port from the one where I plugged the cable, and a monitor that's asleep and didn't notice that it should wake up. Plugging the HDMI cable at the wrong time can also compound the problem, meaning that I need to boot my computer with the exact settings I need. Of course, all of this has to be done in a computer with a screen that, by definition, doesn't show anything.

You know what has gotten me out of this problem more than once? VGA. Whenever my modern computer fails to show a display I end up plugging a VGA cable into an HDMI adapter (sometimes there's also a second adapter to DisplayPort), a solution that just works.

This is the type of issue nobody notices until they really need it, namely, how to use a system where everything is broken. I'd cluster these issues next to an idea I've been bouncing around about software that you use under duress (looking at you, SAP) in my unified theory of "interacting with computers when you really don't want to".

Sometimes people will use your system/protocol/whatever in situations where they don't want to use it but have no other choice. Think of a work meeting where the other part is using a tool you hate (a.k.a. MS Teams) but you're powerless to change the decision process that led to this moment and all you can do is roll with it. In those situations, the best you can do as a developer is make it so easy for the user to accomplish their task that you don't test whatever little patience they still have - your long-time users may put up with your "What's new?" pop-ups, but your other users will be thankful if you spare them.

Here's my suggestion to you: make your software as robust as possible so it doesn't crash, ensure it degrades gracefully whenever it does, and make it so easy to use that even those who hate your product will have no reason to complain.

Making stuff up with LLMs

2023-05-16T00:00:00+02:00

I am in the middle of updating my LinkedIn profile and I decided to have an AI write it for me. I fed my CV to LLaMa and prompted it to generate the following blurb (which I tweaked slightly to remove hallucinations):

Me: Hey, LLaMa, why should I hire this candidate?

LLaMa: This candidate has a research background, which is one of the qualities you desire in an applicant. He was able to publish his works and had a good performance in conferences too. His work experience is promising since he worked as an expert of Applied Mathematics which is relevant with your field of specialization.

Me: Why is this candidate great for my company?

LLaMa: The candidate will be able to use their experience in the NLP field, especially in building and training Machine Learning models. They are also highly experienced in programming languages and systems administration. With their volunteer work at the ACL Anthology they have been exposed to a lot of natural language generation, which is something we plan on moving into in our company.

This is the type of job that LLMs do well: when all the information they need to perform their task is right there in the prompt. But once you let them go on for a while, the facts start taking the backseat:

LLaMa: (...) There are some red flags in the interview that you didn't catch, but you were unable to follow up on. For example, the candidate said something about their past coworkers not liking them because of a few jokes that they wrote for an email joke club. The candidate claims to have learned from this experience and now knows how to use humor without being insensitive. (...) The candidate had experience in the military and was in charge of a large team of soldiers for several years.

Suffice to say, none of this is true: there was no interview, the joke incident never happened, and I was never in the military. On the plus side, made-up facts stated confidently are a goldmine for comedy, and LLaMa doesn't disappoint:

Me: Why is this candidate probably a serial killer?

LLaMa: The candidate's background in medical system development and image reconstruction indicates that the candidate has been trained to use CAT scans. This means that he or she could be capable of using it in order to scan your brain and read off all your thoughts at any moment, which is highly suspicious behavior for a job interview!

Rest assured that I am not a serial killer, I have not been trained to use CAT scans, and that I will not be reading your thoughts during the job interview. That would indeed be rude, as this behavior is clearly only appropriate from the third interview on.

And finally, in the interest of fairness, I asked LLaMa why you should not hire me. The result is not entirely inaccurate, as this post proves by itself:

Me: Why should I not hire this candidate?

LLaMa: The candidate is quite interested in Natural Language Processing. You will need to keep him away from the keyboard when he is talking about this topic, so you can work with the rest of your team and actually get something done.

I tried asking a different way, but looks like LLaMa has my back:

Me: Why is this candidate bad for my company?

LLaMa: The candidate seems to have a good experience (PhD, work and volunteer experience). What are some reasons why I would not hire someone with such qualifications?

I don't know, LLaMa, you tell me!

Returning to Monkey Island

2023-04-07T00:00:00+02:00

I have just finished playing Return to Monkey Island (RtMI). This is the culmination of a 30+ years journey, and it is only fair that I dedicate a couple lines to it. In short: of all the Monkey Islands I played, I just had to pay for the worst one¹. I should have read the Wikipedia page and be done with it.

Also note that Yahtzee from Zero Punctuation made a lot of the same points I'm about to make, but of course he did it better.

Turbulent times ahead

Let's start with the easier stuff: is the game fun? Does it live up to its legacy? And what happened with that controversy about the art?²

Regarding the art, I disagree with the main complaints but I also have my own. On the topic of pixel art I side with series' creator Ron Gilbert: Monkey Island 1 and 2 were not supposed to be "pixel art games", but rather the best that technology could offer at the time. And while the final art style is not for me, I can nonetheless appreciate that the game has a personality. And I definitely disapprove of online abuse in any way. What I do object to, however, is the animation. Characters in RtMI look, feel, and move like paper dolls, a style that is cheap to animate but with the downside of also looking cheap. 1997's Curse of Monkey Island may not have been an "official" Ron Gilbert game, but that game was gorgeous and I'd pick it over RtMI every time.

As for the game itself, I'm struggling figuring out who is it for: it plays like the kind of game I'd recommend a beginner (puzzles are pretty easy), but beginners are not known for jumping straight into episode 5 of a saga. And veterans of the series have cut their teeth with much harder stuff. Sure, the scurvy puzzle is a good return to form, but if you are looking for multi-layered puzzles and rich, entertaining conversations with multiple characters then this is not the game you're looking for. There is no pixel-hunting, which is nice, but this is an innovation we had already seen in Thimbleweed Park.

The story and progression of the game is fine, just not great. There is humor in it, just not as much as I'd like³. As for its legacy, most of your favorites are back but relegated to minor roles. I feel particularly bad about Murray and Elaine - Murray doesn't do much, and Elaine is just here to be a mother figure for Guybrush and say "that's nice, honey" here and there. If this were your first Monkey Island game, the scene where she takes out her sword and fights would probably strike you as entirely out of character.

No game for old men

But my real problem with RtMI is its atmosphere. The game feels done with itself from the very beginning, and I got the constant impression of a game developed by people who would prefer me to do something else with my life⁴. The world of RtMI is falling apart, full of graffiti and derelict buildings. Our beloved characters have moved on and no one outside Guybrush and LeChuck even cares about the Secret of Monkey Island anymore, a fact that they'll repeat at every chance.

Which brings me to the whole reason for buying the game, namely, finding out what the Secret of Monkey Island is (which I won't spoil here). I am not against the conclusion itself - the idea presented in RtMI has been floating around forums for decades, I've made my peace with it long ago, and I had already accepted it as canon. My problem is that it is presented in the meanest and most unsatisfactory way possible. When Guybrush (minor spoiler!) says "I'm ready to go", Elaine's condescending "Good. Me too" sounds less like an ending and more like the game designers telling me to go away. Hell, even Guybrush's son calls them on it.

Interestingly enough, my main (positive) emotional reactions came not from where the game went but rather from where I thought the game would go. More than once I thought that Guybrush would lose Elaine, and I also occasionally worried that my choices would lead me to a bad ending. I'm glad neither of those things happened, and yet I can't shake the feeling that I put more thought into the story than the designers did.

End of an era

I have often told people on the internet to stop buying bad games, a concept that gamers refuse to accept for some reason. And I know that, if you have been waiting like me for a resolution to the Secret of Monkey Island, then nothing I write will dissuade you from playing the game. But honestly, from a fan to another? You might be better reading an online walkthrough and/or watching a video of the last 5 minutes. Nothing in this game is likely to have any influence in the future of the series anyway (assuming there is a future), and honestly I don't think I have it in me to keep listening to my once-beloved characters telling me how much they don't care.

I will always have a soft spot for Monkey Island, and I am glad that we finally got an official resolution. I just wish it was delivered in something other than a tired man telling me to get off his lawn and move on with my life.

Footnotes

Yes, I pirated the other ones, but that's okay - I pledged enough for Thimbleweed Park that I got a Certificate of Absolution out of it.
Make sure to follow this comic and its sequel on the topic.
While I'll always defend that Monkey Island I and II are hilarious, I should publicly accept that one of my favorite jokes is this scene from Curse of Monkey Island (about 40 seconds, Guybrush's third response).
Of course, I don't know what Ron Gilbert was actually thinking, and based on his articles on his website he seems like a nice man. Which makes the whole thing even more baffling: either he's tired of it all and never let on, or he isn't and made a mean game by mistake.

Forcing Spotify to take my money

2023-03-30T00:00:00+02:00

As I mentioned before I have a Spotify premium account that I regularly pay for with gift cards. Spotify does not like that: if they could get ahold of my credit card number they could not only stop worrying that I won't pay them anymore but also they could track me better across the web - people change their e-mail all the time, but their credit card and cell phone numbers tend to last much longer. Or even better, maybe I'll stop using the service altogether but forget to cancel, which is as close as free money as one can get.

Given that Spotify does not like people using gift cards, they don't offer a way for you to redeem your gift cards directly via the app. This is annoying: I do not know my Spotify password (that's what my password manager is for) and therefore renewing my account requires me to go home and sit in front of my PC. If only there were a way for me to redeem gift cards directly from the app...

But guess what? There is. After some probing I have found the following steps that seem to work reliably in Germany:

Open the Spotify app, click on the "Settings" wheel and go to "Premium plan".
Choose any of the available premium plans, as if you were about to subscribe to one of them.
Choose "Credit Card" as payment.
Without entering any information scroll down and click on the "Terms and Conditions" link.
This will take you to a new window. On the top right you'll see your profile picture. Click on it and choose "Support".
This takes you to Spotify's Support page. Type "gift card" (or, in Germany, "Geschenkkarten") and click the first link that comes up.
The link will take you to an article explaining how to redeem gift cards, including a link to https://spotify.com/de/redeem. Click on it.
This will take you to the Gift Card redemption website, with the nice extra that you'll be already logged in. So enter the gift card code and you're done.

This stupidly long series of steps works because Spotify's internal browser already has your credentials (otherwise you'd need a password every time you open the app) and because the T&C page gets you out of the expected workflow that restricts what you can do. And given that we know the app has your account info, all Spotify would have to do to support gift cards is to add an extra button under their Premium plans saying "Redeem Gift Card" taking you there.

It is amazing the kind of thing your apps can do for you once you force them into compliance.

Goodbye Reddit

2023-03-30T00:00:00+02:00

I heard about Reddit for the first time around August 2010, when Digg redesigned its website for various reasons (none of them related to improving the user experience) and lost most of its users (many of whom were already annoyed). The "Digg Exodus" is how I moved to Reddit, and I've used since then as my main source of news.

And yet, the shadow of a bad redesign hangs over Reddit. Even though they have been redesigning the website since 2018, their results have been uniformly awful. A quick test I made about a year ago showed that, while the "old" interface downloaded 410Kb of data and rendered in 81ms, the "new" interface would download 10Mb rendered in about 3.2s. Last week Reddit announced proudly that their team had reduced loading times by 2 seconds, which is still 15 times slower than the performance they had five years ago. The pop-ups, modals, hidden comments and low information density are not helping either.

I rarely saw any of this for one simple reason: i.reddit.com. This interface was a leftover from the earlier times of the mobile web and therefore it was fast, simple, and straight to the point. And precisely because it was so convenient at delivering the content that matters the most to me it had to die, killed by Reddit administrators yesterday and buried deep into this announcement post.

There's a widespread belief that the Reddit website is made awful on purpose with the end goal of exhausting you into using their mobile app where they can get better data about you and where you can no longer block their ads and sponsored posts. I do not know whether this theory is correct, but I can agree with half of it: now that i.reddit.com has shuffled off his mortal coil I am indeed exhausted, and today I am leaving the site for greener pastures. I have already made a GDPR request for all of my data in case I ever need to refer to some old post of mine, and once that's done I'll be gone for good.

Deciding where to go is a bit trickier. For tech news I already have Hacker News which is fine (a bit too obsessed with GPT at the moment, but we've been there before and hopefully this too shall pass). Local news will require some more research. I assume some part of Discord might do the trick, but I'd honestly prefer something more open.

So long, Reddit. It was fun while it lasted.

Fine-tuning a Transformer for text classification

2023-03-07T00:00:00+01:00

I have found myself often enough trying to classify a text that I decided to be lazy and write a solution that I could always rely on.

The following script is an example on how to pre-train a HuggingFace transformer for text classification. This example uses only two classes, identifying whether the sentiment of a text is positive or negative, but nothing stops you from adding more classes as long as you predict just one.

There are no difficult imports that I can tell - I think you can install all the required libraries with the command

pip install transformers datasets evaluate nltk

but I'll double check just in case.

Update: I have changed the script to extract the function get_dataset_from_json_files(). You can use this function as is to replace the calls to get_data(). I also updated the prediction code to give an example of how to make predictions for a lot of texts at once.

import json
import numpy as np
import os
import random
import tempfile
# These are all HuggingFace libraries
from datasets import load_dataset
import evaluate
from transformers import DistilBertTokenizer, DistilBertForSequenceClassification,\
     TrainingArguments, Trainer, TextClassificationPipeline
# KeyDataset is a util that will just output the item we're interested in.
from transformers.pipelines.pt_utils import KeyDataset
# Library for importing a sentiment classification dataset.
# Only here for demo purposes, as you would use your own dataset.
from nltk.corpus import movie_reviews

# The BERT tokenizer is always the same so we declare it here globally.
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-uncased")

def get_data():
    """ Returns a dataset built from a toy corpora.

    Returns
    -------
    Dataset
        An object containing all of our training, validation, and test data.
        This object stores all the information about a text, namely, the raw
        text (which is stored both as text and as numeric tokens) and the
        proper class for every text.

    Notes
    -----
    If you have texts in this format already then you can skip most of this
    function and simply jump to the `get_dataset_from_files` function.

    References
    ----------
    Most of this function is taken from the code in
    https://huggingface.co/course/chapter5/5?fw=pt.
    Note that that link is very confusing at times, so if you read that code
    remember to be patient.
    """
    # We read all of our (toy) data from NLTK. The exception catches the case
    # where we haven't downloaded it yet and downloads it.
    try:
        all_ids = movie_reviews.fileids()
    except LookupError:
        import nltk
        nltk.download('movie_reviews')
        all_ids = movie_reviews.fileids()
    # We identify texts by their ID. Given that we have those IDs now, we
    # proceed to shuffle them and assign each one to a specific split.
    # Note that we fix the random seed to ensure that the datasets are always
    # the same across runs. This is not strictly necessary after creating a
    # dataset object (because we want to reuse it), but it is useful if you
    # delete that dataset and want to recreate it from scratch.
    random.seed(42)
    random.shuffle(all_ids)
    num_records = len(all_ids)
    # This dictionary holds the name of the temporay files we will create.
    tmp_files = dict()
    # We assigne the IDs we read before to their data splits.
    # Note that we use a fixed 80/10/10 fixed split.
    for split in ['train', 'val', 'test']:
        if split == 'train':
            split_ids = all_ids[:int(0.8 * num_records)]
        elif split == 'val':
            split_ids = all_ids[int(0.8 * num_records):int(0.9 * num_records)]
        else:
            split_ids = all_ids[int(0.9 * num_records):]
        all_data = []
        # We read the data one record at the time and store it as dictionaries.
        for text_id in split_ids:
            # We read the text.
            text = movie_reviews.raw(text_id)
            # We read the class and store it as an integer.
            if movie_reviews.categories(text_id)[0] == 'neg':
                label = 0
            else:
                label = 1
            all_data.append({'id': text_id, 'text': text, 'label': label})
        # Step two is to save this data as a list of JSON records in a temporary
        # file. If you use your own data you can simply generate these files as
        # JSON from scratch and save yourself this step.
        # Note that we also save the name of the temporary file in a dictionary
        # so we can re-read it later.
        tmp_file = tempfile.NamedTemporaryFile(delete=False)
        tmp_files[split] = tmp_file.name
        tmp_file.write(json.dumps(all_data).encode())
        tmp_file.close()

    # Step three is to create a data loader that will read the data we just
    # saved to disk.
    dataset = get_dataset_from_json_files(tmp_files['train'], tmp_files['val'], tmp_files['test'])

    # Remove the temporary files and return the tokenized dataset.
    for _, filename in tmp_files.items():
        os.unlink(filename)
    return dataset


def get_dataset_from_json_files(train_file, val_file=None, test_file=None):
    """ Given a set of properly-formatted files, it reads them and returns
    a Dataset object containing all of them.

    Parameters
    ----------
    train_file : str
        Path to a file containing training data.
    val_file : str
        Path to a file containing validation data.
    test_file : str
        Path to a file containing test data.

    Returns
    -------
    Dataset
        An object containing all of our training, validation, and test data.
        This object stores all the information about a text, namely, the raw
        text (which is stored both as text and as numeric tokens) and the
        proper class for every text.

    Notes
    -----
    IMPORTANT: the first time you run this code, the resulting dataset is saved
    to a temporary file. The console will tell you where it is (in Linux it is
    /home/<user>/.cache/huggingface/datasets/...). This temporary location is
    used in all subsequent calls, so if you change your dataset remember to
    remove this cached file first!

    This function is a thin wrapper around the `load_dataset` function where
    we hard-coded the file format to use JSON.
    If you don't want to read that function's documentation, it is enough to
    provide files whose content is simply a JSON list of dictionaries, like so:
    [{'id': '1234', 'text': 'My first text', 'label': 0}, {'id': '5678', 'text': 'My second text', 'label': 1}]
    """
    files_dict = {'train': train_file}
    if val_file is not None:
        files_dict['val'] = val_file
    if test_file is not None:
        files_dict['test'] = test_file
    dataset = load_dataset('json', data_files=files_dict)

    # Tokenize the data
    def tokenize_function(examples):
        return tokenizer(examples['text'], padding='max_length', truncation=True)
    tokenized_dataset = dataset.map(tokenize_function, batched=True)

    # Return the tokenized dataset.
    return tokenized_dataset


if __name__ == '__main__':
    # In which mode we want to use this script
    # 'train': start from a pre-trained model, fine-tune it to a dataset, and then
    #          save the resulting model
    # 'test': evaluate the performance of a fine-tuned model over the test data.
    # 'predict': make predictions for unseen texts.
    mode = 'predict'
    assert mode in ['train', 'test', 'predict'], f'Invalid mode {mode}'
    # We define here the batch size and training epochs we want to use.
    batch_size = 16
    epochs = 10
    # Where to save the trained model.
    trained_model_dir = 'training_output_dir'
    if mode == 'train':
        # Train mode. Code adapted mostly from:
        # https://huggingface.co/docs/transformers/training#train-with-pytorch-trainer.
        # Collect the training data
        inputs = get_data()
        # Since we are starting from scratch we download a pre-trained bert model
        # from HuggingFace. Note that we are hard-coding the number of classes here!
        model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)
        # Define training arguments
        # You can define a loooot more hyperparameters - see them all in
        # https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/trainer#transformers.TrainingArguments
        training_args = TrainingArguments(output_dir=trained_model_dir,
                                          evaluation_strategy='epoch',
                                          per_device_train_batch_size=batch_size,
                                          num_train_epochs=epochs,
                                          learning_rate=1e-5)
        # Define training evaluation metrics
        metric = evaluate.load('accuracy')

        def compute_metrics(eval_pred):
            logits, labels = eval_pred
            predictions = np.argmax(logits, axis=-1)
            return metric.compute(predictions=predictions, references=labels)
        # Define the trainer object and start training.
        # The model will be saved automatically every 500 epochs.
        trainer = Trainer(model=model,
                          args=training_args,
                          train_dataset=inputs['train'],
                          eval_dataset=inputs['val'],
                          compute_metrics=compute_metrics)
        trainer.train()
    elif mode == 'test':
        # Perform a whole run over the validation set.
        # This code is adapted mostly from:
        # https://huggingface.co/docs/evaluate/base_evaluator
        # Some useful information also here:
        # https://huggingface.co/docs/datasets/metrics

        # Collect the validation data
        inputs = get_data()
        # Create our evaluator
        task_evaluator = evaluate.evaluator("text-classification")

        # Use the model we trained before to predict over the validation data.
        # Note that we are providing the name of the classes by hand. We are not
        # using these labels here, but it is useful for the prediction branch.
        model = DistilBertForSequenceClassification.from_pretrained(f'./{trained_model_dir}/checkpoint-1000/',
                                                                    num_labels=2,
                                                                    id2label={0: 'negative', 1: 'positive'})
        # Define the evaluation parameters. We use the common text evaluation
        # measures, but there are plenty more.
        eval_results = task_evaluator.compute(
            model_or_pipeline=model,
            tokenizer=tokenizer,
            data=inputs['test'],
            metric=evaluate.combine(['accuracy', 'precision', 'recall', 'f1']),
            label_mapping={"negative": 0, "positive": 1}
        )
        # `eval_results` is a dictionary with the same keys we defined in
        # the `metric` parameters, plus some time measures.
        print(eval_results)
    elif mode == 'predict':
        # Make predictions for individual texts.
        # Same as above, we use the model we trained before to predict a single text.
        model = DistilBertForSequenceClassification.from_pretrained(f'./{trained_model_dir}/checkpoint-1000/',
                                                                    num_labels=2,
                                                                    id2label={0: 'negative', 1: 'positive'})
        # We build a text classification pipeline.
        # Note that `top_k=None` gives us probabilities for every class while
        # `top_k=1` returns values for the best class only.
        # Inspired on https://discuss.huggingface.co/t/i-have-trained-my-classifier-now-how-do-i-do-predictions/3625
        pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer,
                                          batch_size=batch_size, top_k=None)
        sequences = ['Hello, my dog is cute', 'Hello, my dog is not cute']
        predictions = pipe(sequences)
        print(predictions)
        # The above print shows something like:
        # [[{'label': 'positive', 'score': 0.574},
        #   {'label': 'negative', 'score': 0.426}],
        #  [{'label': 'negative', 'score': 0.868},
        #  {'label': 'positive', 'score': 0.132}]]

        # For a more heavy-duty approach we now classify an entire dataset.
        inputs = get_data()
        test_inputs = inputs['test']
        outputs = []
        # Note that 'text' is the key we defined in the `get_data` function.
        for out in pipe(KeyDataset(test_inputs, 'text'), batch_size=batch_size, truncation=True, max_length=512):
            outputs.append(out)
        # Now that we collected all outputs we massage them a little bit for
        # a more friendly format. For every prediction we will print a line like:
        # 'pos/cv093_13951.txt: positive (0.996)'
        for i in range(len(test_inputs)):
            id = test_inputs[i]['id']
            pred_label = outputs[i][0]['label']
            pred_prob = outputs[i][0]['score']
            print(f'{id}: {pred_label} ({pred_prob})')

Thoughts on ChatGPT, LLMs, and search engines

2023-03-01T00:00:00+01:00

Now that everyone is talking about Large Language Models (LLMs) in general and ChatGPT in particular I thought I would share a couple thoughts I've been having about this technology that I haven't seen anywhere else. But first, let's talk about chess.

Chess is a discipline notorious for coming out on top when the robots came to take its lunch. Once it became clear that even the humblest of PCs can play better than a World Chess Champion, the chess community started using these chess engines to improve their games and learn new tactics. Having adopted these engines as a fact of life, chess is as popular as ever (if not more). Sure, a computer can do a "better" job on a fraction of the time, but who cares? It's not like there's an unmet economic need for industrial-strength chess players.

In contrast, one field that isn't doing as well is digital art. With the advent of diffusion models like Midjourney and Stable Diffusion many artists are worried that their livelihoods are now at stake and are letting the world know, ranging from comics (I like this one more but this one hits closer to home) all the way to class action lawsuits. My quick take is that these models are here to stay and that they won't necessarily destroy art but they will probably cripple the art business the same way cheap restaurants download food pictures from the internet instead of paying a professional photographer.

LLMs are unusual in that sense because they aren't disrupting a field as much as a means of communication. "Chess player" is something one does, and so is "artist"¹. But "individual who uses language" is on a different class altogether, and while some of the effects of these technologies are easy to guess, others not so much.

Random predictions

What are we likely to see in the near future? The easiest prediction is a rise in plagiarism. Students are already using ChatGPT to generate essays regardless of whether the output makes sense or not. And spam will follow closely behind: we are already awash in repost spam and ramblings disguised as recipes, but once people start submitting auto-generated sci-fi stories to magazines we will see garbage in any medium that offers even the slightest of rewards, financial or not. Do you want a high-karma account on Reddit to establish yourself as not-a-spammer and use it to push products? Just put your payment information here and the robots will comment for you. No human interaction needed².

What I find more interesting and likely more disruptive is the replacement of search engines with LLMs. If I ask ChatGPT right now about my PhD advisor I get an answer - this particular answer happens to be all wrong³, but let's pretend that it's not ⁴. This information came from some website, but the system is not telling me which one. And here there's both an opportunity and a risk. The opportunity is the chance of cutting through the spam: if I ask for a recipe for poached eggs and I get a recipe for poached eggs then I no longer have to waddle through long-winded essays on how poached eggs remind someone of evening at their grandma's house. On the other hand, this also means that all the information we collectively placed on the internet would be used for the profit of some company without even the meagre attributions we currently get.

On the long tail, and entering into guessing territory, it would be tragic if people started writing like ChatGPT. These systems have a particular writing style composed of multiple short sentences and it's not hard to imagine that young, impressionable people may start copying this output once it is widespread enough. This has happened before with SMS, so I don't see why it couldn't happen again.

Pointless letters and moving forward

One positive way to move forward would be to accept that a lot of our daily communication is so devoid of content that even a computer with no understanding of the real world can do it and work on that.

When I left my last job I auto-generated a goodbye e-mail with GPT-3, and the result was so incredibly generic that no one would have been able to learn anything from it. On the other hand, I doubt anyone would have noticed: once you've read a hundred references to "the good memories" you no longer stop to wonder whether there were any good memories to begin with. I didn't send that auto-generated e-mail. In fact, I didn't send anything: I had already said goodbye in person to the people that knew me and there was no reason to say anything else. The amount of information that was conveyed was exactly the same, but my solution wasted less of other people's time.

Maybe this is our opportunity to freshen up our writing and start writing interestingly, both in form (long sentences for the win!) and in content. The most straightforward solution would be cursing: these models have to be attractive to would-be investors they are strictly programmed not to use curse words and NSFW content (I just tried). So there's a style that no AI will be copying in the near future.

Footnotes

Note that this is a simplification for the sake of the argument. As someone who often said "being a programmer is both what I do and what I am" I am aware that "artist" (like so many other professions) isn't just a job but also a way of looking at the world.
I have noticed that people on Reddit will upvote anything without reading it first, so this is not a high bar to clear.
The answer mashed together several researchers into one. One could argue that I got more researchers per researcher, which is definitely a take.
The answer doesn't have to be correct - all it takes is for the person using the system to believe that the answer is correct, something we are already seeing despite the overwhelming evidence to the contrary.

Smartening up, Part II

2023-02-21T00:00:00+01:00

In my previous entry about home automation I mentioned that I was ready to start plugging things into other things and connecting everything to a central computer running OpenHab. This is the story about how everything went wrong.

Networking

Things starting going badly from the get go when I discovered that whatever "vectoring" is it has rendered my DSL router obsolete. Only two (available) modems had the right technology: a TP-Link TD-W9960v and a Fritz!Box 7530 AX. I should have gone with TP-Link, but alas, I did not: I decided against the TP-Link because it was not supported by OpenWRT, but failed to notice that the other one was not supported either.

The reason I regret my choice is because the Fritz!Box router provides DSL, Telephone, Wifi, WLAN Mesh, Media server, NAS, some type of IoT compatibility, it updates automatically, and who knows what else out of the box. I have never owned any other device with such a large attack surface, and yet I am expected to put all of my data in it and then plug it to the internet? Yeah, no. I disabled everything instead, plugging all of my devices to my old router (which uses dd-wrt after OpenWrt dropped its support for it), relegated the Fritz!Box to a mere modem, and added every network access restriction I know to keep the later from entering my internal network.

Speaking of my old router, setting up sub-networks in dd-wrt was like pulling teeth thanks to my favorite internet problem: documentation that still refers to older, deprecated versions of software that nonetheless rank stupidly high in search engines because they have been around forever. In my case, all links pointed to this complicated method that has been improved by this much simpler method. Do you know how frustrating it is to debug network errors when bad configuration means you can no longer access the thing you are debugging? Because I do.

Smart lights

This is the only part that worked as intended, only to realize that what I thought I wanted is not what I actually needed.

I configured the lights to use their own Guest WiFi network that, once I'm done, will be disconnected from the internet entirely. That part worked well, and in fact I am typing way after midnight with the "candlelight" setting just because I can. The tricky part was the dilemma that came afterwards.

In order to work properly, the lights remain "on" all the time (to receive commands) and you turn them on/off via software. So what happens if I enter a room without my phone? At this moment I have three choices: * I can go back, pick my phone, unlock it, turn on the light, and then turn it off once I leave * I can turn the light off and then on again, turning the lights on. I then need to leave them on and turn them off via software once I have my phone back. * I can stay in the dark and convince myself that this is fine.

The solution to this problem is a small remote control I can keep on me all the time and turn lights on and off with as many programmable buttons as smart lights. I know this is a good solution because every single compatible remote is currently sold out.

Heating

Even before lights became an issue, my dream was to control my heating to turn it on when I get up and turn it off shortly after. My first step was to buy a cheap Bosch thermostat. This was a bad choice, as Bosch equipment require a central hub with constant internet connection. If you want to turn off your heating you have to send your request to Bosch headquarters, who in turn will tell your hub to tell your heating that it should turn off. Good thing I only bought one.

My second attempt was an Eurotronic Spirit Z-Wave plus. These devices connect via Z-Wave, a protocol that requires a dongle instead of internet connection. My research suggested an Aeotec Z-Stick, which would have been a fine choice if I had stuck to v5. Instead, I bought the newest v7 (newer is always better, right?) which has known compatibility issues and whose firmware can only be upgraded using Windows (a luxury I no longer have). So I bought a v5 and all those problems went away. Both the v5 stick and the Bosch thermostat will be landing on eBay eventually.

OpenHab

All of my equipment was supposed to be controlled with OpenHab, an experience that so far has been mixed: when it works it works great, but when it doesn't it is quite difficult to find out why. That compatibility problem I mentioned above regarding Z-Wave? All I found in the forums (before giving another stick a try) was the suggestion that my thermostat was asleep and that all I had to do was sit next to it and keep it awake for an indeterminate time until it worked. Once I changed the stick, however, everything worked just fine.

I installed OpenHab in my old laptop, where things worked mostly fine. But once I decided to move it to its own Raspberry Pi (using the OpenHabian distro) it all fell into pieces: my Pi is too underpowered, the memory card too slow, and the default Java installer doesn't support my architecture. So not only do I need a new dongle, but also a new Raspberry Pi.

Progress and next steps

The next steps are getting a proper Raspberry Pi, a remote control, and make everything talk to everything else. I still have some hopes for OpenHab - the thermostat have eventually been detected, the lights worked just fine, and the new old Z-Wave dongle does its job. I could look deeper into its voice recognition functions, but since my plan was installing Mycroft I might as well stick to that.

With the working up I have finally been able to re-plug my TV. The TV is old-ish and therefore controlled with a Raspberry Pi 3 running OSMC, with all of its content coming either from YouTube or the NAS (which I have once again turned on). This is not technically part of the smart home, but it is worth mentioning anyway. And no, I am not re-purposing this Raspberry Pi for OpenHab.

And last but not least, a word of caution: living in a non-functioning house (well, apartment) is exhausting, and I do not recommend following my steps until you know things will work fine.

I'll keep you updated.

How my brother's iCloud account was stolen

2023-01-16T00:00:00+01:00

My brother got his iPhone stolen at gunpoint. This is the story of how he lost control of his iCloud account first (along with years of priceless memories, including my nephew's first steps) and how this couldn't have happened without Apple's incompetent support.

But before that, a plea for help: if you know of anyone who can help us get the account back (or rather, the priceless photos locked inside) please get in touch with the links at the end of the article.

Part 1: Locked out of iCloud

The first thing the thieves did (after running away, of course) was changing the phone number associated with the iCloud account. I do not know how they did this - it has been suggested that Apple will send a code to your phone, which the thieves obviously had. In any case, as soon as my brother tried to log into "Find my Phone" he was faced with a screen asking him to verify the phone number associated with the account, which was set to a number we do not recognize. It didn't matter that we still had the proper password for the iCloud account, nor that we still have control of the e-mail associated with the account. As far as Apple is concerned, if you don't know the phone number (which, again, the thieves changed) you cannot access your iCloud account. This is a known issue with iCloud security.

Next we got in touch with Apple, both via phone and Twitter. Phone support was useless, as all they would say was that they couldn't help us unless we knew the phone number. The closest I got to a victory was getting the phone representative to say out loud that she wanted me to provide the phone number the thieves gave, but that's about it. Even if we have the old phone number, the iCloud e-mail, the iCloud password, a police report and multiple IDs, Apple will not budge. Twitter support was even worse: after repeatedly asking them to read what I said two messages ago, I ended up getting this pointless, infuriating response:

We completely understand the concern with how important it is to have this resolved as soon as possible, and we were able to locate the cases from when you had previously reached out. Based on the information that you've provided, and the steps you and your brother have gone through, if your brother is unable to regain access to his Apple ID we would be unable to provide any additional methods to regain access to the account, and we would be unable to change the trusted phone number on the account. If you have any other questions or concerns regarding this issue, the best option would be to reach back out to our Apple ID experts at the link provided in our previous message (Note: that's the phone we called before).

Part 2: Losing iCloud for good

But the story gets even worse (better?). While we were stuck in phone support hell, my brother got a phishing SMS. He didn't recognize it as such, and lost the phone for good. The trick works like this: once you get a new SIM card (which the thieves can tell because the old one stops working) the thieves send you a phishing e-mail pretending to be from Apple. You follow the link, give your iCloud username and password, and now the thieves can unlock your iPhone and resell it. Crucially, this step only works because thieves know that Apple support will not help you: if Apple had been of ANY help then we would have recovered access long before the SMS and my brother wouldn't have followed the link.

According to Gizmodo (in Spanish) the next step would have been a phishing call with spoofed caller ID. But we will never know.

Sidenote: I reported both URLs (https://apple.iforgot-ip.info and https://apple.located-maps.info) to their hosting providers. Results have been mixed. PublicDomainRegistry.com (which belongs to Webhostbox) will not take down the hosting addresses unless they can see the phishing attack themselves (they won't check logs), but good luck getting a one-time link to work twice. UnifiedLayer.com was helpful, and I believe that GoDaddy revoked their domains.

Part 3: No more Apple

The iCloud website promises "all your stuff — photos, files, notes, and more — is safe, up to date, and available wherever you are". You have now seen what "safe" means in this context: it will be in the cloud, yes, but that doesn't mean you'll have access to it.

The question now is: would my brother choose Apple again? An iPhone is not cheap in general, and in Argentina less so. The current price for an iPhone 13 is ca. 400.000 ARS, which roughly translates to 2200 USD or 1300 USD at the unofficial rate (it's complicated). With an average monthly salary of 427 USD (according to Numbeo) you can see that getting a new iPhone is not a choice to take lightly.

What will my brother do? Paraphrasing from a conversation we had: "if I could get my account back I could consider getting a new iPhone. But if I have to start from scratch then it doesn't make sense. If I can't get my data back I'll probably get an Android phone instead".

Part 4: Conclusion and next steps

I have not entirely given up, but I'm not keeping my hopes up either. We are currently looking into whether my brother's wife can get access to his files (they had some kind of shared access), whether his iWatch can be of any use (it was logged into the iCloud account), whether small claims court is likely to help (I know it would work in the EU but Argentina is trickier), and whether anyone in my extended network can reach someone at Apple who is not an AI (thanks for nothing, LinkedIn).

As for next steps, I will be gifting my family access to some cloud storage, but unless I can get a service with competent tech support I may end up setting up a cloud of my own. Hopefully the loss of my nephew's first steps will not be in vain.

Do you have any ideas? Do you work for Apple? Then send me an e-mail or get in touch in Mastodon or even Twitter.

A bag for life

2022-12-26T00:00:00+01:00

About 8 years ago I was searching for a new messenger bag - the one I had was good but too small, its only one big pocket was inconvenient, and the zipper was starting to fail. And since I was planning on bringing this bag with me everywhere, I needed a bag that could survive being dragged on a classroom floor, exposure to sun and rain, and fit under an airplane seat. Anything requiring "gentle care" was immediately out. Backpacks were also ruled out for a simple reason: in my country of origin backpacks are often cut open to steal their contents, so I wanted a bag I could always keep my eyes on.

My first purchase was a Timbuk2 bag which I bought on Amazon because there were no local retailers (looks exactly like this). Their marketing at the time promised "the last bag you'll ever buy" (or something to that effect), which was exactly what I wanted. But you know what the problem with lifetime guarantees is? That you can only judge them with customers who have used them for a lifetime, which disqualifies essentially everything made after at least the 2000s. Luckily (for you) you don't have to wait that long because a year or two after buying my bag the waterproofing started peeling off (along with a bit of the interior). As for the "lifetime guarantee", it turns out it is only valid for customers who bought their bag in an official shop. I got mine from Amazon, so no servicing for me. It has now been relegated to ugly sports bag status.

My second bag is an Aunts & Uncle's leather bag that I bought 7 years ago. The actual model is no longer for sale, but this one is very similar. I chose leather because, unlike more modern materials, I knew this would last forever (we still have bags from the 1800s around after all). And to be fair the bag's exterior is fine - it's a bit beaten up, but that's just part of its charm. The real problem is that the bag doesn't close anymore: the closing system uses Velcro (sorry, "hook and loop") which always made the bag impossibly loud to open. It was loud in daily life, it was loud in a quiet classroom, and it was definitely loud on a train at night. Eventually the material gave up and stopped closing altogether, which made it much quieter (good) at the cost of risking everything falling out (bad). I got in touch with the manufacturer after reading their sustainability compromise to get some proper belts installed, but the only solution they offered was to replace the Velcro and make it loud again.

I'm currently looking into leather working to see whether I could install some belts myself without making a mess. I would love to pay someone to do it, but apparently my city has exactly one woman who could do the job and she's on a sabbatical. I'll also probably peel all of the remaining waterproofing flakes of my other bag, spray it with some generic waterproofing, and make it look at least decent.

What would I recommend to someone who wants "a bag for life"? My suggestion would be to go for something simple that one can fix by themselves: while I currently don't have the leather working skills for doing the job, at least there's a clear path forward for my second bag. But sourcing, cutting and reapplying plastic liner for the first one? Good luck with that.

Further reading: some interesting comments regarding this article on the quality of modern home appliances.

ML models in Flask

2022-11-29T00:00:00+01:00

Does this situation sound familiar to you?

You are a data scientist, you developed an ML model in Python (using PyTorch, TensorFlow, or something like that), and you'd like your users to interact with it,
You would like to make either an API or a web interface to your model,
Your model is big enough (and therefore slow) that you would prefer not to load it from scratch every time a user wants to use it, and
You know a thing or two about servers, but you don't have a deep background, you don't have the time and/or patience to get into it, you don't have the proper server administrator rights, or a combination of all three.

If that's your situation, this post is for you.

Flask is a Python package for the quick and easy creation of APIs that you can use to serve model predictions over the internet. And if your users need a GUI, Dash is a software package built on top of Flask that allows you to quickly create web interfaces - if you are familiar with R, Dash has been described as "ShinyApps for Python".

Typically you would use a "real" web server (Apache, NGINX, etc) to do the heavy lifting, but this post focuses on how to use Flask alone to quickly return results generated from an ML model. In particular, we will focus on how to keep a model in memory between calls so you don't need to restart your model at every turn.

WARNING: Flask is not designed to work this way. The Flask documentation itself tells you not to use their integrated web server for anything other than testing, and if you blindly expose this code to the internet things can get ugly. It will also be much slower than using a proper web server. And yet, I am painfully aware that sometimes you don't have the resources to do things right, and people telling you "there's a way but I'm not telling you because it's ugly" doesn't help. So remember: this solution is ideal for situations where you have low traffic, ideally inside an intranet and/or behind a firewall, and you don't have the technical help you'd need to do it right. But be aware of its limitations!

Method 1: global variables for simple models

Let's start with the simplest of web servers. This code exposes a single API endpoint /helloworld that receives a name and returns a greeting:

from flask import Flask
app = Flask(__name__)

@app.route('/helloworld/<name>')
def hello_world(name):
    return f'Hello, {name}'

if __name__ == '__main__':
    app.run(debug=True)

If you send a request to http://localhost:5000/helloworld/Test, you would get Hello, Test as a result.

Let's say that you now want to return a counter of how many times you have received a request - every time you get a request you simply increase a counter, and then you return that counter. One simple solution is using a global variable, like so:

from flask import Flask
counter = 0
app = Flask(__name__)

@app.route('/helloworld/<name>')
def hello_world(name):
    global counter
    counter += 1
    return f'Hello, {name}, you are request number {counter}'

if __name__ == '__main__':
    app.run(debug=True)

This code does work only in Linux (I think), but under some circumstances it could be all you need - all that would remain is for you to replace the variable initialization with code that loads your model into memory. You would use this method, for instance, when you need to perform a task with a long enough startup time, such as parsing a long list of JSON files. If that's your case, you can leave this server running like so:

Disable debug and open the server to the world by changing the app.run line to app.run(debug=False, host='0.0.0.0').
Install the screen utility (tmux is also good), start it typing simply screen in your console, run your server (python script_name.py) and leave the server running in the background (press Ctrl+A+D). The server will keep running until the computer is restarted.

Unfortunately for some of you, this solution doesn't work under Windows nor does it work if you use a "real" web server instead of the one provided with Flask. More important, it also tends to fail when using some ML libraries that are not happy with the parallel simultaneous access. If that's your case, your best solution is to create a sub-process (ugh) and communicate with your model via IPC (double ugh).

Method 2: Inter-Process Communication (IPC)

Before we jump into the code, we need to understand who is going to talk to whom and how. It goes as follows: the ML model will run in its own process, which we'll call the ML-process. This process can only be reached via a multiprocessing queue, a data structure where you put multiple elements which are later retrieved in the same order in which they were inserted and such that it can be shared with multiple processes.

Whenever you make a request to the API, Flask creates a new process that we will call a request-process. The first thing that this process does is to open a Pipe. You can think of a pipe like a special pair of telephones that can only talk to each other and where sound is only emitted when someone is listening - you can talk for hours into the receiver, but nothing will come out of the other end until someone listens (in which case they'll get all of your talking at once) or until the pipe is full. Whenever a request-process needs to perform a request to the ML-process, it does so as follows:

As we said above, the request-process opens a Pipe. It has two ends which we'll call the 'source' and 'target' ends. Remember, though, that despite their name communication can flow in both directions.
The request-process puts some data in the 'source' end of the pipe. Whenever someone picks up the 'target' end of the pipe they'll receive this data.
Next, the 'target' end of the pipe is put in the multiprocessing queue. If we stick to our analogy, it would be the equivalent of having two cell phones, putting one of them in a box, and mailing it to another person.
And now, we wait.

The ML-process is constantly monitoring the queue, and it will eventually receive the 'target' end of the pipe that we put in the queue. I say "eventually" because other processes are also trying to talk to the ML-process, and therefore every process has to wait for their turn. In our analogy, it is the equivalent of a person receiving package after package, each one containing a cell phone. Once the ML-process receives our 'target' end of the pipe it extracts the data, processes it, and puts the result back into the pipe using the 'target' end it received earlier. This result is then sent back via the pipe, where our request-process retrieves it and where it can be served back to the user that made the original request.

The following code does exactly that:

from flask import Flask
from multiprocessing import Process, SimpleQueue, Pipe

# This is the global queue that we use for communication
# between the ML-process and the request-processes
job_queue = SimpleQueue()
app = Flask(__name__)

# This is the process that will run the server
class MLProcess(Process):
    def init(self, queue):
        super(MLProcess, self).__init__()
        self.queue = queue
        # The slow initialization code should come here.
        # For this example, we just create a really bad cache
        self.cache = dict()

    def run(self):
        # Query the end of the pipe until we tell it to stop
        stop = False
        while not stop:
            # Receive the next message
            incoming = self.queue.get()
            if incoming == 'shutdown':
                # We got the magic value that tells us to stop.
                # Make sure this value doesn't happen by accident!
                stop = True
            else:
                # `incoming` is a pipe and therefore I can read from it
                data = incoming.recv()
                # Do something with the data. In this case, we simply
                # convert it to lower case and store it in the cache,
        # but you would probably call an ML model here
                if data not in self.cache:
                    self.cache[data] = data.lower()
                # Send the result back to the process that requested it
                incoming.send(self.cache[data])
        # If your model requires any shutdown code, you would place it here.
        pass

# This is a normal API endpoint that will communicate with the ML-process
@app.route('/helloworld/<name>')
def hello_world(name):
    # Create both ends of a pipe
    my_end, other_end = Pipe()
    # Send the data through my end
    my_end.send(name)
    # Send the other end of the pipe via the queue
    job_queue.put(other_end)
    # This process will now wait forever for a reply
    # to come via its own end of the pipe
    result = my_end.recv()
    # Return the result from the model
    return 'Hello, {}'.format(result)

if __name__ == '__main__':
    ml_process = MLProcess(job_queue)
    ml_process.start()
    app.run(debug=True)
    job_queue.put('shutdown')
    ml_process.join()

This code works well as long as there is perfect communication between all moving parts. If the ML-process hangs up, for instance, then no more data will be returned and all request-processes will keep waiting forever for a reply that will never come. The same will happen if you send the pipe to the server but you don't put any data in it. You can mitigate these problems by using the poll method of a Pipe (which looks whether there's any data and returns immediately), but you should be aware that synchronization errors are both common and mean to debug.

Note also that we have a special value that we use for instructing the ML-process to shut down - this is necessary to ensure that we clean up everything before exiting our program, but make sure no client can send this special value by accident!

Final thoughts

Is this a good idea? Probably not - if the developers of Flask themselves tell you not to do something, then "don't do it" sounds like solid advice. And we both know that there's nothing more permanent than a temporary solution.

Having said that, and as far as horrible hacks go, I like this one: if you are a data scientist then you are not here to learn how web servers work nor to have long discussions with your system administrator on what CGI means. You are here to get stuff done and getting your ML model in front of users as fast as possible is a great way to achieve that.

I used to know a way to extend this method to work with Apache (you know, a "real" web server) but I honestly can't remember it right now. If you need some help with that then reach out to me and I'll try to figure it out.

Smartening up, Part I

2022-11-01T00:00:00+01:00

I always wanted a smart home. I don't have a particular use for it, I just think it's cool that I can yell at my living room and it will obey me. And since next month I'll be moving to an empty flat, it is truly now or never.

This is the first post detailing what I hope will be a painless experience and what I know will be a long list of frustrations. Today I will detail my general plan, and in future entries I'll let you know how it all goes.

My aspirations for this first stage are modest: I want to be able to control the lights in key spaces just by talking to a device. Not just turn them on and off, mind you, but also dimming the lights to certain levels. I also would like to have a smart mirror with my morning information (to-do and weather, mostly), but that is more of a stretch goal.

Starting from the top, I need smart light bulbs. Originally I planned on going with Ikea's TRÅDFRI light bulbs for their price, but I decided against them because they don't seem to play too well with open platforms and because they require an extra hub. I settled instead for the middle-priced Philips WiZ because they connect directly over WiFi (unlike it's expensive cousins from the Philips Hue line). I would have loved to use the cheap Hama smart lighting options, but my past experience with this company gives me little hope of their protocols being open or, for that matter, good.

Another factor in favor of WiZ was that they are supported by OpenHab thanks to the heroic work of one volunteer. Once it is properly configured I expect I'll be able to add complex commands like "dim the lights to 60% after 17:00 if it's winter" and stuff like that.

The voice commands will be handled by Mycroft, the privacy-focused alternative to Alexa and friends. I would really like to buy a Mark II, but given their delivery times I fear that I'll have to install my own version first (probably in my old notebook) and eventually migrate. Lucky for me, Mycroft and OpenHab are good friends.

The final part is networking. If you are familiar with my blog you may know how much I care for privacy, which is typically a problem when you want to install hardware that monitors your home 24/7. Therefore, all of the above-mentioned services will run in their own isolated LAN with no connection to the internet. Mycroft may get an exception depending on whether I would like to ask it about the weather, but everything else will stay isolated. This would also guarantee that I don't lose control of my lights when my internet is down. I have long ago flashed my WiFi router with dd-wrt which allows me to have multiple networks and define who can talk to whom.

Progress so far

Given that I already have the light bulbs, I tried to set them up using the Android WiZ app. This did not work: one key step of setting up the light bulbs is to register them on the cloud (for whatever reason), and the closed network made this impossible. I am fine in principle with the light bulbs phoning home once and then never again (combined with a VPN, the information they would expose would be minimal), but for that I would need internet and I still don't have any. I have also decided that two rooms will get "dumb" lights: the kitchen and the bathroom. These rooms are not "chill" rooms but rather "be there with a purpose and then leave" rooms, so there's no point in doing much with them.

And finally, one issue I have not yet decided is what to do about the microphone. Placing a Mycroft in the hallway would mean that I always need to yell at it, but I don't like the idea of my neighbors knowing that I turned on my lights at 3 AM and I doubt my neighbors would like it either. My best alternative so far is a small portable microphone - I read an interview sometime in the 90s about how Bill Gates' mansion was controlled with pins you were supposed to wear, and that seems reasonable enough to me. But I have yet to find something small enough.

Next steps

If I decide to go for the smart mirror, this guide seems like the way to go: I already have a Raspberry Pi I'm not using (used to be my NAS server) and an old laptop screen, and it's mirror film approavh is cheaper than those using two-way glass. The annoying part would be finding the appropriate control circuits for the screen, which is a can I've been kicking down the road for a couple years now.

I also would like Mycroft to play my music, but that would require me to install a NAS and set all networking correctly, which was not fun with all devices in the same network and will probably not be fun here.

I'll let you know how this all works out.

Sports movie pitch

2022-10-30T00:00:00+02:00

Here's a movie idea no one asked for:

A group of teenagers have a dream of competing in the Paralympic Games, but their dream is in danger when their beloved coach gets terribly sick. And things don't look promising with their substitute coach either. Sent there as punishment for his failures in the army, he is very strict, belittles them constantly, and has no sense of humor. The coach's name? Darth Vader.

Will the young athletes make it to the Space Paralympic Games? Can the power of friendship prove to be stronger than the Dark Side? Will Darth Vader find his humanity, with his failure at defending the Death Star being the first falling domino in his eventual return to the Light Side? I guess you'll have to watch the movie to find out!

(This is not the dumbest thing I've ever written, but it's probably Top 5)

GMail broke my email

2022-10-16T00:00:00+02:00

In what must be one of the weirdest flexes ever, I can proudly say that I keep my own email server. And like everyone who keeps their own e-mail server can attest, GMail is always annoyingly difficult to work with.

The last version of this problem is that GMail stopped receiving my email because I didn't have a PTR record, an error that's annoyingly difficult to debug for two reasons: because the record is not really called "PTR", making it slightly difficult to figure out what they want for me, and because I know I've had one for years.

Long story short, the problem was that my server started using IPv6 and the outgoing address was not matching the IPv4 PTR record I had. So I solved the way these problems are always solved: I turned off IPv6 and everything works again. I can't remember who gave me this advice for the first time, but I can confirm that it works.

Now, if only I could manage to identify why Apache is not redirecting the LetsEncrypt challenges properly...

Edit: This post is best read when listening to Future Boy's "Computer Shop". Warning: NSFW language!

Pelican broke on me

2022-10-13T00:00:00+02:00

As I mentioned a long time ago I am using Pelican for publishing my blog. I recently did an upgrade which broke the syntax highlighting, and I thought I should take a short time to explain how to fix it because I couldn't find a solution anywhere.

The problem: it used to be the case that you could write your code like this and it would do what you'd expect:

```
:::python
print('Hello world')
```

But since some version this syntax is no longer valid, resulting in a plainly visible syntax identifier and the wrong colors being used in the highlighting:

:::python
print('Hello world')

The solution is simple: change to the other style for writing code in Markdown, like so (note the leading spaces):

    :::python
    print('Hello world')

And that solves the issue.

print('Hello world')

I shouldn't forget to mention that this change forced me to re-indent all of my code samples: the reason I was using the triple backquotes format was because it lets me indent at the 0-th column, while the new syntax forces me to indent everything with four leading spaces. And you know what happens with Python code when you accidentally miss four spaces, i.e., one indentation? It breaks in marvelous and unexpected ways! Ensuring that every single line of code was properly re-indented was not a fun task, but a necessary one nonetheless.

Which brings me to my final point. I thought I was used to the whole "indentation has meaning" thing, but I have to say: only after using Rust at the end of last year and going back to braces to delimit code blocks did I realize how much I've missed them.

Why the music industry still won't take my money?

2022-10-13T00:00:00+02:00

Remember this post where I complained that the music industry won't take my money? Well, it's that time of year again.

As you may remember from following me on Mastodon, I bought an Alexa, set it up in Spanish, and almost went insane due to how bad it was. So I did what every normal person would do in this case and I gifted to my mom. This Alexa slept in an Argentinean drawer for a long time, until one day I finally convinced my family to stop worrying about breaking it. Today it works mostly as a voice-activated music player for my nieces.

We have established before that I am an idiot, and because I'm an idiot I decided to set up Spotify (with a family plan) instead of Apple Music. I already have a rocky situation with Spotify, which is why I am not shocked at all to learn that, just like Amazon and Warner before them, Spotify will not take my money:

I cannot pay for my mom's plan because my card is German and the account is not. And the device's IP is obviously in Argentina.
I cannot use an Argentinean credit card, because I don't have one.
I cannot use the other available methods because they require me to physically go to a store in another continent.
I cannot pay with a gift card - even suggesting this as a possible feature will get your request closed without review.
I can try to set the account to a German one, but as the tech support representative would put it, "you can give it a try, however that is something we are unable to guarantee that will work". Also, I fear this may cause Alexa to start speaking in German, and my mom won't be amused by that.

So once again I am trying to give a company money -- no, scratch that: I am trying to give a company money THAT'S WORTH MORE THAN THE MONEY THEY WANT (Argentinean pesos are not super hot right now), and yet they won't listen to reason. The Spotify forums are full of threads where the best answer you'll get is a "Community Legend" saying that it sucks to be you.

The most likely end of this story is that I'll wire some money over Western Union and one of my relatives will go to a store to pay the bill. My time will be wasted, I'll lose some money in the exchange, my relative's time will be also wasted, and Spotify will receive money in a currency that's devaluating at a 10% monthly rate.

Great job, Spotify.

On physical books and horses

2022-09-25T00:00:00+02:00

I originally envisioned this post as a two-part brain dump, but then I realized that the type of people interested in libraries are also the type of people who aren't bothered by long text. So this post will be longer than usual.

Now that we are almost a quarter into the 21st century it is perhaps time to have the talk: what do we do with our public libraries?

I'd like to start by pointing out that I love libraries. I love the feeling of being surrounded by books, picking one of them at random and reading a couple pages -- sometimes I put the book back after a few seconds, and sometimes I rush to the nearest chair and remain there until it's dark outside. Having said that, I also realize that I'm only focusing on what libraries are instead of what they could be.

Part I: books

Physical books are awesome: paper is relatively inexpensive, lasts a long time, doesn't hurt your eyes, can survive some natural disasters, requires no electricity, an can be easily recycled if a book is really bad. Having said that, paper is also heavy, flammable, and duplicating its content takes work.

A book's physicality is both its blessing and its curse. The New York Society Library acquires 4800 books a year, meaning that they also need a plan for getting rid of 4800 books a year. What happens to those books is a touchy subject. History is full of brilliant artists who didn't get recognized during their time, but 90% of all books are crap. How to strike a balance between both extremes is a full-time job.

I am a fan of physical books. But I wouldn't be honest if I didn't consider that maybe I like them just because that's what I grew up with, the same way my dad's coworkers rejected computers because a calculator was all they ever needed. And I am willing to bet that, about a hundred years ago, someone somewhere must have rejected cars for being cold and boring in comparison to the horse they brushed and cuddled every day.

It isn't hard to imagine a library without physical books. You would simply show up to the reception desk, pick a tablet (or bring your own), sit in one of the multiple available couches (with all the books gone, there's plenty room for them), and browse the virtual catalog. No need for queues, no book needs ever be thrown away, and you could even do it from the comfort of your house.

(Note that "don't throw anything away" sounds suspiciously similar to hoarding, but we won't get into that today)

But this is where I worry: why stop there? If you can check out books from anywhere, why even have a building? Just make it a webpage that everyone can access. All you need to do is add Digital Restrictions Management (DRM) and you no longer need to worry about people making unauthorized copies and/or lending your books to non-members. It is also a very convenient way of getting rid of problematic books: one simple command and voilà!, the book is gone from every library in the country. And as a plus, you don't need to fight anymore with those pesky librarians who refuse to hand over library records without a court order.

Which touches on an important point: that if you were to invent the concept of a library today, you would be labeled a pirate and sued. A library is an inconvenient institution: it is not run for profit, it facilitates copyrighted material, and it provides access to dangerous ideas to anyone. If I were me, I would be scared of tinkering with such an important institution and risk breaking it forever.

Part II: activities

Whenever I think about what an ideal library would look like, I always come back to the same example: the Vancouver Public Library. If all you had was a VPL card and time, you could learn music theory, learn how to play an instrument (which you can borrow on the fourth floor), record a podcast with your progress in the recording booths, design an album cover in their computers (don't know how? there's a book for that), and eventually release your album. Not many other institution offer this much knowledge for free.

The VPL (and others like this, such as the Cologne Library) go straight to the core of a different concept of library: not a place to store books, but rather a place to explore ideas. And while in theory you could also do that with a virtual library, I wouldn't discount the power of moving your physical body somewhere. Once you are inside the library, a part of you knows that you are there with a purpose.

No one I know associates libraries with activities, but they definitely associate them with the idea of a safe, quiet place to study. Which is, once again, a blessing and a curse: the space you need for your activities is space you take away from the books. And just as important for this discussion, it is space: if you have tried to rent even a moderately sized empty room you'll know that it doesn't come cheap.

Perhaps I am conflating too many ideas. Perhaps the world doesn't needs a library-makerspace-coworking space. And yet, the idea of a Library as a "Temple of knowledge" is all my nerd brain ever wanted.

Part III: the Metaverse and Zoom meetings

This is the part where Mark Zuckerberg shows up and says "something something THE METAVERSE". And he could have a point: if you are not at least intrigued by the idea of reading a (virtual) book while sitting (virtually) in the Library of Babel you need to catch up. But until the day comes when we all wear Metaglasses (R) all the time, we could take about their little sibling the smartphone.

Have you tried working in a library in the past months? I have, and I tell you, it's a challenge. This was not a problem 4 years ago, when all I needed was fast internet and power for my notebook. But in a post-pandemic world where everyone and their dog want to rope me into Zoom meetings this is no longer enough: meetings require sound-proof cabins in short notice, and they are very scarce. The Staatsbibliothek Berlins has six such cabins for all of their members (split into 4 and 2 between its two buildings) and trust me, they are not even close to enough.

And so, the only people who can currently accept a phone call in a library are Olympic sprinters who can run down four flight of stairs in less time than it takes for a call to go to voicemail (ask me how I know this!). I imagine libraries will eventually adapt - if anything, those six cabins in the StaaBi did come from somewhere. But again, each one of those cabins takes away space from "regular" book readers, who may argue that libraries are not WeWork and that I should go somewhere else.

Conclusion

I believe books will still be an integral part of our daily lives way into the 22nd century and further. Sure, eventually all new books will be digitized and nothing will be lost ever again, but the value of holding a book in your hands cannot be matched. And in a time when sharing is no longer caring, I am not counting on any better replacement coming anytime soon.

I love my local library - I like the small ones, I like the big ones, and I definitely like the VPL. And precisely because I like them I can't stop wondering whether I'm the old guy who complains about cars while brushing his horse. But what does the future of the library look like? Is it in the comfort of my home? Is it in my pocket? Is it inside incredibly expensive VR glasses? Or is the future of my library the exact same place where it was the last couple hundred years? Sure, sometimes one of them catches fire, but doesn't everything? And it's not like we can't just rebuild them. Lucky for us, manuscripts don't burn.

Microsoft Teams sucks

2022-07-11T00:00:00+02:00

If you told me I was going to die in the next 24hs, asked me to guess what from, and it was a working day of the week, "MS-Teams-induced aneurysm" would definitely be in my top 5 guesses.

I know what you are thinking: "he's exaggerating - no software can be that bad". This reveals one of two possible scenarios: either you don't use Teams in your daily work, or you do use it but you only pretend to work. Don't worry, your secret is safe with me. But just in case, please take a seat and let me tell you all about Teams.

First and foremost, Teams is where information goes to die. If I need a piece of information I know exists, I have to identify the Account (I have 2), the Team (around 25 and growing), the Channel (around 5 per team but only two in actual use) and the media type (was it in a chat? a wiki? a file? If so, in which folder?), all of which are just slow enough to make it frustrating but not slow enough for my office to ditch it (assuming they could and wanted to, which then can't and don't want to). And with a search function that works ~20 percent of the time, it's not unusual for me to just give up. "It's somewhere in our Teams channel" is what I tell people when I don't actually care about whether they find what they need or not. And if I'm looking for something said in a person-to-person Chat, this is different from a Channel chat and I need a different window altogether.

Chat brings us to notifications. Teams is the only tool I know that will display up to four notifications for a single event and leave more than one on. If someone sends a message during a meeting, Teams will display a box in the notification area, play a sound, add an indicator to the task bar, and a second indicator to the meeting windows. Your window will resemble a Christmas tree until you open the chat window, even if you already clicked on the very first notification box. As far as Teams is concerned, a single unread chat is the most important thing that has ever existed, and it will not rest until you've clicked on it.

The system requirements are also an issue, but all things considered they are a minor one. Sure, I cannot calculate the number of hours that I lost waiting for things to load, but I can at least estimate how much my employer had to pay to replace an otherwise perfectly good computer. And yet,if I only consider them a minor issue it's because they pale in comparison to the hours I wasted doing things the Teams way, a burden no hardware upgrade can relieve.

Teams empowers you to go through the motions of work without actually getting anything done. No project can start without yet-another useless channel, a bunch of documents you'll never find, a to-do list hidden at least four levels deep into the interface, and a collaborative editor that ruins your formatting until you open the desktop app. But it takes a strong company leader to stand to Microsoft, and those don't come easy nor cheap. Once your company has willingly chained itself to a Teams-based working environment, your fate is tied to that sinking ship.

And yet, the worst aspect of Teams had escaped me until today when a friend pointed it out: that we started using it because Microsoft imposed it on us right at the beginning of the pandemic. I guess some Microsoft salesman decided that the pandemic was not hard enough on us as it was, and that's why they decided to add a buggy, resource-hungry tool as the cherry on top of the pandemic dessert. I have not yet received my check for working as an alpha tester for a terrible tool I didn't want, and I am starting to believe that I might never get it.

Friends don't let friends implement MS Teams in their organizations. If you are going to choose a collaboration tool that sucks, you might as well choose one that's lighter on resources and less annoying. Even yelling at your coworkers down the hall is better, even if they are in another building.

Just say no to Teams.

I got Covid and I wouldn't recommend it

2022-04-09T00:00:00+02:00

Two years and 44 days after the beginning of the Covid-19 pandemic, I caught it. I haven't felt this sick since at least January 2013 (probably longer), and I therefore spent an entire week lying on my couch. Which is ironic considering that, as I lay there "looking half dead" (not my words), Germany was scraping almost all protective measures.

What happens when you get sick right as the country is ready to move on? In my experience you get sent back and forth between different services who haven't yet noticed that there's a pandemic going on, you rage against "Corona test centers" who don't test sick people, and eventually someone takes pity on you and takes care of it.

My first difficulty was the PCR test: quick tests are not accepted by my employer, my girlfriend's employer, nor my soon-to-be-ex-doctor's office. But getting a PCR test while sick is bafflingly difficult: the test centers suggested by the 116117 Patient service only offer quick tests, and the multiple other test centers that sprung up these years don't accept customers with symptoms. The only choice is a 20 minutes ride to the nearest Hospital, meaning you need either a car (have fun driving while sick!), a driver (ideally someone who is willing to get sick by being in contact with you), or a moderately-full public bus. Fun fact: do you know which kind of people takes a bus on a working day during working hours? Answer: Old people! Have fun at night wondering how many of them you'll put in an intensive station next week.

I then tried to use the Corona Warn-App to tell anonymous people they should get tested, but that didn't work either. The app doesn't let you self-declare as sick, offering only the possibility of scanning your official code. The problem is that I started exhibiting symptoms on Wednesday, got my PCR results on Friday, and yet as of this writing (one week later) the results are still not uploaded and probably never will be. I said it before and I'll say it again: when security researcher Bruce Schneier said that contact tracing apps have no value, he was right.

And finally, the sickness certificate: my soon-to-be-ex-doctor has not yet learned that they are allowed to extend certificates per telephone, but won't see me in person either. I tried calling the 116117 people again for a referral, but they entered a loop and got me nowhere. I guess no one asked them for a doctor before? I next gave the health services of my employer a try, but they were as useless as always. And as I started going through a list of other possible doctors to call my girlfriend lost her patience, took time off work, and went to my soon-to-be-ex-doctor's office in person to get me the stupid certificate.

People, it's been OVER TWO YEARS since we canceled Karneval the first time, and yet we are still in the same spot where we started. I know that Germany is a bureaucratic place, but that's not the issue - if anything, the fact that there is a protocol to follow is a bit of a consolation. But when EVERY.SINGLE.STEP of that protocol is broken, it really makes you wonder what else might be FUBAR and waiting for you to discover it.

So you want to use my PC

2022-04-01T00:00:00+02:00

So you want to use my PC? Sure, no problem, it's right there. But real quick, before you start, there's a couple small tips that you should keep in mind.

If you turn it on and nothing happens, that means the KVM switch is listening on the wrong input. Just press the black button behind the screen to switch it back. Can't find it? Maybe it fell down - it has a short cable and sometimes the other cables pull it back. Or maybe it's underneath all of my drawing stuff. Either way, search for it behind the desk and press it. The screen should turn on.

You may also notice that the computer chair has been relegated to the side and that I'm using a regular wooden chair instead. You can use the fancy one if you want - after a couple weeks it may cause some completely unexpected health issues, but on a short enough scale it's fine.

Now, this computer runs Linux. Technically it runs Windows too, but the screen for choosing a different operating system goes away before the monitor has time to turn on, and we lost too much time to spam the arrow keys anyway. But that's probably for the best: given that I haven't booted Windows in a long time, that Windows loves mandatory updates, and that the computer has a mechanical hard drive, there's a good chance that you would have to wait at least an hour before being able to use it. Why haven't I upgraded to an SSD, you ask? I have considered it, but Windows won't let me for complicated reasons. So Linux it is.

You may see a bunch of text scrolling by. That's fine. If the text stops scrolling for a long time you may want to consider pressing the NumLock key several times. I'm not entirely clear on the details, but something about interruptions and IRQs sometimes keeps the PC from starting if it doesn't detect activity. And since the NumLock key doesn't do anything bad, it is the safest key to spam-press.

Once you've made it to the login screen it is going to ask you for a username and password. That one should be easy - I have been using the same combination for over 20 years, so you probably know it already. Need a hint? It's in the same league as my network name plus my favorite number. It's not exactly hunter2, but it's close.

You may have a bit of trouble reading the letters on screen. Don't worry, the screen works fine - I just keep my brightness and contrast real low, and even the smallest hint of natural light turns them invisible. You could adjust the brightness using the monitor's crappy touch controls, or you could do what I do and close the curtains. I'm not going to tell you which one is right, but I am going to vouch for the one that has kept me glasses-free for more than 25 years of extensive computer use.

If you want a graphical interface, type startx and my Mate Desktop Environment will start. Ignore the error about sticky notes - I have been meaning to fix it for some time, but there's always something else and it doesn't bother me anyway. But before you type anything, a word of caution: the interface is in English, but the keyboard is German and the key mapping is Spanish. That means that regular keys are where you expect them to be (except for Z and Y, which are inverted), but if you want to type a single quote you need to add a space afterwards or you'll get an accented letter otherwise. If you have a password with more than just letters and numbers then you may want to type it in a text editor first and copy-paste.

You probably need a web browser, in which case the Firefox icon is right there. But you probably don't want that: the script blockers almost guarantee that whatever website you are trying to open will not work. For those cases I recommend using the super-outdated Chrome browser instead - the icon is right next to the Firefox one. It won't play Netflix because I didn't install the DRM stuff, but everything else should work fine. If you want audio, remember to plug the headphones first. And that reminds me: I have set the browser to auto-delete all history when closing the browser. So don't close the window until you're done, or you'll have to login again. I hope you didn't close the text editor with your password!

Now, I'll be honest with you: at this point most people give up and resort to their phones. And it's okay, I get it. That's what happens when you keep your tools exactly like you want them to be and suddenly you are asked to share. But most people tend to blame Linux for it, which I think it's unfair - if anything, I'd like to see Windows offer even half the flexibility to do things my way.

Now, let's talk about that WiFi password. But first, one question: do you know how to type the "¿" symbol on your keyboard?

Inspired by this article

On bad deals

2022-02-27T00:00:00+01:00

An article titled Don't talk to Corp Dev has recently showed up in my feed and the HN discussion around it reminded me of this anecdote.

To clarify: this story is based on something that happened in real life, as told to me by someone who witnessed it first-hand. Time has blurred some of the details and I have changed others. So feel free to dismiss the details, but please keep the main point in mind.

Once upon a time there was a small wine producer we'll call Dave. He kept a small shop and was doing well by himself, selling his wine at $15 a bottle. One day a big supermarket chain contacts him -- the chain needs a new wine supplier, and they heard good things about Dave. They have just one condition: they will take care of distribution and advertising, but Dave has to provide them with one month of free wine in return.

Dave is ecstatic: the supermarket chain can spread his brand across the entire country and turn his small operation into a national contender. Sure, giving up one month of profit is not ideal, but the supermarket representative convinces Dave that he'll recover his lost income in no time. Hands are shaken, contracts are signed, and Dave provides them with one free pallet of wine to start the week.

Week two arrives, and the supermarket chain requests two pallets of wine. "Looks like my wine is selling well", Dave thinks, and increases his production targets for the week. But this is not enough: when Dave is asked for five pallets of wine the next week he is forced to hire a new assistant and buy new machinery to keep up. Predictably, week four closes with an even larger order of ten pallets. The loss of 18 pallets of wine is a tough pill to swallow, but at least the month is over and from now on he will get paid.

A new month begins and Dave receives his weekly order: he is asked to deliver a grand total of one pallet of wine. "This must be a mistake", Dave thinks, and calls the supermarket representative. "Last week I gave you ten pallets of wine. So why are you only asking for one today?". "That's easy", says the representative, "up until last week we were selling your wine for a dollar a bottle. This week we will put the price back at $15 and therefore we are not expecting it to sell as well as before".

Facing a large loss of inventory and having invested both in employees and machinery that are no longer needed, Dave goes broke. The supermarket chain finds a new naive wine supplier, and the cycle starts again.

A wise man once taught me that it's not a good idea to enter into negotiations with someone much bigger than you. This is arguably the reason why South American countries formed Mercosur instead of joining theFTAA agreement with North America, but it's also true for simpler activities like office politics.

For the record, I would have totally fallen for this. The first time I heard this story I didn't even consider that someone could do something like this on purpose. Then I started reading news about startups and learned that it happens relatively often.

Consider yourself warned.

Rust Part II: What is it good for?

2022-02-04T00:00:00+01:00

Last time I talked about Rust I mentioned that I wanted to like the language but I couldn't find a good reason for using it. Luckily for me, the last Advent of Code gave me the perfect reason for doubling down on Rust, and here's my updated report.

In case you never heard of it, Advent of Code is an online competition that takes place every year in December. It is structured as an Advent calendar where you get a puzzle every day, and where the puzzles get harder and harder every day. Plenty of people use this competition as the perfect excuse for learning a new language, which is how I ended up programming lots of Rust in my spare time.

So here they are: in no particular order, these are the things I like, dislike, and feel mildly uncomfortable about Rust.

Things I like

The one thing I like the most about Rust is the power of the match operator combined with enums. Unlike in Python, where the implementation of the match statement is pretty dangerous, Rust makes it easy to program the type of code that's easy to write, read, and maintain:

let mut pos = 0;
let mut depth = 0;
for instruction in orders {
        match instruction {
                Instruction::Forward(meters) => pos += meters,
                Instruction::Up(meters) => depth -= meters,
                Instruction::Down(meters) => depth += meters,
        }
}

Then, there are the compiler errors. While not true for external crates (we'll get to it), compiler errors in Rust are generally helpful, identify the actual source of the problem, and sometimes even give you good suggestions on how to solve the issue. Gone are the days in which a compiler error meant "I know an error happened 50 lines above, but I'll complain about it here instead".

And finally, as someone who has been doing mostly Python for the last years, it feels so good not to have to worry about indentation anymore. This doesn't mean that I'll stop indenting my code - instead, it means that I can finally move a function around without worrying about pasting it one indentation to the left and turning a class into a class and multiple pieces of code that don't compile.

Things I hate

I am puzzled by how aggressively unhelpful arrays are. The puzzle for day 25 could be easily solved (spoilers!) by writing

horizontal_row = horizontal_row>>1 && !(horizontal_row || vertical_row)

but I ended up having to implement it with Vectors of booleans instead. Why? Because I didn't know how many bits horizontal_row would have at compile time, and Rust refuses to create arrays with dynamic size. I am sure there is a way to keep a large binary in memory and manipulate it at the bit level - otherwise, you wouldn't be able to use Rust for serious game development. But whatever the method is, it is well hidden.

And on the topic of that puzzle, I come back to one of my main complaints from last time: popularity is not the correct way to decide which library is the best one for the job. Do you know the difference between the bitvec, bit-vec, and bitvector libraries? Can you add either of them to your code without worrying about the developer going rogue? How about the fact that the first result that comes up for rust bit vector is an accepted StackOverflow answer suggesting bit-vec... which is no longer maintained?

Minor annoyances

I still can't make sense of the module system. I mean, sure, I know how to put functionality in sub-directories, but that doesn't really explain why I would choose between lib.rs, day24.rs, or day24/vm/mod.rs. The book could use some improvements on this topic.

If I'm doing something like u16 = u16 + u8 (or even better, u16 += u8), the compiler should cast the last value automatically. u8 += u16? Sure, I get it, that's an overflow waiting to happen. But there is no need for me to get in there and write u16 = u16 + u8 as u16 when we all know the data fits just fine.

The collect function is very finicky. This is a function that I used quite often in constructions like .map(|x| something(x)).collect.to_vec(), but more often than not it will complain about not knowing the type required for collect even though there is only one type that would make sense.

And since we are talking about the compiler, one of the crates I needed (it was either nalgebra or ndarray, where I suffered the same problems I had with bit vectors) had a nasty side effect: if one of your instructions failed to compile, they all stopped compiling. Good luck finding the one line that needs fixing!

And finally, I ran a couple times into functionality that had been deprecated in favor of functionality that doesn't currently exist. Not cool.

Conclusion

Would I use Rust again? Yes.

Is it my most loved language? No. But under the right circumstances I could see it happening.

What is it good for? Last time I jokingly said "writing Rust compilers", and I wasn't that far: it's the right programming language for apps that need performance and memory safety, and where we are willing to spend some time calculating who is borrowing from whom in order to get code with fewer bugs. So it's pretty much C++, only with borrowing replacing memory allocations.

I like the idea of giving my original project another try, but I can't make any promises. The Advent of Code has already pushed forward the date of my next project by a couple months, and the time it's taking me to migrate my infrastructure to Ansible is making everything worse.

How not to sell tickets

2022-01-27T00:00:00+01:00

Let's say you are a band and you want to keep scalpers from buying all of your tickets and reselling them at ridiculous prices.

By now we know what works: you sell tickets with assigned names on them, and check at the door that the name on the person's ID (or, alternatively, credit card) matches the name on the ticket. You can also offer a "+1 option" for bringing someone along, but they must be accompanied by the person who bought the tickets in the first place. No ID, no entrance.

By contrast, here's what doesn't work:

How do I know that it doesn't work? For starters, because all of the regular $40 tickets are sold out:

And because the only available tickets are those sold by scalpers at 10 times their value:

You know what I would do if I were a scalper? I would buy as many tickets as I could, and then I would also sign up for the exchange with as many fake addresses as possible. It doesn't matter that I can't use the exchanged tickets afterwards - as long as the supply is reduced, the value of the tickets I bought first will only go up. Since every overpriced ticket offsets the cost of the other 9 I got early, it all works out for me at the end.

By now it is well known that ticket companies don't get rid of scalpers because they don't want to. The method I mentioned in the first paragraph is proven to work, but why would ticket companies get rid of scalpers when they can reach deals with them and get a second cut on the same ticket?

Which brings me to my final point. I think there are three types of bands:

those that care about their fans and successfully prevent scalpers from getting all the tickets,
those that care about their fans but don't know what to do (and/or do it wrong), and
those that don't care about scalpers because they get paid anyway (or get paid even more)

If you are a member of the first type of band I applaud you for standing up to your fans. And if you know someone who is in the second situation, feel free to forward them this post.

I could share some rough words with those who belong to the third group. But honestly, why bother? It's not like there's a shortage of good musicians. I'll go watch those instead.

April 2022 edit: John Oliver has a funny segment on this topic. You can watch it on YouTube

Ungooglable Azure errors

2021-12-14T00:00:00+01:00

I recently found myself in a situation I haven't experienced in a long time: I got an error message that no one on the internet has had before. Neither DuckDuckGo, Google, nor Bing could find a single hit about the problem, much less a solution. The root cause (spoilers!) ended up being rather boring: I tried to use some features of the Azure cloud platform that are apparently blocked by the IT department of my company.

I don't know exactly why these errors are impossible to find - either they were written by someone in my company's IT department or my employer is the only one who blocks the OAuth authentication flow. Either way, and as a service to the community, I am copying the error messages here.

The first one means that you are trying to use any type of authentication other than the allowed one:

AADSTS1000470: The protocol OAuth2DeviceAuth is blocked for tenant <tenant-id>. Please contact your administrator for assistance.

The second one means that you tried to create a resource that the internal rules don't allow. I know I hit this one when following an ML tutorial, but I can't remember the specifics right now. Hint: If you also got the string "RBAC restrictions" somewhere in the description, then you are seeing the same one:

At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.

code: RequestDisallowedByPolicy
message: Resource '<resource-id>' was disallowed by policy. Policy identifiers: <bunch of json with redacted information>

If you run into these error messages, feel free to get in touch with me and I'll share all of my poor Azure wisdom with you.

Brain dump

2021-11-06T00:00:00+01:00

Here's a list of short thoughts that are too long for a ~~tweet~~ toot but too short for a post.

On old computer hardware

I spent the last month of my life fixing the computers of my family. That meant installing Roblox on a tablet with 1Gb of RAM, fixing antivirus on Windows 7, dealing with Alexa in Spanish, and trying to find cheap ink for printers with DRM. Fun fact: HP uses DRM to forbid you from importing ink, and then stopped delivering ink to my family's city.

Modern hardware can have a long, long life, but this won't happen if software developers don't start optimizing their code even a bit. Sure, Barry Collins may not have a problem with an OS that requires 4Gb of RAM, but I feel I speak for tens of thousands of users when I say that he doesn't know what he's talking about.

On new computer hardware

I know that everyone likes to dump on Mark Zuckerberg, and with good reason: the firm formerly known as Facebook is awful and you should stay away from everything and anything they touch. Having said that, there's a reasonable chance that the moment of VR is finally here. If you are a software developer, I encourage you to at least form an informed opinion before the VR train leaves the station.

On movies

I wasn't expecting to enjoy Ready or not as much as I did. I also wasn't expecting to enjoy a second watch of Inception almost as much as the first time, but those things happened anyway. I was however expecting to enjoy Your name, so no surprises there.

I also got on a discussion about Meat Grinder, a Thai film that is so boring and incoherent that it cured me of bad movies forever. No matter how bad a film is, my brain can always relax and say "sure, it's bad, but at least it's not Meat Grinder". I hold a similar opinion about Funny Games, a movies where even the actors on the poster seem to be ashamed of themselves. At least here I have the backing of cinema critic Mark Kermode, who called it "a really annoying experience". Take that, people from my old blog who said I was the one who didn't "get it".

On books

Michael Lewis' book Liar's Poker is not as good as The Big Short, but if you read the latter without the former you are doing yourself a disservice. I wasn't expecting to become the kind of person who shudders when reading that "the head of mortgage trading at First Boston who helped create the fist CMO, lists it (...) as the most important financial innovation of the 1980s", and yet here we are.

I really, really, really like Roger Zelazny's A night in the lonesome October, which is why I'm surprised at how little I liked his earlier, award-winning book This immortal. I mean, it's not bad, but I wouldn't have tied it with Dune as Hugo Award winner of 1966 for Best Novel. I think it will end up overtaking House of Leaves in the category of books that disappointed me the most.

And finally, I can't make any progress with Katie Hafner and Matthew Lyon's book Where wizards stay up late because every time I try to get back to it I get an irresistible urge to jump onto my computer and start programming. Looks like Masters of Doom will have to keep waiting.

On music

My favorite song that I discovered this year so far is Haunted by Poe. Looking for some music of her I learned about how thoroughly lawyers and the US music industry destroyed her career. This made me pretty angry until I read that her net worth is well into the millions of dollars, so I guess she came out fine after all. And since the album was written as a collaboration with her brother while he was writing "House of Leaves", I guess I did get something out of that book at the end.

Some thoughts on learning about computers

2021-10-30T00:00:00+02:00

Here's a thought that has been rattling around in my head for the last 10 years or so.

My first word processor was WordStar, but I never got to use it fully. After all, what use does a 10 year old have for a word processor in 1994?

I then moved on to MicroCom's PC-Flow for several reasons. First and foremost, it was already installed on my PC. Either that or I pirated it early enough not to make a difference - in either case, it had the major advantage of already being there. Second, it was "graphical" - it may have been a program for flow charts, but that didn't stop me from writing some awful early attempts at writing music that are now lost to the sands of time and 5.25 floppy disks. And third, because it printed - not all of my programs played nice with my printer, but this one did. The end result being that I wrote plenty of silly texts using a program designed for flow charts.

The late 90s were a tumultuous time for my computer skills. At home I had both MS Office '97 and my all-time-favorite Ami Pro, while in high school I had to use a copy of Microsoft Works that was already old at the time. The aughts brought Linux and StarOffice, which would eventually morph into OpenOffice and LibreOffice. Shortly afterwards I also learned LaTeX, making me care about fonts and citations to a degree that I would never have imagined.

And finally, the '10s brought Google Docs and MS Office 365. But the least said about them, the better.

The point of this incomplete stroll through memory lane is to point out the following: if there's a way to mark a text as bold in a document, I've done it. Cryptic command? Check. Button in a status bar that changes places across versions? Check. Textual markers that a pre-processor will remove before generating the final file? Check. Nothing? Also check. If you ever need to figure out how some piece of software expects you to mark something in bold, I'm your man.

And precisely because this function kept jumping around and changing shape throughout my entire life, I have developed a mental model that differentiates between the objective and the way to get there. If I have a reasonable belief that a piece of software can generate bold text, I'll poke around until I find the magic incantation that achieves this objective. But my story is not everybody's story. Most regular people I know learned how to make bold text exactly once, and then they stuck to that one piece of software for as long as possible. For them, text being bold and that one single button in that one single place are indistinguishable, and if you move the button around they'll suddenly panic because someone has moved their cheese.

I have long wondered what's the long-term effect of this "quirk" in our education system. And forecasts are not looking good: according to The Verge, students nowadays are having trouble even understanding what a file is. Instead of teaching people to understand the relation between presentation and content, we have been abstracting the underlying system to the point of incomprehension. The fact that Windows 10 makes it so damn hard to select a folder makes me fear that this might be deliberate - I'm not one to think of shady men ruining entire generations in the name of profit, but it's hard to find a better explanation for this specific case.

Based on my experience learning Assembly, pointers, and debugging, I believe that the best cure to this specific disease is a top-down approach¹ with pen and paper. If I were to teach an "Introduction to computers" class, I would split it in two stages: First, my students would write their intended content down, using their own hands on actual paper. They would then use highlighters to identify headers, text that should be emphasized, sections, and so on. At this stage we would only talk about content while completely ignoring presentation, in order to emphasize that...

... yes, you might end up using bold text both to emphasize a word and for sub-sub headers, but they mean different things.
... once you know what the affordances of a word processor are, all you need to figure out is where the interface has hidden them.

We would then move on to the practical part, using a word processor they have never seen. We would use this interface so the students get a rough idea of what the interface looks like in real life. And finally, my students would then go home and practice with whatever version of MS Office it's installed in their computer. If at least one of them tries to align text with multiple spaces only to feel dirty and re-do it the right way, I will consider my class a success.

Would this work? Pedagogically, I think it would. But I am painfully aware that my students would hate it. And good luck selling a computer course that doesn't interact with a computer. It occurs to me that perhaps it could be done in an interactive program, one that "unlocks" interface perks as you learn them. If I'm ever unemployed and with enough time in my hands, I'll give it a try and let you know.

Footnotes

[1] A top-down approach would be learning the concepts first and the implementation details later. Its counterpart would be bottom-up, in which you first learn how to do something and later on you learn what you did that for. Bottom-up gets your hands dirty earlier, similar to Mr. Miyagi's teaching style, while top-down keeps you from developing bad habits.

My WIP: unsignedch.ar

2021-09-16T00:00:00+02:00

I have been too busy to blog the last month and a half, so I thought I'd take a bit of time to talk about my new project, unsignedch.ar.

Ever since I started taking care of this server I have been worried about the projects I host here - the more scripting languages I install, the higher the chances that someone will find a vulnerability and use my server for mining cryptocurrencies.

Therefore, I have started a new side-project: a new server where I will host all of my coding experiments, knowing full well that I can reinstall the whole thing whenever needed.

The server is currently under construction, but if you're interested in a sneak peak you can access my current draft of a git tutorial following this link. If you have comments on that draft, feel free to reach out to me.

How to recover your GMail account in four months

2021-07-30T00:00:00+02:00

Last time I talked about GMail I mentioned that my account was blocked for sending about 100 e-mails despite following Google's best practices for doing so. On July 23 I got my access back, and I though I would update you all on how that happened since.

Note: This article is rather long because there's plenty of e-mail content. If you don't care about the details, you can jump to the end for a timeline and some final thoughts.

March

After getting my account blocked on March 27, I contacted Google support. Following an automated message receipt, I received this message:

Hello Google user,

Your account has been disabled due to unusual activity being detected. We take security seriously and want to make sure that only you have access to your account.

(...)

How do I regain access to my account?

Sign in to any Google product. If your password is accepted, you'll be asked a set of questions to verify that you are the owner of your account. Once the verification is complete, you can safely continue using your account.

What if my password doesn't work?

If your password is rejected, please visit the [Account support page] and answer all of the prompted questions as best as you can.

If you remember my previous post you might know that none of these suggestions is useful. I replied with a message saying exactly that, but I got no response.

April

Here's an interesting fact: while Google has no obligation to keep me as a customer (or, in this case, product), they are obligated by the General Data Protection Regulation (GDPR) to give me a copy of my personal data. And even though Google provides a tool for downloading a copy of your data, the tool is useless if you can't log in. With this in mind, and with the help of the My Data Done Right tool, I sent the following letter (yes, letter) to Google's Data Controller on April 28:

To Whom it may concern:

I am invoking my right to data portability as specified in Article 20 of the General Data Protection Regulation. In particular, I am requesting Google Ireland Limited ("Google") to either provide me with a copy of my e-mails and other personal data in a structured, commonly used and machine-readable format or to grant me access to existing tools such as Google Takeout so I can do it myself.

I am the owner of the GMail email address <redacted>@gmail.com. For the past two weeks Google has blocked my access to my account and refused all methods of verification. I have provided the correct password, the correct verification e-mail address and a valid telephone number, none of which worked. Both the "Google Takeout" tool and the "Data Access Request Form" mentioned here are unavailable to me for this reason.

I request that Google either restores my access so I can use Google's tools myself or that Google provides me with a copy of my data following the GDPR's Right to Data Portability. I can provide further means of verifying my identity if necessary.

Why a letter? Three reasons:

Because I knew a human would have to process it.
Because I wanted a paper trail in case I decided to hire a lawyer (I paid extra to send it via registered mail).
Because signing as "Dr. Martín Villalba" with blue pen sends the signal that I'm an annoying person and that we would all be better off if they simply fast-tracked my request.

May

On May 4 I got the following reply from Google's Data Protection Office:

Hello,

Thank you for contacting us.

It sounds like you're having some problems with your account.

If you can't sign in to your account: Learn how to [recover your account]

If you're having trouble recovering your account: Try these tips to get [your account back].

Regards,

Google

After following the steps above (once again, they didn't work), I made a mistake. See:

What I should have replied is "this is not an account recovery request, but rather a data access request. While giving me my account back is one way of fulfilling that request, that's not the purpose of my letter".
What I did end up replying was telling them that I tried all of those options and none of them work.

Why was that a mistake? Because this is the reply I got on the same day:

Hello,

Thank you for contacting us.

Please note that this team does not handle account recovery related questions. Please refer to our prior email for more information, as well as follow these steps to recover your account [g.co].

As we are not able to further assist you, we are closing this inquiry.

True to their word, they closed the inquiry and never replied again.

June

Having learned from my mistakes, on June 1st I sent a second letter to politely remind them that it's been more than 30 days since my request. Why? Because 30 days is the period granted by the GDPR to fulfill data access requests like mine.

To Whom it may concern,

I am the owner of the Gmail address <redacted>@gmail.com. I have contacted you on April 28th to request a copy of my personal data as it is my right under Article 20 of the General Data Protection Regulation. It has been more than 30 days since my original request (Internal Ref. <redacted>) and yet I have received neither a copy of said data nor access to a tool where I could download it myself.

I request once again that you provide a copy of all my data (including the content of my e-mails) in a structured, commonly used, and machine-readable format. As a reminder, I have no access to the "Google Takeout" tool and none of the options suggested in the following links grant me access to the data I request. Therefore, I cannot accept suggestions of using these websites as a valid response:

https://g.co/recover

https://support.google.com/accounts/answer/7682439

https://support.google.com/accounts/answer/7299973

To reiterate: this is not an account recovery request - it has been more than two months since Google revoked access to my account and I consider it deactivated for all practical purposes. Instead, I only request a copy of my personal data. For purposes of identification I am still in possession of the current password and the recovery e-mail address, but I would be willing to provide further proof of identity if necessary.

And then I went back to living my Google-free life.

July

On July 16th I opened my e-mail and found this:

Hello,

Thank you for contacting us.

The information you seek may already be available to you via a number of secure online tools we provide to all users to access data. Sign in to your [Google Account] to get an overview of the ways you use Google’s services and access that data. Here are some other actions you can take:

(... long e-mail redacted ...)

To which I replied

Dear Sir or Madam,

thanks for your reply. As I explained before, Google has blocked access to my account. None of those tools work for me because I cannot sign in and no one replies to my account support emails.

Seeing as your office is in charge of data requests, I reiterate my request that my data be provided to me. Suggesting apps I cannot use are not a satisfactory response to my request.

On July 22nd, and coinciding with the anniversary of the most expensive hyphen in history, I finally got a step closer to my goal.

Hello,

We understand that you can’t sign in to <redacted>@gmail.com. You can file a claim and start the process to get back into your account.

To recover your account:

File a claim with the [Google Internal Escalations link]. This is a special link, so please do not share it with anyone.

Important: This link creates a claim so the Google Accounts team can investigate, but doesn't guarantee you'll get your account back. However, please make sure we have the relevant information to investigate.

Here's what I told the Accounts team:

Hello,

my case ID is <redacted>. As a reminder, this is a request for a copy of my personal data - while access to my Google account does fulfill this request, I am just looking for a copy of said data in any electronic format.

And guess what? On July 23 I finally got what I was asking for:

Hello,

To recover access to <redacted>@gmail.com, reset your password.
          [RESET PASSWORD]
The link to reset your Google account password expires in 7 days. If your link already expired, reply to this email to get a new link.

Timeline

March 27: my account is blocked. I fill an online form, but I only get a canned response.
April 28: I sent my first letter.
May 4: I receive an e-mail misunderstanding the problem. My ticket is closed.
June 1: I sent the second letter.
July 16: I receive an email suggesting I use Google Takeout. I reply that this doesn't help because I can't log in.
July 22: I receive a link to escalate my issue.
July 23: I regain control of my account.

Final thoughts

I'd like to once again thank the My Data Done Right people for providing letter templates that I could use and, more important, the mailing address of Google's Data Protection Office. If you are in the EU and you have data access problems, make sure you pay them a visit.

If you don't have as much time as I do, then a lawyer might help you speed up the process. I imagine a certified demand letter from a lawyer might have gotten me a quicker resolution, but now we will never know. Feel free to get your account banned and let me know afterwards how it goes.

And finally: take control of your data. Make sure that what happened to me can't happen to you. You don't have to administer your own e-mail, but you can definitely use a provider with reasonable customer support.

Polar coordinates and circular layouts

2021-07-02T00:00:00+02:00

Here's a "trick" that has saved me a lot of time and that plenty people have never learned.

As those with graphics experience probably know, when you draw something in a computer screen you need to give the x and y Cartesian coordinates where your object will be drawn. This makes drawing squares easy but, on the other hand, makes circular layouts difficult.

This is where polar coordinates come to the rescue. The idea is simple: instead of giving the coordinates of an object as a combination of x and y, you give its position as a combination of a radius (that is, the distance to the center of your graphic) and an angle. This is super convenient for several reasons:

Making a circular layout is really easy - all you need to do is increase the angle value while keeping the radius constant. If you want to draw 10 circles, all you do is increase your angle in increments of 360°/10 = 36° degrees (which, for polar coordinates, translates to an angle of 2*Pi/10 radians).
Drawing a spiral is also easy - all you need to do is increase the radius at every step.
The equations for going from Cartesian coordinates to polar coordinates and back are trivial.

Here's a Python function that draws num_circles around a middle point with a distance of radius to said middle point:

import math

# Gray circle in SVG format
circle_string = '<circle cx="{}" cy="{}" r="10" fill="gray" />'
def draw_circles(middle_x, middle_y, num_circles, radius):
    """ Draws `num_circles` circles around the (circle_x, circle_y) point.
    The circles distance to the center is `radius`.

    Parameters
    ----------
    middle_x : int
        Horizontal coordinates (in pixels) of the center of your graph.
    middle_y : int
        Vertical coordinates (in pixels) of the center of your graph.
    num_circles : int
        How many circles will be drawn.
    radius : int
        Distance (in pixels) from every circle to the center of your graph.
    """
    angle_delta = (2*math.pi) / num_circles
    for step in range(num_circles):
        angle = step * angle_delta
        # Equations to turn polar coordinates into Cartesian
        x = middle_x + (radius * math.cos(angle))
        y = middle_y * (radius * math.cos(angle))
        print(circle_string.format(x, y))

Polar coordinates have also helped me in a 2D car racing game I never finished. By storing polar coordinates for my car, I could save a lot of work:

The direction of the front wheels is stored as an angle - steering is as easy as increasing or decreasing this value.
When drawing the sprite on screen I simply take the base sprite and rotate it by the same angle mentioned above. Now my car is looking in the direction of movement.
The acceleration is the radius. Accelerating is as easy as increasing this one value.
To move my car one frame all I need is to convert the angle and radius to Cartesian coordinates and sum them to the current position of the car.

This is one of my favorite graphics tricks, along with the equations for turning 3D coordinates into 2D. Problems requiring these solutions don't come up every day, but when they do you'll be really happy about knowing them.

Hierarchical loss for multi-label classification - Part II

2021-06-26T00:00:00+02:00

In my previous post on hierarchical loss for multi-label classification I gave an implementation of a specific algorithm for calculating the loss between two trees. I then added a quick edit mentioning that "this algorithm doesn't work too well in practice", and today I want to delve into why.

Imagine you want to predict the cities where someone lived based on some data. The hierarchy of locations is a tree with country at the first level, province or state second, and city at its third level. This tree has ca. 195 nodes on its first level and a lot more as we go down the tree.

Let's now say that I was supposed to choose Argentina.Misiones.Posadas (which corresponds to a city in Argentina) but I predicted Congo.Bouenza.Loutété; (which is the 10th most popular city in the Republic of Congo). The loss for this prediction is 0.01, which is surprisingly low - seeing as I wasn't even close to the real answer, I would have expected something near 1.

As we go deeper into the tree, the loss goes down real quick. If I had predicted Argentina.Chubut.Puerto Madryn (a city 1900km away in one of the other 23 possible provinces) the loss would be 0.00043, and if I had predicted Argentina.Misiones.Wanda (one of the other 64 cities in the correct province) my loss would have been 0.000019. If your tree is deeper than this then you will soon start running into numerical issues.

The problem here is the nature of the problem itself. Because my predictions are multi-label there is no limit to the number of cities where a person may have lived while, simultaneously, there is no limit to how many cities I may predict. If I predict that a person has lived in every single city in America, from Ward Hunt Island Camp in Canada down to Ushuaia in Argentina and everything in between, but it turns out that the person has lived in all other cities in the world, my loss would only then be 1. And if it turns out that the person has briefly lived in Argentina.Misiones.Posadas then my loss goes down to ~0.995 because getting one city right also means that I got the country right.

Now you see why this algorithm is very good in theory but not useful in practice: if you are trying to predict one or two points in a big tree then your losses will always be negligible. No matter how wrong your prediction is, the loss for a "normal" person will never be high enough to be useful.

On the other hand, if you are expecting your predictions to cover a good chunk of the tree then this algorithm is still right for you. Otherwise a good alternative is to use the Jaccard distance instead and represent Argentina.Misiones.Posadas as the set {"Argentina", "Argentina.Misiones", "Argentina.Misiones.Posadas"}. This is not as fair a measure as I would like (it punishes small errors a bit too harshly) but it still works well in practice. You could also look deeper into the paper and see if the non-normalized algorithms work for you.

Good practices on sharing your research with end users

2021-06-17T00:00:00+02:00

So, this is a thing that happened:

I was invited to give a talk to the Social event organized by LatinX in AI during the NAACL 2021 conference.

I talked about best practices for publishing your code on the internet for everyone to see, starting from how to collaborate with your future self (aka "please write comments"), with scientists, with nice APIs who will do the web design for you, and finally directly with final users. I have published the slides in this PDF, and will publish the video (or even better, a transcription) as soon as I get my hands on it.

Update July 11th: the presentation with notes is now available here.

Hierarchical loss for multi-label classification

2021-05-22T00:00:00+02:00

Here's one of those problems that sounds complicated but, when you take a deep dive into it, turns out to be just as complicated as it sounds.

Suppose you build a classifier that takes a book and returns its classification according to the Dewey Decimal System. This classifier would take a book such as "The return of Sherlock Holmes" and classify it as, say, "Fiction".

Of course, life is rarely this easy. This book in particular is more often than not classified as 823.8, "Literature > English > Fiction > Victorian period 1837-1900". The stories, however, were written between 1903 and 1904, meaning that some librarians would rather file it under 823.912, "Literature > English > Fiction > Modern Period > 20th Century > 1901-1945".

Other books are more complicated. Tina Fey's autobiography Bossypants can be classified under any of the following categories:

Arts and Recreation > Amusements and Recreation > Public Entertainments, TV, Movies > Biography And History > Biography
Arts and Recreation > Amusements and Recreation > Stage presentations > Biography And History > Biography
Literature > American And Canadian > Authors, American and American Miscellany > 21st Century

This is known as a hierarchical multi-label classification problem:

It is hierarchical because the expected classification is part of a hierarchy. We could argue whether Sherlock Holmes should be classified as "Victorian" or "Modern", but we would all agree that either case is not as bad as classifying it under "Natural Science and Mathematics > Chemistry".
It is multi-label because there is more than one possible valid class. Tina Fey is both a Public entertainer and an American. There is no need to choose just one.
It is classification because we need to choose the right bin for this book.
It is a problem because I had to solve it this week and it wasn't easy.

There seems to be exactly one paper on this topic, Incremental algorithms for hierarchical classification, and is not as easy to read as one would like (and not just because it refers to Section 4 when in reality should be Section 5). Luckily, this survey on multi-label learning presents a simpler version.

I ended up writing a test implementation to ensure I had understood the solution correctly, and decided that it would be a shame to just throw it away. So here it is. This version separates levels in a tree with '.' characters and is optimized for clarity.

Edit June 17: this algorithm doesn't work too well in practice. I'll write about its shortcomings soon, but until then you should think twice about using it as it is.

Edit June 26: Part II of this article is now up

#!/usr/bin/python
from collections import defaultdict

def parent(node):
    """ Given a node in a tree, returns its parent node.

    Parameters
    ----------
    node : str
        Node whose parent I'm interested in.

    Returns
    -------
    str
        Parent node of the input node or None if the input Node is already a
        root node.

    Notes
    -----
    In truth, returning '' for root nodes would be acceptable. However,
    None values force us to think really hard about our assumptions at every
    moment.
    """
    parent_str = '.'.join(node.split('.')[:-1])
    if parent_str == '':
        parent_str = None
    return parent_str

def nodes_to_cost(taxonomy):
    """ Calculates the costs associated with errors for a specific node in a
    taxonomy.

    Parameters
    ----------
    taxonomy : set
        Set of all subtrees that can be found in a given taxonomy.

    Returns
    -------
    dict
        A cost for every possible node in the taxonomy.

    References
    ----------
    Implements the weight function from
    Cesa-bianchi, N., Zaniboni, L., and Collins, M. "Incremental algorithms for
    hierarchical classification". In Journal of Machine Learning Research,
    pages 31–54. MIT Press, 2004.
    """
    assert taxonomy == all_subtrees(taxonomy), \
           "There are missing subnodes in the input taxonomy"

    # Set of nodes at every depth
    depth_to_nodes = defaultdict(set)
    # How many children does a node have
    num_children = defaultdict(int)
    for node in taxonomy:
        depth = len(node.split('.'))-1
        depth_to_nodes[depth].add(node)
        parent_node = parent(node)
        if parent_node is not None:
            num_children[parent_node] += 1

    cost = dict()
    for curr_depth in range(1+max(depth_to_nodes.keys())):
        for node in depth_to_nodes[curr_depth]:
            if curr_depth == 0:
                # Base case: parent node
                cost[node] = 1.0/len(depth_to_nodes[curr_depth])
            else:
                # General case: node guaranteed to have a parent
                parent_node = parent(node)
                cost[node] = cost[parent_node]/num_children[parent_node]
    return cost

def all_subtrees(leaves):
    """ Given a set of leafs, ensures that all possible subtrees are
    included in the set too.

    Parameters
    ----------
    leaves : set
        A set of selected subtrees from the overall category tree.

    Returns
    -------
    set
        A set containing the original subtrees plus all possible subtrees
        contained in these leaves.

    Notes
    -----
    Example: if leaves = {"01.02", "01.04.05"}, then the returned value is the
    set {"01", "01.02", "01.04", "01.04.05"}.
    """
    full_set = set()
    for leave in leaves:
        parts = leave.split('.')
        for i in range(len(parts)):
            full_set.add('.'.join(parts[:i+1]))
    return full_set

def h_loss(labels1, labels2, node_cost):
    """ Calculates the Hierarchical loss for the given two sets.

    Parameters
    ----------
    labels1 : set
        First set of labels
    labels2 : set
        Second set of labels
    node_cost : dict
        A map between tree nodes and the weight associated with them.

    Notes
    -----
    If you want a loss between 0 and 1, the `nodes_to_cost` function implements
    such a function.

    Returns
    -------
    float
        Loss between the two given sets.

    References
    ----------
    The nicer reference of the algorithm is to be found in
    Sorower, Mohammad S. "A literature survey on algorithms for multi-label
    learning." Oregon State University, Corvallis (2010).
    """
    # We calculate the entire set of subtrees, just in case.
    all_labels1 = all_subtrees(labels1)
    all_labels2 = all_subtrees(labels2)
    # Symmetric difference between sets
    sym_diff = all_labels1.union(all_labels2) - \
               all_labels1.intersection(all_labels2)
    loss = 0
    for node in sym_diff:
        parent_node = parent(node)
        if parent_node not in sym_diff:
            loss += node_cost[node]
    return loss

if __name__ == '__main__':
    # Simple usage example
    taxonomy = set(["01", "01.01", "01.02", "01.03", "01.04", "01.05",
                    "02", "02.01", "02.02", "02.03", "02.03.01"])
    weights = nodes_to_cost(taxonomy)
    node_1=set(['01'])
    node_2=set(['01.01', '02'])
    print(h_loss(node_1, node_2, weights))

The guerrilla guide to looking good online

2021-05-01T00:00:00+02:00

I recently confirmed that high quality audio makes you sound smarter, which is exactly what I always wanted: a way to look smart without having to actually work for it. This discovery led me to an internet rabbit hole on how to look and sound good online, with this guide being the end result. If you are trapped inside online-meeting purgatory like me, hopefully this guide will give you a small edge in your next important meeting.

I divided this guide in three sections:

Video: while video is not as important as audio, it's the easiest one to improve. You may not know exactly how to better equalize your voice, but identifying which part of your face needs light is easy.
Audio: you can live with bad video (or no video at all) but bad audio is a different issue.
Delivery: once you are clearly seen and heard, let's talk about how to improve your message.

Before we start, a couple words of meta-advice: by caring about how you look and sound you are already ahead of everyone else who just turns their computer on and shows up. And since you can't improve when you don't know what is there to improve, your first step is to go get some feedback. If you can't find someone willing to have a meeting with you then you should at least have a test meeting alone. For instructions click here: Zoom, Teams, Jitsi.

Video

As this guide points out you should ensure there is no strong light coming from behind you. Ideally you want a three-point lighting setup but a good compromise is a general strong light source (such as an open window or room light) plus diffuse light behind your monitor (either point a lamp towards the wall behind your monitor or a full-screen, white document opened on your screen). The next step up are ring lights, but we have other, more pressing issues to worry about first.

Your background comes next. Most meeting software nowadays includes a "blurry background" filter that you can use to hide what's going on behind you. These filters don't work as good as I wish they did, but they have nonetheless been a blessing for those of us sitting in shared living rooms. Still: consider re-orienting your camera (or your desk!) to keep a clear, distraction-free background.

Which brings us to the final point: the camera itself. Whichever camera you have around is likely to be fine. A more expensive camera will give better results, but they might not be worth the cost. Pro tip: if you have a DSLR camera laying around, it may also double as an amazing webcam too. Check your manual.

Pay attention to the camera angle. Keep your camera at eye level either by repositioning your webcam (if it's an external one) or by getting yourself a laptop stand (which you can also build out of cardboard). Say no to cameras looking at you from below!

Audio

Audio is tricky: it is more important than video during conferences but it's harder to tune adequately. Let's get the obvious out of the way: ideally you want a quiet room for your meeting, but there's only so much you can do with the rooms your apartment already has. So let's not dwell on that.

Unlike video, where you can get far with what you have, in audio you really, really want to have a better microphone. You don't have to go pro (in fact, an expensive mic can easily backfire by being too sensitive) but you should at least get a decent, dedicated one. If you have no idea of audio then I would recommend a USB mic - I have had bad experiences with microphones picking up line noise and USB should help with that. And since using your speakers is guaranteed to cause echo sooner or later, save yourself the trouble and get some headphones too.

If you want to tweak your voice even more you can try a software equalizer. There are plenty of guides around courtesy of the internet, but getting into details goes beyond this guide.

Delivery

Once you have optimized your environment as much as possible, it is time to talk about delivery. That's a topic by itself, so I'll limit myself to two tips:

Dress appropriately and keep a neat background. Bookshelves are particularly nice. If you show a messy room your audience will assume you are also a messy person with messy ideas, and no one wants that.
You don't have to go and get a voice coach, but it might be worth your time to watch a couple videos on the topic. I have personally learned a lot from the Broadcast Voice Handbook but you might be better served by more casual online courses. Youtubers have created an explosion of content on that area, so it should be easy to find.

How helping Latin American researchers cost me my GMail account

2021-03-30T00:00:00+02:00

Have you heard of the LatinX in AI social event? It is a social event organized by LXAI intended to bring together Latin American researchers in NLP and AI. I joined as a participant in their EMNLP 2020 edition, and I am now volunteering to run the 2021 version to take place soon parallel to EACL 2021.

One of my tasks as part of the organizing committee is to send invitations to those who joined last year. And you can probably see where this is going: even though I have written permission from these 116 participants to contact them and even though I followed Google's best practices for sending emails, my 18-years-old GMail account was nonetheless blocked and it has remained so ever since.

I have now spent several days in Google support hell leaving no stone unturned and no link unclicked. If you have never tried to get support from Google, the following diagram illustrates all the maddening steps I have followed during the account recovery process with no luck so far:

The "Number not accepted" box is particularly annoying: I have always refused to give Google my personal telephone number because there is no guarantee that they won't use it for tracking me like Facebook was caught doing, and you cannot enable 2-factor-authentication without providing one first (trust me, I tried). As a result, Google will not trust any number I give now - it is mildly funny to read that the telephone number of the Fortune Global 500 company where I work "has already been used too many times for verification" even though I had never used it before. Either whoever used my desk before me was a serious spammer, or Google is not being as honest as one would expect.

But you know what hurts the most? That all of this could have been avoided if I hadn't insisted on personalizing the emails. I hate emails addressed to "Dear sir or madam" and therefore went out of my way to write the script that would pull people's names and display it properly. If I had written a generic email instead and dumped 100+ addresses in the web interface I would probably still have my account. I know it isn't much, but that's all I could do to show people that I care about them receiving their invitation. No good deed goes unpunished.

Maybe in the future I will write about all my complaints. One particularly mean example are the emails I still get in my recovery account letting me know that someone has been trying to access my account but that I shouldn't worry because they didn't let them in. But for today, I want to leave you with two parting thoughts.

First: this story is not new, and if you have all your eggs in the Google basket it is only a matter of time before you lose something important with no recourse. Maybe they will remove your browser extension, ruin your startup, kill your game, terminate your Android app, delete your YouTube channel, or who knows what else. So be prepared. I can assure you that if I had not started using my own email domain years ago I would now be truly screwed with no way forward. If you are not willing to leave Google products for good, at the very least get a local copy of your data with Google Takeout and keep it safe.

And second: I still need that account to organize the event. So if you know someone who works for Google, please tell them to write me an email to get this sorted out. I would be slightly sad of losing the epic burn "Google closed my 18-years-old account forever for helping Latin American researchers", but I'll let it go if that means moving this event forward.

GPT-3 is blockchain

2021-03-29T00:00:00+02:00

I need to share with you an epiphany that occurred to me yesterday.

Have you heard of GPT-3? If not, I can tell you that it's a language model that has been showing up everywhere. Having been trained with a lot of data, it can generate text that people find impressive.

If you follow the hype, GPT-3 will revolutionize everything - people have been using it to generate plausible-looking creative fiction, pickup lines, SQL Queries that are sometimes wrong, trivia answers, tweets, and so on.

But you know what no one has generated yet, as far as I know? Something useful. Or even better: something that people always wanted but current technology cannot provide.

People are excited about GPT-3 because it promises to "just work" - you give it the right prompt and you get the right answer. This would solve all of those pesky problems associated with NLP such as "this search terms make no sense", "I hate knowledge bases", "That question has multiple answers", or "I don't want to manually write all possible answers for my system". But this is not what GPT-3 can do, because GPT-3 will not bend to your so-called facts and therefore will not do what you want. As Robert Dale puts it when talking about GPT-2: "driven as it is by information that is ultimately about language use, rather than directly about the real world, it roams untethered to the truth". In other words, people are excited about GPT-3 because they think it solves a different problem that the one it actually does.

If you want a chatbot to tell a patient that the solution to their depression is talking to a professional instead of GPT-3's suggestion that they should just go ahead and kill themselves, you need a way to constrain the system's output. This means that you still need to write the code that interprets the patient's problems, the code that chooses the right solution to that problem, and the code that says exactly what you want, no more and no less. And while turning structured data into human-readable sentences is a valid possible use for GPT-3, the amount of work required to constrain its output to an acceptable error level is comparable to the effort required to write smart templates that guarantee you'll generate exactly what you want.

And so, GPT-3 joins blockchain technology in being a solution searching for a problem. In fact, the parallels are kind of amazing: both technologies are hyped to the extreme, completely misunderstood by the general public, very expensive to run, and products based on them rarely make it out of the proof-of-concept stage.

I would like to leave you with two optimistic thoughts. First, I do think that it is only a matter of time before someone actually finds a good use for GPT-3. I predict it is going to be something marginal, with my best bet being something related to grammatical correctness. Abstractive summarization is also a good candidate, but my faith is lower because inserting unrelated facts is simultaneously what abstractive summarization tries to avoid and what GPT-3 does best.

And second, I want to let you know that there's a great business opportunity here. The blockchain craze reached the point where simply putting "blockchain" in your company name is enough to make your stock price rise by 289 percent. Therefore, if my prediction is correct then all you need to do is either name your own company "GPT-3" or invest in someone else doing it. Sure, their stock will probably tank once investors realize they invested for the wrong reasons, but by then you will have hopefully cashed out and moved on to something else.

Disclaimer: I am not an investment banker, this post does not constitute financial advice, I don't know why anyone would listen to me, and you shouldn't follow advice you find on random blog posts anyway.

Uber in Argentina

2021-03-08T00:00:00+01:00

Uber arrived in Argentina working in a grey legal area, as usual. Word of mouth is that Uber refused to be classified as a transport company and insisted on being classified as a digital services company instead. These legal problems led to them being unable to accept Argentinean credit cards for payment. But Uber kept offering the service at a loss, allowing local drivers to accept cash and adding the debt to their driver profile. According to insiders, the drivers were expected to keep Uber's 25% cut aside and transfer it once in a while themselves. Although Uber eventually managed to get access to credit cards, they kept the cash option available.

The collateral damage of this policy is extensive.

Some drivers decided not to settle their debts with Uber, keeping 100% of the proceeds instead. If and when Uber closes the driver's account they get a new SIM card, send fake documentation, and start with a fresh account that lasts between a week and a month. These drivers accept only cash: they have no bank account data to provide because their data is fake, and they know that it's only a matter of time before their account gets banned anyway.

Because these drivers accept that their account is temporary, none of Uber's typical incentives work. When a passenger pays with credit card the money goes straight to Uber and the driver doesn't see a dime -- it all goes away to settle a debt they had no intention of paying. Therefore, drivers will often contact potential passengers asking how they intend to pay. If the passenger says "credit card", the driver either cancels the trip or straight up ignores the passenger forcing them to cancel. You can take the time and report the driver, but few people do it and all it does is to cause a mild inconvenience to the driver.

And while this is inconvenient for the passenger, it also opens the door to the really shady practices: once you have no way of verifying that the driver is who they claim to be, you are one step away from being robbed by a fake driver (in Spanish).

In short, Uber Argentina has become yet another dysfunctional taxi service. And rival local apps are catching up: not only do they have their paperwork up to date, but they have also incorporated apps into their daily routine. It would be no surprise if Uber were still operating in Argentina just for PR purposes. With a 43% drop in revenue for Latin America last year, and with Uber pinky swearing that they will achieve profitability any time now, the only reason I can see for Uber operating in Argentina is to keep the illusion of "one app for the entire world".

And sure, that's a fair point. But I have no reason to believe that these problems are exclusive to Argentina, and probably neither should you. I wrote this story because I found it interesting and I picked Argentina because that's what I know about, but if you are one of those tourists who blindly gets into an Uber believing that their drivers are more honest than taxi drivers you may be up for a rude awakening. Apps are not well known for solving deep, systemic social problems after all.

Sources

The information for this post came from these threads in Reddit's /r/argentina: Thread 1, Thread 2, Thread 3.

Why won't the music industry take my money?

2021-03-01T00:00:00+01:00

I have tried this week to buy the soundtrack for The Greatest Showman for a gift and let me tell you, it's really hard.

I started naively thinking that, since the album is available on Amazon as MP3, I could just click "Buy" and be done with it. But Amazon, as it turns out, doesn't want my money. Sure, they say they will sell me the album. But once I actually try they reject my credit and debit cards with a mysterious error that, after some digging, may be related to Amazon not having the rights to that album in Germany. I say "may" because Amazon doesn't give me any usable information - all they show is this error:

We were unable to process your purchase with your current payment information. Please enter a valid payment method and an address which are both local.

Seeing that my credit card is valid, my address is local, and the buy page doesn't mention any kind of restrictions, that's my first dead end.

My second stop is Warner Music, who owns the soundtrack. This is also a waste of time: they will gladly sell me physical copies in vinyl, but digital? No luck there.

Next: Apple, the first big company to offer DRM-free music downloads and self-professed champions of user experience. We were off to a rocky start: you can only buy music using iTunes, which is not available in Linux and forces me to boot my Windows 10 PC. One hour later, courtesy of Windows 10 deciding it's a good time for an update, I am faced with this screen:

If you think this well-known and yet unresolved issue stopped me, you are mistaken - I have signed way too many contracts in languages I don't fully grasp to be afraid of what is clearly a credit card details form. Luckily, after giving my password like 6 times, converting m4a files to mp3, and almost two hours later, I am finally the proud temporary owner of this soundtrack.

So let's talk now about Spotify. I reluctantly started using it again because it's one of the few services with an offline mode for Android phones that doesn't require giving my phone number. Seeing as I still object to their collection of private data, I created a fake profile that I regularly renew with gift cards. But do you know what happens when your subscription is about to run out? The answer is "nothing": you get zero notifications, no e-mail, nothing.

What happens when my subscription runs out? First: all of my offline music is deleted, which is the one feature I'm paying for. Since I'm often in offline mode for work, that means no music for me for the rest of the day. And second: just like there is no notification about my balance running out, there is also no option in the app to give a new gift card code. I can easily give my credit card and subscribe forever, but gift cards require extra steps.

What these two infuriating stories have in common is that they are examples of the music industry working both badly and as intended. Amazon, Spotify, and Apple (up to a point) will gladly give me access to the music I'm trying to pay for, but only if I agree to set recurring payments to their walled gardens and access to my private data. Owning my music and keeping my privacy, however, is really hard.

Which brings me to my final point. There is a service with an extensive, high-quality music catalog that's easy to use, works on every platform, let's you keep your privacy, and will take your money but only if you really want to. It's called piracy. And even though it's been almost 10 years since Gabe Newell publicly pointed out how to effectively get rid of piracy for good, we are somehow still living in a world where buying a single music CD takes two hours, Windows, fluency in fictitious languages, and a computer science degree.

At least you can now order your vinyl records via e-mail. Take that, 1980s!

Which apostrophe should I use?

2021-01-16T00:00:00+01:00

As someone who regularly switches between keyboard layouts, I have a problem: I have at least three keys that can be used as an apostrophe, but I don't know which one is the correct one. Compare:

Backtick: Hamlet`s father
ASCII Apostrophe: Hamlet's father
Acute accent: Hamlet́s father
Single closing quote: Hamlet’s father

Do you know which one is the right one? If not, this small guide is for you. But you'll have to endure a lot of theory first.

The problem here is that using your keyboard requires mixing three different concepts: which key you pressed, which character it represents, and how is it visually represented. To explain that in clearer terms, let's take the backtick as an example.

The key itself can move around. The backtick key is located below the tilde (~) in a US keyboard, to the left of the backspace key in a German keyboard, and under the caret (^) in a Spanish keyboard.

Internally, this key is called "Grave accent" in ASCII but programmers know it it as "backquote" or "backtick". When you press it you send a code to your computer that, if you were using ASCII, would be reflected as the 0x60 hexadecimal value. And here we make another distinction: if your computer is configured to do so, this code can be interpreted as a dead key that only exists to modify the next character. If you want to type the è in the French word très (very) you use the combination <Grave accent key> + <e key>. If your computer is not configured in this way then you simply get a backtick.

But here we have a very subtle difference between a backtick and a Grave accent. The Grave accent is a modifier, changing the sound of the letter underneath. Therefore, it cannot exist by itself. If you see the character ` alone then it is not a Grave accent, it's a backtick. They both look the same, but they have different meanings.

Your program is another factor: some programs may replace the character you are using because it's very likely that you are using it wrong. If I type the double quote character (") in Microsoft Word it may be replaced by an Opening double quote (“), a German opening double quote („), an English closing double quote (”), or a German closing double quote (“, which is the same character as the Opening double quote in English).

And then we have the issue of fonts. Some fonts may represent two different characters in the same way, or straight up ignore it. If you have been staring at some of the quotes I mentioned above and see no difference, well, maybe that's why.

So back to our original question: what is each key good for?

The backtick quote (`) has no meaning in typography at all. The only reason a non-programmer would use it would be to type a Grave accent.
The apostrophe (') is the one I need for daily use. In truth, Unicode defines a different character as a "true" apostrophe (we'll get to that soon), but not all keyboards can generate it. Therefore, and just like the double quote example above, it is okay to use it.
The Acute accent (́) should only be used as a modifier. You can see that it insists on modifying whichever character is next to it, because it has no meaning by itself.
The Single closing quote (’) can be generated under certain combinations of keyboard and local configuration. If your keyboard supports it, this is the preferred key to use for apostrophes.

There are plenty other versions of the quote character that can be used. This list of commonly confused characters has been of great help, while this technical description of the apostrophe character also provides more info than you thought you needed. Sure, it may sound like a lot of work. But you'll thank me next time you need to write a phrase like “In a Déjà vu, René corrected ‘It’s x′, not x″’” knowing full well that you are right.

Internet should be an utility

2021-01-14T00:00:00+01:00

Have you heard of Parler? In case you haven't, Parler is (was?) a social network for ~~racists~~ the alt-right that gained notoriety this week. Having allegedly been used to coordinate the storming of the U.S. Capitol, its app was removed from both the Google Play Store and the Apple Store and then pulled from Amazon AWS the next day. With no hosting and no app, Parler has been effectively killed by the tech giants.

The swift removal of Parler from the internet is the incident that, I hope, will bring together the right and the left under a common cause: that the internet should be considered a public utility and that Internet Service Providers (ISP) should be regulated as such.

A public utility is a service that everyone needs (think water and electricity) and where regulation is needed because the high cost of entry discourages competition (so-called natural monopolies). It has been argued that internet should be included in this list too - can you imagine your current daily life without internet? ISPs, on the other hand, are happy setting their own prices and policies, and have resisted for years efforts in this direction.

Parler was effectively removed from the internet by the tech giants under the argument that, as private companies, they have the right to refuse service to anyone they don't want to work with[1]. But ISPs are private companies too, and therefore free to do the same to you - if your live in a country with no ISP regulation, your provider has a right to stop giving you internet access and tank your business with little repercussions.

And here is where I hope both "the left" and "the right" will see that their interests overlap. The left should support ISP regulations (and net neutrality!) because they believe, as Germany and France put it, that free speech should be governed by law and not by tech giants. The right, on the other hand, should realize that they gave tech monopolies all the cards and that they are the only ones getting kicked out of their social media accounts. If the President of the United States himself can be banned from Twitter, Facebook, Snapchat and Youtube, then no one is safe.

Why ISPs?

You might have noticed that everyone I linked above talks about regulating Amazon, Google, and/or Apple. I, on the other hand, would suggest that we focus on ISPs instead for the following reasons.

First, because the internet parallels the history of the telephone almost perfectly: a communication technology that catches on and that, while not biologically required (unlike water and heating), plays a critical role for life in a society. And ISPs are not "like" the telephone companies, they are the telephone companies.

Second, because it makes sense that internet should be provided to everyone without discrimination: imagine a world in which your shower stops working because you said in public that you prefer bottled water, or where your telephone is disconnected because you bad-mouthed someone during a conversation.

And finally, because it's the last step down the technology chain at which you can still survive: there are alternatives to Amazon AWS, and if they won't have you then you could still plug your own server and keep going (ask the Pirate Bay). But if the only ISP in town denies you service, what are you going to do, move your family to the closest town? Ask people to send you letters?

So here's my proposal: make the internet a public utility and, in exchange, give ISPs immunity from what their customers do with it. Let's bring the internet into the 21st century.

Footnotes

[1] Leaving illegal discrimination aside, which doesn't seem to be the case here.

Unity and Internet-Oriented Software Development

2021-01-04T00:00:00+01:00

I am developing a simple HTML5 game with Unity. The experience reminded me of my post about Rust and led me to coin a new term that I'd like to discuss today.

Ladies and gentlemen, in the spirit of Object-Oriented Programming I present to you Internet-Oriented Software Development (IOSD): a style of software development in which the official way to program is by trying something, giving up, asking strangers on the internet, and hoping for the best.

You may wonder: how is this new, seeing as that's what we are all already doing? The keyword here is "official". If you want to program with (say) Keras during a long trip with unreliable internet, you could do it with an offline version of the API reference alone. Sure, getting the offline docs will take a bit of work, but at least there's an official repository of knowledge that you can always go to. Of course you can search on the wider internet for help too, but you don't have to.

IOSD is different: when you release IOSD software you publish an okay guide to your software, and that's it. There is no need for you to keep it up to date nor for it to be useful, because your documentation is Internet-Oriented: if someone has a problem, they can ask the internet, IRC channels, their co-workers, anyone but you.

Rust came to mind because, as I complained before, that's how their intended development cycle apparently works: if you don't know how to do something, you are encouraged to either search the forums or ask the developers. In the later case, "we don't know, we won't do it, and we won't tell you who is doing it right" is a possible response. TensorFlow used to be like that too (ask me about the Seq2Seq tutorial!), but they reversed course and their current official suggestion is that you use Keras (no, seriously).

But Unity is an even better example. For starters, the official docs are essentially useless because they tell you what something does but neither why nor what to use it for. Can you guess in which situations is a Sprite renderer required, and what do you need it for? Because I can't. One might argue that Unity Learn is where you should look for answers, in which case one would be wrong. Taking the first course in the "Game development" section, for instance, gets you this tutorial which is only valid for an outdated version of Unity.

No, the real source of answers are YouTube tutorials. Sure, sometimes they refer to windows that aren't there anymore and/or changed their name, but you can always add a "2019.4" to your search and try again.

I am not entirely a beginner with Unity, as I worked with it for my PhD projects. Even then, the list of resources I needed to complete my current project so far includes 5 YouTube tutorials, 2 forum threads, and zero links to official documentation. Is this a problem? Is IOSD better than the thick manuals we had before? Am I the only one getting outdated answers for trivial problems? I have no idea. So I propose a compromise: I will point at the situation and give it a name, let someone else answer the hard questions, and we will share the credit.

Animations in Gephi

2020-12-13T00:00:00+01:00

Gephi is a tool for graph visualization that's great for plotting big datasets. If you have large, complex graphs to plot, Gephi has plenty of layouts that you can apply to your data and turn a mess of a graph into something that hopefully makes sense.

But there's something Gephi can't do: animations. The layout algorithms at work look awesome, and yet there's no easy way to create a video out of them. Capturing the screen is an alternative, but the final resolution is not really high.

It has been suggested that you could export images of the individual networks at every step and then put them together. Of course, the people who suggested this in the past have also...

... sworn that they have scripts to do it,
... promised that they'll release them after a quick cleanup, and
... disappeared from the internet without having published their solutions.

Therefore, this post.

Note: this post is not an introduction to Gephi, and I'll assume that you know the basics. If you don't know them, these slides look quite good.

Requirements

Gephi (obviously)
ImageMagick (to post-process images in bulk)
FFmpeg (to create the final video)

I am also assuming that you are using Linux. This is not a requirement, but it does make my life easier.

Preparing your data

The first step is to have some data to visualize. For the purposes of this exercise, you can download this zip file containing two files named nodes.csv and edges.csv. They form a simple directed graph that I generated from a sample of directories in my computer.

Now, load them into Gephi:

Start a new Project
Load the nodes table: File → Import spreadsheet → nodes.csv. Follow the steps, but remember to select "Append to existing workspace" in the last one.
Repeat the previous step, this time using the edges.csv file.

Next, choose a layout. In my experience it is better to try them all first, see which one gives the best result, and then generate the final graph from scratch. Generating the final graph will take a while, so it's better to do it only once you are sure about which parameters to use.

Exporting all frames

It is now time to run our first script. If you don't have the Scripting Plugin installed, you can do it now via Tools → Plugins. We will use it to write a very simple Python script that does the following:

Run a single step of a layout
Take note of the relative size of the graph (we'll come back to this)
Export it to a PNG file
Return to step 1.

In case you want to copy-paste it, this is the script I used. Don't forget to remove the comments first, because Gephi doesn't like them.

def make_frames(iters, layout, outdir):
    # As many iterations as frames we want to generate
    for i in range(iters):
        # Run a single step of our layout
        runLayout(layout, iters=1)
        # Calculate the bounding box of this specific graph
        min_x = 0
        max_x = 0
        min_y = 0
        max_y = 0
        for node in g.nodes:
            min_x = min(min_x, node.x)
            max_x = max(max_x, node.x)
            min_y = min(min_y, node.y)
            max_y = max(max_y, node.y)
        width = max_x - min_x
        height = max_y - min_y
        # Generate a file and include the graph's bounding box
        # information in its filename
        exportGraph("%s/%05d-%d-%d.png" % (outdir, i, width, height))

Once you have copied this script into the console, you can generate all animation frames with the command make_frames(100, FruchtermanReingold, "/tmp/"). This will run the FruchtermanReingold layout for 100 iterations, and will save the generated frames in the /tmp/ directory. Of course, you can choose other layouts (see the documentation for more info) and you can run the script for a larger number of steps. You can also customize the layout parameters in the regular Layout tab, and the script will pick them up.

The script will block Gephi entirely, so don't go for a really high number of steps from the beginning. Start with 50-100, and only then move on.
For a nicer effect, make a single run of the "Random" algorithm first. This will put all your nodes in a very small space, and the final effect will be like an explosion of nodes.

Generating the animation

A further issue to deal with is the changing size of the image canvas. If we set Gephi to generate a 1024x1024 output image but we only have two nodes close to each other, those two nodes will look huge. If we have thousands of disperse nodes, however, the individual nodes will be barely visible. Therefore, if you made a video with the images we generated in the previous section directly, you would almost certainly get a zoom effect where the nodes would get bigger and smaller as the graph gets denser or sparser respectively.

To avoid this, we need to scale all pictures proportionally. The Bash script below calculates the maximum theoretical size of our graph (based on those variables we added to the filenames before), scales all images down to the proper size (as defined by canvas_w and canvas_h), and places them in the center of a single-color canvas (see the rgb() call).

# Directory where all individual frames are
SOURCE_DIR=/tmp
# Directory where final frames will be stored
TARGET_DIR=/tmp/outdir

# Obtain the theoretical maximum width and height in the PNG frames
max_width=`ls ${SOURCE_DIR}/*png | cut -f 2 -d '-' | sort -n | tail -1`
max_height=`ls ${SOURCE_DIR}/*png | cut -f 3 -d '-' | cut -f 1 -d '.' | sort -n | tail -1`

# Give your desired canvas size
canvas_w=1024
canvas_h=1024

# Scaling factor for the frames, based on the largest theoretical dimension
if (( $max_width > $max_height ))
then
    factor=`bc -l <<< "$canvas_w/$max_width"`
else
    factor=`bc -l <<< "$canvas_h/$max_height"`
fi

# Generate the new frames
for file in ${SOURCE_DIR}/*png
do
    # Obtain the properties of the image
    frame=`echo $file | xargs -n 1 basename | cut -f 1 -d '-'`
    width=`echo $file | cut -f 2 -d '-'`
    height=`echo $file | cut -f 3 -d '-' | cut -f 1 -d '.'`
    # Calculate how the image should be scaled down
    new_width=`bc -l <<< "scale=0; ($width*$factor)/1"`
    new_height=`bc -l <<< "scale=0; ($height*$factor)/1"`
    # Put it all together
    convert $file -scale ${new_width}x${new_height} png:- | \
        convert - -gravity Center -background "rgb(255,255,255)" \
        -auto-orient -extent ${canvas_w}x${canvas_h} \
        ${TARGET_DIR}/${frame}.png
done

Once you have generated this second set of frames, you can generate your final video going to your TARGET_DIR directory and running the command

ffmpeg -framerate 30 -i %05d.png -c:v libx265 -r 24 final_video.mp4

If your video is too slow, a higher framerate value will do the trick (and vice versa). The final result, heavily compressed for the internet, can be seen below:

Your browser does not support the video tag.

Final thoughts

I hope you'll find this guide useful. I'm not going to say that it's easy to follow all these steps, but at least you can set them up once and forget about it.

Some final points:

For an alternative method of graph generation involving nodes with timestamps, this script looks like the way to go.
I'm interested in unifying everything under a single script - I chose Python because it was easier than Java, but maybe developing a proper multi-platform plugin is the way to go. I can't promise I'll do it, but I'll think about it. In that same vein, perhaps the script should center in a specific node?
If you plan to publish your video on the internet by yourself, this post gives a nice overview of which standards to use. If you want to tweak the video quality, this SE question provides some magic incantations for FFmpeg.
Finally, and speaking of magic incantations, special thanks to this post for providing the right ImageMagick parameters.

Wiki idea: what happens when you press Enter on your browser

2020-11-25T00:00:00+01:00

Here is an idea that I had and that I don't have time to work on. I read somewhere about the following job interview question:

What happens after you write a URL in your browser and press Enter?

If you think about this for a moment, you might realize that what this question really means is "tells us everything you know about computers". I have yet to find a topic that wouldn't be involved in giving a full response. Off the top of my head, and in roughly chronological order, you would have to explain...

... how your keyboard sends signals, including the difference between pressing and releasing a key. Also, how your computer display works.
... how to turn a series of electric impulses into a character.
... how to parse a URL, including the difference between Unicode and ASCII.
... how the internet works: DNS, TCP/IP, IPv4 vs IPv6, routing, etc
... how the browser and server negotiate the type of content they want. It might also include an introduction to the GZIP compression algorithm.
... a primer on HTTPS, including cryptography and handling certificates.
... what is a web server and how it works. Same for load balancers, proxies, and pretty much all modern server infrastructure.
... how your operating system renders anything on screen.
... how your web browser renders content.
... the standards involved in receiving content: HTML, CSS, JavaScript, etc.

I imagine that this would be a great idea for a Wiki: the main page would simply present the general question, and you could go deeper and deeper until you reach your motherboard's buses, your microprocessor's cache, the specifics of BGP, or pretty much anything that was ever used in an internet-connected computer.

I was never asked this question, which is a bit of a missed opportunity: I don't know exactly how many hours I could waste on this question, but I'm willing to bet it would be more than what any reasonable interviewer is willing to spend. More realistically, I imagine the point of the question is both to check whether you know about computers and, more important, whether you know when to stop talking about computers.

Here we go again - encryption and the EU

2020-11-09T00:00:00+01:00

The Council of the European Union has released this Draft in which they call for what is effectively a ban on End-to-End Encryption (E2EE). The document itself is unsurprisingly vague, but if you follow the parallel document about "Exceptional Access" you'll see a bunch of proposed solutions, all of which require the interception of your private communications. As it is to be expected, the documents pinky swears that this is the only way that terrorists and child predators will be stopped.

There are several reasons why this is a stupid idea. Today's post will briefly detail the main two.

First, this is technically impossible. The entire point of E2EE is that no one (not you, not me, not the NSA) can decrypt their content without the right key. And yet, the proposal that has been passed around in the last years is the idea of a "master key", a key that only authorities have and that would be "carefully" used by the authorities to legally decrypt content between two parties that they consider suspicious. So let's assume that WhatsApp implements this idea. They now have a single key that only the EU can access. Well, two keys - Australia has legally mandated backdoors, so they need their own. And China will need one too. The US wouldn't need one, simply because some of WhatsApp servers are in the USA and therefore the NSA can use a National Security Letter to force WhatsApp to reveal the other keys while forbidding everyone to talk about this. As you can see, the "one single key" idea is flawed from the very beginning.

And then there are the hackers: if it comes out that there is a secret key that breaks WhatsApp's encryption, it is now a race between WhatsApp's engineers to keep it safe against every single government in the world trying to break it.

The second main point is: if you ban secure communications, then only criminals will have access to secure communications. We already have unbreakable encryption and it is trivial for any criminal organization to deploy their own. So they are not the ones whose communications will get intercepted. The only wiretapped ones will be us, the law-abiding citizens. Instead of keeping us safe from criminals, the Council of the European Union is delivering us into the data collection efforts of the NSA and friends.

A call for action

Do you remember when the European Unions imposed sanctions against the NSA for their illegal data collection? Me neither, because that didn't happen. And I don't see why this time it would be any different. Well, there is that one time when Angela Merkel told Obama that she was angry he wiretapped her phone. I'm sure he felt really bad about that. But my point is: I wouldn't expect our politicians to stand up for our privacy, in particular when they are the ones creating the problem to begin with.

We have once again a proposal that will not stop any criminals, is technically impossible, and that is being written without asking anyone who knows what they are talking about. If you are in the EU I ask you to contact your representatives - I am not aware at the time of any movement against this, but I bet at least the Pirate Party will have something to say (edit Nov. 25: they do). The tech industry already lost the DRM fight (as exemplified by the ongoing youtube-dl saga) and the fight against Article 13. And there are lobbying efforts underway to bring software patents to Europe.

Don't let your privacy go away too.

Unpaid paid recommendations

2020-11-09T00:00:00+01:00

In today's weekend posting, two recommendations about things that are not free (which is a first) and a rant (which is very much in brand for this blog).

Drawing faces with JLJ

On a previous blog entry I complained that it's very difficult to find a good drawing tutorial because many, many teachers will suggest something as useless as "do whatever comes natural". So imagine my surprise when I found a course on drawing faces that makes none of those mistakes.

The course in question is titled "How to draw a portrait" and is taught by an illustrator from Florida called Joshua L Johnson. The course guides you through the steps of framing your drawing, identifying the main features, refining the details and, finally, adding shadows. The course can be found on Skillshare following this link.

I like this course for a couple reasons. First, each step is actionable: when he wants you to draw an eye, he explains that a generic eye is composed of 7 segments and explains where to place each one. Second, the workflow itself is designed in a smart way, first delimiting "areas" of work and then refining them step by step. The course ends with a 40 minutes, real-time lesson on how to draw a specific face from beginning to end which I found really helpful. So if your faces are as bad as mine, you should consider taking a look.

Solutions and other problems

It is hard for me to express to you how ridiculously funny Allie Brosh is. Her blog Hyperbole and a half is the only website I can remember where I had to stop reading for minutes at a time because I couldn't stop laughing. Some of the most well-known entries are probably This is why I'll never be an adult which gave rise to the "all the things" meme, and the creation of the Alot. Unsurprisingly, her first book collecting some of these stories ended up being a New York Times best-seller.

Perhaps more well-known are her two posts on depression (part 1, part 2) where she manages to put in words the feelings of thousands of people. I have seen an actual therapist recommend these posts to people, and the almost 10K collective comments in those entries alone seem to agree.

And the reason I am bringing up these two sides of her blog is because I recently read her second book, and let me tell you, it is a roller coaster: it is funny, it is sad, and sometimes it's both at the same time. It is the best thing I read all year, and I think everyone should do the same. To say that I recommend it would be an understatement. It would be more accurate for you to imagine me grabbing you by your clothes while yelling "READ THIS BOOK".

Disclaim all of the things

I didn't want to leave this post as it is without complaining about how difficult it is to make an honest recommendation on the internet.

I have a subscription to Skillshare because I like the quality of their courses, but I am really, really annoyed at their marketing showing up everywhere. With so many youtubers doing paid promotions for courses they don't care about, I feel slightly dirty making a recommendation just like them, even if no one is paying me for doing it. I thought for a second about pointing you to a free mirror, but that would be unfair to the course's creator.

Similarly, someone on Allie Brosh's publishing team had the brilliant idea of creating fake Reddit accounts and using them to market the book. People like them make it impossible for me to recommend almost anything in good conscience. I have decided to make an exception for this specific book, but I don't see that happening again anytime soon.

I miss funny Dilbert

2020-11-07T00:00:00+01:00

There was once an article about Jim Davis, creator of Garfield, in which he recognizes the recipe of his success. The trick, it seems, was to make Garfield as inoffensive as possible. No matter what you believe, no matter how delicate your sensitivities are, you can always read Garfield without feeling hurt or offended. Comedians might object that a lot of humor boils down to ridiculing something, so it's worth asking: if Garfield does not offend anyone, how does it manage to keep being funny? The answer should be obvious to Garfield's readers: it doesn't. Because Garfield is not funny.

The reasoning is pretty interesting: Jim Davis' goal was not to be the next greatest American cartoonist, nor to push the boundaries of comic strips as an art form (that would be Bill Watterson). His goal was to make money, and boy did he succeed at that. By being a recognizable, bland, perfectly formulaic icon, Garfield can be adopted by any company or product willing to pay for it. The key, said Davis in this interview, was to make the strip as plain and predictable as possible. "Oh, look,", says the reader, "Garfield is mad because it's Monday". Cue the sound of crickets.

The same, I'm afraid, has happened to Dilbert some time ago. And while it pained me to stop reading after so many years, I've read enough to understand that the Dilbert I liked is gone, replaced by that which he was intended to criticize. Including the archives, I read about 27 years worth of strips, so it was not a decision I took lightly. That was about 4 years ago, and I haven't regretted the decision.

For those who might feel like me, and as a service to the community, I give you the one and only strip you will ever need from now on. It is the culmination of years of Dilbert, and nothing you read in the actual strip will be better than this in the foreseeable future.

Now, in all fairness, congratulations to Scott Adams: he has managed to secure Dilbert in the mind of the public, and he made a lot of money out of it. It was sad to see the old Dilbert go away, but then again, I don't have an animated series nor an (forever in production) upcoming movie to my credit. Having said that, I can only wonder how much more he could have produced if he hadn't rested on his laurels: his Wikipedia achievements have almost entirely peaked around 2010, and he seems to spend most of his time nowadays writing about what an amazing president Donald Trump is. While this is speculation on my part, I believe this might be why his blog is no longer featured on the Dilbert homepage.

I can see why he doesn't need to come with new ideas for Dilbert strips. After all, he has enough money to do whatever he wants. I just wish "make Dilbert funny again" was one of those things he cared about.

Recovering Mercurial code from Bitbucket

2020-10-14T00:00:00+02:00

I received today the type of e-mail that we all know one day will arrive: an e-mail where someone is trying to locate a file that doesn't exist anymore.

The problem is very simple: friends of mine are trying to download code from https://bit.ly/2jIu1Vm to replicate results from an earlier paper, but the link redirects to https://bitbucket.org/villalbamartin/refexp-toolset/src/default/. You may recognize that URL: it belongs to Bitbucket, the company that infamously dropped their support for Mercurial a couple months ago despite being one of the largest Mercurial repositories on the internet.

This is the story of how I searched for that code, and even managed to recover some of it.

Offline backup

Unlike typical stories, several backup copies of this code existed. Like most stories, however, they all suffered terrible fates:

There was a migration of most of our code to Github, but this specific repo was missed because it belongs to our University group (everyone in that group had access to it) but it was not created under the group account.
Three physical copies of this code existed. One lived in a hard drive that died, one lived in a hard drive that may be lost, and the third one lives in my hard drive... but it may be missing a couple commits, because I was not part of that project at that time.

At this point my copy is the better one, and it doesn't seem to be that outdated. But could we do better?

Online repositories

My next step was figuring out whether a copy of this repo still exists on the internet - it is well known that everything online is being mirrored all the time, and it was only a question of figuring out who was more likely to have a copy.

My first stop was Archive Team, from the people behind the Internet Archive. This team famously downloaded 245K public repos from Bitbucket, and therefore they were my first choice when checking whether someone still had a copy of our code.

The experience yielded mixed results: accessing the repository with my browser is impossible because the page throws a large number of errors related to Content Security Policy, missing resources, and deprecated attributes. I imagine no one has looked at it in quite some time, as it is to be expected when dealing with historical content. On the command line, however, it mostly works: I can download the content of my repo with a single command:

hg clone --stream https://web.archive.org/web/2id_/https://bitbucket.org/villalbamartin/refexp-toolset

I say "mostly works" because my repo has a problem: it uses sub-repositories, which apparently Archive Team failed to archive. I can download the root directory of my code, but important subdirectories are missing.

My second stop was the Software Heritage archive, an initiative focused on collecting, preserving, and sharing software code in a universal software storage archive. They partnered up with the Mercurial hosting platform Octobus and produced a second mirror of Bitbucket projects, most of which can be nicely accessed via their properly-working web interface. For reasons I don't entirely get this web interface does not show my repo, but luckily for us the website also provides a second, more comprehensive list of archived repositories where I did find a copy.

As expected, this copy suffers from the same sub-repo problem as the other one. But if you are looking for any of the remaining 99% of code that doesn't use subrepos, you can probably stop reading here.

Deeper into the rabbit hole

At this point, we need to bring out the big guns. Seeing as the SH/Octobus repo is already providing me with the raw files they have, I don't think I can get more out of them than what I currently do. The Internet Archive, on the other hand, could still have something of use: if they crawled the entire interface with a web crawler, I may be able to recover my code from there.

The surprisingly-long process goes like this: first, you go to the Wayback Machine, give them the repository address, and find the date when the repository was crawled (you can see it in their calendar view). Then go to the Kicking the bucket project page, and search for a date that kind of matches that. In my case the repository was crawled on July 6, but the raw files I was looking for where stored in a file titled 20200707003620_2361f623. In order to identify this file I simply went through all files created on or after July 6, downloaded their index (in my case, the one named ITEM CDX INDEX) and used zgrep to check whether the string refexp-toolset (the key part of the repo's name) was contained in any of them. Once I identified the proper file, downloading the raw 28.9 Gb WEB ARCHIVE ZST file took about a day.

Once you downloaded this file, you need to decompress it. This file is compressed with ZST, meaning that you probably need to install the zstd tool or similar (this one worked in Devuan, so it's probably available in Ubuntu and Debian too). But we are not done! See, the ZST standard allows you to use an external dictionary without which you cannot open the WARC file (you get an Decoding error (36) : Dictionary mismatch error). The list of all dictionaries is available at the bottom of this list. How to identify the correct one? In my case, the file I want to decrypt is called bitbucket_20200707003620_2361f623.1592864269.megawarc.warc.zst, so the correct dictionary is the one called archiveteam_bitbucket_dictionary_1592864269.zstdict.zst. This file has a .zst extension, so don't forget to extract it too!

Once you have extracted the dictionaries, found the correct one, and extracted the contents of your warc.zst file (unzstd -D <dictionary> <file>) it is now time to access the file. The Webrecorder Player didn't work too well because the dump is too big, but the warctools package was helpful enough to realize... that the files I need are not in this dump either.

So that was a waste of time. On the plus side, if you ever need to extract files from the Internet Archive, you now know how.

Final thoughts

So far I seem to have exhausted all possibilities. I imagine that someone somewhere has a copy of Bitbucket's raw data, but I haven't managed to track it down yet. I have opened an issue regarding sub-repo cloning, but I don't expect it to be picked up anytime soon.

The main lesson to take away from here is: backups! I'm not saying you need 24/7 NAS mirroring, but you need something. If we had four copies and three of them failed, that should tell you all you need to know about the fragility of your data.

Second, my hat goes off both to the Internet Archive team and to the collaboration between the Software Heritage archive and Octobus. I personally like the later more because their interface is a lot nicer (and functional) than the Internet Archive, but I also appreciate the possibility of downloading everything and sorting it myself.

And finally, I want to suggest that you avoid Atlassian if you can. Atlassian has the type of business mentality that would warm Oracle's heart if they had one. Yes, I know they bought Trello and it's hard to find a better Kanban board, but remember that Atlassian is the company that, in no particular order,

regularly inflicts Jira on developers worldwide,
bought Bitbucket and then gutted it, and
sold everyone on the promise of local hosting and then discontinued it last week for everyone but their wealthiest clients, forcing everyone else to move to the cloud. Did you know that Australia has legally-mandated encryption backdoors? And do you want to take a guess on where Atlassian's headquarters are? Just saying.

Netflix and sound whitewashing

2020-08-14T00:00:00+02:00

Note: I wrote this article in August, but I didn't realize it wasn't published until October. I kept the published date as it was, but if you didn't see it before well, that's why.

Are you familiar with a small streaming company called "Netflix"? If so, you might recognize their opening sound. And even if you don't, you might have seen one of their multiple recent press campaigns regarding this topic. From a recent episode of the Twenty Thousand Hertz podcast on all the sound choices that go into their logo to their announcement that Hans Zimmer has worked on making it longer for cinema productions.

What none of those articles are saying is that this sound is also the sound of Kevin Spacey hitting a desk at the end of Season 2 of House of Cards. Yes, that House of Cards, the critically-acclaimed series that made Netflix' stock jump a 70 percent even before it started and put Netflix on the map. If I were a Netflix executive back then, I would be proud of having the series as part of my corporate identity.

If I were an executive today, however, I would be terrified of people forever remembering that my company's official sound, the one that plays before every show, was first heard in a scene with an actor that has been very publicly accused of sexual assault in 2017. So I can understand why someone would feel that a change is needed, and I'm all for it. No one is blaming Netflix (as far as I know) for not running background checks on their actors.

Having said that, it seems that Netflix has gone all the way to completely erase that any of this ever happened, in what has to be the most pointless history rewrites in some time. In the above-mentioned podcast, a sound engineer talks about all the sounds that came together to compose the current Netflix sound, from a ring on a cabinet to the sound of an anvil, with no mention whatsoever of Kevin Spacey hitting any desks.

Suffice to say, I was confused by this omission, so I dug a bit more and found a Facebook post from August 2019 from the Twenty Thousand Hertz podcast official account, where they posted:

"I'm convinced the @netflix sonic logo was originally built from Frank Underwood banging on the desk at the end of House of Cards Season 2. BUT, I'm dying to know who enhanced it! I can't find anything online! (...)".

I can only conclude that the "it's a ring on a cabinet" story is technically true and a sound engineer has actually used it to enhance Kevin Spacey's desk banging sound, but they conveniently "forgot" to mention the relation between these two facts. One of the answers to this Quora question mentions that "The tapping on the table with his (Kevin Spacey) ring is associated with completing a mission or one of his plans being accomplished", which sheds even more light into why they were banging rings on furniture to begin with. And let's pray that the hand wearing the ring wasn't Kevin Spacey's...

None of this is mentioned in the podcast. As for the longer version composed by Hans Zimmer, it does not include the original soundbite at all. I believe that Netflix is going on a PR campaign to rewrite their history, has convinced the Twenty Thousand Hertz podcast people to just go with it, and have so far been very successful.

And yet, I have to ask... why? Was it so difficult to come out and say "we don't want to be associated with this sound anymore, and therefore we are releasing a new one"? I honestly don't care about Netflix nor House of Cards (which I have not seen), but I am kind of annoyed at such a transparent attempt to hide their history behind a PR campaign. Or even worse, that they seem to have gotten away with it.

A hot-dog is a sandwich, but you can't call it that

2020-08-09T00:00:00+02:00

I remember seeing two main camps in the old debate on whether a hot-dog is a sandwich: those that argued "it's meat between two pieces of bread, therefore it's a sandwich" and those that counterargued "if I asked whether you wanted a sandwich and then gave you a hot-dog, you would be surprised". And both sides are right! That said, one of them is more right from a language-theoretical point of view, and that's the point of today's post.

(Spoiler: the "not a sandwich" camp is right. Sorry, pro-sandwichers!)

According to Herbert Clark a dialogue is a cooperative activity. It is a joint activity in which both speakers try to accomplish a common objective, ranging from something as formal as "make me understand where the train station is" to as vague an objective as "let's kill some time". And because it is cooperative, we do not expect the other person to be deliberately obtuse. If I ask "Can I get you something to drink?" and someone replies "I don't know, can you?", nobody would assume that this person has a genuine interest in my capacity for carrying drinks. Instead, we would immediately see this for what it is: that this person is not cooperating, that any reasonable person would have understood what I meant, and that this person stopped cooperating on purpose. Whether he did it to make a joke or because he's a jerk, that's a topic for a future discussion.

There is also a principle called the "maxim of quantity" (one of Grice's four maxims) according to which a person will always give as much information as possible, but not so much that it breaks the dialogue. If someone asks me where I come from, my answer can be as precise as a specific neighborhood or as vague as "somewhere near the border with Brazil". My answer will depend on how familiar I believe the other person to be with South-American geography, because I don't want to give them excessive information that they cannot handle. Again, I'm cooperating.

Which brings us to the hot-dog debate. From a taxonomic point of view, a hot-dog is a sandwich: it is composed of two pieces of processed meat between two pieces of bread, which is as clear as it gets. But this is only half the story.

The definition of sandwich came long after the sandwich itself. It is an artificial construct designed to model and understand a set of real-life language usages. Spoken language, on the other hand, is the real deal. Language rules and word definitions model how we speak, and not the other way around. All definitions are artificial and, therefore, may not always reflect the way we actually use those words.

Bringing this all together, I would only use the word "sandwich" to describe a hot-dog if I had a reason to believe the other person doesn't know what a hot-dog is. If you know what a hot-dog is and I know you know what a hot-dog is, using the word "sandwich" to speak about a hot-dog is neither maximally informative (I am giving you less information than I could) nor cooperative (I know which word would help us the most, but I'm not using it). "Sandwich" is a catch-all word that can only be used when no better word exists.

Even worse: you know we both know what a hot-dog is. By choosing "sandwich", I am actively leading you to believe that I want to offer something that can be described as a "sandwich" but not as a "hot-dog". Fans of malicious compliance will argue that this is not technically untrue, but you and I both know that there's little practical difference between "I told you a lie" and "I told you something that any sane person in the world would understand in a certain way while secretly using a different, opposite interpretation that I kept to myself".

So there you have it. A hot-dog is a sandwich if you stick to rigid categories created by researchers with a tenuous grasp of the real world at best (you know, people like me), but you are only allowed to use it if you talk to people who never heard about hot-dogs before. Using the word "sandwich" for a hot-dog in any other context is uncooperative, mildly dishonest, and kind of a jerk move. People do not use the word "sandwich" like that and, since spoken language is where "true" language usage lies, they are the ones who are using it right.

I'm running out of forums

2020-07-26T00:00:00+02:00

As so many ~~immigrants~~ expats around the world, I like to follow the news regarding what's happening back in my home country. But getting complete, reliable information has been getting more and more difficult every year, and 2020 is the year in which I finally ran out of news sources.

Unlike my parent's generation, I don't consider newspapers a reasonable source of reliable information. The problem is that, following the example set by Fox News, the largest newspapers at home have substituted fact for opinions, extremely biased articles, and outrage has replaced objectivity as the main selling point. All of this seasoned with local celebrity gossip, of course.

If despite my best judgment I decide to check what's going on based on newspapers, I currently start with the biggest newspaper (which is very right-leaning) and then compensate with their main opposition (which is, as expected, very left-leaning). I then figure out which news are common to both, and decide which version is more likely to be true - one newspaper's fair trial is the other newspaper's witch hunt, and one newspaper's smart move is the other newspaper's national betrayal. Finally, I check which news have been mentioned in only one of them, and decide on a reasonable narrative for why only one of them is talking about it. Suffice to say, doing this in the morning on my phone takes a lot of effort.

In my case, this was one problem (probably the only one) solved by the news aggregator Reddit. The sub-reddit for my country used to be relatively good at highlighting the main issues in the public conscience, and reading the main posts used to be enough to get a good, updated picture. But not anymore: extremely lax moderation and general apathy have turned the forum into a meme-laden wasteland where all information has been replaced by bad jokes, political outrage, and the occasional "kill all poors" post. So I quit it for good, and haven't looked back.

Which leads me to today. I have not yet found any source of news that I can trust to inform me. Sure, I can easily get distracted, entertained, outraged, tricked, and lied to, but informed? Good luck with that. I have remained uninformed for a couple months now, and I don't see that changing anytime soon.

Being a programmer who has spent a bit of time toying around with NLP, I have been working on a solution - a small news aggregator where I can get a proper sense of what's going on. But it hasn't been easy: news agencies no longer offer RSS feeds, important data is locked behind proprietary formats made intentionally hard to read, and there is barely any corpora around in Spanish that I could use to train models.

I don't have a deeper point. I can't promise that my solution will work, because it's entirely possible that it never makes it out of vaporware. I don't have a website to recommend, because websites are getting bad faster than good ones are propping up. And I'm not going to talk about the cesspool that are unmoderated forums because we all know about that already.

I just wanted to say: I am very unhappy with this situation.

How to draw

2020-07-19T00:00:00+02:00

I am okay at drawing. That means: I am probably better at drawing than a random person walking down the street but I'm far, far behind the type of artists that regularly post in Instagram. I am also mostly self-taught: I took some initial lessons via mail from the well-known (at the time) Modern Schools, gave up for a couple years, and picked it back up in my late teens when I needed something to do besides programming and not having friends. Some of my drawings have been published, and one in particular has been stolen countless time by people who thinks copying things from the internet without attribution is fine.

I was recently asked what I would recommend to someone who wants to learn how to draw. This question took me by surprise for two reasons: one, because I was never asked this before, and two, because my answer was surprisingly useless even to me:

Any tutorial you find online will give you the right steps. But you'll only understand them after you already know how to draw.

This is a pointless answer, which also happens to be 100% correct. This post is my attempt at giving a slightly clearer answer, explaining why anyone would think that my advice makes sense and hopefully give beginners some good points on where to start.

Note 1: this post contains links to drawings of naked people. If you are not comfortable with drawn nudity, you should probably not follow the links and definitely reconsider whether figure drawing is good for you.

The boring advice

All drawing is, at its core, more or less the same. Whether you are interested into realistic drawing, comic drawing, manga drawing (a term I hate), webcomics or editorial cartoons, the art of representing human figures in 2D is based on 90% the same rules. Sure, US comics have more muscles and japanese manga characters have no nose, but the fundamentals are the same. A typical drawing curriculum should include:

How to sketch a human figure. This guide is relatively good, while this one sucks for reasons I'll explain later on. If you've seen those wooden figures, they are useful for getting the hang of this step.
The proportions of the human body. More specifically, this guide on how many heads you need to draw a full body.
The proportions of the face. This is annoying enough that it often warants a section on its own.
How perspective works. One, two, and three points perspective are the typical ones.
How shadows work. Getting it perfect will take a long time, but "dark part is dark" will get you far with little effort.

Once you reach this point, you can either start learning about muscles and improve your anatomy (have you ever stopped to think about how weird knees look?), or become a caricaturist and call it a day.

The number of books and tutorials out there convering all these points is virtually infinite, and therefore any book you choose it's going to be probably fine. If you want some more specific advice, multiple generations of artists have learned with Andrew Loomis' books, which are freely available on the Internet Archive. You should start with Figure drawing for all it's worth, follow up with Drawing the head and hands, and fresh up your perspective with the first half of Successful drawing.

Practical advice I: Keep drawing

There are two extra pieces of advice worth discussing.

The standard advice says "keep drawing until you become good at it", which is technically true but only barely. The full, honest version should say:

Start with one drawing. It will suck, and that's fine. Once you're finished, look at it objectively and enumerate its defects. For your next drawing, focus on solving those defects. Repeat until you consider yourself good enough¹.

In other terms: you can draw circles all day and all night for years, but that won't make you any better at drawing squares. If you want to get better at drawing, you first need to be aware of what's there to improve.

That doesn't mean that you can't be happy about something you just drew. Few things are as rewarding as putting your art supplies to the side, looking at your drawing, and admiring something knowing that you made it. All I'm saying is: you need to know what your blind spots are. If you are like me and your eyes are always sliiiiightly out of alignment, it is perfectly fine to still be happy about that portrait you just made. But if you are not honest and accept that yes, that one eye looks weird, then you will never learn how to fix it².

Practical advice II: Copy other people

As the quote goes, "Good artists copy, great artists steal". Therefore, it is your duty as aspiring artist to copy as much as you can. Most self-taught artists I know started the same way, copying drawings over and over until they felt comfortable enough to start doing their own.

My suggestion: find an artist you like. Pick one of their drawings and copy it. Add the final work to your sketch folder. Repeat. This exercise serves several purposes:

It will improve your pencil grip, make your lines stronger, and improve your technique overall.
It allows you to focus on a sub-part of the problem (drawing a figure) without having to worry about the complicated stuff - you don't need to think about perspective, shadows or posture because the artist already did it for you.
It helps you to build your personal portfolio. It will help you visualize your progress, and gives you something to brag about whenever someone learns you are drawing and asks you to see something you've done. Plus, it's not like you wanted to throw those drawings away, right?
It will help you answer questions you didn't know you had. Do you want to know how to draw a feminine-looking nose? Copy one of Phil Noto's illustrations. Would you like to know how does a professional go from zero to done? You can watch professionals like Jim Lee do a couple pieces in real time online and even explain their process as they go. Are you wondering how much attention to pay to clothes and background? Once you notice that classical painters couldn't care less about whatever is below your shoulders, maybe you won't lose your sleep about it either.

Eventually, you'll start noticing that different artists have different skills to offer. Maybe that guy draws cool hands, that other artist draws clothes very well, and that third other one has very expressive faces. Copying their work helps you understand the tricks they are using, and adding them to your repertoire helps you develop your own style.

Rest of the owl

The final point is both super important and really difficult to explain to beginners.

Are you familiar with the how to draw an owl meme? This picture is very popular in amateur art circles because it goes straight to the core issue: that most tutorials will take your hand and guide you step-by-step, but then they will let go at a critical step and you'll fall down a metaphorical cliff.

The root of the problem, I think, is that one step where the book tells you to "do what feels natural" or to "just keep going". What these people forget, however, is that learning what feels natural takes a lot of practice!

This tutorial I mentioned above is as bad as it gets: the instructions tell you to "Draw some vertical and horizontal lines to plan your drawing", which is completely useless advice that only makes sense once you know which lines to draw and where. Whoever wrote that guide has forgotten what it was to be a beginner, and their advice is really not helping.

When that happens, you have two choices. You can look for a better tutorial, or you can keep going, and see how far you make it. There is no shame in trying and failing, and who knows? maybe you'll still make it. Truth be told, there is a point at which no tutorial can help you and all that's left for you to do is to just draw. But that only applies for specific, advanced tutorials. It is the sad truth that, as a beginner, you will often recognize bad tutorials only once you are stuck in them.

Nobody said the life of an artist was easy.

Closing remarks

This guide ended up being longer than I intended, and half as long as it should be. That's always going to be a problem: the average artist does not let structure get in the way of their vision, and any attempt at a "formal" answer will stop halfway (as I have complained before). That said, if you would still appreciate a more structured approach, I have heard good things about Betty Edward's book Drawing on the right side of the brain.

And finally: have fun. All of this advice is useful for when you want to get objectively better, but there's a lot to be said in favor of simply drawing because you enjoy it.

Happy drawing!

Footnotes

Fair warning: in my experience, most artists never feel that they are "good enough". This is a well-known bug of art.
I believe the process of "find defect, correct defect, repeat" is why most artists I know are never happy about their work. Seriously, go to an artist and tell them you like a particular drawing of them - there's a good chance that they'll give some excuse for why the drawing sucks.

Why programming is hard

2020-07-12T00:00:00+02:00

I have been giving programming languages a lot of thought recently. And it has ocurred to me that the reason why (reportedly) lots of people fail at learning how to program is because they are introduced to it at entirely the wrong level.

If you as a beginner search "Python tutorial" right now, you will get lots of very detailed, completely correct, very polished tutorials that will teach you how to program in Python, but that will not teach you how to program. Conversely, if you search for "how to program", the first results will be either completely useless advice such as "decide what you would like to do with your programming knowledge" or they ask you to choose a programming language. You might choose Python, in which case you are now back to square one.

One of the founding principles of my field is that "Computer Science is no more about computers than astronomy is about telescopes". In other words, programming is a skill that it's expressed typicall using programming languages, but it's not exclusively about them. And programming, the skill underlying all of these programming languages, is hard.

To be a good programmer, you need to master three related skills:

Understand how to convert messy real-life problems into a clearer version with less ambiguities.
Understand what the best practical approach to this problem is, and choose one as a possible solution.
Understand how to use programming languages and data structures to implement that solution.

Mastering the first skill requires an analytical mind, and in particular forces you to see the world in a different way. If someone asks you for a program to keep track of how many people are inside a room, you need to stop thinking in terms of people and rooms and think in terms of numbers and averages. You also need to account for badly-defined situations: if a woman gives birth inside the room, is your solution good enough to increase the room count by one?

This part alone is quite hard. Some people make a living out of it as software requirements engineers, meeting with clients and discussing some approaches that would make sense. It also requires at least a surface level understanding of the type of solutions that one could realistically employ. If you ever wondered why your high-school math teacher always asked you to turn apples and trains into equations and solving for x, well, this is why: they were teaching you how to solve real-world problems with simpler methods.

In order to master the second skill "choose a viable solution", you need to read a lot about which problems are easy and which ones are hard. There are some problems that a programmer solves daily, and some problems for which the best known solution would still take thousands of years. If you think that finding new solutions to problems is interesting, I encourage you to go knock at the Math department of your nearest university. They do this for a living, and will be very excited to have you around.

Finally, the third and last skill "implement a solution" requires you to write it down in a way that computers can understand. Half the job requires understanding common concepts for structuring programs (variables, databases, data structures, networks, and so on), and the other half requires learning the syntax of your preferred programming language.

And here we reach the core of this post's answer. If you type "Python tutorial" right now, what you'll get are very detailed guides on how to acquire the second half of the third skill, also known as "the unimportant one". Sure, programmers love discussing which programming language is better and how not to write code, but here's a little secret: in the larger scale of things, it rarely matters. Some programming languages are better suited for specific tasks, true, but the best programming language is not going to be of any help if you don't know what you are trying to build.

At its core, programming is learning how to solve problems with a specific set of tools. And while you do need to understand how to use those tools, they are completely useless if no one explains to you how to solve problems with them. If knowing how to use a pen doesn't make you a writer, and knowing how to use a wrench doesn't make you a mechanic, teaching you a programming language and expecting you to become a programmer overnight is just going to leave you confused and frustrated. But remember: it's not you, it's them.

Appendix I: what does problem solving looks like?

Let's say someone asks me to "write a program to know who I have been in contact with in any given day", a problem known as contact tracing that has been in the news in the past weeks. How would the skills above come into play?

(Note: for the sake of simplicity, I am going to solve this problem badly. It's a toy example, so don't @ me!)

The first step is to model this situation in a formal, more structured way. Real people are difficult to work with, but luckily we don't care about most things that make them human - all we care about is where they have been at any point in time. Therefore, we replace those real people with "points", keep track of their GPS coordinates at all times, and throw all of their remaining attributes away.

We have now turned our problem into "tell me which GPS coordinates (i.e., points) have been close to my GPS coordinates at any given time". We can simplify the problem further by defining what "close to me" means, and we turn the problem into "give me a list of points that have been 1 meter or closer to me at any given time".

Next, we need to find a way to efficiently identify which points have been close enough to our coordinates. Since there is a lot of people in the world, we start by crudely removing all points that are more than 10km. away from me - this can be done very quickly, and it probably won't affect our results too badly.

We now need to refine our search, and therefore we take a dive into the geometry literature. After a quick look, I decided that building a Quadtree is the best solution for what I want to build. Note that I only have a passing knowledge of what Quadtrees are, but that's fine: once I have a hint of where the solution might be, I can search further and learn the details as I go.

And finally we get down to writing code. If our programming language doesn't already include an implementation of a Quadtree data structure, we might have to do it ourselves. If we choose Python, for instance, we need to understand how to create a class, how to use lists of objects, and all those other implementation details that our Python tutorial has taught us. Similarly, storing the list of points will probably require a database. Each database has a different strong point but, as I said earlier, knowing which database to use is not as important as knowing that some database is the right tool.

Let's now picture the same exercise in revers: imagine I come to you and say "Write me a program to know who I have been in contact with in any given day. Here's a guide on how to use Quadtrees in Python". You wouldn't find that last bit of any use, would you?

Moon landing in Basic

2020-06-23T00:00:00+02:00

As many other people around my age, I learned programming from a book. In particular, I started programming with a 3-books series called Computación para niños (Computers for kids). Volume 3 was dedicated to programming in BASIC, and it opened the door to what is now both my profession and hobby.

That said, that book was also the source of a 25-ish-years-long frustration, and that story is the point of today's article.

In the ancient times known as "the 90s", it was still common to get printed code for games that you had to to type yourself. This book, in particular, included a game called "Lunar rocket" in which you were supposed to (surprise!) land a rocket on the moon. For context, this is how the game was sold to me:

And this is what the code looks like:

Suffice to say, the program never worked and I couldn't understand why. I spent weeks trying to tweak it to no avail, getting different errors but never making a full run. And no matter how hard I tried, I could never get a single picture to show on screen. Eventually I gave up, but the weeks I spent trying to understand what I did wrong have been on my mind ever since.

That is, until last Sunday, when I realized that I should go back to this program and establish, once and for all, whose fault all of this was.

Problem number one was that the book shows pretty pictures, but the program is text-only. That one is on me, but only partially. Sure, it was too optimistic of me to expect any kind of graphics from such a short code. But I distinctly remember giving it the benefit of the doubt, and thinking "the pictures are probably included with BASIC, and one or two of these lines are loading them". That was dead wrong, but I'll argue that young me at least had half of a good idea. A few years later I would learn that the artwork and the game rarely had anything to do with each other, a problem that has not entirely gone away.

Now, problem number two... That code I showed above would never, ever work. My current best guess is that someone wrote it in a rush leaving some bugs in, and someone else typed it and introduced a lot more. In no particular order,

Syntax errors: Line 130 has a typo, the variable name "NÚMERO" is invalid because of the accent, and line 150 is plain wrong. The code also uses ";" to write multiple instructions in a single line, but as far as I know that's not valid BASIC syntax.
The typist sometimes confuses "0" with "O" and ":" with ";" and " ". This introduced bugs on its own. Line 150 (again) shows all mistakes at once.
Error handling is a mess: if you enter the wrong input, you are simply asked to enter it again. No notification of any kind that you did something wrong.
The logic itself is very convoluted. GOTOs everywhere. Line 440 is particularly bad, and could be easily improved.
Some of the syntax may be valid, but it was definitely not valid in my version of Basic. And seeing how my interpreter came included with the book, I feel justified in not taking the blame for that one.

And so I set out to get this to run once and for all. The following listing shows what a short rewrite looks like:

LET DISTANCE=365000
LET BARRELS=100
LET MASS=1000
LET TIME=60
LET A0=0
LET VO0=0
LET V0=0
LET EO0=0
LET E=0
LET F=0
LET THROWS=0
DO WHILE BARRELS>0 AND E<=364900 AND THROWS <= 30 
    LET REMAINING=DISTANCE-E
    PRINT "DISTANCE SO FAR="; E
    PRINT "DISTANCE TO GO="; REMAINING
    PRINT "SPEED"; V
    PRINT "BARRELS LEFT="; BARRELS
    PRINT "THROWS LEFT(MAX 30)="; THROWS
    PRINT "-----"
    INPUT "NUMBER OF BARRELS?"; NUMBER
    INPUT "DO YOU WANT TO BRAKE (Y/N)"; RESP$
    IF NUMBER>BARRELS THEN
        PRINT "NOT ENOUGH BARRELS!"
    ELSEIF RESP$ <> "Y" AND RESP$ <> "N" THEN
        PRINT "INVALID INPUT!"
    ELSE        
        LET THROWS=THROWS+1
        LET BARRELS=BARRELS-NUMBER
        IF RESP$="N" THEN
            LET F=(F+NUMBER*1000)*0.5
        ELSE
            LET F=(F-NUMBER*1000)*0.5
        END IF
        LET A=F/MASS
        LET V=VO+A*TIME
        LET E=EO+VO*TIME+0.5*A*TIME*TIME
        LET EO=E
        LET VO=V
    END IF
LOOP
IF BARRELS <= 0 THEN PRINT "MISSION FAILED - NOT ENOUGH FUEL"
IF THROWS > 30 THEN PRINT "MISSION FAILED - NOT ENOUGH OXYGEN"
IF E>364900 THEN
    IF V<5 THEN
        PRINT "MISSION ACCOMPLISHED. ROCKET LANDED ON THE MOON."
    ELSE
        PRINT "MISSION FAILED. ROCKET CRASHED AGAINST THE MOON SURFACE."
    END IF
END IF

I think this version is much better for beginners. The code now runs in a loop with three clearly-defined stages (showing information, input validation, and game status update), making it easier to reason about it. And now that the GOTOs are gone, so are the line numbers. However, and in order to keep that old-time charm, I kept all strings in uppercase and added no comments whatsoever.

I also added some input validation: the BASIC interpreter I'm using (Bywater Basic) will still crash if you enter a letter when a number is expected, but that's outside what I can fix. At least you now get a message when you use too many barrels and/or you choose other than "Y" or "N".

It is only fair to point out something that I do like about the original code: that the variable names are descriptive, and in particular that the physics equations use the proper terms. If you are familiar with the physics involved here, those equations will jump at you immediately.

If I had time, I would still tie a couple loose ends in my version. A proper rewrite would ensure that the new code behaves exactly like the old one, bugs and all. And there's a good chance that I have introduced some new bugs too, given that I barely tested it. I also feel like making a graphical version, using the original artwork and adding some simple animations on top.

But even then, I finally feel vindicated knowing that younger me had no chance of making this work. Even better: the next exercise, a car race game, just gave you a couple pointers on how to draw something on the screen, and then left you on your own. That one would take me some time today.

Next on my list: finally read the source code of Gorilla.bas. I know I tried really hard to understand it when I was 10, so maybe I should get closure for that one too.

Move fast and break things. In particular, cars.

2020-06-05T00:00:00+02:00

Once again, and as seen on this video, a Tesla car driving alone on its lane on a clear day runs straight into a 100% visible, giant overturned truck. I say "once again" because Tesla had already made the news in 2018 when one of its self-driving cars ran into a stopped firetruck.

This is not an unknown bug - this is by design. As Wired reported back then, the Tesla manual itself reads:

Traffic-Aware Cruise Control cannot detect all objects and may not brake/decelerate for stationary vehicles, especially in situations when you are driving over 50 mph (80 km/h) and a vehicle you are following moves out of your driving path and a stationary vehicle or object is in front of you instead.

The theory, as it goes, is that a giant stationary truck stopped in the middle of a highway is so an unlikely event that the system considers it a misclassification and ignores it. If your car is a 2018 Volvo, it may even accelerate.

There is a popular argument that often surfaces when people try to point out how completely ridiculous this situation is: that the driver should always be alert, and that they should be prepared to take control of the vehicle at any time. And if your car is equipped with "lane assist", then that's fine: the name of the feature itself is telling you that the technology is only there to ensure that you stay in your lane, and anything else that might happen is your responsibility.

But when your promotional materials have big, bold letters with the words "Autopilot" and your promotional video shows a man prominently resting his hands on his legs, you cannot hide behind a single sentence saying "Current Autopilot features require active driver supervision and do not make the vehicle autonomous". Why? First, because we both know this is a lie - if you weren't intending on deceiving people into believing their car is autonomous, you wouldn't have called the feature "Autopilot" and you wouldn't have made such a video. Second, and more important, virtually the entire literature on attention will tell you that the driver will not be in the right state of mind to make a split-second decision out of the blue. No one believes that someone will activate their Autopilot™ and remain perfectly still and attentive their entire time, because human attention simply doesn't work like that.

Hopefully, some Tesla engineer will come up with a feature called "do not run into the clearly-visible obstacle at full speed" in the near future. Until then, all of you people drinking and/or sleeping in your moving cars should seriously consider not doing that anymore.

Write-only code

2020-05-13T00:00:00+02:00

The compiler as we know it is generally attributed to Grace Hopper, who also popularized the notion of machine-independent programming languages and served as technical consultant in 1959 in the project that would become the COBOL programming language. The second part is not important for today's post, but not enough people know how awesome Grace Hopper was and that's unfair.

It's been at least 60 years since we moved from assembly-only code into what we now call "good software engineering practices". Sure, punching assembly code into perforated cards was a lot of fun, and you could always add comments with a pen, right there on the cardboard like well-educated cavemen and cavewomen (cavepeople?). Or, and hear me out, we could use a well-designed programming language instead with fancy features like comments, functions, modules, and even a type system if you're feeling fancy.

None of these things will make our code run faster. But I'm going to let you into a tiny secret: the time programmers spend actually coding pales in comparison to the time programmers spend thinking about what their code should do. And that time is dwarfed by the time programmers spend cursing other people who couldn't add a comment to save their life, using variables named var and cramming lines of code as tightly as possible because they think it's good for the environment.

The type of code that keeps other people from strangling you is what we call "good code". And we can't talk about "good code" without it's antithesis: "write-only" code. The term is used to describe languages whose syntax is, according to Wikipedia, "sufficiently dense and bizarre that any routine of significant size is too difficult to understand by other programmers and cannot be safely edited". Perl was heralded for a long time as the most popular "write-only" language, and it's hard to argue against it:

open my $fh, '<', $filename or die "error opening $filename: $!";
my $data = do { local $/; <$fh> };

This is not by far the worse when it comes to Perl, but it highlights the type of code you get when readability is put aside in favor of shorter, tighter code.

Some languages are more propense to this problem than others. The International Obfuscated C Code Contest is a prime example of the type of code that can be written when you really, really want to write something badly. And yet, I am willing to give C a pass (and even to Perl, sometimes) for a couple reasons:

C was always supposed to be a thin layer on top of assembly, and was designed to run in computers with limited capabilities. It is a language for people who really, really need to save a couple CPU cycles, readability be damned.
We do have good practices for writing C code. It is possible to write okay code in C, and it will run reasonably fast.
All modern C compilers have to remain backwards compatible. While some edge cases tend to go away with newer releases, C wouldn't be C without its wildest, foot-meet-gun features, and old code still needs to work.

Modern programming languages, on the other hand, don't get such an easy pass: if they are allowed to have as many abstraction layers and RAM as they want, have no backwards compatibility to worry about, and are free to follow 60+ years of research in good practices, then it's unforgivable to introduce the type of features that lead to write-only code.

Which takes us to our first stop: Rust. Take a look at the following code:

let f = File::open("hello.txt");
let mut f = match f {
    Ok(file) => file,
    Err(e) => return Err(e),
};

This code is relatively simple to understand: the variable f contains a file descriptor to the hello.txt file. The operation can either succeed or fail. If it succeeded, you can read the file's contents by extracting the file descriptor from Ok(file), and if it failed you can either do something with the error e or further propagate Err(e). If you have seen functional programming before, this concept may sound familiar to you. But more important: this code makes sense even if you have never programmed with Rust before.

But once we introduce the ? operator, all that clarity is thrown off the window:

let mut f = File::open("hello.txt")?;

All the explicit error handling that we saw before is now hidden from you. In order to save 3 lines of code, we have now put our error handling logic behind an easy-to-overlook, hard-to-google ? symbol. It's literally there to make the code easier to write, even if it makes it harder to read.

And let's not forget that the operator also facilitates the "hot potato" style of catching exceptions¹, in which you simply... don't:

File::open("hello.txt")?.read_to_string(&mut s)?;

Python is perhaps the poster child of "readability over conciseness". The Zen of Python explicitly states, among others, that "readability counts" and that "sparser is better than dense". The Zen of Python is not only a great programming language design document, it is a great design document, period.

Which is why I'm still completely puzzled that both f-strings and the infamous walrus operator have made it into Python 3.6 and 3.8 respectively.

I can probably be convinced of adopting f-strings. At its core, they are designed to bring variables closer to where they are used, which makes sense:

"Hello, {}. You are {}.".format(name, age)
f"Hello, {name}. You are {age}."

This seems to me like a perfectly sane idea, although not one without drawbacks. For instance, the fact that the f is both important and easy to overlook. Or that there's no way to know what the = here does:

some_string = "Test"
print(f"{some_string=}")

(for the record: it will print some_string='Test'). I also hate that you can now mix variables, functions, and formatting in a way that's almost designed to introduce subtle bugs:

print(f"Diameter {2 * r:.2f}")

But this all pales in comparison to the walrus operator, an operator designed to save one line of code²:

# Before
myvar = some_value
if my_var > 3:
    print("my_var is larger than 3")

# After
if (myvar := some_value) > 3:
    print("my_var is larger than 3)

And what an expensive line of code it was! In order to save one or two variables, you need a new operator that behaves unexpectedly if you forget parenthesis, has enough edge cases that even the official documentation brings them up, and led to an infamous dispute that ended up with Python's creator taking a "permanent vacation" from his role. As a bonus, it also opens the door to questions like this one, which is answered with (paraphrasing) "those two cases behave differently, but in ways you wouldn't notice".

I think software development is hard enough as it is. I cannot convince the Rust community that explicit error handling is a good thing, but I hope I can at least persuade you to really, really use these type of constructions only when they are the only alternative that makes sense.

Source code is not for machines - they are machines, and therefore they couldn't care less whether we use tabs, spaces, one operator, or ten. So let your code breath. Make the purpose of your code obvious. Life is too short to figure out whatever it is that the K programming language is trying to do.

Footnotes

1: Or rather "exceptions", as mentioned in the RFC
2: If you're not familiar with the walrus operator, this link gives a comprehensive list of reasons both for and against.

Building a family tree

2020-04-28T00:00:00+02:00

I have been not-so-recently tasked with the job of putting together my family's family tree. It has occurred to me that other people might want to give the task a try, so here is my experience as a programmer.

First steps

The first important step was collecting my family information. I did this the old-fashioned way, sitting down with my mom and writing down in a spreadsheet names and relations. We didn't go beyond "married to" and "son/daughter of" because it was not important for us, but you could extend this information with birthdays, marriage dates, and so on.

The spreadsheet was good for collecting data fast, but getting the graph done required better tools: like all good programmers I'm lazy, and I'm not going to sit down and hand-place over 100 nodes if I can avoid it. I therefore turned to Gramps, an open-source software for Genealogical research. This software offers a lot of functionality and may seem daunting at times, but fear not: the software is clearly done by people who understand the problem very well, and it lets you record as much or as little information as you want. My favorite example: you can add a person with no name and no birth date, and the software will not even complain. I think it also auto-fills a person's gender based on their name.

Gramps has plenty of visualization options, and if you're not a software developer you can use one of them and stop reading here. Luckily I am a software developer, and un-luckily I couldn't find any visualization that did what I wanted. I was looking for a compact graph that I could fit in an A4 page, but all I could generate in Gramps' GUI were trees with very generous white space between nodes. This is in no way a point against Gramps, by the way: no software package can be everything for everyone, and I don't think I've ever had a graphic design project that I didn't need to hand-tweak one way or another. But one way or another, it was now time for massaging some data.

Tweaking the graph

After exporting the Gramps database to a CSV file, I decided to use Graphviz for automatic node placement. Graphviz is my go-to tool for drawing complex graphs, and I knew it could get the job done, but first I would have to convert my CSV file to Graphviz' DOT format. So I wrote the following script, which generates a DOT version of the input file.

#!/usr/bin/python3

import csv

# Counter with my own internal IDs, for fake nodes
counter = 1000
# Maps people to their intermediate relation nodes
relation = dict()
# Maps between Gramps IDs and my own internal ones
old2new = dict()

print("digraph G {")

# This CSV file should contain the following columns:
# 1: Name, 2: Last name, 4: Father ID, 5: Mother ID, 6: Partner
# A default Gramps export should be already structured like this.
with open('gramps_database.csv') as csvfile:
    datareader = csv.reader(csvfile)
    for row in datareader:
        id = row[0]
        # By default, you are in a relation with yourself
        # Useful for single parents
        relation[id] = counter
        counter += 1
        name = row[1] + " " + row[2]
        parents = None
        partner = None
        if row[4]:
            father = row[4]
            parents = relation[father]
        if row[5]:
            mother = row[5]
            parents = relation[mother]
        if row[6]:
            partner = row[6]
            if partner not in relation:
                relation[partner] = relation[id]
            else:
                relation[id] = relation[partner] 
        # Note that we have all information we need for this person,
        # we are not dependent on a future iteration.
        # Therefore, we can print this section of the tree.
        # First, the name
        print("{}[label=\"{}\"];".format(id,name))
        # Now, relations
        if parents is not None:
            print("{} -> {};".format(id, parents))
        if partner is not None:
            print("{} -> {} [dir=none];".format(id, relation[partner]))
print("}")

I then fed this output (tree.dot) to Graphviz. After playing with different visualization options, I settled for the following options:

sfdp -Tpdf -Gsplines=true -Goverlap=false tree.dot -o tree_output.pdf

which ended up looking like this:

This is still not perfect, but it's 85% there and it's something I can easily work with. I imported this graph into Inkscape, moved nodes around until they fit as tightly as I wanted, and added some colors.

Border

One of my known weaknesses is that I'm very picky on the graphic design of my projects: why stop at doing something right, when you can do it right and good looking? For this reason, I am constantly collecting pictures of cool posters, certificates, book covers, t-shirts, and pretty much everything that can be designed for print. You never know when something might come useful as inspiration!

Interesting enough, my collection of museum family trees was not as helpful as I expected: while very nice to look at, I found most trees difficult to understand, too simple, or too much illustration and not enough content (most artists are really invested into the "tree" metaphor). What did help me, however, was my collection of old-fashioned documents: the border of this tree is based on a 1923 Share Certificate from the "Charlottenburger Wasser- und Industriewerke AG", as seen in Berlin's Museum in Alten Wasserwerk.

Using it as a template, I created a vector version in Inkscape and placed it around my tree, which gives it a nicer "feel" and leads us to this version (with names removed intentionally):

The space on the right was originally reserved for legends, references, and whatever else, but so far I haven't come up with anything. And of course, I also need some space for future family members.

Final steps

The final step was to share it with my family, which was done with a combination of e-mail and telephone calls. About once a month I get a list of corrections from family members, which I do directly in Inkscape because at this point it's faster. I still keep the Gramps database updated, though, as I never know which future projects might require it.

The final tree can be printed in A4, as desired, but I wouldn't recommend anything smaller than A3. As a service for my readers, and painfully aware of how difficult it is to find good frames, you can download the empty template in .SVG format following this link to use in your own projects.

Benchmarking Python loops

2020-04-14T00:00:00+02:00

April 21: see the "Update" section at the end for a couple extra details.

A very common operation when programming is iterating over elements with a nested loop: iterate over all entities in a collection and, for each element, perform a second iteration. A simple example in a bank setting would be a job where, for each customer in our database, we want to sum the balance of each individual operation that the user performed. In pseudocode, it could be understood as:

for each customer in database:
    customer_balance=0
    for each operation in that user:
        customer_balance = customer_balance + operation.value
    # At this point, we have the balance for this one customer

Of all the things that Python does well, this is the one at which Python makes it very easy for users to do it wrong. But for new users, it might not be entirely clear why. In this article we'll explore what the right way is and why, by following a simple experiment: we create an n-by-n list of random values, measure how long it takes to sum all elements three times, and display the average time in seconds to see which algorithm is the fastest.

values = [[random.random() for _ in range(n)] for _ in range(n)]

We will try several algorithms for the sum, and see how they improve over each other. Method 1 is the naive one, in which we implement a nested for-loop using variables as indices:

for i in range(n):
    for j in range(n):
        acum += values[i][j]

This method takes 42.9 seconds for n=20000, which is very bad. The main problem here is the use of the i and j variables. Python's dynamic types and duck typing means that, at every iteration, the interpreter needs to check...

... what the type of i is
... what the type of j is
... whether values is a list
... whether values[i][j] is a valid list entry, and what its type is
... what the type of acum is
... whether values[i][j] and acum can be summed and, if so, how - summing two strings is different from summing two integers, which is also different from summing an integer and a float.

All of these checks make Python easy to use, but it also makes it slow. If we want to get a reasonable performance, we need to get rid of as many variables as possible.

Method 2 still uses a nested loop, but now we got rid of the indices and replaced them with list comprehension

for row in values:
    for cell in row:
        acum += cell

This method takes 17.2 seconds, which is a lot better but still kind of bad. We have reduced the number of type checks (from 4 to 3), we removed two unnecesary objects (by getting rid of range), and acum += cell only needs one type check. Given that we still need checking for cell and row, we should consider getting rid of them too. Method 3 and Method 4 are alternatives to using even less variables:

# Method 3
for i in range(n):
    acum += sum(values[i])

# Method 4
for row in values:
    acum += sum(row)

Method 3 takes 1.31 seconds, and Method 4 pushes it even further with 1.27 seconds. Once again, removing the i variable speed things up, but it's the "sum" function where the performance gain comes from.

Method 5 replaces the first loop entirely with the map function.

acum = sum(map(lambda x: sum(x), values))

This doesn't really do much, but it's still good: at 1.30 seconds, it is faster than Method 3 (although barely). We also don't have much code left to optimize, which means it's time for the big guns.

NumPy is a Python library for scientific applications. NumPy has a stronger type check (goodbye duck typing!), which makes it not as easy to use as "regular" Python. In exchange, you get to extract a lot of performance out of your hardware.

NumPy is not magic, though. Method 6 replaces the nested list values defined above with a NumPy array, but uses it in a dumb way.

array_values = np.random.rand(n,n)
for i in range(n):
    for j in range(n):
        acum += array_values[i][j]

This method takes an astonishing 108 seconds, making it by far the worst performing of all. But fear not! If we make it just slightly smarter, the results will definitely pay off. Take a look at Method 7, which looks a lot like Method 5:

acum = sum(sum(array_values))

This method takes 0.29 seconds, comfortably taking the first place. And even then, Method 8 can do better with even less code:

acum = numpy.sum(array_values)

This brings the code down to 0.16 seconds, which is as good as it gets without any esoteric optimizations.

As a baseline, I've also measured the same code in single-threaded C code. Method 9 implements the naive method:

float **values;
// Initialization of 'values' skipped

for(i=0; i<n; i++)
{
    for(j=0; j<n; j++)
    {
    acum += values[i][j];
    }
}

Method 9 takes 0.9 seconds, which the compiler can optimize to 0.4 seconds if we compile with the -O3 flag (listed in the results as Method 9b).

All of these results are listed in the following table, along with all the values of n I've tried. While results can jump a bit depending on circumstances (memory usage, initialization, etc), I'd say they look fairly stable.

	N=10	N=100	N=1000	N=10000	N=20000
Method 1	0.00001	0.00078	0.07922	8.12818	42.96835
Method 2	0.00001	0.00043	0.04230	4.34343	17.18522
Method 3	0.00000	0.00004	0.00347	0.33048	1.30787
Method 4	0.00000	0.00004	0.00329	0.32733	1.27049
Method 5	0.00000	0.00004	0.00329	0.32677	1.30128
Method 6	0.00003	0.00269	0.26630	26.61225	108.61357
Method 7	0.00001	0.00006	0.00121	0.06803	0.29462
Method 8	0.00001	0.00001	0.00031	0.03640	0.15836
Method 9	0.00000	0.00011	0.00273	0.22410	0.89991
Method 9b	0.00000	0.00006	0.00169	0.09978	0.40069

Final thoughts

I honestly don't know how to convey to Python beginners what the right way to do loops in Python is. With Python being beginner-friendly and Method 1 being the most natural way to write a loop, running into this problem is not a matter of if, but when. And any discussion of Python that includes terms like "type inference" is likely to go poorly with the crowd that needs it the most. I've also seen advice of the type "you have to do it like this because I say so and I'm right" which is technically correct but still unconvincing.

Until I figure that out, I hope at least this short article will be useful for intermediate programmers like me who stare at their blank screen and wonder "two minutes to sum a simple array? There has to be a better way!".

April 21 Update

A couple good points brought up by my friend Baco:

The results between Methods 3, 4, and 5 are not really statistically significant. I've measured them against each other and the best I got was a marginal statistical difference between Methods 3 and 5, also known as "not worth it".
Given that they are effectively the same, you should probably go for Method 4, which is the easiest one to read out of those three.
If you really want to benchmark Python, you should try something more challenging than a simple sum. Matrix multiplication alone will give you different times depending on whether you use liblapack3 or libopenblas as a dependency. Feel free to give it a try!

Working asynchronously - the one tip for successfully working from home

2020-04-06T00:00:00+02:00

If you are one of the lucky ones who didn't lose their jobs during the current pandemic, you may also be one of the involuntary volunteers taking part in the world's largest work-from-home experiment.

There are several well-written guides on how different companies and individuals do it. Software developers, who have been more or less working from home for the last decade, have particularly good tips - if you haven't yet read any of those guides, you can find a great thread here with practical suggestions.

A less-talked point, and the topic of this article, is learning how to work asynchronously with your team. That means: adapting your team's work style not to rely on other people being available to you right now and, conversely, not being "on call" for everyone all the time.

Synchronous work in the time of Cholera

In a regular, sane office setting the rules are clear: if my colleagues need something from me, they can come by and ask. If I'm busy they'll notice it, either because I'm with someone or because my door is closed. But otherwise, answering their question is literally my job, and when they ask me they get an immediate response. This is a synchronous situation.

Remote work, on the other hand, does not lend itself well to this. There can be a multitude of invisible reasons why I can't answer your question right now: I could be on the phone, stretching from hours sitting on my uncomfortable chair, discussing lunch with my (also at home) partner, answering the door, or multiple other situations that rarely happen in an office setting and cannot be conveyed with a status icon.

For the synchronous worker, this situation is unacceptable. They will send you an e-mail and five minutes later call to ask whether you received it, or are the type of bosses who film their employees to check whether they are working. Working synchronously while remote means being constantly dragged to pointless calls ("hey, quick question!"). You dread leaving your desk because you know there will be endless redundant notifications waiting for you. And you end the day exhausted from switching tasks all the time.

A new paradigm

For a term that's so difficult to write, working asynchronously is rather simple.

As part of a team, you need to embrace the idea that you cannot know whether your coworkers are busy. For all you know, they are in a different time zone and asleep. If you need someone's input you send them an e-mail or an instant message (see below) and then... you move on to something else. Your coworker will reply when it's convenient for them to do so. And meetings must be agreed beforehand - no more calling out of the blue!

As an individual, you need to mute your notifications. I recommend that you turn off all notifications except for when someone "tags" you (that is, they write your username and you get notified) for important topics that require your input. That doesn't mean that you can go fishing during work hours - you will still work, reply within a reasonable time frame, and be reachable when time is of the essence. But you no longer have to reply right now.

Working like this presents many advantages. Important points that would previously be discussed and forgotten are now being communicated on a need-to-know basis via e-mail; workers can easily step away from their desks knowing that nothing critical will be missed, and will stop spending valuable time catching up on notifications. And did you know that longer periods of interruption-free work can save up to 40% of your productivity?

"Asynchronous" does not mean "hermit"

The most common argument against asynchronous work is that you lose "human contact". And, while misguided, that's not a bad point: loneliness is a big factor for lots of people right now, and talking to their coworkers helps them cope in these uncertain times.

Luckily, there are solutions. A classic one is the "random" channel: a virtual board where people are encouraged to post non-work-related topics and discuss about their day. Another one is the "virtual office lunch" where everyone gathers for an hour with their cameras on and just talk.

And this is perfectly fine! I'm not saying "do not use Skype ever again". But I hope you'll agree that there's a difference between "I'm in this meeting because we planned it", "I'm in this meeting to waste some time because it's fun", and "I'm in this meeting to waste my time because someone thinks my time is worthless".

Working asynchronously is a team effort, where everyone needs to be on the same page. But it's not hard to pull off, and it can make your day more manageable. It might not be as helpful as toilet paper or flour, but it's a good start nonetheless.

I need your help liking Rust

2020-03-04T00:00:00+01:00

If you're a software developer, you know the rules: new year, new programming language. For 2020 I chose Rust because it has a lot going for it:

Memory safe and high performance
Designed for low-level tasks
Backed by Mozilla, of which I'm a fan
Named "most loved programming language" for the fourth year in a row by the Stack Overflow Annual Survey

With all of these advantages in mind, I set to build something concrete: an implementation of the Aho-Corasick algorithm. This algorithm, at its most basic, builds a Trie and then converts it into an automaton, with the final result being efficient search of sub-strings (why? I hope I can write about why in the near future). It also seemed like the type of problem you'd like to tackle with Rust: implementing a Trie in C requires some liberal use of pointers, a task for which I had expected Rust to be the right tool (memory safety!). And since I need to run a lot of text through it, I need it to be as fast as possible.

So how did I fare? Two weeks into this project, Rust and I have... issues. More specifically, I'm having real trouble figuring out what is Rust good for.

Part I: Pointers and Strings are too complicated

Dealing with pointers is straight up painful, because allocating a piece of memory and linking it to something else gets very difficult very fast. I followed this book, titled "Learn Rust With Entirely Too Many Linked Lists", and the opening alone warns me that programming a linked list requires learning "the following pointer types: &, &mut, Box, Rc, Arc, *const, and *mut". A Reddit thread, on the other hand, suggests that a doubly-linked list is straightforward - all you need to do is declare your type as Option<Weak<RefCell<Node<T>>>>. Please note that neither Option, weak, nor RefCell are mentioned in the previous implementation...

So, pointers are out as killer feature. If optimizing memory usage is not its strong point, then maybe "regular" programming is? Could I do the rest of my text handling with Rust? Sadly, dealing with Strings is not great either. Sure, I get it, Unicode is weird. And I can understand why the difference between characters and graphemes is there. But if the Rust developers thought long and hard about this, why is "get me the first grapheme of this String" so difficult? And why isn't such a common operation part of the standard library?

For the record, this is a rhetorical question - the answer to "how do I iterate over graphemes" (found here) teaches us that...

... the developers don't want to commit to a specific method of doing this, because Unicode is complicated and they don't want to have to support it forever. If you want to do it, you have to pick an external library. But it won't be part of the standard library anytime soon. At the same time, ...
... they don't want to "play favorites" with any specific library over any other, meaning that no trace of a specific method is to be found in the official documentation.

The result, then, is puzzling: the experts who designed the system don't want to take care of it, the official doc won't tell you who is doing it right (or, more critical, who is doing it wrong and should be avoided), and you are essentially on your own.

Part II: the community

If we've learn anything from the String case, is that "just Google it" is a valid development strategy when dealing with Rust. This leads us inevitably to A sad day for Rust, an event that took place earlier this year and highlighted how bad the Reddit side of the community can be. To quote the previous article,

the Rust subreddit has ~87,000 subscribers (...) while Reddit is not official, and so not linked to by any official resources, it’s still a very large group of people, and so to suggest it’s "not the Rust community" in some way is both true and very not true.

So, why did I bring this up? Because the Reddit thread I mentioned above displays two hallmarks of the type of community I don't want to be a member of:

the attitude of "it's very simple, all you need to create a new node is self.last.as_ref().unwrap().borrow().next.as_ref().unwrap().clone()"
the other attitude, where the highest rated comment is the one that includes nice bits like "The only surprising thing about this blog post is that even though it's 2018 and there are many Rust resources available for beginners, people are still try to learn the language by implementing a high-performance pointer chasing trie data-structure". The fact that people may come to Rust because that's the type of projects a systems language is supposedly good for seems to escape them.

If you're a beginner like me, now you know: there is a good community out there. And it would be unfair for me to ignore that other forums, both official and not, are much more welcoming and understanding. But you need to double check.

Part III: minor annoyances

I really, really wish Rust would stop using new terms for concepts that already exist: abstract methods are "traits", static methods are "associated functions", "variables" are by default not-variable (and not to be confused with constants), and any non-trivial data type is actually a struct with implementation blocks.

And down to the very, very end of the scale, the trailing commas at the end of match expressions, the 4-spaces indentation, and the official endorsement of 1TBS instead of K&R (namely, 4-spaces instead of Tabs) are just plain ugly. Unlike Python, however, Rust does get extra points for allowing other, less-wrong styles.

Part IV: not all is hopeless

Unlike previous rants, I want to point out something very important: I want to like Rust. I'm sure it's very good at something, and I really, really want to find what it is. It would be very sad if the answer to "what is Rust good for?" ended up being "writing Rust compilers".

Now, the official documentation (namely, the book) closes with a tutorial on how to build a multi-threaded web server, which is probably the key point: if Rust claims that error handling and memory safety are its main priorities are true, then multi-threaded code could be the main use case. So there's hope that everything will get easier once I manage to get my strings inside my Trie, and iterating over Gb of text will be amazingly easy.

I'll keep you updated.

A new and improved homeopathy

2020-03-02T00:00:00+01:00

You might have heard of this new, alternative take on medicine called Homeopathy. If you haven't, the basic idea is that you take a (possibly active) substance, dilute it with alcohol or distilled water, and repeat the process until only the "vital energy" of the original substance remains. According to Hahnemann, the creator of Homeopathy (or, to be precise, according to the Wikipedia article), each dilution increases the potency of the preparation while ensuring that all traces of the original substance are effectively gone.

The efficacy of this practice has been called into question several times, which to me sounds less like a problem and more like an opportunity: how do we bring Homeopathy into the 21st century?

Enter the Nocebo effect. Unlike it's big brother the Placebo effect, the nocebo effect is at play when a treatment has a negative effect simply because the patient believes it to be so - the common example being patients that suffer from "side effects" when receiving an inert substance. While precise numbers are impossible to obtain, around 5% of all patients are considered susceptible to this nocebo effect.

If a nocebo "weakens" a patient's positive response to a medication, and Homeopathy is based on diluting substances, we can combine them both! In what I have decided to call "Martinopathy" in honor of its creator (me), I suggest the following clinical procedure: when a patient is prescribed a Martinopathic treatment for (say) common cold, they are first directed to a standard pharmacy, where they buy a common, over the counter, non-homeopathic common cold drug. They are then sent to their physician. The Doctor will take a look at the medicine, repeat to the patient "this medicine will not work" around 20 times, after which the patient is free to continue their treatment with their now-martinopathic medication. In this way the effect of the regular medicine has been "diluted" down to homeopathic standards, but this time in a scientifically sound way.

There is still some room for improvements. If costs are an issue, they could be kept low by martinopathing the medicine at the source - instead of yelling at a patient, a medical professional could yell at the boxes directly in the factory floor. It is not entirely clear whether the medical professional would have to be certified in this new treatment or not. But those are small details that we can sort after I get my Nobel prize.

Untitled Bash blog - source code is now online

2020-02-27T00:00:00+01:00

As I mentioned before, I have now replaced my blogging engine with Pelican. Now that I'm mostly done dealing with ensuring that everything behaves more-or-less as before, it's time to talk about Bash.

I don't really remember why I wanted to write my blog using Bash. In general, I think it was a combination of the following factors:

I didn't want to install PHP. I have way too many memories of script-kiddies constantly probing my server, and that was never fun. Sure, I keep my server up-to-date, but why risking it? The less entry points for wannabe crackers, the better.
I didn't really thought it would be possible. Sure, writing it at first was kind of fun, but once I started writing triple-nested-quotes (all of which needed to be escaped properly) things got weird. I probably should have called it a day at that point, but I was close enough to my goal to make it worth it to continue till the end.
It was a good conversation starter - if you want to get a conversation rolling, bringing your terrible idea up is not the worst way to do it.

Of course, just because it was not the right tool for the job it doesn't mean that it was that bad. If anything, once I accepted that the heavy lifting should be made in a "proper" programming language (in this case, perl), using Bash to glue everything together worked surprisingly well.

With all of that in mind, I have now finally published the source code. I also plan on some light editing to make it friendlier, and an installation guide in case you really want to blog in a system where you have no permissions to install anything. The current template (just as my current blog) is based on Yahoo's Pure.css library library. In case you're not familiar with it, Pure.css is a set of CSS modules to make your project look good and responsive without a lot of effort, similarly to ~~that other library whose name escapes me right now~~ Bootstrap. I chose it specifically because I like to explore alternatives to the most popular projects, and Pure.css ended up being one of my favorites.

Music for programming

2020-02-14T00:00:00+01:00

Like many programmers, I am a night owl. Also, as many other programmers, I have a day job that forces me to be there at 8. These two characteristics interact badly with each other.

For most programmers, this is the type of problem normally solved with coffee. But not being a coffee drinker in general (I think it's just okay) and with what I can only assume is a natural immunity to caffeine, my to-go alternative solution is music: a good pair of headphones and epic, upbeat music makes wonders for my concentration until lunch time, when all productivity dies.

2019 was a great year for me to both catch up with songs I didn't listen to in many years and to discover new ones. The following is a list of songs to which I return every week, divided into three sections: Full albums, Instrumental songs (no words), and Individual songs (with words).

Full albums

There are two full albums that I have often listened entirely during long coding sessions, and that I definitely recommend:

For no one's surprise, Daft Punk's soundtrack for TRON: Legacy makes the list. Too bad the rest of the movie was not as good.
I haven't seen The Exorcist yet, so I never considered this album "creepy", but if you have seen it then you might recognize the opening of Tubular Bells. I found that the song's rhytm perfectly syncs with my internal rhytm, and it is not unusual for me to realize that I need to take a break right as the album comes to an end.

Instrumental Songs

It has been common knowledge for some time now that movie music is ideal for focusing on a task - you don't want the music to pull you out of a movie, the same way I don't want my music to pull me out of my work. For this reason alone, the first three items in this list are pulled straight out of Hollywood blockbusters:

The Pacific Rim main theme,
The Planet Sakaar theme from Thor Ragnarok, and
In the Hall of the Mountain King, specifically the version heard in the movie "The Social Network"

Moving onto TV, the next two songs are taken from the Japanese series "Kill la Kill": Naming Sense Gata Boshi Gokuseifuku, which I could swear I never heard in the series itself, and Nui Harime's theme.

Finally, and cheating a little bit, the theme from "The Good, the bad, and the ugly" as performed by the Danish National Symphony Orchestra is the one piece of music that got me to actually, physically buy music in many years.

Individual Songs

Individual songs are always tricky, because it takes a lot of listening to them before you learn to ignore the lyrics and let them blend in the background. That said, if you are looking for songs to listen over and over again, here's a bunch:

The least controversial song in this list are The greatest show on Earth and Ghost Love Score, both by Nightwish. They have long instrumental-only sections, and they are epic enough to give you an extra push while working.
Both Heldenzeit and Guten Tag by the German band "Wir sind Helden" are the perfect example of a great band that you discover long after they have disbanded. If you are a geek, the videoclip for Analogpunk (performed by the singer of "Wir sind Helden") is full of easter eggs.
The theme of "Revolutionary Girl Utena", Rinbu Revolution, is really good. There are not that many series where seeing the opening over and over is a plus, but Utena manages it.

Honorable mentions

I feel John Butler's "Ocean" deserves a spot in this list. It didn't make it into the official selection simply because I couldn't decide which version to include. I'm partial to the live version because it's the first one I heard, but the 2012 studio version is not bad at all.

Blog update!

2020-02-08T00:00:00+01:00

It is tradition to start every new year with a blog post lamenting why I haven't posted more. Instead, I have decided to kickstart 2020 with a list of the changes I've made to the blog to ensure I write more, why I've made them, and what interesting tools I've found along the way.

A big problem in my blog has always been how difficult it is to actually get something published. As I've mentioned in the past, my blog is powered by a bunch of command-line tools and Bash. This works fine when I want to work from home, but makes it very difficult when I want to blog something spontaneously: writing a post involves SSHing into my server, and there's exactly one computer from which I can do that. Converting an entry to the final HTML is not hard, either, but is friction enough that I need to be really motivated to get into it.

Enter 2020. I am more and more concerned about the state of the modern web, where "the internet" has become a synonym for Facebook, Google, and not much more. At the same time, I realize that I'm part of the problem: I may not be putting content in Facebook, but I'm not really putting content anywhere. The little content I'm putting out is not particularly useful, either. Clearly, something had to change.

And thus, a plan for 2020 was born. In order to simplify blogging, I have now thrown away my custom blog engine and moved to Pelican. I've also embraced Markdown as document format, meaning I no longer have to worry about things looking ugly after I've written them down. All my previous content has been migrated from .html to .md using Pandoc.

On the content generation side, I've also decided to try something new. Rather than wait until inspiration strikes, I'll try to blog weekly about small problems and how I solved them. This will often involve talking about Bash and Python, so I'll have to pay special attention to other topics that could be interesting.

In the meantime, here are some changes you'll notice from the migration:

Some entries will look a bit different. My custom footnote CSS no longer works with Markdown, so I'll probably substitute them with regular, boring footnotes.
A couple entries with custom html will look weird for a while. It will take a bit to get them looking like before, and I'd rather avoid delaying the update until then.
The RSS feed will probably break a little bit. Seeing as the RSS will now be generated by Pelican, I imagine your RSS reader will panic a little bit.

Next in the pipeline: a showcase of my now-almost-defunct blog engine.

Bad App!

2020-01-28T00:00:00+01:00

Random story from a random Android App: the "Photo Manager" app that I was using in my phone was updated a couple weeks ago. Opening my photos now requires the following steps:

Tap the icon to open the app.
Deny access to my location, because you don't need that to show me photos.
Deny access to my calendar, because you don't need that either.
Get sent back to the main screen, because the app will not run without this information.

This is a typical marketing tactic to try and sell my personal data for a higher price, but I was surprised at how brazen it was: most apps would still work, as selling me bad ads is better that selling none, but apparently the cost of me having their app installed is too high. Or maybe it was bad error handling. There's no way to know.

I was ready to uninstall it (which I did) and call it a day, but then I looked a bit deeper. It turns out that I was wrong, and the app is not developed by some obscure adware company in an effort to catch some low-hanging fruit. Instead, it was (probably) sold to a company first, which then repacked it with ads in an effort to catch some low-hanging fruit. Do you have an app you'd like to sell them? Here's a link. I'm willing to bet this specific app was sold around January 2019, as that would explain why they have gotten (almost) nothing but bad reviews for a year straight.

I don't really have a deeper point. I was just surprised at how quickly the app went to hell, and thought entrepreneurial people would appreciate it as a case study on what not to do.

What traveling every other week taught me

2019-11-24T00:00:00+01:00

I travel a lot. In the last two years I've taken easily over 100 trains (with half of those trips lasting 7hs or more), 20 overnight buses (12hs in each direction), two intercontinental flights, and I even biked to work once.

I would have to be an idiot to travel this much without learning anything. And while the jury is still out on whether I'm an idiot, I did learn some tips and tricks that I'd like to share with you.

Mentality

Before you even leave your place, it is important to put yourself in the right state of mind. So close your eyes, breath deeply, and tell yourself:

"I don't know how, why, or when, but of this I am certain: someone is going to fuck it up. I have made my peace with this, and I will not let it ruin my day."

You would be surprised of how useful this mentality is. It will lower your blood pressure, make you less angry, and more importantly, will naturally lead you to "safer" choices: if you already assume your future plane will be delayed due to acts of nature, you won't book that risky 30-min airplane connection.

Bag

The secret to a good trip (or, at least, a less miserable one) is to be prepared, and your bag is where it's at. Unlike common wisdom, I'm going to suggest you get a small-ish one. How small? Enough to fit underneath the seat in front of you in a plane: boarding a plane is stressful enough without having to fight for overhead storage, and the last you want is traveling without your gear because the plane is full and your big bag was sent to cargo. And traveling light is always good.

A controversial opinion: check your allowed bag dimensions and weight, and stick to them. I know neither passengers nor airlines give a damn, but once in a blue moon an employee decides to check and it will probably happen in your trip (see previous section). I know of a bus company that makes a non-insignificant amount of extra money simply by enforcing their rules.

If you are traveling internationally and/or by plane, I recommend you also get a travel document holder like these ones. Keeping all of your documents in one place (plane tickets, ID, Passport, itinerary, emergency money, credit card, ...) will simplify your travel, and can be taken in and out of your bag in a second. I recommend one that fits A4 pages, since most tickets are printed nowadays at home.

Gear

If your have an overnight trip, then your list should start with...

Earplugs and sleep mask: they take almost no space, are very cheap, and are ridiculously helpful for overnight trips.
Water bottle: nothing fancy, just enough for when you wake up at night with a dry mouth thanks to the AC.
Good travel pillow: notice that I said a good pillow. Forget about that U-shaped garbage they sell in airports - if it doesn't close at the front, you'll wake up several times either when your head falls forward or when the pillow slides away. I got one from this list, and I'm quite happy with my choice.

For long trips in general, I also like to have a good set of headphones. Over-ear (and, even better, noise canceling) are my favorites because they block outside noise by default allowing me to use a lower volume setting, a small act for which my hearing will be thankful 20 years from now. They can be bulky, however, so I encourage you to shop around for the right model for you.

Seating

First-timers will always take the window seat. This is a perfectly reasonable choice: they want to enjoy the view. Veterans, on the other hand, know that the hallway seats are where it's at: windows lose their charm after about an hour, but standing up and walking whenever you want never gets old. And unless your plane/bus/train is full, other passangers would rather try other rows than voluntarily sitting between the wall and someone they don't know.

You should also seriously consider taking your shoes off. You can find some light slippers to bring along, but at this point a warning is mandatory: you must ensure (and I shouldn't even have to say it) that you are wearing clean socks and that your feet don't stink. Don't be that guy!

Finally, boredom is your enemy. Forget that difficult novel, your work notebook, and that expense report you've swearing you'll start once you have free time. Bring a book you like, that series you've been meaning to watch, or pretty much anything you already know you'll enjoy.

WhatsApp follows your links

2019-11-14T00:00:00+01:00

This is not a secret, but I found it curious and thought about sharing it here: earlier today I needed to share a video file that I was hosting in my server, and I decided to send the link over WhatsApp. As I was looking at the log files at the time, I managed to catch this curious set of requests:

7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:49 +0100] "GET / HTTP/1.1" 200 5059 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:51 +0100] "GET / HTTP/1.1" 200 1806 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:53 +0100] "GET /fi HTTP/1.1" 404 549 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:53 +0100] "GET /fil HTTP/1.1" 404 688 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:54 +0100] "GET /file HTTP/1.1" 404 689 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:54 +0100] "GET /files HTTP/1.1" 404 690 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:55 +0100] "GET /files3 HTTP/1.1" 404 691 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:02:57 +0100] "GET /files3.mpr HTTP/1.1" 404 695 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:03:00 +0100] "GET /files3.mp HTTP/1.1" 404 694 "-" "WhatsApp/2.19.305 A"
7c0h.com:443 203.0.113.19 - - [14/Nov/2019:20:03:01 +0100] "GET /files3.mp4 HTTP/1.1" 200 219785 "-" "WhatsApp/2.19.305 A"

In simple terms: ever since WhatsApp identified that I was sending a link (that is, after I wrote https://7c0h.com/), WhatsApp generated a new request for every letter I typed (including the "mpr" typo). It's a good thing they only downloaded 215Kb of data too, because the request came straight from my phone. I guess there is no reason to optimize your code when you don't pay for the bandwidth.

Comics are unrealistic

2019-07-11T00:00:00+02:00

I know this is not in the usual style of this Blog, but I found something so out of touch with reality in Batgirl #36 that I need to share it with you. Please follow me on this journey.

(Warning: spoilers for Batgirl #34–#36. Also, I'm assuming you know who Batgirl is.)

Our story starts in Batgirl #34 with Barbara Gordon (aka Batgirl) missing a meeting with her business partner Alysia and some of her company's investors due to bat-related obligations:

As it turns out, the investors are not happy. They bought plenty of shares in the company, and want to use their position to push Barbara Gordon out:

Unfortunately for them, they don't have enough shares to kick her out. Even Alysia tells them so. But they wouldn't be important shareholders if they didn't have a backup plan, right?

You see: many issues ago, Barbara hired Pamela Isley (aka Poison Ivy, known eco-terrorist and occasional villain) to develop bee-friendly plant fertilizer for their company (it's a long story - see "Batgirl and the Birds of Prey" #12 for details).

The investors have now found out, and want to use this information to blackmail Alysia into kicking her friend out of the company. If Alysia does not agree to fire Barbara, they'll make this information public, tank the company, and put all their employees on the street:

But here's the thing: those investors have acquired a lot of the company. That is, they own it because they paid for it. Do you know what would happen to that investment if they went along with their plan? Yup, you guessed it: gone. All of it. Millions of dollars lost just to spite one of the company's founders. And it's not like they can sell their stock either: doing so in this situation is a clear-cut case of insider trading. Barbara Gordon may end up on the streets, but those investors would end up in jail.

You would expect a smart executive like Alysia to thank the investors for their valuable input, promise to call them back, get a good night sleep, and (at worst) negotiate a good separation agreement for her friend. A less smart person (but smart nonetheless) would have called their bluff: Mutually Assured Destruction is a good stalling tactic, but only an idiot would actually follow through.

Of course, none of these perfectly reasonable suggestions occur to Alysia. She kicks Barbara out of the company instead, leaving her broke and homeless.

And here is where I draw the line. See, I'm fine with the idea of an early-20s super-genius with photographic memory jumping from rooftops and fighting crime without the Police Commisioner noticing that she looks exactly like his daughter. But now you want me to believe that executives from a million-dollar company wouldn't have the simplest grasp of economy? Hell, do you want me to believe that you can organize a hostile takeover, fire a company founder, and freeze their assets in less than 3 hours?! Talk about nonsense.

With that out of the way, and as a final point, I would like to call your attention to Alysia's last words to Barbara. I think you'll agree that her frustration is a little... misplaced.

No, Alysia, Barbara isn't letting "them" take away her life's work. That one's entirely on you.

Lessons from EMNLP and INLG 2018

2018-12-06T00:00:00+01:00

This post is a condensed version of a talk I gave at my research group.

I've been lucky enough to attend both EMNLP 2018 and INLG 2018, and I thought it would be useful to share my main impressions of which cool things happened during the conference. I've split them into four sections: General tendencies about the direction that modern NLG research is taking, Practical considerations to be aware of for your own daily research, Exciting research areas where interesting results are being produced, and Fun ideas that I stumbled upon while walking around.

General tendencies

For me, one of the most important aspects of going to conferences is getting a sense of what has changed and what is coming. In that spirit, this conference season taught me that...

It is now okay to make a presentation full of memes: Are you leading a half-day tutorial for over 1K researchers? Put some minions on your slides, it's cool.

The text produced by neural networks is fine: For the last years, computational linguists have been worrying that, while neural networks produce text that looks okay-ish, it is still far from natural. This year we have finally decided not to worry about this anymore. Or, as described in Yoav Goldberg's slides, "Yay it looks readable!".

BLEU is bad: It is not every day that you see presenters apologizing for the metric they've used in their paper. It's even less common when they do it unprompted, and yet here we are. BLEU is a poor fit for modern NLG tasks, and yet everyone is still waiting for someone to come up with something better.
Further reading: BLEU is not suitable for the evaluation of text simplification, and ChatEval: A tool for the systematic evaluation of Chatbots for a human-evaluation platform.

Need something done? Templates are your friends: We are all having a lot of fun with Neural this and Deep that. The companies that are actually using NLP, though? Templates, all of them. So don't discount them yet.
Further reading: Learning Neural Templates for text generation.

Practical considerations

Classical NLP is still cool, as long as you call it anything else: Everyone knows that real scientists train end-to-end neural networks. But did you know that tweaking your data just a bit is still fine? All you have to do is pick one of the many principles that the NLP community developed in the last 50+ years, call it something else, and you're done.
Further reading: Handling rare items in data-to-text generation.

Be up to date in your embeddings: It is now widely accepted that you should know what fastText and Glove embeddings are. And ELMO and BERT are here to stay too. So if you're not up to date with those (or, at the very least, my post on Word2Vec), then it's time to start reading.
Further reading: want to use fastText, but at a fraction of the cost? Generalizing word embeddings using bag of subwords.

Data is a problem, and transfer learning is here to help: Transfer Learning is a technique in which you take a previously-trained model and tweak it to work on something else. Seeing as how difficult it is to collect data for specific domains, starting from a simpler domain may be more feasible that training everything end-to-end yourself.

Exciting research areas

If you are working on NLG, which I do, then you might be interested in a couple specific research directions:

Understanding what neural networks do: This topic has been going on for a couple years, and shows no end in sight. With neural methods everywhere, it only makes sense to try and understand what is it exactly that our morels are learning.
Further reading: Want to take a look at the kind of nonsense that a neural network might do? Pathologies of Neural Models Make Interpretations Difficult.

Copy Networks and Coverage: The concepts of Copy Networks (a neural network that can choose between generating a new word or copying one from the input) and Coverage (mark which sections of the input have already been used) where very well put together in a summarization paper titled "Get to the point! Summarization with Pointer-Generator Networks. Those techniques are still being explored, and everyone working on summarization should be at least familiar with them.
Further reading: On the abstractiveness of Neural Document Summarization explores what are copy networks actually doing.

AMR Generation is coming: Abstract Meaning Representation (AMR) is a type of parsing in which we obtain formal representations of the meaning of a sentence. No one has managed yet to successfully parse an entire document (as far as I know), but once it's done the following steps are mostly obvious: obtain the main nodes on the text, feed them to a neural network, and obtain a summary of your document. Work on this has already began, and I look forward to it.
Further reading: Guided Neural Language Generation for Abstractive Summarization using AMR.

Fun ideas

I don't want to finish this post without including a couple extra papers that caught my eye:

Here's an interesting idea: why doing a single pass of the input through your encoder, when a multi-pass approach may do better? If you like the idea, here's a paper for you: Iterative Document Representation learning towards summarization with polishing.
How do you detect insults when the people insulting do not use the actual insulting words? These two papers have ideas on how to do it: Determining code words in euphemistic hate speech using Word Embedding Networks and Improving moderation of Online Discussions via Interpretable Neural Models.
As told by someone during the conference: "The job of news agencies is to publish news in an objective, unbiased way, and the role of newspapers is to add bias. This paper does the opposite of that": Learning to flip the bias of news headlines.
This paper, titled Assisted nominalization for Academic English Writing, is an example of two interesting yet unrelated phenomena. First: you can do fun things with language tools. Second: just because you can, it doesn't mean you should. Active voice, everyone!

A more polite Taylor Swift with NLP and word vectors

2018-08-25T00:00:00+02:00

My relation with Taylor Swift is complicated: I don't hate her — in fact, she seems like a very nice person. But I definitely hate her songs: her public persona always comes up to me as entitled, abusive, and/or an unpleasant person overall. But what if she didn't have to be? What if we could take her songs and make them more polite? What would that be like?

In today's post we will use the power of science to answer this question. In particular, the power of Natural Language Processing (NLP) and word embeddings.

The first step is deciding on a way to model songs. We will reach into our NLP toolbox and take out Distributional semantics, a research area that investigates whether words that show up in similar contexts also have similar meanings. This research introduced the idea that once you treat a word like a number (a vector, to be precise, called the embedding of the word), you can apply regular math operations to it and obtain results that make sense. The classical example is a result shown in this paper, where Mikolov and his team managed to represent words in such a way that the result of the operation King - man + woman ended up being very close to Queen.

The picture below shows an example. If we apply this technique to all the Sherlock Holmes novels, we can see that the names of the main characters are placed in a way that intuitively makes sense if you also plot the locations for "good", "neutral", and "evil" as I've done. Mycroft, Sherlock Holmes' brother, barely cares about anything and therefore is neutral; Sherlock, on the other hand, is much "gooder" than his brother. Watson and his wife Mary are the least morally-corrupt characters, while the criminals end up together in their own corner. "Holmes" is an interesting case: the few sentences where people refer to the detective by saying just "Sherlock" are friendly scenes, while the scenes where they call him "Mr. Holmes" are usually tense, serious, or may even refer to his brother. As a result, the world "Sherlock" ends up with a positive connotation that "Holmes" doesn't have.

This technique is implemented by word2vec, a series of models that receive documents as input and turn their words into vectors. For this project, I've chosen the gensim Python library. This library does not only implement word2vec but also doc2vec, a model that will do all the heavy-lifting for us when it comes to turn a list of words into a song.

This model needs data to be trained, and here our choices are a bit limited. The biggest corpus of publicly available lyrics right now is (probably) the musiXmatch Dataset, a dataset containing information for 327K+ songs. Unfortunately, and thanks to copyright laws, working with this dataset is complicated. Therefore, our next best bet is this corpus of 55K+ songs in English, which is much easier to get and work with.

The next steps are more or less standard: for each song we take their words, convert them into vectors, and define a "song" as a special word whose meaning is a combination of its individual words. But once we have that, we can start performing some tests. The following code does all of this, and then asks an important question: what would happen if we took Aerosmith's song Amazing, removed the amazing part, and chose the song that's most similar to the result?


import csv
import gzip
from gensim.models import Doc2Vec
from gensim.models.doc2vec import TaggedDocument

documents = []
with gzip.open('songlyrics.zip', 'r') as f:
    csv_reader = csv.DictReader(f)
    counter = 0
    # Read the lyrics, turn them into documents,
    # and pre-process the words
    for row in csv_reader:
        words = simple_preprocess(row['text'])
        doc = TaggedDocument(words, ['SONG_{}'.format(counter)})
        documents.append(doc)
        counter += 1

# Train a Doc2Vec model
model = Doc2Vec(documents, size=150, window=10, min_count=2, workers=10)
model.train(document, total_examples=len(documents), epochs=10)

# Apply some simple math to a song, and obtain a list of the 10
# most similar songs to the result.
# In our lyrics database, song 22993 is "Amazing", by Aerosmith
song = model['SONG_22993']
query_vector = song - model['amazing']
for song, vector in model.docvecs.most_similar([query_vector]):
    print(song)

One would expect that Amazing minus amazing would be... well, boring. And you would be right! Predictably, when we do exactly that we end up with...

...Margarita, a song about a man who meets a woman in a bar and cooks soup with her.
...Alligator, a song about an alligator lying by the river.
...Pony Express, a song about a mailman delivering mail.

We can use this same model to answer all kind of important questions I didn't know I had:

Have you ever wondered what would be "amazingly lame"? I can tell you! Amazing + lame = History in the making, a song where a rapper tells us how much money he has.
Don't you think sometimes "I like We are the World, but I wish it had more violence?". If so, Blood on the World's hands is the song for you.
What if we take Roxette's You don't understand me and add understanding to it? As it turns out, we end up with It's you, a song where a man breaks up with his wife/girlfriend because he can't be the man she's looking for. I guess he does understand her now but still: dude, harsh.
On the topic of hypotheticals: if we take John Lennon's Imagine and we take away the imagination, all that's left is George Gershwin's Strike up the band, a song about nothing but having "fun, fun, fun". On the other hand, if we added even more imagination we end up with Just my imagination, dreaming all day of a person who doesn't even know us.

This is all very nice, but what about our original question: what if we took Taylor Swift's songs and removed all the meanness? We can start with her Grammy-winning songs, and the results are actually amazing: the song that best captures the essence of Mean minus the meanness is Blues is my middle name, going from a song where a woman swears vengeance to a song where a man quietly laments his life and hopes that one day things will come his way. Adding politeness to We are never coming back together results in Everybody knows, a song where a man lets a woman know he's breaking up with her in a very calm and poetic way. The change is even more apparent when the bitter Christmases when you were mine turns into the (slightly too) sweet memories of Christmas brought by Something about December.

Finally, and on the other side, White Horse works better with the anger in. While this song is about a woman enraged at a man who let her down, taking the meanness out results in the hopeless laments of Yesterday's Hymn.

So there you have it. I hope it's clear that these are completely accurate results, that everything I've done here is perfectly scientific, and that any kind of criticism from Ms. Swift's fans can be safely disregarded. But on a more serious note: I hope it's clear that this is only the tip of the iceberg, and that you can take the ideas I've presented here in many cool directions. Need a hand? Let me know!

Eye-tracking and visual salience

2018-06-26T00:00:00+02:00

This article is the fifth of a series in which I explain what my research is about in (I hope) a simple and straightforward manner. For more details, feel free to check the Research section.

In my last post we faced a hard problem: If a person visits a museum, for instance, we could give them information on the piece they are looking at. But computers don't have eyes! We could use a camera, sure, but that only works if there is only one art piece nearby. If there are several paintings close to each other, how do we decide which one of them is the interesting one?

One way is through what we call eye-tracking. This technology works like a regular camera, but with a catch: it doesn't only look forward, but it also looks backwards, at you! If you wear one of these so-called eye-trackers, it follows the movement of your eyes and records not only the entire scene (like a regular camera) but also a tiny dot that points out what you were looking at. Some colleagues and I found that eye-movement gives you a very good guess at what has captured someone's attention. After all, if you are interested in something, you are probably looking at it.

But there's a complication: eye-trackers are bulky, expensive, and take a long time to set up. And most people feel uncomfortable knowing that someone is recording their activity all the time. It is safe to say that we won't be wearing eye-trackers for fun anytime soon, and that's not great: what good are our results, if no one wants to use them?

Luckily, two men named John Kelleher and Josef van Genabith came up with a smart idea: whenever we are interested in an object, we look at it and get closer. He then applied this idea backwards: if we are looking in a certain direction and walking towards it, all we need to do is figure out what is right in front of us - that must be the object we care about. This technique is called visual salience, and it's a good alternative to an eye-tracker: rather than wearing expensive glasses, all we need to know is the direction in which they are walking. It might not be as effective, but it's good enough for us.

Following people's attention is important if we want our computers to cooperate with us: if a computer asks you to turn on the lights, but you start walking towards the fire alarm, it should warn you immediately that you are about to make a mistake. How to correct that mistake, however, is the topic of the next (and final) article.

Sentiments are the new Spam - Part 2: user groups

2018-03-09T00:00:00+01:00

So, you have successfully created an online community. People seem genuinely engaged, and you have interesting discussions going on. And then one day I show up, decide that "it would be a shame if something were to happen to your little communnity", and start harrassing your users because... well, because. Call it 4chan, Gamergate, MRA or trolls, there's always a group ready to drag a community into the ground.

Like I said last time, one of the main characteristics about the internet is that you can't block me, you can only block my user. So let's focus, from the simplest to the more complex, in how could you keep me from being annoying and/or harrassing other people in your community.

Privileged users

The first step I suggest you take is a hierarchical scale of users. It doesn't have to be too complex - I'd start with something like this:

Anonymous users are those that have not yet logged in. Usually they are allowed read-only access to the site, but in some cases not even that. As a counter-example Slashdot is known for allowing anonymous users to post and comment on the site, although with a catch that I'll discuss later.

New users should have limited posting capabilities - maybe they can only vote but not comment, or their comments are given partial visibility by default. Getting out of this category should be relatively easy for a "good" user (although time-consuming - no less than an hour, perhaps even days), but it should definitely annoy those that are only "giving the website a try".

Your regular users are the ones that actually use your site as intended. They can post and comment at will. And finally, the power users are allowed some extra permissions - usually this mean they can edit or remove other people's posts. This level should be pretty hard to achieve.

The iron fist of justice

Now that you have user levels, new users are your main concern: it is not unusual for trolls to create thousands of accounts (automatically, of course) and use them to assault a particular user. Remember: any regular user should be able to stop the noise in a simple and straightforward way - otherwise you risk becoming an online harrassing platform, and you'll have to publicly apologize like Twitter's CEO often does.

Our first moderation tool will be karma points. Each time a user contributes to our website, other users can rate this contribution positively or negatively. Contributions with "high karma" will be given a predominant position, while contributions with "low karma" will be buried. This is how Slashdot can allow anonymous contributions without being buried in dumb comments: every comment posted anonymously will have very low karma by default, but if enough users vote it up, it will eventually be seen by everyone else. Similarly, Hacker News will not allow users to vote negatively if they haven't yet reached a certain karma threshold.

Sidenote: you don't want to rank your posts/comments simply based on who has the highest number of votes. Instead, take a look at reddit's comment sorting system.

Another tool you'll find useful is the good old ban. A temporary ban means that a given user cannot post for a given period of time, while a permaban (permanent ban) means that the user is kicked out forever. This is a standard tool in every forum, but we can still do better: given that nothing stops a banned user from creating a new account and continue their toxic behavior (and remember, now they are pissed for being banned), you can use a hellban. When a user is hellbanned, no one but them can see their activity. The user can still log in, comment and post, but this activity is invisible to everyone else. From their point of view, it looks as if no one cares about them anymore, and it's not unusual for them to just leave.

Finally, you might also want to consider a "report" button, through which users can report unruly behavior. This should be more or less automated, but you cannot blindly trust these reports: you risk trolls banding together and reporting users at will. To prevent this, an automated recourse method should be enough - a moderator is notified, and the user is not fully banned until a final decision is reached. And finally, if you want to go the extra mile, you could have a "protected" flag that keeps certain users from being reported.

That's about all you can do at this level. There are no new ideas here, which is good - now you know that these concepts have been tried and tested before. In next two posts I'll be discussing about things that might not make as much sense, so stay tuned.

Sentiments are the new Spam - Prologue

2018-02-21T00:00:00+01:00

Once upon a time, you would create an e-mail account and use it for a long time without receiving spam. In fact, whenever you received your first spam message, you'd know exactly who to blame: that one cousin of yours who'd send you every single motivational powerpoint she came across, along with a list of 1500 other e-mail addresses. We could argue about who's the spammer in this situation, but that discussion will have to wait.

That kind of control over your account is no longer possible: even if you never share your account with anyone, you will at some point get spam. It's just the way things are, the "background radiation" of the internet. Luckily for us, things got so bad that a lot of smart people sat down to think really hard about this, and came up with Bayesian filtering, a technique so effective that most of us don't even bother checking our Spam folders anymore.

So we^1 succeeded once. It's a good thing to remember, because we have a much harder battle to fight now: trolling, and it's ugly cousin, online harrasment.

Let's say you post a message on an online board. These are some of the things that could happen, in no particular order:

You could get an interesting, well thought reply (note that "well thought" doesn't mean "agrees with you"). It happens.
You could be modded down by people that disagree with what you just posted, even if the rules say they shouldn't.
You could be flooded by negative messages, because a certain group decided to impose their point of view. This is called brigading, by the way, and it's usually not personal - they oppose your point of view, but not you.
You could be flooded by negative messages, because a group has decided to target you online for something you said, or did, or are.
You could be posting in behalf of a company, in order to speak in favor of your products posting as anyone-but-an-employee. This is called being a shill, and most websites either pretend that it doesn't happen or they don't care.
You could be trying to derail a discussion, in order to make sure a certain point is not brought to light, or is drowned in the noise. This usually implies that you work for a government agency, it's being done right now, and it works.

We used to believe that everyone on the internet would eventually behave nicely, and that we could build our services based on trusting the 95% of users that have no hidden agenda. This is sadly not so, because

... people have not behaved nicely on the Internet since September 1993.
... 5% of very loud users are a lot more noticeable than 95% of the quiet ones. A post-mortem of a DARPA Challenge showed that a single person can sabotage the work of thousands of well-meaning volunteers.

In the follow-up articles I'm going to comment on what I perceive to be three main points in which this issue could be attacked. They are

Anonymity: there's no way of taking measures against a person, only against a user. This is by design, and I'm not arguing that we should get rid of anonymity. We should instead focus on identifying toxic users, which I think can be done implementing user groups.
Flamewars: derailing discussions in order to kill them. This may be a job for pattern matching, identifying when the shape of a discussion is tending towards known anti-patterns. We might also want to add clustering, in order to identify brigades.
Harrassment: perhaps the harder one, requires sentiment analysis techniques to identify negative comments and kill them before they reach their destination.

In the follow-up essays I'll present some papers about how one would go about attacking each point. I have no reason to believe that this techniques are unknown (some of them are already implemented), but I post them hoping that, much like Bayesian filtering, someone will read them and have an "oh, wait" moment).

Coming up next: anonymous users and user groups.

Footnotes

^1 Of course, by "we" I mean "the computer science community in general". I did not create Bayesian filtering.

Genius MousePen i608 in Debian Linux

2017-11-15T00:00:00+01:00

I'm the proud owner of a Genius MousePen i608 graphic tablet (also known as UC-LOGIC Tablet WP8060U). This tablet is quite old and cheap, which is more often than not a recipe for headaches.

One very specific problem that I have: my tablet has an aspect ratio of 4:3, like old computers did, but both my desktop and laptop's screens have an aspect ratio of 16:9. Why is this a problem? Because my computer believes that the tablet and screen have the same aspect ratio, and whenever I draw a circle on my tablet it comes up on screen as an oval.

There are two possible solutions to this issue. One is changing my screen's resolution to match the 4:3 aspect ratio, which is annoying: I have to change the screen settings, then fiddle with my actual, physical screen so it doesn't stretch the image, and then I have to undo both steps once I'm done. The second solution requires a bit more calculations, but it's the right way: we'll configure the tablet in such a way that Linux recognizes the difference in ratios.

To be more precise: We will define a rectangle with the same height as the screen and a proportional width (sticking to the 4:3 ratio between width and height), we will position that rectangle in the center of the screen, and all movements in our tablet will only apply to that section of the screen. All movements on the tablet will translate to this rectangle without distortion, and if we need to interact with the screen outside this area we can still use our mouse.

The following code will run all the numbers for us. In essence, it will calculate the required set of parameters, and then it will modify the property Coordinate Transformation Matrix of xinit accordingly:

# Get the current screen resolution
resolution=`xrandr | grep '*' | cut -f 4 -d ' '`
width=`echo ${resolution} | cut -f 1 -d 'x'`
height=`echo ${resolution} | cut -f 2 -d 'x'`

# Get the proper tablet width, according to the 4:3 proportion
tablet_width=`echo "${height} 3 / 4 * n" | dc`

# We need to calculate four parameters c0, c1, c2, c3. For that, we use the
# 'dc' utility, which uses postfix notation (i.e., you write "7/3" as "7 3 /").
#
# Note: if you want to move the usable section of the screen left or right,
# take a look at the 'x offset' value. Also note that, since we are using the
# entire height of the screen, the 'y offset' is simply 0.

# Touch area width / width
c0=0`echo "7 k ${tablet_width} ${width} / n" | dc`
# Touch area x offset / width
c1=0`echo "7 k ${width} ${tablet_width} - 2 / ${width} / n" | dc`
# Touch area height / height
c2=1.0
# Touch area y offset / height
c3=0.0

# Obtain the device ID for the graphics tablet. Note that UC-LOGIC is my device
# ID, but yours may be different
device=`xinput | grep UC-LOGIC | head -n 1 | cut -f 2 -d '=' | cut -f 1`
# Set the Coordinate Transformation Matrix
xinput set-prop ${device} --type=float "Coordinate Transformation Matrix" ${c0} 0 ${c1} 0 ${c2} ${c3} 0 0 1

And that's it! It happens to me often that the transformation doesn't work straight away, in which case unplugging and plugging the tablet again solves the problem. A second issue with every reinstall is that the X server sometimes refuses to recognize my tablet. I solved that problem by adding the following lines to the /etc/X11/xorg.conf file:

Section "InputClass"
    Identifier "evdev tablet catchall"
    MatchIsTablet "on"
    MatchDevicePath "/dev/input/event*"
    Driver "evdev"
EndSection

The problem with music recommendations

2017-02-17T00:00:00+01:00

There's a popular song, written by an Argentinean musician called Charly García, called "Los Dinosaurios" ("The Dinosaurs"). The song was released in 1983 in the album "Clics Modernos", and you can listen to it in all its vinyl glory here.

This song represents for me an interesting problem: it is by far my favourite song from this author, and I would like to listen to something similar. But so far all recommendation systems have failed me. Here are some of the reasons why.

A first approach could be to pick something from the same author, or even the same album. This approach, sadly, doesn't work: while Charly García is certainly a prolific author, with 41 published records and countless guest appearances, his main style is oriented towards electronic music, and it doesn't really fit the style of this specific song. If anything, this song is more fitting for his earlier albums, which limits us quite a bit - out of those 41 soloist albums, "Clics Modernos" is the second one.

We could instead assume that this song was written in a certain context, and that looking at authors from a similar context we can obtain similar music. Again, this doesn't entirely work: if we pick "Argentinean songs from the 80's", we would end up with a list of songs that fit perfectly the style of the other 8 songs on this album, but not this one specifically^1. Grouping the song into "Latin American music", as some systems do, only exacerbates the problem: there is no relation at all between this song and, say, a Cuban bachata.

If we look at the actual lyrics, the situation gets even worse: "Los dinosaurios" is a thinly veiled critique of the military dictatorship that ruled the country between 1976 and 1983. A lyrics-based systems would most likely fail on two fronts: either it wouldn't understand the references made in the song and label it as "nonsense/fantastical", or it would understand the reference and recommend politically charged songs. Neither approach seems really right - while "The times they are a-Changin" could be a viable candidate for a recommendation, neither "The Revolution will not be televised" nor "Redemption song" fit the bill.

All of these approaches fail for the same reason: they apply a network-oriented measure to a song that doesn't fit the popular rhythm of the time and place in which it was produced, and which doesn't fit the overall style of the author either.

So what exactly am I looking for? A non-technical answer would be "I need a song that contains simple vocals, a piano as it's main instrument, and with raising tension towards the end". Or in the words of the author, a song that "adapts the English sound to Tango". As far as I know, the only system that applied a similarity measure capable of detecting this would be Pandora, but with their system closed to Europe, I cannot tell whether this works or not.

Footnotes

^1 How to obtain a digital archive of Argentinean songs from the early 80's is left as an exercise for the reader.

Voting in Argentina

2017-02-16T00:00:00+01:00

For the longest time, I thought that elections all over the world were more or less the same. Finding out that this is not the case was surprising, but not as much as what came afterwards: out of all the typical complaints, the Argentinean voting system manages to avoid most common pitfalls. So here's a quick overview.

Technicalities

First of all, and this should be a no-brainer, election day in Argentina is a public holiday, and always takes place on Sunday. The act of voting itself can take somewhere between 5 minutes and an hour, depending on whether you show up during rush hour or not. Voting is mandatory, too, and it always takes place as close to home as possible, so you really have no excuse not to vote.

On election day you show up with your ID to your designated place (usually the nearest school), and you queue at your assigned table. Eventually you present your ID and receive in return an empty envelope. Then you enter the so-called "dark room" alone, and close the door. Inside you'll find piles of ballots with the name and photo of each candidate. You pick one, seal the envelope, put it inside the urn, receive your ID back, and off you go.

Running the voting process is a (paid!) civic duty, and as such anyone over the age of 21 can be chosen. If you are selected, you get a letter informing you of your role (either President, Vice President, or backup), and where and when to receive a free training course. Aside from these voting authorities (three per table), there's an unspecified number of volunteers from political parties. These volunteers' main job is to make sure everything is run fair and square, but the table President can kick them out if they are out of line. Should (s)he need it, the army is always there to lend a hand.

What goes well, what doesn't

What is there to like about this system? A lot. Mandatory voting, public holiday, and short-ish queues ensure that everyone can vote. And while jerk bosses exist everywhere, it's an accepted part of the culture that, on election day, you vote. The system is also simple enough that anyone can understand it.

Large scale fraud is notoriously difficult, and expensive. Voter supression is also reasonably dealt with, as evidenced by the 80% turnover rate. Fun fact: every election, a non-zero number of fugitives end up arrested when they show up to their assigned voting place.

The Achiles heel of this system are the ballots themselves. For starters they can be stolen (and they are), so political parties must print ten times the required number to account for this (and even then, at times they just give up). They can also be replaced by fake ones, so they are thrown away during the vote count for being couterfeits. There's literally no good reason for not implementing a single voting ballot, which is probably why new projects introducing this change are shot down over and over by the ruling parties. At a more general level, the argentinean political system exhibits all known defects from the "first past the poll" system.

In conclusion

I'm really impressed by the vision the founding fathers of Argentina showed in several aspects, and elections is one of them (the other main one being the Constitution). There is no doubt that argentinean politicians are not the best the country has to offer, but that's hardly the voting procedure's fault.

And then again, who hasn't elected a clown for President here and there?

So, what did you say you do?

2016-09-09T00:00:00+02:00

Tell me if the following sounds familiar:

Oh, hi! It's been such a long time! They told me that you are a researcher now, right? What are you working on?
Me? Oh, well, ...

... I am developing a new carbulator theory that can hiperstat a maximum-entrophy logarithmic equation.
... it's something complicated. Have you ever heard of carbulators? No? Don't worry, no one ever does.
... you don't really want to hear that. It's super boring.

I am guilty of giving all of those answers at some point in my life. And while I am used to people not caring about my work, I'm not happy about it. Of course, I'm a nerd, so "doing boring things that no one cares about" is what I do. And I'm not saying everyone should be pasionate about Dungeons and Dragons^1. But I do think that, when I give a completely useless answer like the ones above, I'm contributing more to the problem than to the solution.

It is a fact that a lot of what we programmers and researchers do is considered boring by lots of people. But think about it, do you think it's boring? If the answer is "no", then I bet you could explain to me what's exciting about your job, why does it matter and/or what are you expecting to achieve. So all we need to do now is to better transmit this excitement to those around us. And yes, by "around us" I mean people who doesn't know what a carbulator is, have never heard the term in their life, and are probably none the worse for it.

I think a good start is my research section, in which I've listed some articles where I give a simple explanation of what I do. Not because I'm expecting my relatives to check my personal homepage, but because writing the articles has made me think really hard about what might be hard to grasp to non-technical readers, and next time I'll have a good script to begin with.

I can't tell you why I feel so strongly about this. Perhaps it's because the last time I was asked this question all I had were links to published papers, and that's unacceptable. Or because the time before that I straight up lied about it. Or maybe because I'm thousands of kilometers away from my family, and yet they don't have a clue about why I think it's worth it. And I have yet to find any downside to making knowledge more accessible.

So, what did you say that you do?

Footnotes

^1 I still can't find a Dungeon Master near me, though. And if you don't know what that is, no, it's not a sex thing. It's an old game...

Experiment report

2016-07-25T00:00:00+02:00

Hypothesis

People who claim being able to identify brands of cola by taste are usually right

Resources

5 different kinds of cola drinks (Pepsi, Coke, Coke Light, Diet Coke, Freeway), 5 similar cups, sticky paper, pens, even more paper

Description

So this is something that definitely happened, mainly because I think things too much and my girlfriend doesn't know when to stop me. The setup was: we kept all soft drinks at room temperature (to keep the taste unaltered), and then we split up. In step 1, one of us poured a cup of each softdrink into a numbered cup, and then left the room. In step 2, the other one replaced the numbers with different ones (Roman numerals), and mixed the cup order. This is a single blind experiment, which is as scientific as we could get without involving third parties.

Then the tasting began, which we did literally blind to keep any effect from identifying the color of a given softdrink (just in case). The task was to accurately identify which drink is which. Here are our highlights:

Soft drinks at room temperature taste awful.
Coke light is terrible even when blindfolded.
The Coke brands were identified without a hitch, while one of us mixed up Pepsi and Freeway. This points to our hypothesis being right.
If you don't think that softdrinks pack enough sugar, try tasting them at room temperature and blindfloded. We had to implement water glasses in between, because after the second cup they all tasted just like pure, unadultered sweetness.
It is curiously difficult to find Pepsi in Berlin. Seriously, give it a try.
It would be quite easy to fake an experiment to prove that people prefer a certain brand over the other. Just make sure your brand's drink is cold while the other one is not.

Conclusion and Future work

It is quite possible to identify a brand of cola by taste only. In future work we intend to repeat the experiment, but this time rating drinks in order of preference instead, to find out whether we really prefer the soft drink we claim to prefer.
We also found that the claim "people prefer X over Y in blind tastes" ranges from a filthy lie to useless at best. This result might come as a shock to some of our reviewers. We suggest those reviewers to leave their houses more often.

I am an idiot (a Windows Phone development story)

2016-05-13T00:00:00+02:00

I was warned. I was told this wasn't a good idea. I mean, even the price should have pointed out that something was not what it seemed. But no, I had to go ahead. I had to be cooler than the other guys, and I was certainly going to prove everyone that you can't go wrong with a Nokia phone. So I got myself a Windows Phone 8.1 (again), and today I say to you, that was a stupid choice and I'm stupid for going with it.

I've chronicled in a previous blog^1 how much of a pain it was to try to develop for a Windows Phone (WP). That alone should have kept me away from buying a smartphone with a too-good-to-be-true price. But on I went, tricked by memories of me saying "this is not so bad after all", carefully ignoring that I only said that before I tried to write my own app. And here we are again.

This is, step by step, what my experience trying to develop a very simple app for my (Lumia 530) Windows Phone was like. Yours might vary, of course, but I'm not betting on it.

Install Visual Studio Community 2015 (free), following the Official WP develop guidelines
Problem: all versions that could work in Windows 7 (the one I have) are deprecated for WP development one way or another: they don't have required tools, cannot be activated because the servers are gone, or just won't work. So get Windows 10 and that comes bundled with it want it or not, and try again.
Update your Windows and Install Visual Studio Community 2015 (free), following the official WP develop guidelines
Problem: the installer doesn't actually install the WP tools. Go back to the installer and modify your requirements.
Plug your phone to enable development
Problem: you need to unlock your phone first.
Sign up to Microsoft's Dev Network to register as a developer to get your phone unlocked
Problem: you need to pay €14, because reasons.

3.1 (Optional) Sign up on Dreamspark as a student to get a free developer account
Problem: the automatic verification process is broken. You need instead to send a copy of your ID, your student ID, and a transcript of your grades to Microsoft over unencrypted email. But don't worry, this data will be destroyed after they use it. "Anna" promised me so over email.

Start Visual Studio, create your first project, and add a couple text fields
Problem: bugs! The auto-complete is buggy, running my app once would force me to restart every time, and adding a control on the GUI while the text cursor is in the wrong place can (and will) erase every other control. But hey, at least we are finally developing something.
Add a "select date" field to your project
Problem: you need the DatePicker control to do this, but it's not included - I guess people don't select dates in phone apps. You have to follow several steps to get it working. However, as there's a library incompatibility somewhere, you'll get stuck anyway.

If you have been following all the steps you'll notice that, after several GiB of downloads and a lot of hours spent on internet forums, I have not yet managed to finish the first screen. I spent three days trying to get things running, which is the time I budgeted for the whole app, I'm out €14, and yet I haven't even managed to add a control that should have been there anyway. I also gave Microsoft quite some money, along with a lot of my personal information.

But the worst part is that I knew this is what development would be like, and yet I insisted on giving it another try. This is why I'm an idiot, and if you think your experience will be any better, it is my opinion that you are deluding yourself too.

Listen to my advice, dear reader: Windows Phone? Not even once.

Footnotes

^1 As of now I haven't restored the backup anywhere, but once I do you should see a link to that post here.

My plan to save online advertising

2016-05-02T00:00:00+02:00

After my last blog post arguing that advertisers should fall on a well and die, I've given the issue some more thought. After all, how would this little blog survive without ads? How would I keep up my product reviews without advertisers sending them to me for free?

Therefore, I came up with a plan to save the advertising industry from the scourge known as "ad-blockers". It's not precisely cheap, but we can still make it work in volume.

First of all, the Central Bureau of Advertising (or similar agency) has to announce publicly that they intend to make a raffle. The prizes have to be pretty good - The main prize should be at least a car, and the minor prizes should be iPhones (or even better, access to a yet-unreleased version), tablets, smartwatches and the like. You'll need a lot of them, so this is the expensive part. Ad agencies have to create quite a buzz, but then again, that's what they do for a living anyway.

Once we've assembled what is essentially the coolest contest on Earth, we announce the rules. In a nutshell, they will be:

Everyone is allowed to claim a prize
Winners will be selected via ads, served through the usual channels.
Other than reading "you've won", there's no requirement about what the ads will look like
The campaign will go on as long as there are prizes left. New prizes can be added at any moment

Here's the genius part: the winning ads will look as sketchy as possible. I suggest flashing gifs straight out of the 90s. Think about it: if the buzz for the campaign is high enough, we won't just have thousands (millions?) of users disabling their ad-blockers; we'll have effectively trained them to click on anything, no matter how suspicious it looks! All you need is to drag the contest long enough for little Johnny to think "I wish I could disable the ads, but then I might miss my chance to win". Once smaller companies start running their own contests in the same way, it's game over for ad-blockers.

There are of course some details to sort out, but I think you get the main idea. Some people will argue that this is dishonest, and that will lead to thousands of virus infections from rogue ads. Don't listen to these people: it's not your problem if some schmuck fails to protect his/her computer, and then again, if you cared about "dishonesty" your industry wouldn't be in this problem anyway.

I'm confident you'll find my scheme worth trying, and I hope this will clear any remaining bad blood between us. No need to thank me.

I'm not that angry

2016-01-11T00:00:00+01:00

One of the most enjoyable aspects of gaming for me is to try and pretend I'm the protagonist. That includes making choices the way I'd do them in real life. Of course, I understand that games are escapism, and I'm not blaming those that use games as an opportunity to murder pretty much anything that can be murdered. It's just not my style. I'm more the kind of gamer that's constantly being chased around by the guards I didn't kill because they didn't do anything.

Having said that, I'm having trouble with Angry Birds. I know, I shouldn't expect moral lessons from what is essentially a group of suicide birds bombing an enemy. But am I the only one who has trouble with a game that asks you to bomb a playground, including the kids playing there?

Did you enjoy committing a crime against humanity? Do you need more genocide? Then good news! You can bomb the skate park, somebody's house, and even a cemetery. Not only you get to kill your enemies again, but this time you can get their friends and family too!

Maybe it's because I'm having a bad day. Maybe it's because I'm putting too much thought into it. Maybe it's because I'm sick and tired of the phrase "collateral damage", or maybe I'm just missing some black humor. But in any case, I found out I can't bring myself to finish the game.

I hope that says something nice about me.

Screw your ad-supported internet

2015-10-01T00:00:00+02:00

So, it's that time of the year again. Apple has unveiled its new ad-blocking technology for the latest iPhone, and people on the internet are losing their minds about why ads are good for you, and how this will be the end of the internet as we know it.

To which I reply: no, they are not. And good riddance. Don't let the door hit you on your way out.

I can't properly outline the mess that online ads have brought into internet users. Slower loading times, annoyingly loud sounds, erosion of our privacy, malware, and constant surveillance of our browsing habits. "But wait", says the marketer, "if we track you constantly, then we can sell you better stuff. If ads are tailored to you, then you wouldn't be annoyed by them - in fact, you would be grateful". To this, I usually reply by not replying, because arguing with someone who thinks I enjoy getting sales pitches is a person I don't bother discussing with.

The technical point is more interesting. If the ad industry where to collapse today, bringing down every ad-supported website along with it, the resulting meltdown is something I'd love to witness. Imagine the ad bubble blows up tomorrow. Thousands of websites disappear overnight. Panic. Chaos. Bloggers on the streets yelling "LIKE ME!" to random pedestrians. Desperate tweens crying in restaurants because they don't know what to do with their food pictures. But then we'd all cool down our collective heads, and realize that most of the stuff we enjoy is already produced for free, is not supported by ads, or both. Without the marketers, the internet goes on.

I still remember the "older days" in which advertising was restricted to a couple pop-ups here and there. And I can tell you: that early internet was anything but "dead". There was an incredible amount of content long before the advent of advertising. Which is why I cannot understand how anyone would believe that the internet needs cash to survive. There was plenty of content before ads, and there will be plenty of content afterwards, too

In a following article I'll detail replacements for popular websites and apps based on technology we already have, to show that ads are more a convenience than a necessity. But I want to repeat that a volunteer-run internet would be incredible. Would we lose several of those San Francisco startups burning cash as fast as they can? Yes. But I'm still not sure why should I even care. Marketers are free to continue spreading the notion that they are the saviors of the internet if they want to. But I'm also free to keep recommending Adblock Edge and Disconnect to everyone who wants to enjoy the internet as it was meant to be.

A final thought: The fact that the side with more money has not prevailed in this discussion makes me a little less cynical. If their rooms full of paid commenters have not squashed this discussion yet, then we must be doing something right.

A neural network in Javascript

2015-09-03T00:00:00+02:00

A couple days ago I found a very interesting article, titled A neural network in 11 lines of Python. I always wanted to get a visualization of how a neural network works, so I took this as my opportunity. Taking that article as a base, I created a nice visualization of the simple case, and it ended up looking very nice.

This post is a small version of that. I'm going to show you how a neural network does its magic, using the magic of Javascript and SVG graphics.

Inputs			Output
0	0	1	0
1	1	1	0
1	0	1	1
0	1	1	1

This is the function we'll learn. Implementing XOR is pretty much the "Hello, World" of neural networks, so we'll be doing the same here.

Also, note that the third input column is all 1's. This will be our bias unit (i.e., a column that always equals 1).

Now, let's plug this into our network.

This is a graphical representation of our neural network. The weights have been randomly initialized (you can check that by reloading the page). Neurons 0-2 are input, neurons 3-4 are the hidden layer, and neuron 5 is output. Given that we have 4 training examples, we'll follow each training example individually.

The computation should proceed as follows: for neuron 3, we'll first multiply neurons 0-2 by the value of the edge that connects both neurons, sum those three values, and apply the sigmoid function to the result. In formal terms, $$n_3 = sigmoid(n_0 w_{0,3} + n_1 w_{1,3} + n_2 w_{2,3})$$ where n_i is the i-th neuron, and w_i,j is the weight that connects the neuron w_i with the neuron n_j.
The sigmoid function guarantees that the value for neuron 3 will be a value between 0 and 1. We repeat the same process for neurons 4 and 5.

So here we can see what our network is actually computing. If you have not yet read the article to the end, there's a very good chance that our network is returning random garbage, and that's fine - we haven't trained it yet, so of course the output makes no sense. This is called the forward step, in which I test my network and see what it's being computed.
For the second step, backpropagation, we'll need to write a couple tables, and see how bad our results are.

Output (expected)	Output (network)	Error
0	0	0
0	1	-1
1	1	0
1	0	1

Here is an interesting fact. We will call the difference between the value I expected and the one I actually got the "error" of the network. Similarly, sig(x) will represent the sigmoid function, and sig'(x) will be its derivative. Having defined that, the following equation $$error*sig'(output)$$ tells me how much should I correct my weights, and in which direction (whether they should be bigger or smaller). I won't delve in the math for that now, but if you need more details there are some links at the end that can help you.

So we have now applied our corrections to the green weights, but how about the red and blue ones? We'll also correct those by applying a variation of the same principle: once I corrected the value for the output, I have to distribute the amount of error into every weight that contributed to its computation. This will allow me to correct the values for neurons 3 and 4, which I'll finally use to correct the values for the remaining weights.
This process is called backpropagation, because I'm analyzing the network backwards to correct the errors I made when computing forwards.

Now all that remains for the training process is for me to repeat these steps over and over, until the er ror (shown underneath) is small enough.

Error
0
0
1
1

You can click any of the buttons here. One of them will perform one step of the calculation (both forward and backpropagation), while the other ones will perform one hundred and one thousand steps respectively. Using this, you can verify that the network eventually learns how to perform XOR, and then it stabilizes. Reloading the page will start all over again, but with different random weights for the network.

This is a very simple neural network (although not the simplest), implemented in Javascript. I've skipped some details in order to make the whole process simpler - If you want something slightly more challenging, I definitely suggest you to read the original article, which goes into the right level of detail.

I still want to improve on this, but I'm not entirely sure how. I think it would be nice to see how some neurons are given more (or less) weight, but I'm not sure how this should look like. If you have any ideas, feel free to tweet me.

Note: If you are quick enough, you might have noticed that the bias unit is missing in the hidden layer. The short version is: yes, it is. I only noticed once it was too late. I'll try and fix it in future revisions of this article.

It's not me, Spotify, it's you

2015-08-21T00:00:00+02:00

Dear Spotify,

I think it's time to realize that you are not the service you once where. At first it was subtle, like that time when you changed the shade of green of your logo to the ugly one you are using now. Then there was that issue with offline mode, in which I lost a whole playlist because your synchronization with Windows Phone doesn't work. I guess I should have seen the signals back then.

But now... now you changed. More specifically, you change your Terms of Use, and I can only use you if I agree for you to collect my pictures and track my location, among others. And that's where I have to draw the line. Spotify, I'm breaking up with you.

Wait, let me rephrase that: I already broke up with you 10 minutes ago, when I canceled my paid subscription. This is just me being polite.

Let's be honest here: I was not paying for music. I can get free music pretty much everywhere - call it Youtube, Vimeo, MP3 forums or torrents, finding free music is not particularly difficult. I was paying those € 10 because I preferred that to paying with my data, like so many other services. But if you are going to build a profile of me anyway that is related not to what I like to listen (which is what you are supposed to care about) but about what I do in my daily life (which is none of your business), then what's the point? I was paying to get away of the claws of marketing, and that is now gone. And so am I.

I guess I'll just go back to the old way, building my own music collection and listening to it wherever and however I want. I may even get back to my old idea of a streaming server. I know we had our issues before, like when I kept looking for videogame music and you kept showing me crappy piano versions of them. Or when you wouldn't change the title of Fabiana Cantilo's misspelled album even after I pointed it out repeatedly. But this time it's different. This time I'm gone for good.

Bye, Spotify. I'll show up later on to collect the titles on my playlist, so I can download them later. You can keep my e-mail address. It was a throwaway anyway.

Guitar music and I

2015-07-15T00:00:00+02:00

Here's a joke I heard once at a music academy:

How do you keep a pianist from playing? You take away their music sheet.

How do you keep a guitarist from playing? You give them a music sheet.

This joke rings oh-so-true because it highlights a key point for those of us who tried to learn guitar by ourselves: the typical amateur guitar player doesn't know how to read music, and (s)he doesn't care about it. Somehow they manage, but how they do that remains a mistery to me.

Typical guitar tabs (those you buy at a shop or download from the internet) contain therefore little more than the lyrics for a song and the points at which you are supposed to switch from one chord to another. This works pretty well for your left hand, but how about the right one? Should I just move it up and down? And at which speed? "Well", says the guitar book, "you should just do whatever feels natural". This is of course useless - what am I, the song whisperer? What if nothing feels natural? Do I just sit there in silence?

Let's take the following example, which I borrowed from Ultimate-guitar.com

RHYTHM FOR INTRO AND VERSE:
    D                       G       A
E|--2--2-2-0-2---2-3-2-0-2--3--3-3--0--0--0-0-0-0-0---|
B|--3--3-3-3-3---3-3-3-3-3--3--3-3--2--2--2-3-2-2-2---|
G|--0--0-0-0-0---0-0-0-0-0--0--0-0--2--2--2-2-2-2-2---|
D|--------------------------2--2-2--0--0--0-0-0-0-0---|
A|--------------------------3--3-3--------------------|

INTRO
D   G  A   [play twice]

VERSE 1:
       D
You'll say
G      A           D
we got nothing in common
   G      A         D
no common ground to start from
    G       A      D     G  A
and we're falling apart

This is actually a fairly complete piece: it shows the lyrics and notes (lower half) along with an attempt at explaining how should the strumming (i.e., what to do with your right hand) be performed. But here's the thing: it's not clear at all which strokes should be "up" and "down", nor the duration and silences between them. You cannot derive rhytm from this information, which is pretty bad for a section titled "Rhytm for intro and verse". And here's an extra fact: the "D" section should actually be played exactly the same as the "G A" section, but good luck discovering that from this notation. This is a known bug of guitar tabs, and yet I have several books with songs that don't even include such a section, either because they don't care or because they realized it's useless.

This is one of those very few problems that is currently solved by, of all things, Youtube. It's not too hard to find a "How to play Breakfast at Tiffany's" video tutorial, where some dude will spend some time showing you slow motion strumming, so you can play the whole thing. But how come youtubers have fixed this problem so fast, while guitar books have remained the same for decades? Why isn't everybody complaining? My theory is that the typical amateur guitarist picks up a guitar, downloads one of these tabs, fails at getting anything out of it, and quits guitar forever saying "guitars are hard".

I don't really have a good solution, because any attempt at formalizing the strumming will undoubtedly require some knowledge about rhytm, and guitar players seem to hate that. Perhaps that's how we ended up in this mess in the first place. Or perhaps there's a super easy, totally intuitive method that I've always missed for one reason or another.

But then again: how can a method do any good if it's never taught?

The Semantic and Observational models

2015-06-01T00:00:00+02:00

This article is the fourth of a series in which I explain what my research is about in (I hope) a simple and straightforward manner. For more details, feel free to check the Research section.

Let's continue with our idea of guiding people around like I mentioned in the previous article. It turns out that people usually make mistakes, either because the instruction we gave was confusing, or because they weren't paying attention. How can I prevent those mistakes?

For my first research project at the University of Potsdam, we designed a system that took two things into account: how clear an instruction was, and what did the player do after hearing it. Let's focus on those points.

For the first part, which we called the Semantic Model, a system tries to guess what will the user understand after hearing an instruction. If the instruction says "open the door", and there's only one door nearby, then you'll probably open that one. But what if I tell you "press the red button" and there are two red buttons? Which one will you press? In this case, the model tells us "this instruction is confusing, so I don't know what the user will do", and we can use that to make a better instruction.

For the second part, which we called the Observational Model, a second system tries to guess what are your intentions based on what you are doing now. For instance, if you are walking towards the door with your arm extended, then there's a good chance you are going to open that door. Similarly, if you were walking towards a button, but then you stopped, looked around and walked away, then I'm sure you wanted at first to press that button but changed your mind.

When we put both models together, they are pretty good at guessing what you are trying to do: when the first one says "I'm sure you'll press one of the red buttons" and the second one says "I'm sure you'll press either this blue button or that red one", we combine them both and get "We are sure you'll press that red button". Even though neither of them were absolutely sure about what you'd do, together they can deduct the right answer.

Each system takes into account different clues to make their guess. The semantic model pays attention mostly to what the instruction says: did I mention a color? Is there any direction, such as "in front of"? Did I mention just one thing or several? And which buttons were visible when you heard the instruction? The other model, on the other hand, takes into account what you are doing: how fast you are moving, in which direction, which buttons are getting closer, and which ones are you ignoring, among others.

Something that both models like to consider is which buttons were more likely to call your attention, either because you looked at them for a long time or because one of them is more interesting. But there's a catch: computer's don't have eyes! They don't know what you are really looking at, right? Finding a way of solving this problem is what my next article will be about.