Month: February 2014

On the Perils of Failing Silently

Last week we discovered a bug in production where a certain user flow would crash. Naturally, since this was a high-profile flow, the bug was discovered on Thursday evening (the end of the work week in Israel). And naturally, we could quickly reproduce the error in our live site, but it would not reproduce in any non-production environment. Par for the course.

Diving into the issue, we saw that under this set of circumstances, the client was receiving a(n inappropriate) 403, with no further information. Since the flow did not require authentication, this was particularly suspect. We started iterating through the various usual suspects – could our nginx be at fault? Is our load-balancer denying this particular flow? A broken SSL certificate perhaps? Or was something in the code broken, or just deep-six’ing the request with no further info?

Our usual log systems did not show any errors, and upon further inspection we realized that the breaking flow did not result in any applicative logging at all, which was especially odd since we make heavy use of logs and error-log anything that breaks. The lack of applicative logs along with the silent 403 led to the expected back-and-forth between the application code developers and the dev-ops infra people, where each side assumed it’s the other’s fault. Hours spent, dollars lost.

Eventually we traced it down to a particular legacy before_filter (we ride on Rails) which was affected by recent site configuration changes. This before_filter denied the request, but did so extremely silently:

render :nothing => true, :status => 403 and return

Once found, we could quickly verify this was indeed the issue and fix the before_filter to match the configuration changes, making everything go back to normal. However, it had taken us precious time to debug this error – hours with the flow broken, affecting users.

This could have been solved much, much quicker if the before_filter would not be so silent. The act of denial could be appropriately logged, but that’s not so straightforward – we wouldn’t want to log it as an error (since a truthful denial/403 is not an error), and any non-error logging would not be immediately obvious (what we would be looking for, anyway?).

A good way to solve this would be to simply return a non-empty error message to the client. Say:

render :text => "Action denied, Sorry-O.", :status => 403 and return

With a non-empty error message as above, a simple grep would quickly locate the root problem, enabling near-immmediate response.

$ grep "Action denied, Sorry-O." -ir .

Don’t give out compromising information, but *say something*, so your error is identifiable. Rendering ‘nothing’ is the equivalent of saying “Why? Because Fuck You, that’s why.”

Moral of the story: Never fail silently. Just like kids are taught: if you see something wrong, SAY SOMETHING.


Git for Smart Noobs

One of the problems with technical topics is that the oftentimes experts have a very hard time explaining the complicated details to others with less of an expertise. The explanations, aiming for exactness and truthfulness, go over the student’s head and are lost.

In my personal (slow, arduous) journey to learn the basics of OS code development and Git, whenever I manage to reach a more-than-basic understanding of a more-than-basic topic I try to write a post to explain that topic. This (hopefully) results in:

  1. Others (other noobs) benefitting from a down-to-earth explanation in terms they can understand, and
  2. My better understanding of the topic, for the best way to understand something is to explain it.

I’ve been writing using ‘Medium’, a new blogging software, which is aesthetically very pretty and does a good job at it’s main claimed purpose, “getting out of the way” and allowing you to spill your ideas easily.

So far I’ve written on how to use git rebase and git reset. Check it out.

Optimal Notes Strategy For Productivity Freaks

I’ve got lots of stuff to remember so I use notes and I needed a system that was better than any existing tool that I knew of, so I spent a some time thinking about it and now I have a system and it is BOSS.

Use-cases are:

  1. Find a note by name (like ‘git’ or ‘deploy_process’ or ‘electricity bill’)
  2. Find a note by content
  3. Edit, save, sync note to cloud
  4. Access anywhere, anytime, any device
  5. Port all my notes somewhere else in 2 years

The requirements (must be…):

  1. Super-fast to find a note by searching for name.
  2. Super-fast to find a note content.
  3. Super-fast and easy to edit a note and sync to cloud, all without using mouse.
  4. Available online via web, all platforms
  5. Easily exportable (No lock-in)
  6. Complete revision history

Failed attempts: Evernote and GoogleDocs. Both are too heavy, cumbersome, and slow. GoogleDocs is bottlenecked by the network and is in-browser, which is crowded enough. Evernote has a lock-in, no revision history, and is basically ├╝ber cumbersome. They might work for you if you’re infinitely patient but I am kind of the opposite.

Best solution:

  1. Open a folder in your Dropbox folder. Call it ‘notes’.
  2. Use your favorite text editor (which should be Sublime Text) to find, add, and edit simple text notes.

Shazam, you’re done. Let’s examine how this answers the requirements and use-cases.

  1. Find a note by name: ST’s fuzzy-search for file names is famously boss.
  2. Find a note by content: Ditto, ST’s search-function is famously sweet.
  3. Editing notes: By definition, it’s already your favorite text editor. Saving syncs it automatically to the cloud (dropbox).
  4. Access anywhere, anyhow:, and you’re golden.
  5. Exportable: it’s just a folder of text files! Evernote, Dropbox, and Google[Docs] may come and go (not to mention other app solutions like SimpleNote, etc.), but text files will be here for a long time. Whenever you want you can copy the whole folder off to your folder, or your SugarSync magic briefcase, and edit the files with vim instead of Sublime. No lock-in, ever.
  6. Dropbox saves all your revision history.

As you can see, these cover the requirements perfectly. To me, this clearly outclasses using GoogleDocs or Evernote for any personal use. I’ve been riding this for about 4 months at time of writing, and it’s been a dream. Anything I need to remember can easily be jotted down to the relevant note and retrieved at the speed of light (well, at the speed of Sublime Text, but it’s pretty damn close). Zero upkeep. Totally free. And I know that when I grow tired of Sublime (and/or switch computer, OS, polar meltdown, etc.) my notes will be there for me. This very post’s draft was written in my ‘blog’ note, where I keep a list of topics and first drafts of posts.

Basically this is has just been riding on the awesomeness of Sublime Text and Dropbox. You lose tiny things like clickable links and rich-text formatting, but those are for girls anyway.

Extra points for using ST+Dropbox for notes::

  • Use a specific Sublime theme for your ‘notes’ folder, so you can easily tell the ‘notes’ Sublime instance from code instances. I use Solarized (Light) because I like the light/dark contrast with code’s Monokai.
  • Set a syntax highlighting that complements free-text notes. I use AppleScript for no reason at all.
  • Organize your files into folders, if you feel like it, but with Sublime’s great search capabilities – it doesn’t really matter. I’ve got 150+ notes by now on everything from Apple to ZenDesk and don’t need the folders.

So: never remember anything again. You’ve got your computer to remember, now that you’ve got your llama-whooping notes mechanism set to retrieve and update notes at whiplash speed.

Now, go forth and multiply. Or at least use efficient notes.