Show me da code

Homerow Mods in OS X

Sun, 03 Dec 2023 12:00:00 +0000

Since I got the Advantage 360 keyboard I’ve been experimenting with keyboard customizations, and I’ve found homerow mods to be a very comfortable way to use shortcuts at scale without having to move your hands from the home row. The idea is use the asdf and jkl; keys as modifiers when pressed in combination with other keys but as regular keys when pressed alone (longer explanation here). Once you get used to it, you feel you have one hand tied to your back when you use a regular keyboard.

Unfortunately, I cannot customize the integrated keyboard of my MacBook laptop using ZMK or QMK, so I had to find another way. I found KMonad, a program that intercepts your keyboard taps and modifies them according to a configuration file written in pseudo-lisp.

Setup procedure

Install Stack. Stack is a (the?) Haskell build tool. KMonad is written in Haskell as you can guess from such a name. You can install it using brew install haskell-stack.
KMonad piggybacks on the kernel extension that Karabiner-Elements uses to intercept keyboard events. Install the Karabiner driver or the full thing.

Activate the driver:

/Applications/.Karabiner-VirtualHIDDevice-Manager.app/Contents/MacOS/Karabiner-VirtualHIDDevice-Manager activate

Clone the KMonad repo and build it:

git clone --recursive https://github.com/vosaica/kmonad.git
cd kmonad
git checkout Upgrade-DriverKit-VirtualHIDDevice
git submodule update --init
stack build --flag kmonad:dext --extra-include-dirs=c_src/mac/Karabiner-DriverKit-VirtualHIDDevice/include/pqrs/karabiner/driverkit:c_src/mac/Karabiner-DriverKit-VirtualHIDDevice/src/Client/vendor/include

Create a configuration file at ~/.kmonad.kbd. You can start from mine and modify it to your liking after taking a look at the KMonad tutorial. I’m using Command-Alt-Shift-Ctrl as my mapping for the left hand (and reversed for the right one) but other people use different combinations. See the Precondition post for more details.

(defcfg
    ;; Limit KMonad to the integrated keybboard, use (iokit-name) to apply to all
    input  (iokit-name "Apple Internal Keyboard / Trackpad")
    output (kext)
       
    ;; Comment this if you want unhandled events not to be emitted
    fallthrough true
       
    ;; Set this to false to disable any command-execution in KMonad
    allow-cmd true
)
   
;; lmet -> cmd
;; lalt -> option
;; fn   -> fn

(defsrc
    esc  f1   f2   f3   f4   f5   f6   f7   f8   f9   f10  f11  f12
    grv  1    2    3    4    5    6    7    8    9    0    -    =    bspc
    tab  q    w    e    r    t    y    u    i    o    p    [    ]    \
    caps a    s    d    f    g    h    j    k    l    ;    '    ret
    lsft z    x    c    v    b    n    m    ,    .    /    rsft up
    fn   lctl lalt lmet           spc            rmet ralt left down rght
)
   
(defalias
    met_a (tap-hold-next-release 200 a lmet)
    alt_s (tap-hold-next-release 200 s lalt)
    sft_d (tap-hold-next-release 200 d lsft)
    ctl_f (tap-hold-next-release 200 f lctl)
   
    ctl_j (tap-hold-next-release 200 j rctl)
    sft_k (tap-hold-next-release 200 k rsft)
    alt_l (tap-hold-next-release 200 l lalt)
    met_; (tap-hold-next-release 200 ; rmet)
   
    ;; let's use caps as control unless double-tapped
    ccaps (multi-tap 300 lctl caps)
   
    fn (around (layer-toggle function) fn)
)
   
(deflayer default
  _       f1      f2      f3      f4      f5      f6      f7      f8      f9      f10     f11     f12
  _       _       _       _       _       _       _       _       _       _       _       _       _    _
  _       _       _       _       _       _       _       _       _       _       _       _       _    _
  @ccaps  @met_a  @alt_s  @sft_d  @ctl_f  _       _       @ctl_j  @sft_k  @alt_l  @met_;  _       _
  _       _       _       _       _       _       _       _       _       _       _       _       _
  @fn     _       _       _                       _                       _       _       _       _    _
)
   
;; We need to explicitly map the function keys to their original function because
;; KMonad precludes the default configuration. As a bonus, enable vim-like cursor when pressing fn.
(deflayer function
  _       brdn    brup    lp      mctl    bldn    blup    prev    pp      next    mute    vold    volu
  _       _       _       _       _       _       _       _       _       _       _       _       _    _
  _       _       _       _       _       _       _       _       _       _       _       _       _    _
  _       _       _       _       _       _       left    down    up      right   _       _       _
  _       _       _       _       _       _       _       _       _       _       _       _       _
  _       _       _       _                       _                       _       _       _       _    _
)

Go to Settings > Privacy & Security > Input Monitoring and add /bin/launchctl and the kmonad binary.
Test the configuration:
```
sudo kmonad ~/.kmonad.kbd
```

If everything works as expected, you can install KMonad as a service by creating a launch config at /Library/LaunchDaemons/local.kmonad.plist (adjust paths if you installed kmonad elsewhere):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>local.kmonad</string>
    <key>Program</key>
    <string>/Users/your_username/.local/bin/kmonad</string>
    <key>ProgramArguments</key>
    <array>
      <string>/Users/your_username/.local/bin/kmonad</string>
      <string>/Users/your_username/.kmonad.kbd</string>
    </array>
    <key>RunAtLoad</key>
    <true />
    <key>StandardOutPath</key>
    <string>/tmp/kmonad.stdout</string>
    <key>StandardErrorPath</key>
    <string>/tmp/kmonad.stderr</string>
  </dict>
</plist>

Restart or start/stop the daemon manually with these commands:

sudo launchctl load -w /Library/LaunchDaemons/local.kmonad.plist
sudo launchctl unload /Library/LaunchDaemons/local.kmonad.plist

If you get § rather than backtick it might help getting a custom input source. Use this one (source) by copying it to /Library/Keyboard Layouts and selecting it in System Preferences > Keyboard > Input Sources.

References

Precondition post on homerow mods
KMonad tutorial
Example homerow mods config and tutorial
Post on how to run KMonad as a service

A bicycle for your memory

Sun, 27 Mar 2022 12:00:00 +0000

Computers have the need to store huge amounts of information and, at the same time, being able to access that memory at blazingly fast rates. This poses the problem of how to get a fast and large memory when hardware memories are either fast, like registers or L1 cache lines; or big, like hard drives or solid state drives.

Memory	Size	Speed	Comment
CPU register	A few bytes	One CPU cycle (200-300 picoseconds)	A register holds the data the CPU can operate on
RAM memory	A few gigabytes	100 nanoseconds	Applications need their data to be here to be responsive
Hard drive	Many gigabytes	150 microseconds	All your applications and data being kept locally have a copy here
Internet	Infinite (for all purposes)	10-40 milliseconds	Technically other people's hard drives

This is solved in computer architecture by the interplay of two ideas: a memory hierarchy and virtual memory. The memory hierarchy consists of a pyramid with the fastest and smallest memories on the tip and slower but bigger ones at the base. This arrangement allows you to have a complete copy of the information at the base, which is the biggest. Then, you can keep copies of the actively used information closer to the pyramid apex for better performance. Virtual memory is the illusion of having a single memory space which is the result of the collaboration of specific hardware (MMU, CPU features) with the operative system to transparently move data across the levels of the pyramid. The result is that, for most purposes and most of the time, programs running in your computer get to use a single memory space that feels as big as the base of the pyramid and almost as fast as its top.

Simplified memory hierarchy

Apart from computers, humans are rich information processors¹ and memory plays a paramount role in what it means to be a human. On one hand it is an important element of personal identity. Imagine a movie villain using technology or black magic² to swap the memories of two twin brothers. In some philosophical sense we are what we remember, and we will have a hard time deciding who is who in this contrived situation.

On the other hand, memories are needed for our minds to work because without them, we will have only the raw input of our senses and no other concept or idea to work with. The extent to which we can understand the world, communicate with others and act depends on using those mental building blocks or chunks.

Another aspect of being human not so directly related to information processing is tool making and tool use. Thanks to these, we have come a long way since we harnessed the power of fire and stone. Steve Jobs eloquently explained in this talk how humans will lose in almost all categories³ of an Olympic Games in which other animals were allowed to take part. We are not the fastest, we cannot travel the furthest, nor in the most efficient way. However, any cyclist is more efficient in energy per distance than a Condor, the most efficient animal. Then, he declared computers to be like a bicycle for the mind. Back then, personal computing was being rolled out, and we were oblivious to the directions these new machines would cybernetically extend us.

The Cyborg Memory Hierarchy

At this point we have all the ingredients to define what I call the cyborg memory hierarchy: extending human capabilities with technology (cyborg) so we can enhance in-brain memory and extend it with external information (memory hierarchy).

Cyborg memory hierarchy

Level 0. Working memory

Our working memory is composed of a phonological loop, where you can keep repeating a phone number while you search for pen and paper; the visuo-spatial sketchpad, where you can mentally manipulate shapes; and some very limited number of slots that can hold ideas, or chunks, for manipulation. Early George A. Miller work estimated that the typical mind was able to hold 5 ± 2 chunks⁴, but more recent work suggests we are limited to about 4 chunks.

I doubt we will cybernetically extend our working memory until brain-computer interfaces become a reality, but I have some good news in the methodology front. The reason Miller got initially confused with the amount of mental slots is that we can do more with less by having more elaborate and high level chunks. For example, I might be able to work in my mind with only three digits (e.g. 7, 1, 4) but if they are dates, then I can hold many more digits (e.g. 1492, 2001, 1991).

In fact, you can see expertise in any field as the slow and effortful learning of ever higher-level chunks and mental models (read Peak by Anders Ericsson). My advice: hunt all the important concepts in your field of interest and commit them to long-term memory (see next section).

Level 1. Long-term memory

These are memories that we can use fluently, regardless of our smartphone batteries being dead. These are critical for anything we do with fluency like speaking a language, being expert at anything, or being creative, which implies recombination of ideas that should be at the same time in our minds.

Some of these memories are declarative and can be expressed as answers to explicit questions. “What’s the capital of Georgia? Tbilisi or Atlanta”. We can extend our declarative memories by reading and studying for which there is a wealth of technologies and channels but the main enemy to beat is how fast those new memories fade away from us.

We know thanks to Hermann Ebbinghaus and his tedious self-experimentation with memorizing long sequences of nonsense words that facts decay exponentially in our minds unless reviewed and inter-related to other memories. His work was published in the XIX century, but no specific software was created to optimize when to review which piece of knowledge until Piotr Wozniak wrote the first spaced repetition software (SRS) in 1987. The underlying idea of this kind of software is that you encode what you have already understood and learned and want to keep in you memory as flash cards (short questions and answers) and the system keeps track of when to do reviews. Due to Ebbinghaus forgetting curve, the interval between reviews grows exponentially, so you see the typical card again after months or years. This means that you can keep many tens of thousands of cards in your level-zero memory at the cost of a few minutes of daily review.

For a nice explanation of how to implement this with Anki, one of the most popular SRS nowadays, I recommend reading Augmenting Long-term Memory by Michael Nielsen. He is also experimenting with articles that embed an SRS like this one about quantum computing.

If you want to read more about good practices with SRS, Wozniak’s Twenty Rules of Formulating Knowledge is the most authoritative source and Soren Bjornstad’s Rules for Designing Precise Anki Cards is a great complement.

Paraphrasing Nielsen, SRS makes declarative memory a choice. If we ever think that keeping a fact or a concept available in our mind is more valuable than 5 minutes of our time⁵ we can do it⁶.

What about non-declarative memory? Muscle memory cannot be reduced to declarative facts, no one is able to learn how to bike by reading How to ride a bike. However, some creative thinking can help us to bridge the gap in some cases. For example, I use Anki to learn application shortcuts declaratively and then, as opportunity to use them in the day to day happens I build the muscle memory.

Level 2. Personal notes and annotations

When we process information we tend to take notes or perform annotations in the medium itself, so we can come back to them without needing long-term memorization. Note that technically, pen and notebook are already cybernetic extensions for this purpose.

These notes are indeed very useful for us to solidify our understanding and organization of ideas, specially if we do a write-up like a blog post. The mere act of writing, even at book marginalia, also improves our initial memorization.

However, the value of these notes is limited unless they are connected and easily searchable. To get that extra value we need to up both our methodological and technological games.

The quintessential technology to fill this gap is the personal wiki in which you can seamlessly write notes, keep them inter-linked and securely access this second brain from all your devices. There is an inexhaustible offer of wikis out there (open source and SaaS). My recommendation is to pick one and try to use it unless blocked by a showstopper. A very common failure mode is to keep tuning your wiki configuration without really using it.

I personally use TiddlyWiki⁷ with the amazing Soren’s TiddlyRemember plugin to define SRS cards directly from within my wiki notes. When reviewing them in Anki I get a valuable link to their original context.

On the methodology front, I recommend the philosophy of Zettelkasten that was introduced by Niklas Luhmann, a very prolific German social scientist that implemented the logical equivalent to a wiki for his research notes using Victorian technology. Luhmann was carefully digesting other academics books and articles into chunks or ideas written in index cards that were linked, tagged and annotated to form a web of knowledge that he traversed in a myriad ways, leading to serendipitous connections bringing a deeper understanding—and many publications! By the way, zettel is German for note while kasten means box.

How a personal wiki looks like in steampunk style

Soren Bjornstad adapted TiddlyWiki for Zettelkasten and he extensively explains the result in this video. The template is available for anyone to use.

Level 3. Selected sources and everything else

At this level, we have all the source material—books⁸, articles—we collected and totally or partially read and quite likely you took notes and made flashcards about. At the mid-level of technology, you can picture cozy bookshelves at home but to give you more value, you need to get it organized and searchable.

For this purpose I use Calibre, which was presented to me as the iTunes of ebooks. Any other ebook organizer will do as long as you have a nice desktop search engine, so you can easily get back to the tidbits of your books that are neither in you head nor your notes. The spotlight search of Mac OS X is a good starting point for this.

Apart from these selected sources, we have ad-hoc information needs which we satisfy by using general search engines. Google, Bing, DuckDuckGo… choose your poison but once chosen, learn to use it properly (1, 2).

Does it make sense a partial implementation of this Cyborg Memory Hierarchy?

I think that any missing piece is problematic.

If you have SRS without a personal wiki you will tend to either create flashcards for things that are not really meant for that, and you might burn out; or otherwise, you might let go many valuable things.

If you have a wiki without SRS you won’t be fluid enough to take advantage of many things. For example, you can have great coding design patterns documented in your notes and never remember to apply them when the right conditions apply.

If you only have a search engine, you might not even know what to search about to solve a problem and your thinking will tend to be more shallow (related book: The Shallows: What the Internet is Doing to Out Brains).

How Neuralink or some other technology will change this?

I have no idea. I can only speculate that tighter integration with our brains can collapse some levels or open new ones that we do not even understand today. Maybe the Neuralink-enabled Anki feels exactly like Neo’s learning Kung Fu in The Matrix.

Your personal cyborg memory hierarchy

If you have some concept or implementation similar to the cyborg memory hierarchy or this post has inspired you to implement one for yourself I am very interested in knowing about your experiences. Please write a comment about it!

This used to be a false dichotomy as computer was a job title for humans! Until the invention of the modern electronic computer, a computer was a person—most likely a woman—skilled enough at mathematical computation to, for example, slowly tabulate a logarithm table. ↩
“Any sufficiently advanced technology is indistinguishable from magic” (Arthur C. Clarke) ↩
One exception is that we can keep walking and running sustainably way longer than other animals. When we were hunter-gatherers persistence hunting—hunting by exhausting your prey—was a thing. Another exception is our eye-to-hand coordination for throwing rocks and spears. I suppose people less apt at these things were wiped out from the gene pool by Mother Evolution. ↩
This is the origin of the magic number 7 (5 + 2) that is used to prescribe no more than 7 levels or class inheritance, no more than 7 attributes per class, no more than… ↩
Gwern estimates in about 2 minutes the total time that it takes remember a fact foreverish with Anki. I am doubling that estimate to compensate form the rest of us not being as bright as him. ↩
What a great pity not having had SRS in my tool belt during my university years. I would now remember more of the original content, and I would not have needed re-learning it years after my graduation. ↩
One of the strong points of TiddlyWiki is how you can deeply customize it for your purposes without leaving its markup language and dropping to JavaScript. Its plugin mechanism is also based in the same customizations that a regular user can afford and that means the small community around it has published a huge amount of plugins that you can mix and match. ↩
A bookshelf was considered disruptive technology in times of Plato and Plato considered books will make people forgetful. We can forgive him because there was no SRS back them to compensate for this effect. ↩

How to waste less time and space with Terraform

Mon, 24 May 2021 11:46:00 +0000

I’ve been using Terraform quite extensively at work for some time and, over time, I’ve been interacting with more and more modules. At the same time, I need to work with a rather small HDD for modern standards where every GB counts.

Terraform’s default configuration wastes a lot of space by keeping a separate copy of the provider binary in each .terraform/plugins. In my case that was about 20 GB.

$ find . -name .terraform -exec echo {}/plugins \; | xargs du -hs --total
19G

However, you can configure Terraform CLI to use a centalized cache directory as long as your OS supports hardlinks. To do so, create ~/.terraformrc with contents plugin_cache_dir = "$HOME/Library/Caches/terraform/plugin-cache" or some other directory that makes more sense to you. Read more about this configuration.

After that, you can take advantage of the consolidated cache.

$ mkdir -p ~/Library/Caches/terraform/plugin-cache
$ find . -name .terraform -exec rm -r {}/plugins \;
$ ...some terraform inits...
$ du -hs ~/Library/Caches/terraform/plugin-cache
819M

On top of saved space you will have faster terraform inits.

Let DNS do what DNS does

Thu, 18 Feb 2021 00:00:00 +0000

tl;dr: fight entropy in your Kafka Connect connector configurations by resolving cluster nodes with my connect-dns-provider plugin

Thermodynamics presents us with the ruthless teaching that global entropy or disorder is constantly growing and that locally you can only fence it off by expending energy.

For living beings, this means the need to ingest enough food over time to sustain themselves. For the world of software development, this means the will to refactor and to introduce new organizing principles over time. The alternative to paying this price is a deteriorating codebase and supporting artefacts that exposes you to risks and reduces the development pace over time. Eventually, you reach the point in which you add one bug per bug removed and you can declare the thermal dead of the system.

How do you keep that entropy at bay? Ruthlessly remove what is no longer needed and introduce structure to support what is actually needed. Bonus points if you avoid falling for the shiny object syndrome and choose the appropriate proven technology even if it’s not the latest hype.

A personal recent example of this for me is related to working with a Kafka ecosystem in which we use Connect connectors to move events in and out of a main Kafka cluster. Connectors are created, modified and removed by means of some scripts your can launch as Jenkins pipelines.

A pain point of this setup is that the configuration for some connectors require of a list of servers and ports of other clusters. For example, if you are using the Cassandra source connector, you are expected to configure connect.cassandra.contact.points=server1:port1,server2:port2. When you rotate or redimension those nodes, you need to update the scripts and run the pipelines anew.

At this point, there are many different options to attack this problem but many of them add unnecessary complexity. You might be tempted to integrate with a shiny new configuration parameter store or try to parse remote Terraform states and whatnot.

However, there is an old, well-proven technology for this purpose: DNS. In fact, there is a specific kind of DNS record for this purpose. It’s called SRV and it was introduced more than 20 years ago. Applied to the previous example, you will have one SRV record per Cassandra node, for example:

_cassandra._tcp.example.internal.   300 IN SRV 0 0 9042 c-123.example.internal.
_cassandra._tcp.example.internal.   300 IN SRV 0 0 9042 c-333.example.internal.
_cassandra._tcp.example.internal.   300 IN SRV 0 0 9042 c-234.example.internal.

These records are read in this way:

Name. DNS name to use when resolving.
Timeout in seconds. In this case, 5 minutes in seconds.
Class. It’s always IN for SRV records.
Type. SRV!
Priority. For clusters like Cassandra it makes sense to have the same priority for all nodes. The lower, the most priority.
Weight. This is useful to assign different fractions of the load to different servers but, as with priority, it makes sense to use the same value for all nodes of a Cassandra cluster.
Port.
Host name.

You can keep these records updated in many ways. For example, if you use Terraform, Cloud Formation or any other software defined infrastructure, you can define the SRV records as linked to the node names. That way, any infrastructure change will automatically change the records.

Another common use case nowadays are services managed within Kubernetes. Because Kubernetes embraces DNS as an integral part to service discovery, you get SRV records for free for services.

The only missing piece in the puzzle is that our connectors resolve the SRV names. Fortunately, we don’t need to modify existing connectors but we can write a ConfigProvider implementation to add support for any connector.

As you can see, a simple single class is able to do the work. Because this might be useful for more people and I never before published libraries to Maven Central I took the pain to properly publishing it (GPG setup, opening JIRA tickets¹). You can find usage instructions at the repository.

Remember, let DNS do what DNS does. Even if it’s a “boring” technology.

In one of the many official videos explaining the “simple process”, there was an amazing comment of someone comparing the process US immigration redtape ↩

Improv: lessons for life

Thu, 26 Dec 2019 00:00:00 +0000

Why are some conversations engaging, smoothly moving from point to point while others die off in a couple of round-trips and a final monosyllable? Why can you rapidly build chemistry with some people and have false starts with others?

I've been taking classes here ☝️

Lately I’ve been taking some improvisation theater classes, improv for connoisseurs, and the experience has shed some light on these questions and proven useful for the real-life world™.

First, improv is more about self-expression and play than about performing theater in front of an audience, at least at the amateur level. Your typical class has an initial warm-up phase in which some exercises wake you up by physically moving, interacting, or maybe shouting and then you play some games in which you practice the different skills you need to improvise scenes. Finally, you play some improvisation games in which everything is unpredictable but within some guidelines.

For example, one popular improv game called the statues consists of having someone freeze in a pose and then a new actor steps in and starts a scene somehow using such a pose to justify it. At some random point, the teacher will clap and both actors will freeze. Then the cycle restarts when a third actor replaces one of the frozen ones. This exercise is a constant surprise as the photo finish of a scene that started with skiing becomes a do-you-wanna-marry-me one and so on.

This game is representative of elements in improv: some structure or rules, a ton of unpredictability, a sense of building something together and the joy of letting yourself play like you were a child again.

As I’m a total noob and the course is introductory, we focus on the fundamentals that can be summarized as accepting and adding or with the motto yes and…. This was surprising to me as I was expecting something more related to acting but it makes a lot of sense. Usually you improvise in pairs or bigger groups and when someone proposes something it’s of paramount importance to be generous and selfless in order to accept that proposal, not doing so breaks the flow and destroys the spirit of the game. In improv’s jargon it’s called negation and recognizing and avoiding it is one of the first things you are taught.

Negation is the ultimate wet-blanket. Imagine that one actor braces herself and say “It… it’s veery cold up here in the mo-mountain”; then the other negates her with “I’m a fish in the ocean.” The first actor will be disoriented not knowing what to do… even angry at her companion.

Compare this with the opposite to negation: accepting and building on top of that. “It… it’s veery cold up here in the mo-mountain” followed by “Don’t falter now that we are so close to top of the Everest and to win the bet”. Now, the first actor can fill in more details about that enigmatic bet.

Apart from having learned some jargon like negation, PROL¹, bouncing; and from having had a ton of fun I got a lesson to apply back to my regular life: yes and… is the missing ingredient to make conversations more engaging and to build chemistry faster in some other situations. Think about this: who do you prefer to go on in a holiday trip with? With someone that accept other’s proposals and build on top of them or with someone that systematically rejects anything but his or her own verbatim will?

PROL: Persona, Role, Objective and Locale ↩

Can't someone else do it?

Sat, 17 Aug 2019 00:00:00 +0000

An automatic formatting tool is better than quibbling at formatting issues in pull requests. Why paying attention to something that can be automatically minded for? Can’t someone else do it? That’s the reason we use scalafmt where I work.

Why formatting the code yourself when someone else can do it (automatically)?

Now IntelliJ has direct support for scalafmt, not so long ago I needed to use a separate plugin for that. But there is a small problem. There are small corner cases in which the IDE-triggered reformat doesn’t reformat stuff than the SBT-triggered one would do. Therefore, from time to time, the Jenkins build complains about my pull requests for having files incorrectly formatted…

Not anymore!

Let’s remove the frustration of needed to wait again for the Jenkins build from time to time! (This instructions are tailored for OS X, feel free to replace brew with apt-get or equivalent for your system).

First, we are installing nailgun, a protocol and server that solves the problem of long initialization times for JVM executables. With nailgun you have a background JVM process that is already started and ready to execute scalafmt or any other program with nailgun support.

$ brew install nailgun

Then, we are installing scalafmt standalone with nailgun support so that we can reformat or check formatting from other scripts:

$ sudo coursier bootstrap --standalone org.scalameta:scalafmt-cli_2.12:2.0.1 \
  -r sonatype:snapshots \
  -o /usr/local/bin/scalafmt_ng -f --main com.martiansoftware.nailgun.NGServer

To guarantee that the server is always running whenever we login we can use a launchd agent (launchd is the evolution and replacement of initd in the Darwin family of Unix). To do so, we should create ~/Library/LaunchAgents/scalafmt_ng.plist with this contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" 
          "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>KeepAlive</key>
  <true/>
  <key>Label</key>
  <string>scalafmt_ng</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/scalafmt_ng</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

If we restart our session or run launchctl load scalafmt_ng.plist the nailgun server will be ready for service.

Finally, we can write a git hook to check that our files are properly formatted:

#!/bin/sh
set -e
# Check but don't reformat sources with scalafmt
ng org.scalafmt.cli.Cli \
    --non-interactive --quiet \
    --git true \
    --test \
    -c .scalafmt.conf \
    --exclude target \
    `find . -iname scala -type d | grep -v target | grep -v bloop`

I have this file at ~/.githooks/scalafmt-check to avoid duplication. To enable these checks for a repo I simply link it to the pre-commit and pre-push hooks:

$ cd repo
$ ln -s ~/.githooks/scalafmt-check .git/hooks/pre-commit
$ ln -s ~/.githooks/scalafmt-check .git/hooks/pre-push

Now we can instantly know whether we are trying to commit/push code that is not well formatted:

$ git commit -a -m"Test commit"
Looking for unformatted files...
All files are formatted with scalafmt :)

If you prefer to reformat on commit you can go with almost the same script excluding the --test flag.

The moral of the story

Remove frustration and friction from your daily life. You’ll be happier and more productive!

A way out of the Jar Hell

Sat, 27 Jul 2019 00:00:00 +0000

A couple of weeks ago, a friend retweeted the release announcement of Debian 10:

Debian 10 buster has been released! https://t.co/Qm1Oc7SXiF
— The Debian Project (@debian) July 7, 2019

First thought: it has taken a couple of years but here it is.

Second thought: these people know how to curate software… wait, my spider-sense is tingling crazy.

And then I realized the connection between how the Debian project solves certain projects around software ecosystem curation and some of the least happy parts of the life of the Scala developer: the trouble with libraries, mainly binary compatibility problems.

Library galore perils

It can be argued that the single factor that makes modern development–with Scala or otherwise–more productive is the extensive use of tons and tons of open source libraries. What in other times would have required months of development is today a handful of lines of code depending on tens of libraries¹.

However, this powerful practice carries the seeds of important problems that can curtail its benefits. Out of the many possible perils of the profuse use of libraries, most of them are given short shrift by the development team as if they were not our responsibility. Take a look at the skeletons in our closet:

Licensing. There are many kinds of software licenses and more than one definition of open source out there². Some of the licenses impose restrictions on the systems those libraries integrate to (e.g. GPLv3) and we cannot equate “I can download it from Maven central” with “I can use it freely”.

There is an SBT plugin to quickly check the licenses of your dependencies called sbt-license-report. Most of the licenses are detected and classified (e.g. Apache, GPL with Classpath Exception) but in some cases, the information is missing from Maven Central and you will need to do a manual check.
Library quality. Not all the code that is released as libraries meets the same standards of quality. Even when meeting high standards, the assumptions or the focus of the library can be far from the design of your application, making you work around the library design doing circuitous designs. Sometimes we want a banana, but the library we have found has a giant gorilla holding the banana. Think about this gorilla as concepts unrelated with your application and bugs (in the library or its transitive dependencies).

Adopting libraries should not be done blithely but after a cost-benefit analysis. Remember the Sturgeon’s revelation: 90% of anything, including libraries, is crap.
Security. It’s not just that we are inheriting bugs and design warts from the libraries we use, we are increasing the surface area for security.

A cautionary tale from last year happened around NPM, the main library repository for the Javascript ecosystem. Dominic Tarr, the maintainer of a widely used library, decided to open the project to more people because he wasn’t even using it anymore, he was doing it for free and someone volunteered to do it for him. Unfortunately, the person volunteering was a black hat hacker that released a version of the library with some code targeting Copay, a Bitcoin wallet, that indeed got released with the malicious code.

This kind of horror story is less likely to happen for libraries better staffed in which you need to have a track record to become a committer. However, in the long tail of libraries out there, don’t expect all projects to have these practices. In the end, the chain is as weak as the security of the weakest library you depend upon.

While it’s impossible to avoid not yet known security problems, aka zero-day vulnerabilities, these are reported by the security community to the CVE database that can be checked automatically from SBT with the sbt-dependency-check plugin. You can even make your build fail with the dependencyCheckFailBuildOnCVSS command.
Compatibility issues. Unlike the previous ones, this problem can’t be overlooked because when we have a library compatibility problem, like the typical binary compatibility issue like the feared IncompatibleClassChangeError our code refuses to work. You can ignore that one of your libraries has a license you are violating but not fixing a conflict of versions is like spurring a dead horse. Congratulations, you are enjoying a JAR Hell! Way more about this in the next section.

All these problems get amplified by the fact that when we decide to marry one library we are not marring just that library but all its family: potentially many transitive dependencies. You need to check the licenses of all those libs, evaluate their quality, monitor its CVE’s and pray for them not to have a version conflict with the other libraries you are using.

Compatibility problems

To illustrate why we have this library compatibility problem let’s use a simplified example. Imagine you have an application that has two direct dependencies libraries a and b. Both direct dependencies depend on library c but–to our despair–not the same version.

Dependencies as intended

When we run our program, we cannot have two versions of the same library (unless using OSGi) and version 1.1 of c is evicted in favor of the other one.

Dependencies in runtime

When a dependency eviction is a problem? When the assumptions made by the code depending on the evicted library are not met by the one used in runtime. There are two types of problems:

Binary compatibility problems. Our Java/Scala code gets compiled to bytecode that invokes specific methods with some number of parameters of some definite types. If c 2.2 adds a new argument to some method that a is invoking, a NoSuchMethodException will be raised. If the whole class has been renamed or removed, then a ClassNotFoundException one. And if a static method is converted to a normal one, IncompatibleClassChangeError.
Semantic compatibility problems. Even if the public signatures of library c don’t change we can face compatibility problems because some assumptions are not captured by method signatures. For example, we might have a class in c with a constructor with default values that might change between versions completely changing what library a is asking c to do. This is trickier to diagnose that an IncompatibleClassChangeErrors.

In an ideal world, we will have libraries with semantic versioning, which is a convention directed towards making our lives as consumers of libraries easier:

Given a version number MAJOR.MINOR.PATCH, increment the:

  MAJOR version when you make incompatible API changes,
  MINOR version when you add functionality in a backwards-compatible manner, and
  PATCH version when you make backwards-compatible bug fixes.

If all libraries were following this versioning system it would be easier to manage evictions as you’ll be sure that c 1.2 could be safely replaced by c 1.9. However, it’s difficult that this ever happens because you can check binary compatibility in an automated fashion³ but semantic compatibility is more elusive.

Another practice that helps mitigate this problem is to reduce the number of dependencies by:

Not using libraries unless they pay for its salt.
Use of granular libraries. Having a “commons” library that depends on many, many other things is a really bad idea as you are bringing all those even if you just wanted a small piece of the functionality. It’s better to have “commons-streaming”, “commons-kafka”, etc.
Relying on the platform. Can you use java.time instead of a third-party library? Is anything solving my problem in the Scala collections library? By platform I don’t just mean the JDK and Scala’s library, this might include the libraries bundled with Spark or any other platform.

Solving compatibility problems

The typical modus operandi when facing a library compatibility problem is first, finding the libraries whose dependencies get evicted. Second, trying to bump or downgrade one of them in order for the dependencies to get close enough the problem is fixed. Third, if that doesn’t work, fork one of them to produce a custom version whose dependencies are aligned with the rest of our microcosm⁴.

Another solution consists of actually allowing two versions of a library in runtime. That is possible in the JVM because class management is delegated to class loaders. OSGi is a framework that leverages this to allow multiple versions of a library with some caveats. With OSGi, you need to annotate which packages of the library are public, being the rest private and not accessible (this plays very bad with reflection-based frameworks). As long as a library don’t expose any type of its dependencies in its public part that dependency can be hidden from other modules and you can have multiple versions of it.

Is then OSGi a solution? It is for most cases but you are paying for it with more complexity in dependency management, will have problems with anything relying on reflection and for some cases, you will need to still fork some libraries.

Another possible mitigation that doesn’t solve this problem but can greatly reduce its impact is the use of microservice architectures. In such architectures, each service has a narrow business purpose and that corresponds to a subset of the dependencies that the equivalent monolith whould have had. However, this is not automatically true as some microservice ecosystems tend to share many, many shared libraries and you get the worst of both worlds: lots of dependency conflicts and coupling like in the monolith case and you need to cope with distributed systems complexities as in any microservice ecosystem. This anti-pattern is known as the distributed monolith.

Is there any way out of the JAR Hell?

Lately, I’ve been wasting quite some time impeded by binary compatibility problems and I became close to convinced that they were an inevitable part of the life of the Scala programmer very much like NPEs for the Java programmer. Is there any solution? Should we suffer these problems as a way to atone our promiscuous library sins?

Should we act like the JAR Hell is inevitable?

A workable approach to these problems was hidden in plain sight! When I saw the Debian 10 release announcement I was thunderstruck by the revelation: Debian has been solving these problems for decades!

If you think about it, a Debian release is a set of packages (libraries) that comply with some license requirements⁵ and work well together. Those packages are curated by a bunch of maintainers that form a community and you need to be verified in person to get into (so it’s unlikely that something like the NPM problem happens to them).

A release of Debian is created by getting fresh versions of the core packages (kernel, libc, buildtools) and then, aligning everything else. They have tooling that makes very easy to manage patches on top of the original code (“upstream” in Debian parlance). Those patches are used to fix compatibility problems, make default behavior more Debian-like or fix other problems.

After the release is out, packages are frozen except for security patches. These are, by design, things you can trust to update automatically with confidence that nothing is going to break. Interfaces won’t change, configuration formats won’t change. You just get security patches. These are sometimes provided by the upstream developers and, other times by the Debian maintainers.

What happens if I need the latest version of a library for a release of Debian that has an older version? You can always compile it against what is shipped in that release. However, as this is very common, the work of changing dependencies and fixing compilation is shared in dedicated “backport” repositories.

Cool. Can I have this for Java/Scala? We have examples of limited versions of this. For instance, the Sprint Boot community does this kind of curation for Spring-based applications. They have dozens of “starter” modules that are a selection of concrete libraries in versions that work well with each other. For instance, mix in spring-boot-starter-data-couchbase and spring-boot-starter-data-cassandra and you won’t have compatibility problems. However, it is useful only with the premise of building a Spring application and limited in scope.

A promising tool that is in development for the Scala ecosystem is Jon Pretty’s Fury, a replacement for SBT. An integral concept on Fury is the layer, “a collection of related projects you are working on” and each project can reference the upstream repository of an external library or an internal artifact. Layers can be potentially shared to form the basis for a curation similar to what the Debian project does.

Time to mature

Getting out of the JAR hell is a matter of maturity. Using so many libraries so liberally has multiplied developer productivity to a large extent but without the restraint of some curation culture like the one in Debian, its risks and practical problems might cancel out the advantages of having access to this galore of functionality.

Updated on 2019-08-17: mention microservices as a mitigation inspired by trylks’s comment

And it’s transitive dependencies! It’s not uncommon to have hundreds of them in a single small project. ↩
The original definition of free software by Richard Stallman and the FSF, and the more corporate-friendly, open source definition by the OSI. ↩
For example, the SBT’s plugin mima allows you to check for binary compatibility against previous versions of the library you are developing ↩
If you do this, please use good conventions to avoid confusing your coworkers. For example, if you fork some library in version 1.2.2 to align versions, release it on your private artifact repo as 1.2.2-our-company instead of 1.2.3. The former is obviously some private customization and the later is bound to collide with the real thing soon enough. ↩
Open source. What happens with closed software? It’s not integrated by the Debian community but it benefits greatly of this model of releases because you just need to compile your libraries/programs against each recent Debian release and provide a repository URL. No need to create a binary that is statically linked against all libraries (this is analogous to the fat-jar anti-pattern). ↩

Finatra meets Ammonite

Sun, 03 Feb 2019 00:00:00 +0000

Li Haoyi’s Ammonite is the most user-friendly Scala REPL out there. It has a more than decent syntax highlight, you can edit multi-line snippets, search the command history, auto-complete expressions and much more!

It’s remarkable how you can fiddle with a library without the hassle of creating a SBT project thanks to the magic $ivy imports. Let’s say you want to try out some scalaz 7.2.23 for some reason:

$ amm
Loading...
Welcome to the Ammonite Repl 1.1.2
(Scala 2.12.6 Java 1.8.0_161)
If you like Ammonite, please support our development at www.patreon.com/lihaoyi
@ import $ivy.`org.scalaz::scalaz-core:7.2.23`, scalaz._, Scalaz._
import $ivy.$                               , scalaz._, Scalaz._

@ Map('a -> 1, 'b -> 2) |+| Map('a -> 2, 'c -> 1)
res1: Map[Symbol, Int] = Map('a -> 3, 'c -> 1, 'b -> 2)

@

Importing $ivy.`org.scalaz::scalaz-core:7.2.23` is equivalent to adding "org.scalaz" %% "scalaz-core" % "7.2.23" to SBT’s libraryDependencies.

Ammonite can be embedded in other applications and I find it useful to so when developing Finatra services.

There are two interesting ways of doing the integration: as a local development console for debugging and exploration or as a mechanism to inspect or operate against remote services.

Give your Finatra service a development console

If you have ever used Rails and enjoyed it, you will be offended when facing development environments in which you cannot start a console and tinker with domain services, the database, etc. The first step to get Ammonite integrated into Finatra a la Rails is to depend on Ammonite with test scope so add to your build.sbt this:

libraryDependencies ++= Seq(
  "com.lihaoyi" % "ammonite" % versions.ammonite % Test cross CrossVersion.full
)

where versions.ammonite is a recent version (e.g. 1.4.0).

The idea is to have an alternative main class under src/test that will start the service and the console. To make it easier, you might create an alias in your build.sbt:

fork in (Test, run) := true
addCommandAlias("run-with-console", "test:runMain ammonite.integration.ConsoleLauncher")

As you can see, we are using the ammonite.integration package. This is important because we need to use some definitions that are package-private and only available from such package.

Then, you should create the launcher class. It should similar to your actual entry point (usually a server class extending from com.twitter.finatra.http.HttpServer and a handful of Guice modules) but instantiating Ammonite.

Let’s take a look at ConsoleLauncher bit by bit.

package ammonite.integration

import com.google.inject.Module

import com.yourcompany.project.infrastructure.Server
import com.yourcompany.project.modules.FooModule
import com.yourcompany.project.modules.BarModule

object ConsoleLauncher extends Server {

  override protected def modules: Seq[Module] =
    super.modules ++ List(
      FooModule,
      BarModule
    )

Nothing special up to here.

  private val prelude = """
import scala.concurrent._
import scala.concurrent.duration._

implicit class ExtendedFuture[A](val f: Future[A]) {
  def await: A = Await.result(f, Duration.Inf)
}
"""

We can add a custom prelude to Ammonite. You can add common imports or even functions and implicit conversions here. This prelude allows me to easily use services returning Future in the console.

  override protected def postWarmup(): Unit = {
    super.postWarmup()
    try ammonite
      .Main(predefCode = prelude, verboseOutput = true)
      .run(
        "injector" -> injector,
        "foo" -> injector.instance[FooService]
      )
    finally close()
  }
}

Finally, we instantiate the console. Note that we are exposing the Guice dependency injector so we can lookup any component of our service in the console. For common things we can directly bind then to a name, exactly like the sample FooService is bound to foo.

$ sbt run-with-console
[info] Loading settings from idea.sbt,plugins.sbt ...
[info] Loading global plugins from /Users/sebastian.ortega/.sbt/1.0/plugins
[info] Updating ProjectRef(uri("file:/Users/sebastian.ortega/.sbt/1.0/plugins/"), "global-plugins")...
[info] Done updating.
[info] Loading settings from plugins.sbt ...
[info] Loading project definition from /Users/sebastian.ortega/Repositories/foo-service/project
[info] Loading settings from build.sbt ...
[info] Set current project to custom-feed (in build file:/Users/sebastian.ortega/Repositories/foo-service/)
[info] Compiling 12 Scala sources and 0 Java sources to /Users/sebastian.ortega/Repositories/foo-service/target/scala-2.12/classes ...
[info] Done packaging.
Compiling (synthetic)/ammonite/predef/replBridge.sc
Compiling (synthetic)/ammonite/predef/DefaultPredef.sc
Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
Compiling /Users/sebastian.ortega/Repositories/letgo/custom-feed/(console)
Welcome to the Ammonite Repl 1.4.0
(Scala 2.12.7 Java 1.8.0_161)
If you like Ammonite, please support our development at www.patreon.com/lihaoyi

@  foo.checkHealth.await
res1: Health = Health.Sick("cannot reach server")

@

Et voilà, no need to miss the Rails console anymore.

Remote Ammonite powers

The other interesting way to integrate Ammonite into your service is as a remote console. That way you can connect to a remote server in the stating or even production machines and fiddle with them. Maybe not a very good idea… but power to the people!

I’m going to demonstrate this approach by introducing a pluggable Guice module that you can include or not in your server to enable or disable the server.

We will need an additional dependency but this time will have compile scope instead of test scope.

libraryDependencies ++= Seq(
  "com.lihaoyi"  % "ammonite-sshd" % versions.ammonite cross CrossVersion.full
)

And a module named AmmoniteServerModule under src/main/scala:

package ammonite.integration

import javax.inject.Singleton

import ammonite.sshd.{SshServerConfig, SshdRepl}
import ammonite.util.Bind
import com.google.inject.Provides
import com.twitter.inject.{Injector, TwitterModule}
import org.apache.sshd.server.config.keys.DefaultAuthorizedKeysAuthenticator

import com.yourcompany.project.modules.FooModule

object AmmoniteServerModule extends TwitterModule {

  private val prelude = """
import scala.concurrent._
import scala.concurrent.duration._

implicit class ExtendedFuture[A](val f: Future[A]) {
  def await: A = Await.result(f, Duration.Inf)
}
"""

We have some elements in common: same prelude and the class is in the ammonite.integration package. However, this is an independent Guice module.

  @Provides @Singleton
  def providesAmmoniteServer(injector: Injector, foo: FooModule): SshdRepl = {
    val config = SshServerConfig(
      address = "localhost",
      port = 22222,
      publicKeyAuthenticator = Some(new DefaultAuthorizedKeysAuthenticator(true))
    )
    new SshdRepl(
      config,
      predef = prelude,
      replArgs = List(
        Bind("injector", injector),
        Bind("foo", foo)
      )
    )
  }

The SSH server is defined as an application singleton (@Provides @Singleton) and configured to listen in localhost on the port 22222 honoring ~/.ssh/authorized_keys as system’s SSH server. Note also, how the injector and some services can be also bound for the session.

  override def singletonPostWarmupComplete(injector: Injector): Unit = {
    super.singletonPostWarmupComplete(injector)
    injector.instance[SshdRepl].start()
  }

  override def singletonShutdown(injector: Injector): Unit = {
    injector.instance[SshdRepl].stop()
    super.singletonShutdown(injector)
  }
}

We need also to start/stop the server using some lifecycle callback methods.

With this in place, we can SSH and enjoy of the same interactive experience as before but potentially from the other side of the world:

$ ssh localhost -p 22222
Compiling (synthetic)/ammonite/predef/interpBridge.sc
Compiling (synthetic)/ammonite/predef/replBridge.sc
Compiling (synthetic)/ammonite/predef/DefaultPredef.sc
Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
Compiling /Users/sebastian.ortega/Repositories/letgo/custom-feed/target/pack/(console)
Welcome to the Ammonite Repl 1.4.0
(Scala 2.12.7 Java 1.8.0_161)
If you like Ammonite, please support our development at www.patreon.com/lihaoyi

@ val bar = injector.instance[BarService]
bar: BarService = com.foo.BarService@26cfe278

@ bar.someAction.await
res1: String = "hello world"

Scala mangling versus Mockito

Fri, 19 Oct 2018 00:00:00 +0000

One of the main strengths of Scala is that you can parasite the ecosystem of libraries of Java and infrastructure and tooling of the JVM¹. Of course, this is not a free lunch as some gotchas of Scala come from that permeability to Java (like nullability, universal equality and the like).

Interop, or the ability or use Java code from Scala and vice versa, works reasonably well in the Scala-calling-Java direction and not so well in the opposite direction and that is a conscious design decision of the language. To understand why, we need to deep dive in how Scala and Java share the JVM.

Name mangling

In order to execute in the JVM, scalac is forced to compile to the same bytecode and artefacts that javac. Scala concepts that can be mapped directly to Java are no surprise. For instance, you are going to see something like def square(n: Int): Int as int square(int n) from Java. But, what about concepts that cannot be cleanly mapped?

final case class Point(x: Int, y: Int)

object Point {
  val Origin = Point(0, 0)
}

This little snipped is going to be compiled into two classes: an unsurprising Point.class and a more interesting Point$.class. Scala’s compiler needs to generate a class for the companion object and it’s using the reserved character $² to avoid colliding with user defined names. This process is called name mangling and is at least as older as compilers are.

Using javap we can take a look at the class files:

$ javap 'Point$.class'
Compiled from "Point.scala"
public final class mockpost.Point$ implements scala.Serializable {
  public static mockpost.Point$ MODULE$;
  public static {};
  public mockpost.Point Origin();
  public mockpost.Point apply(int, int);
  public scala.Option<scala.Tuple2<java.lang.Object, java.lang.Object>>
    unapply(mockpost.Point);
}

After looking at this it is evident³ how to use the companion object from Java.

Point origin = Point$.MODULE$.Origin();
System.out.println(String.format(
        "Origin is at x=%d y=%d", origin.x(), origin.y()));

Leaky abstractions are going to leak

When mangling is a leaky abstraction? If you limit yourself to calling Java code from Scala you need to consider mangling almost never. Almost.

Recently I got bitten by this leaky abstraction⁴ while using Mockito from Scala. For the sake of simplicity (and to avoid exposing my employer’s IP), let’s see that we have this service interface:

trait Service {
  def request(id: String,
              tags: Map[String, String] = Map.empty): List[Int]
}

We want to unit test this class that gets one instance of the service injected:

final class OtherService(service: Service) {
  def request(id: String, maybeTags: Option[String]): Int =
    (maybeTags match {
      case Some(tags) => service.request(id, parseTags(tags))
      case None       => service.request(id)
    }).sum

  private def parseTags(tags: String): Map[String, String] =
    (for {
      pair <- tags.split(",").filter(_.nonEmpty)
      Array(key, value) = pair.split(":")
    } yield key.trim -> value.trim).toMap
}

Note that we have two cases to test: when tags are passed explicitly and when a default argument is used. Let’s say we write two test cases to cover both cases:

"Mocking service" should "work when all arguments are explicitly passed" in {
  val service = mock[Service]
  when(service.request("id1", Map("k1" -> "v1", "k2" -> "v2")))
    .thenReturn(List(42))
  val otherService = new OtherService(service)

  otherService.request("id1", maybeTags = Some("k1:v1, k2:v2")) shouldBe 42
}

it should "work when using default arguments" in {
  val service = mock[Service]
  when(service.request("id1", Map.empty)).thenReturn(List(42))
  val otherService = new OtherService(service)

  otherService.request("id1", maybeTags = None) shouldBe 42
}

What do we get when running them? The first passes but the second produces a null pointer exception out of the blue.

[info] Mocking service
[info] - should work when all arguments are explicitly passed
[info] - should work when using default arguments *** FAILED ***
[info]   java.lang.NullPointerException:
...

This kind of problem can be a huge time hole as the source of the exception is literally invisible, in code we cannot see. Let’s use javap on the service (slightly cleaned up for readability):

public interface Service {
  public abstract List<Object> request(String id, Map<String, String> tags);
  public static Map request$default$2$(Service service);
  public Map<String, String> request$default$2();
}

Wow, default arguments become new “invisible” methods with a mangled name in the form <method_name>$default$<argument_pos>$⁵. And we are not telling Mockito to what to return in that case… so we get null.

Workarounds and solutions

There are several unsatisfactory workarounds we can use to overcome this gotcha:

We can avoid using default arguments in our production code. This is difficult to enforce and frankly depressing.
We can expect nulls when mocking the main method of the service. Be prepared to write an apologizing comment in your test.

it should "work when using default arguments" in {
  val service = mock[Service]
  // We are going to receive null instead of the default argument.
  // Sorry 'bout this. Trust me!
  when(service.request("id1", null)).thenReturn(List(42))
  val otherService = new OtherService(service)

  otherService.request("id1", maybeTags = None) shouldBe 42
}

We can mock the default argument… from Java! I’m mentioning it just for completeness (makes my eyes bleed!).

public class MockitoHelper {
    public static void setupDefaultArguments(Service mock) {
        Mockito.when(mock.request$default$2())
            .thenReturn(Map$.MODULE$.empty());
    }
}

However, the real solution to this problem⁶ is to use a mocking library designed for Scala rather than Java. With ScalaMock both test cases will work seamlessly:

"Mocking service" should "work when all arguments are explicitly passed" in {
  val service = mock[Service]
  (service.request _)
    .expects("id1", Map("k1" -> "v1", "k2" -> "v2"))
    .returning(List(42))
  val otherService = new OtherService(service)

  otherService.request("id1", maybeTags = Some("k1:v1, k2:v2")) shouldBe 42
}

it should "work when using default arguments" ignore {
  val service = mock[Service]
  (service.request _)
    .expects("id1", Map.empty[String, String])
    .returning(List(42))
  val otherService = new OtherService(service)

  otherService.request("id1", maybeTags = None) shouldBe 42
}

Check out this repo if you want to run this and the previous sample snippets.

In conclusion, it’s wonderful to piggy-back on the JVM ecosystem but sometimes we got bitten by leaky abstractions. It’s in those cases when knowing what’s under the hood can save you hours of frustration.

You can also parasite the Javascript ecosystem with scala.js or fly by yourself with scala native ↩
You can actually try to use it in your class and variable names but it might interfere with mangling. We can say that it is “reserved” in the spirit of JavaScript’s undefined ↩
Evident but “ugly as a bare foot” as we say in Spanish ↩
Almost all abstractions are leaky. Almost. ↩
These names can grow so large when you have lambdas within lambdas that you might hit the path length limit of Linux ↩
Apart from not using mocks… ↩

Viterbi search

Sat, 29 Apr 2017 00:00:00 +0000

The Viterbi algorithm is an interesting example of a dynamic programming algorithm that lies between memoized Fibonacci and Levenshtein distance in complexity. Like any other dynamic programming algorithm it is characterized by having an asymptotic complexity way lower than the naïve approach thanks to being able to divide the task at hand into subproblems that tend to be repeated and are cached.

In this case, the context of the Viterbi algorithm is a Markov chain that models some system as a directed graph of states with some transition probabilities. Reimplementing the example at the relevant wikipedia page in Scala, let’s say that a patient can be healthy or afflicted by a fever:

sealed trait Health
object Health {
  case object Good extends Health
  case object Feverish extends Health
}

However, we are not in a hospital and we cannot know for sure if the person is really healthy or not, but we can observe the symptoms the person experiments day by day:

sealed trait Symptom
object Symptom {
  case object Normal extends Symptom
  case object Cold extends Symptom
  case object Dizzy extends Symptom
}

The Viterbi algorithm can be used only when we made some simplifying assumptions. First, the link between the health status and the symptoms can be modelled as a simple discrete probability distribution. Second, the health status only depends on the status of the previous day (no more memory) by another simple discrete probability. This is known as a hidden Markov chain and it is amenable to beautiful¹ charts:

Note how states are hidden because of the indirection layer that the symptoms represent.

We can represent the same setup not so beautifully by means of some DSLish modelling in Scala:

val hidden = HiddenMarkovChain[Health, Symptom](
  prevalence = Distro(Health.Good -> 0.6,
                      Health.Feverish -> 0.4),
  hiddenChain = Map(
    Health.Good -> Distro(Health.Good -> 0.7d,
                          Health.Feverish -> 0.3d),
    Health.Feverish -> Distro(Health.Good -> 0.4d,
                              Health.Feverish -> 0.6d)
  ),
  observationDistros = Map(
    Health.Good -> Distro(Symptom.Normal -> 0.5,
                          Symptom.Cold -> 0.4,
                          Symptom.Dizzy -> 0.1),
    Health.Feverish -> Distro(Symptom.Normal -> 0.1,
                              Symptom.Cold -> 0.3,
                              Symptom.Dizzy -> 0.6)
  )
)

Now we have all the pieces in place to really explain what the Viterbi algorithm does: given a sequence of symptoms, it searches for the most likely sequence of hidden states. For instance:

hidden.viterbiPath(List(Symptom.Normal,
                        Symptom.Cold,
                        Symptom.Dizzy)) shouldBe
  List(Health.Good, Health.Good, Health.Feverish)

We might solve this problem by brute force by just running all the possible sequences of states (2³, sⁿ in general) to know their probabilities, then finding the probability of each sequence to produce the actual observations. Too bad we are talking about exponential complexity making it impractical except for the smallest cases.

The Viterbi search runs in just O(ns²) where n is the length of the sequence and s is the number of possible hidden states, a polynomial bargain!

The trick, as in other dynamic programming algorithms is avoiding rework while doing the search. The central fact making it possible is the Markov memoryless property: state transitions depend just on the previous state. This means that we can decompose the problem into:

Computing the most probable paths of length n-1 ending in each one of the possible states.
Finding out what is the most likely extension of a path getting to each one of the possible states.

If we memoize this recursive function we get that attractive polynomial execution time.

An interesting point about the memoryless property of the Markov chain is that new evidence can change our interpretation of the previous states even if each state depends just on the previous state. Let’s illustrate it with a chain with different, more extreme probabilities:

In this scenario, having low fever for a day is not evidence strong enough to think that the patient is ill:

hidden.viterbiPath(List(Symptom.Normal,
                        Symptom.LowFever)) shouldBe
  List(Health.Good, Health.Good)

However, if there is high fever the third day our interpretation of what is most likely changes:

hidden.viterbiPath(List(Symptom.Normal,
                        Symptom.LowFever,
                        Symptom.HighFever)) shouldBe
  List(Health.Good, Health.Incubating, Health.Ill)

The implementation of the algorithm is surprisingly brief if you have a monadic discrete probability class to support it. Check the code, or better, implement it yourself to solidify your understanding of it.

At least it is easier to grasp than loads of tables with numbers. ↩