ls,h

Working from home today. Pretty efficient.
Yesterday I sent a PR to Levin and to Felipe, since Levin is in Israel, and can review my stuff while I'm sleeping. He did not, but still reviewed it pretty early, and I merged it. As a consolation, sent another PR to Felipe. And another to Tom. And another to Josh.

Then Xuehai woke up in Seattle, and approved my PR that I had sent him yesterday. And I merged. It was about adding `Long` type to the list of standard types. Weird that we did not have it. Now he writes: "please deply the code to Dev". Sorry, what? You had a problem in pulling new kind of data, do you want to run your tests first in your branch and make sure that they pass? No, he writes, we'll do it "in production". Hmm, I said. I'm not on pager duty today, let's find who is, and talk to that person. It turned out it's Xuehai. I believe now he needs some training, not sure what kind, and whether he will bother to listen to anybody who is not his manager. Fuck.

Ok, and I was cutting out smaller prs from my big one, but suddenly Tom notifies me that integration tests are failing. Sorry, what? Not on my computer... ok, while I was running stuff with other PRs, my fix with adding `Long` opened a small Pandora box. We had an idiot, about two years ago, a French guy who wrote tons of complicated shit; the complexity was due to his not knowing Scala. Like this:

  val stuff = Try { getStuffOrNullFromSomewhere(key) }.toOption.filter (null !=) // make sure we don't have Option(null)

The right solution is val stuff = Option(getStuffOrNullFromSomewhere(key))

And so on. But the problem I bumped into was that he transformed some data and compared it to a sample binary in test resource. Now that we added `long`, that binary is no good. And I spent a couple of hours trying to produce a new binary - the problem though was that our spark partitioning feature, while working in my tests, does not always do what I expected it to do. I wanted to repartition it, to have just one `avro` file, and could not. So that's another story. I'm afraid I don't have a test case for forced repartitioning, and for the case when just one partition is needed. Have to add it.

Anyway, fixed that problem by comparing string data and ignoring the `long` part. Ouch.

Got pretty exhausted by 7pm. But well, I did a lot when nobody was distracting me all day.