Fulfilling a Pikedream: the ups of downs of porting 50k lines of C++ to Go.

Rob Pike, one of the creators of the Go language, stated that he expected the language to be adopted by C++ programmers, a prediction that hasn’t been realised. Recently however at the HFT firm where I work, the success of a team’s move from Python to Go for some pieces of non-speed-critical infrastructure led to the decision to attempt a slimmed-down Go rewrite of a somewhat-throughput-critical 50k LOC C++ server. The old C++ server used the same techniques and libraries used in our latency-critical C++ trading software, where every microsecond matters, and this degree of performance simply wasn’t needed. It was hence thought that a rewrite in Go, using the language’s native scheduler rather than the hyper-optimised C++ framework used by the autotraders, would be easier to maintain. I was tasked with the rewrite.

The tl;dr
In business terms, the project was a success: completed ahead of schedule, performing acceptably, and less than 10k LOC long (this massive LOC reduction was of course partially due to the removal of features that were deprecated or not needed by the team behind the rewrite). In personal terms however I feel the outcome was suboptimal, in the sense that I wrote two to three times as much code as would have been needed in a language with parametric polymorphism. Some of this was due to type-safety: Go forces a tradeoff to be made between verbosity and type-safety, and I settled somewhere in the middle; it could have used less code and been less type-safe, or used more code and been more type-safe.

Now, the pros and cons, starting with the pros.

The pros

Emacs! With plugins for autocomplete, jump-to-definition, error checking upon save, intelligent refactoring, and GoTest integration, programming Go in Emacs offers practically everything one’d expect from a good IDE, with the bonus of super-easy customisation and extensibility via Elisp. Since one of my reasons for getting into programming was the opportunity to get paid to use Emacs, this is definitely a huge plus.

Goroutines! Go makes message-passing based concurrency, which I personally find the easiest form of concurrency to reason about, super simple to use. It also allows parallel/async code to be written in the exact same way as concurrent code, simply by setting GOMAXPROCS to 1. The only other languages I know with built-in lightweight thread schedulers are Erlang/Elixir and Haskell, with the former lacking static typing and the latter lacking management-willing-to-use-ability.

No inheritance. I’ve personally come to view inheritance-based OO as somewhat of an antipattern in many cases, bloating and obscuring code for little benefit, and Go makes this kind of code impossible to write. I suspect this was Rob Pike et al’s motivation for designing Go the way they did: there was a bunch of Java/C++ at Google that was written as if Enterprise Fizzbuzz was a positive role-model, and they wanted to spare themselves from having to deal with such code in future. That being said, in spite of being legacy code the use of inheritance in the old C++ server was pretty sane, and it could easily have been rewritten to use a more modern style.

Readability. I always found the Go code I encountered quite easy to read and understand, both our code and external code. Some of the C++ I encountered, in contrast, took hours to fully comprehend. Go also forced me to write readable code: the language makes it impossible to think something like “hey, the >8=3 operator in this obscure paper on ouroboromorphic sapphotriplets could save me 10 lines of code, I’d better include it. My coworkers won’t have trouble understanding it as the meaning is clearly expressed in the type signature: (PrimMonad W, PoshFunctor Y, ReichsLens S) => W Y S ((I -> W) -> Y) -> G -> Bool”.

Simple, regular syntax. When I found myself desiring to add the name of the enclosing function to the start of every log string, an Emacs regexp find-replace was sufficient, whereas more complex languages would require use of a parser to achieve this. The simple syntax also makes code-generation a breeze, be it generation by Emacs macros or Go templates. Emacs + Go == parametric polymorphism: not only can macros be used to speed up the process of generating the “copy-paste” code that Go’s lack of parametric polymorphism requires, if functions are written right then regex can also be used to update all “copy-pasted” functions simultaneously, making updating the code for fooInt, fooFloat and fooDouble almost as easy as updating foo<t> in a language that supports <t>. The downside is that, while Emacs macros and regex can write and modify Go code in such a manner as to emulate parametric polymorphism, it’s still not as readable or concise as actually-polymorphic code, and of course is not easily maintainable by someone lacking familiarity with regex or an extensible editor like Emacs.

Built-in, effective templating. Go’s text/template package can easily be used to Generate new Go code. This allows IO to be used during code generation: we had for instance a library for interacting with a particular service that was generated from an XML schema, making the code perfectly type-safe, with different functions for each datatype. In C++, IO cannot be performed at compile time, so such schema-driven code generation would not be possible. Languages allowing compile time IO include F#, which has compile time IO via Type Providers, Idris, which also has Type Providers, Lisps, which can do IO in macros, Haskell, which has an IO -> Q compile time IO function in Template Haskell, D, which can use `import` to read files at compile time, Nimrod, which has functions for compile time file IO, Elixir (and possibly Erlang?) which can do arbitrary IO via macros, and Rust, which can use libsyntax to perform arbitrary computations and IO at compile time.

The cons

Stockholm syndrome. I just argued above that generating Go code with templates is superior to compile time metaprogramming in C++ due to allowing IO, which of course is a stupid argument, since one could just as easily generate C++ code using a separate C++ program that does IO.

Lack of parametric polymorphism! I’ve read many people saying this isn’t a problem in practice, well in this particular case it was a huge problem. I’m confident that a C++ translation of the new Go code would be less than half the LOC of the Go version and more type-safe, due to C++’s polymorphic functions and types. A Haskell rewrite would need even fewer LOC, and if I’d been allowed to write it in Clojure I suspect the whole thing could have been expressed in fewer than 1000 lines of macros (although I’m not sure how debuggable or maintainable that would have been…).

Sacrificing type safety. We use extension attributes for the various protobuffer messages that the server handles, and I originally intended to distinctly type these, so that for instance a FooExtensionAttribute could not be used on a Bar. Go’s lack of parametric polymorphism and generic types however meant that this would have involved a significant amount of code duplication, so I ended up settling with just a single ExtensionAttribute type, with the type system not checking that it was used to extend the appropriate message.

Binary sizes. If one uses code generation to Generate a type-safe API, with distinctly typed accessors and whatnot for each datatype, then one can easily wind up with over 100,000 lines of Go and 30mb+ binaries. Compile times are also slower; over 10 seconds in this case, although this wasn’t a significant issue as the library could just be compiled once to a static library and then statically linked.

Kernel compatibility. I may well be the first to make this complaint, but when for Kafkaesque reasons you have to deploy to an old kernel, it’s somewhat disappointing when the newest Go version requires features not supported by the kernel and you’re forced to stick to an older, slower version of Go.

Conclusion

Go is a double-edged sword: it forbids complex abstraction, both bad and good. The worse the abstraction that you and your colleagues are likely to use, the better a language is Go, and vice-verse (ultimately depending on what is considered ‘good’ and ‘bad’ abstraction).

Advertisements
This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

20 Responses to Fulfilling a Pikedream: the ups of downs of porting 50k lines of C++ to Go.

  1. Mike S. says:

    Thanks for the real world example. Cool. How has the performance been? I realize you said the code isn’t performance-critical, but I wonder what you lost in the move from C++ to an older version of Go (since you’re running on a kernel that doesn’t support the latest version).

    I’ve become a big fan of Clojure, but I haven’t found a good application for it at work and I don’t have the energy to work on it in my spare time.

    • logicchains says:

      The throughput of the Go program is quite competitive with the C++ one, although the server’s IO-bound so most of the time is just spent in socket write/read syscalls. The latency is at least an order of magnitude worse, due to Go’s garbage collector, which is amplified by the use of an older Go version. If the server was latency-critical I don’t think it could have been written in Go, at least not until the new GC planned for 1.5 or 1.6 is released (assuming we could upgrade to a newer kernel by the time its released).

      I think Clojure would be great for where you can imagine exactly what code you want to write, and it’s lots of similar code operating on similar data structures. Then you could use macros to just generate the code you were going to write, instead of writing it yourself.

  2. Just felt like saying that you mentioned that it would not be possible in Rust to do code generation as you noted, but I am pretty sure you are wrong.

    Rusts macros (shown with the ! at the end of an ident) do not have that power, but using libsyntax (which is unstable, so cant be used in stable Rust, unfortunately for now), you can basically do whatever you want in compile time, including eat someones laundry, if given the correct API.

  3. Alan Osborne says:

    Take home message: Rejoice if the pointy haired boss would like you to use Go instead of one of the other entrenched corporate friendly languages, but stick to good languages if you have a choice.

  4. Panda says:

    Regarding D, doesn’t import(file) do what you’re asking for?

    string someXML = import(“foo.xml”);
    // ctfe functions can work with someXML

    Otherwise, while it doesn’t have goroutines per se it does have fibers, concurrency message passing, *powerful* template and metaprogramming support (the euphism being “compile-time polymorphism”), and is subjectively generally nicer to work with than C++. :/

    • logicchains says:

      I wasn’t aware import could be used like that; I’ve removed that mention of D and added it to the languages that can do compile time IO.

      D has fibers but it doesn’t yet have the Vibe.d scheduler in the stdlib. As far as I’m aware it also doesn’t yet have pre-emptive scheduling, only cooperative scheduling, requiring “yield” statements to be added manually.

  5. Hello says:

    Do you publish your Emacs config anywhere? I’m interested in how you’ve configured it for Go.

  6. Alex says:

    I’m no Go expert but I think gofmt allows more syntactically aware refactoring than you’d be able to achieve with regexp. A quick search for a couple of good examples that I’d read didn’t turn them up but here’s a starter

    http://golang.org/cmd/gofmt/

    • logicchains says:

      I used that for standard refactoring, renaming etc., but I couldn’t find a way to do something like add the function name to the start of every log string using gofmt.

  7. thanks for this awesome write-up.
    binary size is a real issue & nuisance IMO and hope that they will fix soon.

    the witty title made me smile twice 🙂

  8. Idris is another language with compile-time IO. It’s type provider feature lets you run arbitrary effectful computations during type checking, and splice the resulting value into your code.

    • logicchains says:

      I didn’t realise that; I’ve added it to the languages with compile-time IO.

      Any idea how many years away from 1.0 Idris is? I played with it once and it was pretty fun, although it seemed to be lacking a vector type backed by contiguous memory (as opposed to a linked list), and some of my code silently stopped working when I upgraded to the new release (the one with partial evaluation).

  9. anlhord says:

    I want to see the code that would be simpler with parametric polymorphism

    I’m an author of an “unofficial” parametric polymorphism implementation to go.

    It’s available at http://our-gol-842.appspot.com/

    • logicchains says:

      An example, imagine you’ve got a library that takes a table type (an enum) as input, and returns a generic tableData interface that must then be cast to a fooTableData, barTableData or whatever based on the table type passed in. If you have n table types, you’ll need n functions for casting tableData to the n concreteTableDatas and returning the concreteTableDatas (one function for each concrete type).

  10. kirbyfan64 says:

    I have pity on you…

    Did you ever think of trying Nim or Felix? Nim is a Pascal-Python twist that supports Lisp-style macros (and compile-time IO), and Felix has some Go-like concurrency features (channels) but is very unique in its own way. I would prefer them to Go any day.

    As my new email quote says:

    If I were in a 10-story building glass-sided building and forced to write either Go or APL, I’d jump out a window.

    • logicchains says:

      I wasn’t personally responsible for the decision to use Go. I think however Go could only be chosen because we already had a team working and developing APIs for our internal services in Go. There’d be a lot of political resistance to introducing a new language, especially one less well-known.

  11. Thanks for this awesome article!

    >> when the newest Go version requires features not supported by the kernel

    Having not encountered such a limitation makes me wonder what your use case is. Can you give an example perhaps?

    • logicchains says:

      An example: some important piece of software running on production machines works best on an old version of RHEL, Go 1.4 doesn’t build on the kernel of this version of RHEL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s