OCaml Packaging: Down the Rabbit Hole

It probably doesn’t come as a surprise that I am a big fan of the OCaml language. I’m using it for most of my scientific research as can be seen on Github. When I started out working with OCaml back in the old days, there was not much more than the Ocaml compiler. This required you to write extensive and essentially – what I like to call write-only – Makefiles like this one here.

Also, there was no good way of handling dependencies of packages which required us at the time to use git submodules of some sort without any particular support which didn’t make distribution, dependency management and compilation any easier.

So when Martin Lange and I decided to give PGSolver a major overhaul recently, I decided to see if time has caught up with OCaml. Boy was I in for an adventure.

First, I wanted to figure out compilation. Instead of writing these extensive Makefiles, there is a tool called OCamlbuild that can do most of the compilation for you, particularly finding all required source files and compiling and including them in the right order. Now how do you install OCamlbuild? Looking at their Github page, it seems like you need another tool called OPAM in order to install OCamlbuild.

OPAM turns out to be a package manager for OCaml, so it is reasonable to install that.

Just using OCamlbuild seemed good enough for updating PGSolver, as I didn’t want to change too many things in one go. Since package managers often want your git repositories to be somehow in sync with the package registry, I didn’t wanna mess with the submodules just yet by turning them into packages.

However, there is a single repository, Camldiets, that seems like the perfect test case for figuring out how to create and publish a package. It only has a single module and no dependencies on its own. Should be very straight-forward to create a package for this, right?

Well, not so fast. There also seems to be the so-called OASIS project that, according to their website, allows you to integrate a configure, build and install system in your OCaml projects. While this is vague enough that I’m not completely sure what that means exactly, it does sound awesome somehow, so what the heck.

So let’s set up a OASIS project from scratch, starting out with our Camldiets repository. After installing OASIS (via OPAM, of course), we let it generate a bunch of boilerplate files by calling oasis quickstart and answering a few questions. When it asks you which plugins you wanna install, you have the choice of installing METAStdFiles and DevFiles. Well, no idea, probably none, so let’s continue. It then asks whether – among other things – I wanna create a library or a source repository. Well, no idea either, as long as my package is properly compiled and linked to a depending project, I don’t care. Let’s try library for now.

This creates an _oasis file that essentially contains all the information you just supplied during quickstart. Next, you’re supposed to call oasis setup. This now generates a whole slew of files. Hard to say whether they should be gitignored or not, but I guess I’ll worry about it later. Then, you’re supposed to call ocaml setup.ml -configure which seems to check all kinds of things. Moving on. Finally, we call ocaml setup.ml -build which seems to use ocamlbuild to build our library. Whatever that means exactly.

By now we have so many generated files in the repository that I’m starting to worry. Some of them must be ignored, I’m sure, particularly the _build folder that is being generated by ocamlbuild. I’m now including the “official” gitignore file for OCaml.

So what did I want to do again? Package everything up in OPAM, right. Does OPAM accept my OASIS-enriched repository? Unfortunately it doesn’t, and it requires it’s own set of proprietary meta data. Great. Luckily, there is another tool called oasis2opam that is supposed to convert your freshly created OASIS project into an OPAM-compatible packaging. So we install oasis2opam via OPAM and run it via oasis2opam –local. It immediately complains about our OASIS configuration leaving much to be desired, so we shamefully try to add some more key-value pairs to it and run it again.

This results in a syntax error, and – as it is common with OCaml – the error message is not particularly useful. It is just telling us that there is a syntax error somewhere in the OASIS file. Sigh. Once this is cleared up after some googling, the OPAM files are being generated.

Let’s see if this at least works locally now. According to the docs, we can do this by calling opam pin add camldiets . -n. Now we want to include the package somewhere else. We change into an empty folder and try to install it via opam install camldiets –verbose, but not so fast: Cannot find file ‘src/META’ for findlib library Camldiets

What? But why? Apparently, there is another tool called findlib that needs its own meta data describing the repository. Christ. Down the rabbit hole once again. Okay. So how do I now get the META file? After doing some research, it seems like adding META to the Plugins section of OASIS might seem to do the trick. We’ll go through the chain of generating commands again, and try to install the package one more time, and now it seems to work. We can even include it in a local test project. Success!

Now how do we get this package into OPAM? Obviously, we need another tool for this, opam-publish, that we install via OPAM. How deep can this hole be? Publishing repositories via OPAM seems to require a couple of steps for sure.

First, tag the repository on Github. Then call opam-publish prepare https://github.com/tcsprojects/camldiets/archive/v0.2.tar.gz which generates an awkward subfolder called Camldiets.0.2. You’re then supposed to submit this folder by calling opam publish submit ./Camldiets.0.2 which in turn asks you for your full Github credentials. Yikes, that’s giving me the chills! According to most sources I was able to find online, this call usually fails in any case. And sure enough it failed. Would have been too easy, wouldn’t it?

The manual way is to fork the full OPAM registry repository, add the generated folder to it and issue a pull request. Jeez. Okay. Once this is done, a variety of continuous integration tests kick in and you’re left waiting for them to finish and for somebody from OPAM to have a look at it. Fingers crossed that this can be pulled in as is. I’ll update this post once the pull request is updated.

Publishing and packaging things with OCaml seems much less straight-forward than most build-and-packaging systems for other languages, but I suppose that’s just the way it is. After all, this is definitely an improvement over the fugly Makefiles and submodules we were using a decade ago. Hopefully, updating packages will be less of a fuss than creating a new one from scratch.

Update: that pretty much worked.

Leave a Reply