Rust dependencies scare

(This is the fourth instalment of my “Newsboat Rust rewrite” series. If you’re new to the series, start here.)

Newsbeuter have been mature and stable for a long time, and Newsboat inherited that. With that stability also comes a long “tail of support”: our latest release can be built with an old C++ compiler and linked to old libraries, all orchestrated by any version of GNU Make. And it’s not just a theoretical brag: with 111 Newsboat packages known to Repology, you can bet your bottom dollar that some of them weren’t built with the latest GCC 9, weren’t linked against the latest cURL, and didn’t have docs generated by an up-to-date Asciidoc.

Unfortunately, we can’t have that sort of support with Rust: the core language and library alone reached its 1.0 milestone a year after the release of the oldest C++ compiler we support. The rest of the ecosystem is necessarily even less mature.

And yet, I tried to provide that level of support. Mozilla forced distributions and package repositories to adopt Rust in order to build Firefox, and “everybody” ships Firefox, right? Thus, I knew I can count on some version of Rust being available. What I didn’t know, though, is how those versions are going to be updated. Will distros keep up with the six-week train, or will they update only once in a while, or what?

Another consideration was point-release distros, like Debian stable. Once shipped, those will only receive security updates. It’s inevitable that some Newsboat user will be running that, and will try to build the program with a two-year-old compiler. (I should know; we do receive questions on how to build Newsboat on some half-forgotten CentOS that’s stuck with GCC 4.8).

That’s why I started with the lowest version of Rust that worked (1.26.0) and stuck with it for 1.5 years. In my mind, the downsides of this decision were pretty clear and manageable:

• we miss out on new Rust features;
• as a consequence, we miss out on new features in crates, because they can’t be built with older compilers;
• we increase the risk of build breakage, as new compilers might refuse to build older crates e.g. due to soundness holes being patched. (If this statement doesn’t make sense to you, read up on Cargo.lock. In a nutshell, every Rust application ships a list of known-good versions of its dependencies, and the end-users build with those.)

I think this worked out well for us. Apart from a few private discussions with people who didn’t want Rust in the build chain at all, we haven’t received any negative feedback. Seems like everyone managed to get a new-enough compiler.

Over the last 1.5 years, two of the aforementioned downsides did realize themselves. We—or me, at least—miss some of the conveniences of newer Rust releases, and I anticipate we’ll miss async when we get to port our feed-fetcher. We are still with Nom 4, and probably lag on other updates as well (I don’t even track them¹ since we can’t update). Fortunately, we haven’t seen any breakage—yet.

What we did see were warnings. Completely obvious in retrospect, but I somehow managed to miss that possibility. Rust 1.33.0 deprecated a few functions on str, and ever since then, we’ve been looking at a screenful of warnings in all our builds. Recently, I got fed up with them and decided to look at the compiler situation again.

As it turns out, these days most distros ship one of the three latest Rust releases. Great! Perhaps Newsboat can start bumping its minimum supported Rust version on a schedule now? I proposed that to other contributors, and the discussion unearthed a figurative can of worms.

Newsboat today is an iceberg: it directly depends on 19 crates, but with transitive ones the count reaches 102. One hundred and two! A quick survey of the direct ones shows that at least one crate specifies minimum Rust version wa-ay above our own requirement; in other words, it’s a miracle it builds. Many others don’t specify requirements at all, and test only a few recent Rust releases on CI. Yet again, it’s a miracle those crates work with our older compiler.

This is what inspired the title of this post. I am moderately worried about the number of dependencies, but their state is absolutely scary. The fact that my software builds partly by chance is scary. The thought that tomorrow, some crate will break compatibility with Rust 1.26, and we’ll have to suddenly bump our requirements to within a few releases of current stable, is worrying. The possibility of it being forced by a security issue is depressing.

To me, that drove the point home: Rust ecosystem is nowhere as mature as other production-level languages (d’oh!). Even with a reasonably small number of dependencies, the “tail of support” is something like 18 weeks, because many crates are only tested against the latest three releases. Some only test the latest stable, so the “tail” can be as short as 6 weeks. Nowhere near those five years we had with C++.

Rust was conceived as an industrial language, though, and industries generally want stability; thus I’m pretty sure the situation will improve over time. For now, I’m limiting Newsboat’s exposure by going through with the “scheduled bumps” plan. It’ll be properly announced at the end of March with the 2.19 release. We’re actually going to switch master branch to Rust 1.40 simultaneously with the release, so I really hope there won’t be any negative feedback on this—otherwise, we might have to undo some of the work done for 2.20, as it’ll depend on “new enough” compiler. (I’m saying that with such confidence because one of the outstanding PRs seem to require Nom 5, which in turn requires Rust 1.31+).

I hope this won’t dissuade anyone from porting their existing project to, or starting a new one in Rust. I certainly am not stopping the Newsboat rewrite because of this. The point of this tale is to better inform your decisions, so you’re more prepared than I was.

Update 02.03.2020: Discussion on Mastodon.

Update 03.03.2020: Discussion on Lobsters.

Update 05.03.2020: I have to admit that the discussion on Lobsters sewn a seed of doubt in me, but I’m not ready to expand on that right now. Suffice it to say that the Rust project is aware of the issue, and have been pondering it for a long time (kudos to BurntSushi for all the links):

1. Aaron Turon provides an overview of this and related problems;

2. here’s a Rust LTS proposal, which was ultimately postponed (with an extremely articulate summary);

3. here’s a tracking issue for “minimum supported Rust version” (MSRV) field to Cargo.toml. That field won’t solve the problem I describe, but it will alleviate some of the maintenance pain and will provide a nice stepping stone to true LTS;

4. finally, here’s a Red Hat announcement about Rust in RHEL. Instead of freezing some Rust version in time for ten years, Red Hat opted to update it every three months.

Also worth noting is a curious interaction between Rust updates and semantic versioning. If a new version of some software requires a newer compiler, should the said software bump its major version? Opinions differ.