Improve build reproducibility for Meet

meet
sdk

#1

Jitsi Meet uses a various third-party dependencies. These dependencies are obtained from various sources, mostly: the npmjs registry, and Github.

It turns out that Github download locations are not as stable as you’d think. In the past few months, we’ve had at least two dependencies that stopped to be available on their Github location. Although the Jitsi team is pretty fast in finding solutions, these solution are typically applied to HEAD only. Reproduction of build artifacts of older code is, and remains broken.

To me, this is a major problem, as it requires one to re-base development efforts of our proprietary products. Not only do we need to retest to see if our platform is compatible with all other changes that we pull in, we’re often pulling in unrelated, and possibly undesirable changes, or things that simply break our build and require a lot of effort to resolve (I’m currently battling two: the JSCore update, as well as the React Native version upgrade, which breaks proprietary code for a reason I still have to identify) In all, this causes us to spend a lot of energy, which competes with other efforts (that each of their own priorities and deadlines).

I realize that this all comes with the territory of doing software development. Also, there are good reasons that we’d eventually want to rebase our code anyway - but the current status forces us to do so immediately, as we cannot effectively continue our development efforts before we’ve restored a functional build. Instead, we’d love to make the “when and where” choice ourselves.

The text above is to illustrate how important the reproducibility of the Jitsi Meet project is over time, to us (and likely, others). I’d love to see this being improved. To do so, I’d love for the project to stop using Github repositories that are not under control of the Jitsi team. Making this change would not necessarily be hard: at the time of writing, there are only a handful of dependencies that are obtained from a non-Jitsi Github repository. If these were to be forked into the Jitsi repository, and the package.json modified to obtain these dependencies from there, things would be improved dramatically.

Additionally, a nightly build in the continuous integration system for each release could be created. This would allow us to identify reproducibility-issues, and perhaps allow for LTS-like branches to be created.

The latter is a long shot on my part - I’d be perfectly content with just the Github change, as that will prevent a bucketload of trouble on my end.


#2

What do you mean with this, what kind of nightly build?


#3

I suggest to periodically verify that we can still build the project from the source that has been marked as ‘release’ (for instance, to check if none of the dependencies have become unavailable).

Jenkins can trigger a build from scratch (including having to download all dependencies). This build would be, unlike “normal” CI-builds, based on a schedule (every night, or every week), instead of being triggered by a code change.


#4

I see, thanks for the explanation.


#5

I’m really sorry about that. We were about to release an updated SDK when the acquisition happened so things fell through the cracks a bit. In addition, I had to take over that task (publishing the SDK to our maven repo) and had no prior experience, so I’m figuring things out as I go.

Definitely agreed here. We’ve done this for some, but failed to apply it for all dependencies. I’ll get on this.


#6

To clarify my post: I love what you guys are doing, and I’m probably one of the first to admit that things like these happen, in software development. Heck, I, myself, am the cause of many such issues in other projects.

I wrote down my current bottlenecks (both of which have been fixed by now, by the way), only to illustrate the benefit of my proposal: let’s try to avoid a situation where people are forced to upgrade their code base, because some third-party bit of code becomes unexpectedly unavailable.