[jitsi-dev] Reproducible build (with git submodules)


#1

See below

Freundliche Grüsse,
Ingo Bauersachs

-- Sent from my tablet

···

Begin forwarded message:

From: Ingo Bauersachs <ingo@bauersachs.me>
Date: 30. Dezember 2015 um 10:14:34 GMT+13
To: Jitsi Developers <dev@jitsi.org>
Subject: Re: [jitsi-dev] Reproducible build (with git submodules)

On 30.12.2015, at 04:48, Boris Grozev <boris@jitsi.org> wrote:

On 28/12/15 20:21, Etienne Champetier wrote:
Hi,

I'm trying to make jitsi (jvb & jicofo) reproducible, ie build all
"SNAPSHOT" packages from sources
I'm using git submodules and a master pom file and i'm now able to build
almost every jitsi package from sources, with no changes in the repos or pom

I've 4 PR pending:
https://github.com/jitsi/jain-sip/pull/1
https://github.com/jitsi/jitsi-universe/pull/3
https://github.com/jitsi/smack_3_2_2/pull/1
https://github.com/jitsi/jicoco/pull/2

And i'm also looking for the source of 3 artifacts (callstats, fmj, jnsapi):

Callstats: https://github.com/callstats-io/callstats.java
FMJ: http://sourceforge.net/p/fmj/code/HEAD/tree/

I don't know about jnsapi.

https://github.com/jitsi/jitsi-maven-repository/issues/3

to try (see my PR when it fails to compile):
git clone --recursive https://github.com/champtar/jitsi-submodules
cd jitsi-submodules
rm -rf ~/.m2
mvn install -DskipTests -Dmaven.javadoc.skip=true -Dgpg.skip=true
-Dassembly.skipAssembly=false

The idea for this repo is to be auto updated and tagged (with jenkins?)
when one of the submodules is updated,
so we can reproduce jvb version 532 if we want

This is very interesting! Thank you for working on it and sharing your efforts!

A related update from our side:
We had a discussion with Lyubomir and other people from the team, and we agree that reproducible builds are something we want to have (again).

The idea we have for fixing the problems with the current maven based builds, as far as I understand it, is to pin the dependency versions in pom.xml and have an automated way of bumping them. One problem we was with this was the number of useless "version bump" commits.

I don't understand why that has to be automatic. Some of our libraries are used externally as well and I think we should start to think about a versioning scheme instead of just releasing build numbers.

We have not considered using git submodules so far (at least I don't know that we have), but they seem to solve alot of problems in a simple way.

Not sure if they are a long term solution (they might be, I've never used them). With all that versioning we should also keep our Debian goals in mind.

Boris, can you please verify Etienne's CLA and merge the bccontrib PR in jitsi-universe if it's ok?

Regards,
Boris

Ingo

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#2

Hey,

Sorry for the delayed response on this one.

See below

Freundliche Grüsse,
Ingo Bauersachs

-- Sent from my tablet

*From:* Ingo Bauersachs <ingo@bauersachs.me <mailto:ingo@bauersachs.me>>
*Date:* 30. Dezember 2015 um 10:14:34 GMT+13
*To:* Jitsi Developers <dev@jitsi.org <mailto:dev@jitsi.org>>
*Subject:* *Re: [jitsi-dev] Reproducible build (with git submodules)*

Hi,

I'm trying to make jitsi (jvb & jicofo) reproducible, ie build all
"SNAPSHOT" packages from sources
I'm using git submodules and a master pom file and i'm now able to build
almost every jitsi package from sources, with no changes in the
repos or pom

I've 4 PR pending:
https://github.com/jitsi/jain-sip/pull/1
https://github.com/jitsi/jitsi-universe/pull/3
https://github.com/jitsi/smack_3_2_2/pull/1
https://github.com/jitsi/jicoco/pull/2

And i'm also looking for the source of 3 artifacts (callstats, fmj,
jnsapi):

Callstats: https://github.com/callstats-io/callstats.java
FMJ: http://sourceforge.net/p/fmj/code/HEAD/tree/

I don't know about jnsapi.

https://github.com/jitsi/jitsi-maven-repository/issues/3

to try (see my PR when it fails to compile):
git clone --recursive https://github.com/champtar/jitsi-submodules
cd jitsi-submodules
rm -rf ~/.m2
mvn install -DskipTests -Dmaven.javadoc.skip=true -Dgpg.skip=true
-Dassembly.skipAssembly=false

The idea for this repo is to be auto updated and tagged (with jenkins?)
when one of the submodules is updated,
so we can reproduce jvb version 532 if we want

This is very interesting! Thank you for working on it and sharing
your efforts!

A related update from our side:
We had a discussion with Lyubomir and other people from the team, and
we agree that reproducible builds are something we want to have (again).

The idea we have for fixing the problems with the current maven based
builds, as far as I understand it, is to pin the dependency versions
in pom.xml and have an automated way of bumping them. One problem we
was with this was the number of useless "version bump" commits.

I don't understand why that has to be automatic. Some of our libraries
are used externally as well and I think we should start to think about
a versioning scheme instead of just releasing build numbers.

Without automatic updates, manually updating the less-often-used projects (jigasi, jirecon, jitsi-hammer, etc.) becomes a burden, because the changes accumulate. Automatic updates would force us to fix any issues as they arise (and so when they are still in someone's context, and more clearly separated).

I don't understand how the libraries being used externally is related -- people can choose to use a fixed version or not.

We have not considered using git submodules so far (at least I don't
know that we have), but they seem to solve alot of problems in a
simple way.

Not sure if they are a long term solution (they might be, I've never
used them). With all that versioning we should also keep our Debian
goals in mind.

Boris, can you please verify Etienne's CLA and merge the bccontrib PR
in jitsi-universe if it's ok?

I don't see the CLA yet (I think these have to be added manually still). I'll keep an eye and merge the PR when we get it.

Regards,
Boris

···

On 29/12/15 23:15, Ingo Bauersachs wrote:

Begin forwarded message:

On 30.12.2015, at 04:48, Boris Grozev <boris@jitsi.org >>> <mailto:boris@jitsi.org>> wrote:

On 28/12/15 20:21, Etienne Champetier wrote:


#3

Hi,

Hey,

Sorry for the delayed response on this one.

See below

[...]

Boris, can you please verify Etienne's CLA and merge the bccontrib PR
in jitsi-universe if it's ok?

I don't see the CLA yet (I think these have to be added manually still).
I'll keep an eye and merge the PR when we get it.

I haven't sent the "manually" signed version of the cla to
contributions@atlassian.com,
as the digitally signed version seems to be automatically sent by docusign

···

2016-01-04 11:35 GMT+01:00 Boris Grozev <boris@jitsi.org>:

On 29/12/15 23:15, Ingo Bauersachs wrote:

Regards,
Boris


#4

Hi,

Hey,

Sorry for the delayed response on this one.

See below

Freundliche Grüsse,
Ingo Bauersachs

-- Sent from my tablet

*From:* Ingo Bauersachs <ingo@bauersachs.me <mailto:ingo@bauersachs.me>>

*Date:* 30. Dezember 2015 um 10:14:34 GMT+13
*To:* Jitsi Developers <dev@jitsi.org <mailto:dev@jitsi.org>>
*Subject:* *Re: [jitsi-dev] Reproducible build (with git submodules)*

Hi,

I'm trying to make jitsi (jvb & jicofo) reproducible, ie build all
"SNAPSHOT" packages from sources
I'm using git submodules and a master pom file and i'm now able to
build
almost every jitsi package from sources, with no changes in the
repos or pom

I've 4 PR pending:
https://github.com/jitsi/jain-sip/pull/1
https://github.com/jitsi/jitsi-universe/pull/3
https://github.com/jitsi/smack_3_2_2/pull/1
https://github.com/jitsi/jicoco/pull/2

And i'm also looking for the source of 3 artifacts (callstats, fmj,
jnsapi):

Callstats: https://github.com/callstats-io/callstats.java
FMJ: http://sourceforge.net/p/fmj/code/HEAD/tree/

I don't know about jnsapi.

https://github.com/jitsi/jitsi-maven-repository/issues/3

to try (see my PR when it fails to compile):
git clone --recursive https://github.com/champtar/jitsi-submodules
cd jitsi-submodules
rm -rf ~/.m2
mvn install -DskipTests -Dmaven.javadoc.skip=true -Dgpg.skip=true
-Dassembly.skipAssembly=false

The idea for this repo is to be auto updated and tagged (with jenkins?)
when one of the submodules is updated,
so we can reproduce jvb version 532 if we want

This is very interesting! Thank you for working on it and sharing
your efforts!

A related update from our side:
We had a discussion with Lyubomir and other people from the team, and
we agree that reproducible builds are something we want to have (again).

The idea we have for fixing the problems with the current maven based
builds, as far as I understand it, is to pin the dependency versions
in pom.xml and have an automated way of bumping them. One problem we
was with this was the number of useless "version bump" commits.

I don't understand why that has to be automatic. Some of our libraries
are used externally as well and I think we should start to think about
a versioning scheme instead of just releasing build numbers.

Without automatic updates, manually updating the less-often-used projects
(jigasi, jirecon, jitsi-hammer, etc.) becomes a burden, because the changes
accumulate. Automatic updates would force us to fix any issues as they
arise (and so when they are still in someone's context, and more clearly
separated).

I don't understand how the libraries being used externally is related --
people can choose to use a fixed version or not.

Releasing and having fixed version everywhere is the best option, but it
can be a lot of work, so we need to find the right tool for the job.
I just found
https://github.com/danielflower/multi-module-maven-release-plugin,
http://www.mojohaus.org/versions-maven-plugin/ can also be interesting.
Is someone using any of these 2?

···

2016-01-04 11:35 GMT+01:00 Boris Grozev <boris@jitsi.org>:

On 29/12/15 23:15, Ingo Bauersachs wrote:

Begin forwarded message:

On 30.12.2015, at 04:48, Boris Grozev <boris@jitsi.org >>>> <mailto:boris@jitsi.org>> wrote:

On 28/12/15 20:21, Etienne Champetier wrote:

We have not considered using git submodules so far (at least I don't
know that we have), but they seem to solve alot of problems in a
simple way.

Not sure if they are a long term solution (they might be, I've never
used them). With all that versioning we should also keep our Debian
goals in mind.

Boris, can you please verify Etienne's CLA and merge the bccontrib PR
in jitsi-universe if it's ok?

I don't see the CLA yet (I think these have to be added manually still).
I'll keep an eye and merge the PR when we get it.

Regards,
Boris


#5

Yes, what I meant was that someone on our end has to manually approve it.

Regards,
Boris

···

On 04/01/16 14:22, Etienne Champetier wrote:

Hi,

2016-01-04 11:35 GMT+01:00 Boris Grozev <boris@jitsi.org
<mailto:boris@jitsi.org>>:

    Hey,

    Sorry for the delayed response on this one.

    On 29/12/15 23:15, Ingo Bauersachs wrote:

        See below

[...]

        Boris, can you please verify Etienne's CLA and merge the
        bccontrib PR
        in jitsi-universe if it's ok?

    I don't see the CLA yet (I think these have to be added manually
    still). I'll keep an eye and merge the PR when we get it.

I haven't sent the "manually" signed version of the cla to
contributions@atlassian.com <mailto:contributions@atlassian.com>,
as the digitally signed version seems to be automatically sent by docusign


#6

Has the jitsi team come to any conclusion on supporting builds that don't
pull down the latest (SNAPSHOT) versions of all dependencies each time?
It's making work on our end a bit of a pain when we are forced to rebase
(when perhaps we would prefer not to yet, i.e. we're in the middle of
working on something) whenever we restart our development bridges and they
pull things down that break the build.

Is it still a question of principle? Or just a question of resources?

···

On Mon, Jan 4, 2016 at 4:59 AM, Boris Grozev <boris@jitsi.org> wrote:

On 04/01/16 14:22, Etienne Champetier wrote:

Hi,

2016-01-04 11:35 GMT+01:00 Boris Grozev <boris@jitsi.org
<mailto:boris@jitsi.org>>:

    Hey,

    Sorry for the delayed response on this one.

    On 29/12/15 23:15, Ingo Bauersachs wrote:

        See below

[...]

        Boris, can you please verify Etienne's CLA and merge the
        bccontrib PR
        in jitsi-universe if it's ok?

    I don't see the CLA yet (I think these have to be added manually
    still). I'll keep an eye and merge the PR when we get it.

I haven't sent the "manually" signed version of the cla to
contributions@atlassian.com <mailto:contributions@atlassian.com>,
as the digitally signed version seems to be automatically sent by docusign

Yes, what I meant was that someone on our end has to manually approve it.

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#7

Hey Brian,

Has the jitsi team come to any conclusion on supporting builds that
don't pull down the latest (SNAPSHOT) versions of all dependencies each
time? It's making work on our end a bit of a pain when we are forced to
rebase (when perhaps we would prefer not to yet, i.e. we're in the
middle of working on something) whenever we restart our development
bridges and they pull things down that break the build.

Is it still a question of principle? Or just a question of resources?

We have agreed to proceed with changing to fixed versions for the dependencies. In the long term, we will automatically update the versions, but as a first step we will just convert to fixed versions and change them manually. I hope that we will get this first step done next week (we keep running into this issue as well).

Regards,
Boris

···

On 29/01/16 12:18, Brian Baldino wrote:


#8

Resources.

Ingo

Freundliche Grüsse,
Ingo Bauersachs

-- sent from my mobile

···

On 30.01.2016, at 07:19, Brian Baldino <brian@highfive.com> wrote:

Has the jitsi team come to any conclusion on supporting builds that don't pull down the latest (SNAPSHOT) versions of all dependencies each time? It's making work on our end a bit of a pain when we are forced to rebase (when perhaps we would prefer not to yet, i.e. we're in the middle of working on something) whenever we restart our development bridges and they pull things down that break the build.

Is it still a question of principle? Or just a question of resources?

On Mon, Jan 4, 2016 at 4:59 AM, Boris Grozev <boris@jitsi.org> wrote:

On 04/01/16 14:22, Etienne Champetier wrote:
Hi,

2016-01-04 11:35 GMT+01:00 Boris Grozev <boris@jitsi.org
<mailto:boris@jitsi.org>>:

    Hey,

    Sorry for the delayed response on this one.

    On 29/12/15 23:15, Ingo Bauersachs wrote:

        See below

[...]

        Boris, can you please verify Etienne's CLA and merge the
        bccontrib PR
        in jitsi-universe if it's ok?

    I don't see the CLA yet (I think these have to be added manually
    still). I'll keep an eye and merge the PR when we get it.

I haven't sent the "manually" signed version of the cla to
contributions@atlassian.com <mailto:contributions@atlassian.com>,
as the digitally signed version seems to be automatically sent by docusign

Yes, what I meant was that someone on our end has to manually approve it.

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#9

Hi ,
I tried to reproduce the build jvb 594 and jicofo 185 using https://github.com/champtar/jitsi-submodules/ but its build fail (its throwing the dependency issue ) .
Steps I followed
1) Cloned jitsi-submodules2) I followed (http://git-scm.com/book/en/v2/Git-Tools-Submodules) to update the sub modules git submodule init
git submodule update ( it clone the sub modules )
3) I changed the sub module to specific version
4) I followed the step as per the previous email cd jitsi-submodules
rm -rf ~/.m2
mvn install -DskipTests -Dmaven.javadoc.skip=true -Dgpg.skip=true -Dassembly.skipAssembly=false

Please suggest me to proceed further .
Thanks and Regards,Kannan R

···

From: Boris Grozev <boris@jitsi.org>
To: Jitsi Developers <dev@jitsi.org>
Sent: Saturday, 30 January 2016 6:50 AM
Subject: Re: [jitsi-dev] Reproducible build (with git submodules)
   
Hey Brian,

On 29/01/16 12:18, Brian Baldino wrote:

Has the jitsi team come to any conclusion on supporting builds that
don't pull down the latest (SNAPSHOT) versions of all dependencies each
time? It's making work on our end a bit of a pain when we are forced to
rebase (when perhaps we would prefer not to yet, i.e. we're in the
middle of working on something) whenever we restart our development
bridges and they pull things down that break the build.

Is it still a question of principle? Or just a question of resources?

We have agreed to proceed with changing to fixed versions for the
dependencies. In the long term, we will automatically update the
versions, but as a first step we will just convert to fixed versions and
change them manually. I hope that we will get this first step done next
week (we keep running into this issue as well).

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#10

That's great news.

Johannes

···

________________________________________
From: dev <dev-bounces@jitsi.org> on behalf of Boris Grozev <boris@jitsi.org>
Sent: Saturday, January 30, 2016 2:20 AM
To: Jitsi Developers
Subject: Re: [jitsi-dev] Reproducible build (with git submodules)

Hey Brian,

On 29/01/16 12:18, Brian Baldino wrote:

Has the jitsi team come to any conclusion on supporting builds that
don't pull down the latest (SNAPSHOT) versions of all dependencies each
time? It's making work on our end a bit of a pain when we are forced to
rebase (when perhaps we would prefer not to yet, i.e. we're in the
middle of working on something) whenever we restart our development
bridges and they pull things down that break the build.

Is it still a question of principle? Or just a question of resources?

We have agreed to proceed with changing to fixed versions for the
dependencies. In the long term, we will automatically update the
versions, but as a first step we will just convert to fixed versions and
change them manually. I hope that we will get this first step done next
week (we keep running into this issue as well).

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev