I have recently started working on porting LibJitsi to work with the BC-FIPS version of BouncyCastle.
So far, I have managed to build the VideoBridge with my modified libjtsi jar, and deploy a server, and even with much effort get as far as getting a DTLS-SRTP feed, and stats via the DtlsPacketTransformer.
Basically sometimes everything seems to work correctly, fingerprints are verified, keys are exchanged, packets begin flowing, and a peer connection is established on the client.
The issue is, it seems to intermittently fail, and in general I am getting lockups inside of BouncyCastle, particularly in the clientHandshake method of DTLSClientProtocol, which is being called from connect in the runInConnectThread method of DtlsPacketTransformer.
In all likelihood this is something more to do with the BouncyCastle FIPS implementation that with Jitsi, but I imagine that you guys have a lot of experience debugging the VideoBridge and these sorts of things, and I was hoping maybe someone here would have some insight or ideas about what exactly might be causing this, or how to debug it.
So far I have been using jdb on my server to basically trace through libjitsi, and it seems like it’s stopping in the clientHandshake portion, sometimes indefinitely, and I suspect that it would be easier to debug by looking at the network side of things, and I thought perhaps you guys maybe already had some tools or mechanisms for doing this.
I seem to remember there is a way to dump a pcap file or something along those lines?
Also, I was wondering if anyone else, or the Jitsi team, would be interested in this work?
So far it is only a proof of concept at best, and will need some cleaning up once it’s working entirely.
Any help or pointers would be greatly appreciated.