Unsupported LibJitsi Recording Method Help

#1

I realize you guys intend to disable the LibJitsi (Colibri) method of recording eventually in favor of Jibri.

I used it several years ago, and due to recent work in LibJitsi noticed most of the components are still available there.

It just so happens we have a very important use-case for the existing recording infrastructure that is in LibJitsi, and we will eventually probably decide to backport it, or extract it because it is a perfect fit for our existing video recording and processing infrastructure, if you guys decide to remote it entirely.

I’ve got it mostly working, but I am running into an issue with the SilenceEffect, which looks like it’s meant for inserting silent audio and timestamps into the audio track when packets are lost (Which is really great because it’s a feature we have wanted to implement in our current recording infrastructure).

The bad part is, it’s getting an exception there, and as a result we only get video webm files output.

JVB 2019-02-11 22:46:15.161 SEVERE: [173] net.sf.fmj.media.Log.error() Failed to build a graph for the given custom options.

JVB 2019-02-11 22:46:15.161 SEVERE: [173] net.sf.fmj.media.Log.error() Failed to realize: net.sf.fmj.media.ProcessEngine@3c6b8b3c

JVB 2019-02-11 22:46:15.161 SEVERE: [173] net.sf.fmj.media.Log.error()   Cannot build a flow graph with the customized options:

JVB 2019-02-11 22:46:15.161 SEVERE: [173] net.sf.fmj.media.Log.error()     Unable to add customed codecs:

JVB 2019-02-11 22:46:15.161 SEVERE: [173] net.sf.fmj.media.Log.error()       org.jitsi.impl.neomedia.codec.SilenceEffect@53247af4

The videobridge is being run inside a docker container. Does this JMF piece rely on an underlying audio system like pulseaudio or something like that?

I know I’ve used these pieces in conjunction with Jigasi, and maybe even Jibri in the past, but I’m just hoping for a helpful nudging in the right direction on what could be causing the above exception setting up this filtergraph.

0 Likes

#2

Hi Jason,

I have a vague memory of seeing this at some point, but I don’t remember why. Is the opus library (libjnopus.so) loaded? Otherwise, I can’t think of anything other than debugging FMJ to see why the failure happened (it looks like most of the related code is in ProcessEngine).

Regards,

Boris

0 Likes

#3

I was chasing similar issue I had with jigasi, at the end the problem I was seeing is that jigasi in translator mode was still trying to create players and failing to setup them and play anything … This was the fix I needed for jigasi https://github.com/jitsi/libjitsi/pull/465/commits/912998b9099513f3e793473e9147addf997d84ee
This maybe help you, but I think you are in different case … but yeah I was trying to debug fmj (if you can say that putting prints in fmj is debugging :)) it is a nightmare :slight_smile:

0 Likes

#4

Any tips on debugging FMJ? I have found some hints on the interwebs that more granular logging can be enabled, I assume maybe through a -D flag like regular java logging?

I will check the OPUS .so, and damencho’s patch, and see if I can figure out how to debug FMJ… Putting prints in it sounds pretty wonderful right now given this mysterious failure and its resistance so far to provide more granular debug information.

0 Likes

#5

@Boris_Grozev So, I see that jnopus is loaded by Opus.java— I don’t see any logs that indicate it failed loading, or that it even was attempted to be loaded.

However— the library isn’t present in my docker image at all… So I assume the normal debian package installation of jitsi-meet maybe isn’t installing it.

Is it installed by jigasi’s package possibly?

0 Likes

#6

The so file is embedded in a jar either libjitsi or jitsi-lgpl-dependencies.

0 Likes

#7

Thanks! I’ll check there, just saved me a bunch of time trying to copy those libraries into my image.

0 Likes

#8

Yeah— I checked my jar, and all of those .so files are bundled in linux-x86 and linux-x86-64

I don’t see anything indicating that they aren’t loading or aren’t capable of loading, so debugging FMJ seems like the best route to success now.

0 Likes

#9

@damencho Can you give me any pointers on where to compile FMJ?

Is this still a thing?

It looks like maybe there are jitsi specific patches, but they don’t exist on the jitsi org github as far as I can tell

0 Likes

#10

So I am hitting:

fmj/src/net/sf/fmj/media/PlaybackEngine.java:1148

Which is inside the doRealize1 method of PlaybackEngine.

            } else if (trackControls[i].isCustomized())
            {
                // If the track is being customized and we cannot
                // build a graph for that track, we won't attempt to
                // build the rest of the tracks.
                Log.error(realizeError);
                trackControls[i].prError();
                player.processError = genericProcessorError;
                return false;
            } else

Basically building the tracks fails if a member of trackControls has isCustomized set…

fmj/src/net/sf/fmj/media/ProcessEngine.java:815

        @Override
        public boolean isCustomized()
        {
            return formatWanted != null || codecChainWanted != null
                    || rendererWanted != null;
        }

It seems that codecChainWanted is set to a chain with one codec in it… I can’t see where codecChainWanted is cleared.

 trackControls[i].codecChainWanted[0] = "org.jitsi.impl.neomedia.codec.SilenceEffect@68b96cb3"

Digging deeper now, but if anyone has a clue, let me know.

0 Likes

#11

I think inserting the SilenceEffect makes it customized, so that’s
expected and it works in other cases (jigasi). It must be something in
addition to being customized.

Boris

0 Likes

#12

Yeah, it’s actually somewhat obvious that statement is only fallen into because it fails in buildTrackFromGraph.

Digging deeper.

0 Likes

#13

Finally got back to working on this, it seems like the SilenceEffect may not be the issue, but instead I am seeing errors when trying to find a compatible format for the SpeexResampler:

In FMJ:
src/net/sf/fmj/filtergraph/SimpleGraphBuilder.java:356

    static public Format verifyOutput(PlugIn p, Format out)
    {
        if (p instanceof Codec) {
            Format f = ((Codec) p).setOutputFormat(out);
            if(f == null) {
                 System.out.println("JRT: VERIFY OUTPUT FAILED Plugin: [" + p.getName() + "] [" + out + "] FORMAT: [" + f + "]");
            }

            return f;
        }
        return null;
    }
JRT: VERIFY OUTPUT FAILED Plugin: [Speex Resampler] [LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed, class [S] FORMAT: [null]
JRT: Verify output failed: org.jitsi.impl.neomedia.codec.audio.speex.SpeexResampler@7b34cc33
JRT:   with: LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed, class [S

Looking in libjitsi, it seems like 48khz, mono, signed LE should be supported… maybe?

Maybe this is a red herring…

  • Jason
0 Likes

#14

If I remember correctly 16 bit signed LE, 48kHz is the output of our opus decoder, so I don’t understand why a resampler is necessary.

Boris

0 Likes

#15

It seems like SilenceEffect gets past setting an input and output format without error…

And you’re right, they are the same, 48khz, so I’m not sure what is happening.

I see SilenceEffect set input and output succeed with 48000 khz signed LE.

Then the SpeexResampler immediately after with it setting the input to 48khz then failing to set the output to 48khz.

Where exactly does it determine that it has to insert or use the resampler?

Is it possible that since I have jigasi installed it’s doing something for jigasi?.

Any resources that might explain to me how this works would be greatly appreciated.

0 Likes

#16

Just an example of what I am seeing. I set some traces in AbstractCodec2 to catch input/output formats being set.

One oddity is that the toString for the format that it is checking of the SpeexResampler can be set to has some weird cruft at the end of it ‘, class [S’ which doesn’t match the toString for the matching format…

JRT: No matching output format returning null for: LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed, class [S

The Speex Resampler gets inserted right after the Silence Effect, and tries to set to the same 48khz signed le format as the SilenceEffect, and on setOutput it fails to find a matching output.

Modifying AbstractCodec2 to try to pin this down:
src/org/jitsi/impl/neomedia/codec/AbstractCodec2.java:506

    @Override
    public Format setOutputFormat(Format format)
    {
        if(getName().contains("Silence Effect") || getName().contains("Speex Resampler")) {
            System.out.println("JRT: SETTING OUTPUT FORMAT " + getName() + " TO " + format.toString());
        }

        if (!formatClass.isInstance(format)
                || (matches(format, getMatchingOutputFormats(inputFormat))
                        == null)) {
            if(getName().contains("Speex Resampler")) {
                System.out.println("JRT: No matching output format returning null for: " + format.toString());
                // for(Format f : supportedOutputFormats) {
                //     System.out.println("JRT: SUPPORTED OUTPUT FORMAT: " + f.toString());
                // }
            }

            return null;
        }

        return super.setOutputFormat(format);
    }
JRT: SETTING INPUT FORMAT Silence Effect TO LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed
JRT: SETTING OUTPUT FORMAT Silence Effect TO LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed
JRT: SETTING INPUT FORMAT Speex Resampler TO LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed
JRT: SETTING OUTPUT FORMAT Speex Resampler TO LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed
0 Likes

#17

Here is a stack trace that explains how SpeexResampler is being constructed.

It’s really odd because we add the SilenceEffect to the codec list, and call realize, and somehow… somewhere, SpeexResampler is inserted into the graph…

java.lang.Exception
        at org.jitsi.impl.neomedia.codec.audio.speex.SpeexResampler.<init>(SpeexResampler.java:147)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at net.sf.fmj.filtergraph.SimpleGraphBuilder.createPlugIn(SimpleGraphBuilder.java:36)
        at net.sf.fmj.filtergraph.SimpleGraphBuilder.getPlugInNode(SimpleGraphBuilder.java:202)
        at net.sf.fmj.filtergraph.SimpleGraphBuilder.doBuildGraph(SimpleGraphBuilder.java:627)
        at net.sf.fmj.filtergraph.SimpleGraphBuilder.buildGraph(SimpleGraphBuilder.java:478)
        at net.sf.fmj.media.ProcessEngine$ProcGraphBuilder.buildCustomGraph(ProcessEngine.java:109)
        at net.sf.fmj.media.ProcessEngine$ProcGraphBuilder.buildCustomGraph(ProcessEngine.java:238)
        at net.sf.fmj.media.ProcessEngine$ProcGraphBuilder.buildGraph(ProcessEngine.java:252)
        at net.sf.fmj.media.ProcessEngine$ProcTControl.buildTrack(ProcessEngine.java:688)
        at net.sf.fmj.media.PlaybackEngine.doRealize1(PlaybackEngine.java:1135)
        at net.sf.fmj.media.ProcessEngine.doRealize(ProcessEngine.java:1197)
        at net.sf.fmj.media.RealizeWorkThread.process(BasicController.java:1145)
        at net.sf.fmj.media.StateTransitionWorkThread.run(BasicController.java:1224)
0 Likes

#18

So, the Speex stuff is a total red herring. I managed to disable it entirely by removing it from the list of CUSTOM_CODECs in src/org/jitsi/impl/neomedia/codec/FMJPlugInConfiguration.java:48.

Setting the input and output formats doesn’t fail for the SilenceEffect, so… it must be failing later… :man_facepalming:

0 Likes

#19

So, for JNIDecoder, it goes through the graph builder logic, and finds the SilenceEffect as a target.

Once it is building the graph for SilenceEffect, it goes through the logic in SimpleGraphBuilder.java:
filtergraph/SimpleGraphBuilder.java:555

        GraphNode n;
        if ((n = findTarget(node)) != null)
        {
            // We are done!
//            Log.write(
//                    "Found target: "
//                        + ((n.plugin != null) ? n.plugin : n.cname));
            System.out.println("JRT: Found target: "
                       + ((n.plugin != null) ? n.plugin : n.cname));

            indent = oldIndent;
            Log.setIndent(indent);
            System.out.println("JRT: Returning target.");
            return n;
        }
        else if(isSilence) {
            System.out.println("JRT: Finding target is null for Silence Effect");
        }

findTarget always returns null for SilenceEffect, and so, it ends up going through this logic further down in doBuildGraph, which looks like it tries to find a target, which is why SpeexResampler was being initialized.

I imagine it should readily find a target in the above logic?

Any idea what that target should be?

0 Likes

#20

So, disabling the silence effect entirely, I see the following error.

Maybe it’s as boris suggested, and something is wrong with loading libopus?

Unable to handle format: opus/rtp, 48000.0 Hz
0 Likes