Jigasi open source alternative of Google Speech to Text

Yeah sure, to whoever wants to start this implementation. This is what Nik said:

I would say the implementation would involve two major steps:

  1. Implement/find an existing library for Mozilla DeepSpeech which is capable of:
    * running on a GPU instance
    * offer a form of authentication
    * receive a continuous stream of audio
    * use VAD to filter out non-usable audio)
    * respond with a continuous stream of transcriptions
  2. Implement a TranscriptionService [1] in Jigasi [2] which connects to the DeepSpeech server described in 1)
    [1] = https://github.com/nikvaessen/jigasi/blob/master/src/main/java/org/jitsi/jigasi/transcription/TranscriptionService.java
    [2] = https://github.com/nikvaessen/jigasi

I’m unfamiliar whether existing open software for the tasks in 1) exists. If 1) is trivially solved, it shouldn’t be that much work.

If I ever start working on this implementation, I will share my progress, but right now I don’t think I have enough experience to do this. So these are the tips that Nik gave me. Thank you @Nik_V

3 Likes

We have implemented Jigasi integration with Vosk speech recognition toolkit some time ago:

you can use it pretty easily, just build jigasi with the patch.

To use Vosk speech recognition server
start the server with a docker:

docker run -d -p 2700:2700 alphacep/kaldi-en:latest

Then configure the transcription class with the following properly in ~/jigasi/jigasi-home/sip-communicator.properties:

org.jitsi.jigasi.transcription.customService=org.jitsi.jigasi.transcription.VoskTranscriptionService

and thats it, transcription will work.

Vosk is an offline open source speech recognition toolkit. It enables speech recognition models for 16 languages and dialects - English, Indian English, German, French, Spanish, Portuguese, Chinese, Russian, Turkish, Vietnamese, Italian, Dutch, Catalan, Arabic, Greek, Farsi. Vosk models provide continuous large vocabulary transcription, zero-latency response with streaming API, reconfigurable vocabulary and speaker identification.

I wish that patch can be integrated sooner.

3 Likes

Thank you for heling this community.

can you please provide step by step guide from beginning for the non docker Jitsi.
Thank you.

2 Likes

There are just 3 easy steps listed above. Try and let me know!

If you are uncertain, you probably have to wait till the pull request is merged.

Can you share other env and properties information for jigasi?

@mhmmderen3 I’d be happy to share but unfortunately I don’t quite understand the request. What is required exactly?

Hi @Nickolay_Shmyrev, I’m trying to integrate Vosk transcription in Jitsi with Docker, following your instructions, but the situation is the following:

  • Transcriber joins correctly when I click on CC button, but it doesn’t join as hidden
  • Captions don’t appear
  • The URL provided in the chat leads to a 404

You can see a screenshot here:

I’ve built jigasi from source.

Jigasi logs:

Jigasi 2020-12-02 15:19:41.423 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.424 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.424 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.427 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.430 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.453 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.461 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.491 SEVERE: [19] impl.packetlogging.PacketLoggingServiceImpl.run().845 Error writing packet to file
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/log/jitsi0.pcap
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.getFileNames(PacketLoggingServiceImpl.java:197)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.savePacket(PacketLoggingServiceImpl.java:582)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl.access$100(PacketLoggingServiceImpl.java:35)
        at net.java.sip.communicator.impl.packetlogging.PacketLoggingServiceImpl$SaverThread.run(PacketLoggingServiceImpl.java:834)
Jigasi 2020-12-02 15:19:41.497 INFO: [35] org.jitsi.jigasi.SipGateway.registrationStateChanged().120 REG STATE CHANGE ProtocolProviderServiceSipImpl(SIP:test@sip2sip.info) -> RegistrationStateChangeEvent[ oldState=Unregistered; newState=RegistrationState=Registering; userRequest=false; reasonCode=-1; reason=null]
Jigasi 2020-12-02 15:19:42.134 INFO: [42] impl.protocol.jabber.OperationSetBasicTelephonyJabberImpl.registrationStateChanged().127 Jingle : ON
Jigasi 2020-12-02 15:19:42.325 INFO: [42] impl.protocol.jabber.ProtocolProviderServiceJabberImpl.authenticated().2423 Authenticated: false
Jigasi 2020-12-02 15:19:42.396 INFO: [42] impl.protocol.jabber.OperationSetPersistentPresenceJabberImpl.createAccountPhotoPresenceInterceptor().1987 Can not retrieve account avatar for jigasi@auth.meet.ags-it.com/jitsi-hu8rdi: XMPPError: item-not-found - cancel
Jigasi 2020-12-02 15:19:42.488 SEVERE: [60] util.AvatarCacheUtils.getLocallyStoredAvatar().219 Could not read avatar image from file avatarcache/Jabber&_cojigasi@auth.meet.ags-it.com/Jabber&_cojigasi@auth.meet.ags-it.com
java.lang.SecurityException: Insufficient rights to access this file in current user's home directory: /config/avatarcache/Jabber&_cojigasi@auth.meet.ags-it.com/Jabber&_cojigasi@auth.meet.ags-it.com
        at org.jitsi.impl.fileaccess.FileAccessServiceImpl.getPrivatePersistentFile(FileAccessServiceImpl.java:171)
        at net.java.sip.communicator.util.AvatarCacheUtils.getLocallyStoredAvatar(AvatarCacheUtils.java:190)
        at net.java.sip.communicator.util.AvatarCacheUtils.getCachedAvatar(AvatarCacheUtils.java:77)
        at net.java.sip.communicator.impl.globaldisplaydetails.GlobalDisplayDetailsImpl$UpdateAccountInfo.run(GlobalDisplayDetailsImpl.java:378)
Jigasi 2020-12-02 15:19:42.531 INFO: [56] impl.protocol.jabber.JingleNodesServiceDiscovery.run().111 Start Jingle Nodes discovery!
Jigasi 2020-12-02 15:19:42.549 INFO: [56] impl.protocol.jabber.JingleNodesServiceDiscovery.run().151 Jingle Nodes discovery terminated!
Jigasi 2020-12-02 15:19:42.555 INFO: [56] impl.protocol.jabber.JingleNodesServiceDiscovery.run().152 Found 0 Jingle Nodes relay for account: jigasi@auth.meet.ags-it.com in 96 ms.
Jigasi 2020-12-02 15:19:42.557 INFO: [42] org.jitsi.jigasi.xmpp.CallControlMucActivator.joinCommonRoom().299 Joining call control room: jigasibrewery@internal.auth.meet.ags-it.com pps:ProtocolProviderServiceJabberImpl(Jabber:jigasi@auth.meet.ags-it.com)
Jigasi 2020-12-02 15:19:42.574 SEVERE: [60] impl.protocol.jabber.InfoRetreiver.retrieveDetails().369 Cannot load details for contact jigasi@auth.meet.ags-it.com : XMPPError: item-not-found - cancel
Jigasi 2020-12-02 15:19:42.704 INFO: [51] impl.protocol.jabber.ChatRoomJabberImpl.joined().1256 jigasibrewery@internal.auth.meet.ags-it.com/focus has joined the jigasibrewery@internal.auth.meet.ags-it.com chat room.
Jigasi 2020-12-02 15:20:45.870 INFO: [22] org.jitsi.jigasi.SipGateway.registrationStateChanged().120 REG STATE CHANGE ProtocolProviderServiceSipImpl(SIP:test@sip2sip.info) -> RegistrationStateChangeEvent[ oldState=Registering; newState=RegistrationState=Connection Failed; userRequest=false; reasonCode=-1; reason=A timeout occurred while trying to connect to the server.]
Jigasi 2020-12-02 15:20:45.873 INFO: [22] impl.protocol.sip.SipLogger.logInfo().196 Info from the JAIN-SIP stack: the sip stack timer gov.nist.javax.sip.stack.timers.DefaultSipTimer has been stopped
Jigasi 2020-12-02 15:20:46.874 INFO: [22] impl.protocol.sip.SipLogger.logInfo().196 Info from the JAIN-SIP stack: the sip stack timer gov.nist.javax.sip.stack.timers.DefaultSipTimer has been stopped

Can you help me please?
Thank you

That transcriber thing means you have an old version of something (jitsi/jigasi). Are you sure you are using the latest one?

Thank you @Nickolay_Shmyrev for you answer, in the meantime I’ve solved most issues:

  • Regarding the version you’re right: in the Dockerfile I just copied the jar, not the full zip, so libraries were not updated. After copying the entire content of the zip under /usr/share/jigasi above errors were gone
  • In order to make the dockerized jigasi communicate with vosk I had to add an additional property to sip-communicator.properties:

org.jitsi.jigasi.transcription.vosk.websocket_url=ws://{{ .Env.DOCKER_HOST_ADDRESS }}:2700

and of course add DOCKER_HOST_ADDRESS environment variable to jigasi.yml

After making these changes, transcription works but the transcriber is still seen as an actual participant (not hidden). Do you know how to make it access the conference in hidden mode?

Thanks
Luca

For hidden mode this comment is relevant I think:

Dear Damencho,

I want to add an alternative language into jigasi for transcription. Please let me know how to add the list of alternative languages.

private RecognitionConfig getRecognitionConfig(TranscriptionRequest request)
throws UnsupportedOperationException
{
RecognitionConfig.Builder builder = RecognitionConfig.newBuilder();

    // Set the sampling rate and encoding of the audio
    AudioFormat format = request.getFormat();
    builder.setSampleRateHertz(new Double(format.getSampleRate())
        .intValue());
    switch(format.getEncoding())
    {
        case "LINEAR":
            builder.setEncoding(RecognitionConfig.AudioEncoding.LINEAR16);
            break;
        default:
            throw new IllegalArgumentException("Given AudioFormat" +
                "has unexpected" +
                "encoding");
    }

    // set the model to use. It will default to a cheaper model with
    // lower performance when not set.
    if(useVideoModel)
    {
        if(logger.isDebugEnabled())
        {
            logger.debug("Using the more expensive video model");
        }
        builder.setModel("video");
    }

    // set the Language tag
    String languageTag = request.getLocale().toLanguageTag();
    validateLanguageTag(languageTag);
    builder.setLanguageCode(languageTag);

    addSpeechContexts(builder);

    // set the requested alternatives
    builder.setMaxAlternatives(MAXIMUM_DESIRED_ALTERNATIVES);

    return builder.build();
}

Mozilla DeepSpeech is an open source Tensorflow-based speech-to-text processor that has reasonably high accuracy. DeepSpeech, the speech recognition tool has a remarkable per-word error rate of near about 6.5%. Uses open source code algorithms and TensorFlow machine learning toolkit to build its STT engine. mybk-experience.onl mc-d.uno

Dear Damencho
it may be work but my requirements are different. Normal transcription is working fine but an issue occurred when the user speak two different languages while talking like English and Hindi. in this case it’s working fine for English but not working for Hindi. I have integrated google cloud speech to text API and it has the option to set the alternative language as a list(Detecting language spoken automatically) but in jigasi code, I didn’t find the option to add the alternate language.

In jigasi, cloud dependency seems too old and which is does not have an option for alternate language list.
Please let me know if there is any option.
Thanks.

1 Like

This is not available in jigasi, any PRs are welcome.

Does this service require calling out to a server? I’m looking for something that can be run totally in-house

It works without internet connection, you can run service in-house.

We are facing the same issue, here many locations are bilingual and we’d need to support such a feature with [Catalan|Galician|Basque] + Spanish.

We should never require more than 2 languages on the same conference, and probably we don’t even need to configure the same deployment being able to support more than 1 or 2 languages for transcription, so setting that into Jigasi’s config would be okay in our case.

@damencho would that approach be okay to you or should we let it be configured on UI or some JS file on jitsi-meet?

Well per participant UI would be best if you ask me :slight_smile:

But would you accept a PR that only allows Jigasi to be configured in such a way through its own config file ? hahahaha

Nobody is paying for this development …

Thanks!

Kenneth

Missatge de Damian Minkov via Jitsi Community Forum - developers & users <jitsi@discoursemail.com> del dia dg., 24 de gen. 2021 a les 17:58: