Colibri with raw UDP, possible?

Hi there,

I am trying to use the COLIBRI REST Api ( https://github.com/jitsi/jitsi-videobridge/blob/master/doc/rest-colibri.md ) but I’m quite stuck.

What I am trying to do :

2 users (Alice & Bob) are in a conference using jitsi-meet.
I want to add a 3rd user (Bot), but this user is a bot, so it does not have a browser and it cannot use the jitsi-lib.
My plan was to use COLIBRI to PATCH the conference and add the media channels of my bot.
I already know in advance the ports so that my bot will use so I don’t need to do complicated things with ICE, i want to use RAW UDP.

I don’t care if the bot is not visible from the conference, it’s job is only to send/receive media through the conference.

So, here is what I’m doing now:

1: I do a GET request to /colibri/conferences/, so I can get the conference ID.
2: I send a PATCH request with my conference ID to /colibri/conferences/

Here is my PATCH content:

{
  "id": "c12778b7ae722355",
  "contents": [
    {
      "name": "audio",
      "channels": [
        {
          "expire": 60,
          "initiator": false,
          "endpoint": "test-endpoint",
          "direction": "sendrecv",
          "channel-bundle-id": "test-endpoint",
          "rtp-level-relay-type": "mixer",
          "transport": {
                        "xmlns": "urn:xmpp:jingle:transports:raw-udp:1"
                    }
        }
      ]
    }
  ],
  "channel-bundles": [
    {
      "id": "test-endpoint",
      "transport": {
        "xmlns": "urn:xmpp:jingle:transports:raw-udp:1",
        "rtcp-mux": false
      }
    }
  ]
}

After that, I was hoping that I could start to send audio from my bot to jitsi using the candidates port that i get in the reply, but it is not working.
Also the response seems to not care of the transport:raw-udp that I requested:

{
    "channel-bundles": [
        {
            "id": "test-endpoint",
            "transport": {
                "candidates": [
                    {
                        "generation": "0",
                        "component": "1",
                        "protocol": "udp",
                        "port": "10000",
                        "ip": "172.17.14.82",
                        "foundation": "1",
                        "id": "48873be6b8bc5f204b667d6",
                        "priority": "2130706431",
                        "type": "host",
                        "network": "0"
                    },
                    {
                        "generation": "0",
                        "rel-port": "10000",
                        "component": "1",
                        "protocol": "udp",
                        "port": "10000",
                        "ip": "REDACTED (public ip)",
                        "foundation": "2",
                        "id": "75defb9d6b8bc5f206b5e48dc",
                        "rel-addr": "172.17.14.82",
                        "priority": "1694498815",
                        "type": "srflx",
                        "network": "0"
                    }
                ],
                "xmlns": "urn:xmpp:jingle:transports:ice-udp:1",
                "ufrag": "60gaa1eb3b930p",
                "rtcp-mux": true,
                "pwd": "1qrrtoq8np0md8tijeobj9ntao",
                "fingerprints": [
                    {
                        "fingerprint": "8A:CD:B1:C3:B9:4A:B8:E3:3A:83:88:CF:BE:63:0C:A0:7D:1A:26:64:D6:1D:EA:95:62:92:F9:E3:F8:8D:99:32",
                        "setup": "actpass",
                        "hash": "sha-256"
                    }
                ]
            }
        }
    ],
    "endpoints": [
        {
            "id": "68df1c3d",
            "stats-id": "Jules-aBN"
        },
        {
            "id": "test-endpoint"
        },
        {
            "id": "326b14d4",
            "stats-id": "Jules-aBN"
        }
    ],
    "contents": [
        {
            "channels": [
                {
                    "endpoint": "test-endpoint",
                    "channel-bundle-id": "test-endpoint",
                    "sources": [
                        2239110493
                    ],
                    "rtp-level-relay-type": "translator",
                    "initiator": true,
                    "id": "a5c5d4e96e15ddb0",
                    "receive-simulcast-layer": null,
                    "direction": "sendrecv",
                    "last-n": -1
                }
            ],
            "name": "audio"
        }
    ],
    "id": "c12778b7ae722355"
}

At this point I get the ID returned by the videobridge and try to patch again the conference, but it is not working too, bot to jitsi and jitsi to bot audio is not working.

Here is my second PATCH:

{
  "id": "c12778b7ae722355",
  "contents": [
    {
      "name": "audio",
      "channels": [
        {
          "id": "a5c5d4e96e15ddb0",
          "expire": 60,
          "initiator": false,
          "endpoint": "test-endpoint",
          "direction": "sendrecv",
          "sources": [
            2239110493
          ],
          "channel-bundle-id": "test-endpoint",
          "rtp-level-relay-type": "mixer",
          "ssrc-groups":  [
                        2850903072,
                        3279733364,
                        2239110493
                    ],
          "payload-types": [
            {
              "id": 8,
              "name": "alaw",
              "clockrate": 8000,
              "channels": 1,
              "parameters": {
                "minptime": 20
              }
            }
          ],
          "rtp-hdrexts": [
            {
              "id": 1,
              "uri": "urn:ietf:params:rtp-hdrext:ssrc-audio-level"
            }
          ]
        }
      ]
    }
  ],
  "channel-bundles": [
    {
      "id": "test-endpoint",
      "transport": {
        "candidates": [
          {
                "foundation": 3031090232,
                "component": 1,
                "transport": "udp",
                "priority": 2122260223,
                "ip": "172.17.203.166",
                "port": 23138,
                "type": "host",
                "generation": 0,
                "network-id": 1,
                "network-cost": 10,
                "id": 3031090232,
                "network": 1,
                "protocol": "udp"
              }
        ],
        "fingerprints": [
          {
            "fingerprint": "8A:CD:B1:C3:B9:4A:B8:E3:3A:83:88:CF:BE:63:0C:A0:7D:1A:26:64:D6:1D:EA:95:62:92:F9:E3:F8:8D:99:32",
            "hash": "sha-256"
          }
        ],
        "ufrag": "CBdMLgEmFdIJ9xjJ",
        "xmlns": "urn:xmpp:jingle:transports:raw-udp:1",
        "rtcp-mux": true
      }
    }
  ]
}

And here is the reply:

{
    "channel-bundles": [
        {
            "id": "test-endpoint",
            "transport": {
                "candidates": [
                    {
                        "generation": "0",
                        "component": "1",
                        "protocol": "udp",
                        "port": "10000",
                        "ip": "172.17.14.82",
                        "foundation": "1",
                        "id": "dfa2fea6b8bc5f204b667d6",
                        "priority": "2130706431",
                        "type": "host",
                        "network": "0"
                    },
                    {
                        "generation": "0",
                        "rel-port": "10000",
                        "component": "1",
                        "protocol": "udp",
                        "port": "10000",
                        "ip": "REDACTED (public ip)",
                        "foundation": "2",
                        "id": "1cc2f7d06b8bc5f206b5e48dc",
                        "rel-addr": "172.17.14.82",
                        "priority": "1694498815",
                        "type": "srflx",
                        "network": "0"
                    }
                ],
                "xmlns": "urn:xmpp:jingle:transports:ice-udp:1",
                "ufrag": "60gaa1eb3b930p",
                "rtcp-mux": true,
                "pwd": "1qrrtoq8np0md8tijeobj9ntao",
                "fingerprints": [
                    {
                        "fingerprint": "8A:CD:B1:C3:B9:4A:B8:E3:3A:83:88:CF:BE:63:0C:A0:7D:1A:26:64:D6:1D:EA:95:62:92:F9:E3:F8:8D:99:32",
                        "setup": "actpass",
                        "hash": "sha-256"
                    }
                ]
            }
        }
    ],
    "endpoints": [
        {
            "id": "68df1c3d",
            "stats-id": "Jules-aBN"
        },
        {
            "id": "test-endpoint"
        },
        {
            "id": "326b14d4",
            "stats-id": "Jules-aBN"
        }
    ],
    "contents": [
        {
            "channels": [
                {
                    "endpoint": "test-endpoint",
                    "channel-bundle-id": "test-endpoint",
                    "sources": [
                        2239110493
                    ],
                    "rtp-level-relay-type": "translator",
                    "initiator": true,
                    "ssrcs": [
                        2239110493
                    ],
                    "id": "284966d4928e697e",
                    "receive-simulcast-layer": null,
                    "direction": "sendrecv",
                    "last-n": -1
                }
            ],
            "name": "audio"
        }
    ],
    "id": "c12778b7ae722355"
}

I’m very lost at this point, any hints?

A couple things here:

Your biggest problem is that the bridge no longer supports a raw-udp mode for clients, I’m afraid. It could be implemented…I don’t think it would be terrible but I wouldn’t say it’s trivial either. Right now endpoints create an IceTransport class that acts as the bottom layer (it creates an ICE agent and receives data from the underlying ICE socket). You’d have to change Endpoint (or create a new subclass of AbstractEndpoint) which, instead of setting up an IceTransport created a RawUdpTransport (which doesn’t currently exist, but I don’t think would be terrible to write). I don’t think we’d be interested in merging full support for a raw UDP transport endpoint, but, if there was some refactoring you did to make the use of IceTransport less hard-coded and more flexible, I do think that would be welcome (I have some thoughts around being able to put in a fake transport there in order to feed fake data to test the bridge, so it would be useful).

Second, there would need to be some signaling changes. For example, you use "rtp-level-relay-type": "mixer", but rtp-level-relay-type is obsolete (mixer mode is not supported anymore, so only translator support is left, therefore no need to signal it). Other than that, I think what you’ve got there is about right, but, like I said…the raw UDP support is your main hurdle.

If you’re not willing to dig into adding raw UDP support (which I wouldn’t blame you for), maybe you could use ICE, at least for now?

I’ll also add that we are starting work that will eventually lead to a (hopefully) much improved API for controlling the bridge (a replacement for the current giant colibri message) which will be HTTP based (so a replacement for the current REST api). I don’t have a timeline for you, though. The process involves migrating away from XMPP for this API in Jicofo, so it will take some time.

Hi bbaldino,

Thanks for taking the time to explain why it doesn’t work :slight_smile:

I was gathering informations mainly from this board and with the documentation, so my example is not polished as it was a POC testing.

I wish I could modify the application code to implement support for RAW UDP, but unfortunately I don’t have the skills (I’m not a developer).

My bot is another complicated application, it’s capability to send/receive audio is limited to raw udp without any support for ICE&DTLS, and sadly for me, I can’t change the code too.

Maybe those informations would help someone else in the future.

And for the improved API for JVB, that’s great news, I hope that one day it will be possible with this change to easily use the JVB with others applications/devices :slight_smile: