[sip-comm-dev] Desktop sharing and keyboard interactions


#1

Hello all,

I am currently working on a desktop sharing feature (with remote control) with SIP (and later with XMPP when JIngle will be ready). The mouse interaction is working fine (move, click, dragged and wheel) but I face some problems with keyboards.

In the first implementation I use java.awt.Robot to do all the stuff (mouse and keyboards). The keyPress and keyRelease methods use a keycode and all available keycodes are located in KeyEvent classes such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be different from native keycode of the OS).
So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in the messages to control remote desktop. It works fine... except for some key such as @, /, ... that has no keycode returned by KeyEvent.getKeyCode().
And if keyboard mapping are different between server (desktop sharer) and client (remote desktop controller) some keycode may be map to different letter. Typically on a Mac OS X that act as server (french keyboard) and a Linux client, if I type "a" on Linux, "q" will be displayed on Mac OS X.
Another problem is that AltGr key (that is found on some european keyboards) is not properly generated. Other keys combinations or other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...

Now I try another solution. It is to sent directly the character typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) and I use a JNI to do the regeneration of keys. This solution can be used (I think) to display latin character and also cyrillic or other. For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I read somewhere it will not changed accross keyboard mapping) and regenerates key with Robot. For AltGr key we handle it in the JNI directly in the JNI code.

Technical details of native functions used:
- On Windows, use of VkKeyScan to find the Virtual key from a letter, MapVirtualKey to find the scancode and keybd_event to regenerate key press or release. AltGr is obtained by CTRL+ALT keys press/release;

- On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 keycode from a letter and XTestFakeKeyEvent to regenerate key press or release. For special character (like @, /, \, ...) a map is used with the X11 string representation (@ => "at", # => "numbersign", "altgr" => "ISO_Level3_Shift");

- Mac OS X, use of code from http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, CGEventCreateKeyboardEvent to create the key press/release event and CGPostEvent to regenerate key press or release.

Now some problems with this solution:
- Some character like "�", "�", ... on X11 and Mac OS X does not have a native keycode, Typically NoSymbol is returned from Unix's XStringToKeysym() (and thus JNI cannot regenerates key press/release) but Windows does.

- A problem related with keyboard mapping. Some letters may be generated with a key combination on a specific mapping ("/" can be obtained by SHIFT+: on a french keyboard) and others may not. Typically on a french Mac OS X keyboard mapping "#" is obtained by SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the character "#" will be sent to Mac OS X, the JNI will find the right key but as no SHIFT has been pressed, "@" will be displayed instead of "#".

- Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, backspace... produce a printable character and have to be process as if they do not.

Has someone another idea on how to handle keyboard interactions in a desktop sharing sessions or solutions that can work accross keyboard mapping ?

Regards,

···

--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

Hi,

Yesterday I study a little bit what is done in RFB protocol (in other word VNC). The specification is here http://www.realvnc.com/docs/rfbproto.pdf.

The data that is transmitted in KeyEvent message are X11 keysym (note that it does not correspond to a keycode) and the X11/keysymdef.h file contains all the mapping from latin (1, 2, 4, 8, 9, ...), Arabic, Cyrillic, ... to their respective keysims. The RealVNC Java viewer has a class that can translate from ascii/unicode to keysyms (based on a 742 elements map). Note that the standard latin 1 character (a, b, ...) keysyms are the same as there ascii code.

I have to look more closely on Windows RealVNC server and Mac OS X ChickenOnVNC to see how they convert X11 keysyms to their own keycode.

In page 23 of the specification, it also raise up the problem of character that requires "shift" character to be print out correctly on some keyboard mapping and not on others. They propose to fake the additionnal key (for example a "shift"). Again I need to look at server implementation to see how they know (if they could) the combination to do to obtain the right character.

I wonder if we should do like VNC (transfer X11 keysyms and then perform X11 keysym to native keycode). Maybe we should be able to print the characters that cannot return valid keysym using XStringToKeysym() in current implementation. Again we should test if non latin character works with the OS native function we used (X11's XStringToKeysym() and Windows's VkKeyScan) for example with a cyrillic ones. If it does not work we should probably used a solution based on X11 keysym (at least it should work on Unix-based system).

What do you think ?

Regards,

···

--
Seb

Le 04/08/2010 09:08, Sebastien Vincent a �crit :

Hello all,

I am currently working on a desktop sharing feature (with remote control) with SIP (and later with XMPP when JIngle will be ready). The mouse interaction is working fine (move, click, dragged and wheel) but I face some problems with keyboards.

In the first implementation I use java.awt.Robot to do all the stuff (mouse and keyboards). The keyPress and keyRelease methods use a keycode and all available keycodes are located in KeyEvent classes such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be different from native keycode of the OS).
So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in the messages to control remote desktop. It works fine... except for some key such as @, /, ... that has no keycode returned by KeyEvent.getKeyCode().
And if keyboard mapping are different between server (desktop sharer) and client (remote desktop controller) some keycode may be map to different letter. Typically on a Mac OS X that act as server (french keyboard) and a Linux client, if I type "a" on Linux, "q" will be displayed on Mac OS X.
Another problem is that AltGr key (that is found on some european keyboards) is not properly generated. Other keys combinations or other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...

Now I try another solution. It is to sent directly the character typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) and I use a JNI to do the regeneration of keys. This solution can be used (I think) to display latin character and also cyrillic or other. For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I read somewhere it will not changed accross keyboard mapping) and regenerates key with Robot. For AltGr key we handle it in the JNI directly in the JNI code.

Technical details of native functions used:
- On Windows, use of VkKeyScan to find the Virtual key from a letter, MapVirtualKey to find the scancode and keybd_event to regenerate key press or release. AltGr is obtained by CTRL+ALT keys press/release;

- On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 keycode from a letter and XTestFakeKeyEvent to regenerate key press or release. For special character (like @, /, \, ...) a map is used with the X11 string representation (@ => "at", # => "numbersign", "altgr" => "ISO_Level3_Shift");

- Mac OS X, use of code from http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, CGEventCreateKeyboardEvent to create the key press/release event and CGPostEvent to regenerate key press or release.

Now some problems with this solution:
- Some character like "�", "�", ... on X11 and Mac OS X does not have a native keycode, Typically NoSymbol is returned from Unix's XStringToKeysym() (and thus JNI cannot regenerates key press/release) but Windows does.

- A problem related with keyboard mapping. Some letters may be generated with a key combination on a specific mapping ("/" can be obtained by SHIFT+: on a french keyboard) and others may not. Typically on a french Mac OS X keyboard mapping "#" is obtained by SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the character "#" will be sent to Mac OS X, the JNI will find the right key but as no SHIFT has been pressed, "@" will be displayed instead of "#".

- Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, backspace... produce a printable character and have to be process as if they do not.

Has someone another idea on how to handle keyboard interactions in a desktop sharing sessions or solutions that can work accross keyboard mapping ?

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Hi,

When receiving an RFB's KeyEvent message (with the X11 keysym), here basically the behavior of VNC servers:

- X11 based: generate keycode with XKeysymToKeycode();

- OSX: find directly the CGKeycode in a map of Keysym/CGKeycode correspondance. => US only so no character like �, �, ...

- Windows VNC: Look at some acute/circumflex/diaresis character keysym (�, �, �, �, ...) and generate two characters (� = XK_grave + XK_A), if this is an latin ascii character like a, b, c it is passed to VkScanCode to find the keycode and then regenerate via keybd_event. Nothing about cyrillic or arabic specific code.

The VNC Java Client converts the ascii/unicode directly to X11 keysym (see rfb/UnicodeToKeysym.java class).

For information, current implementation of the SIP Communicator's desktop sharing on Windows (server side aka sharer) works fine since all character from my keyboard are printed correctly (including the characters like �, �, ...). So basically a Linux/Windows controller _with_ same keyboard layout (or at least same character is obtained by one key press or same key combination for both layout) can interact perfectly with a Windows desktop.

As I said Linux/Mac desktop sharer (with french keyboard layout) could not display some acute/grave/circumflex/diaresis character. To solve this problem, we could do as VNC, that is to have a map of ascii character and key or key combination _but_ we should do it for all keyboard layouts to be complete. I am not sure there is a programmatic way to find the keycode combination (or not) to obtain a specific keys (A => SHIFT + A).
Having a map can also solve the problem of having different key combinations or not for same key (french Mac OS X keyboard layout and standard Windows/Linux ones) and Windows code should also have its own ascii characters/keycode(s)

What do you think ?

Regards,

···

--
Seb

Le 06/08/2010 09:30, Sebastien Vincent a �crit :

Hi,

Yesterday I study a little bit what is done in RFB protocol (in other word VNC). The specification is here http://www.realvnc.com/docs/rfbproto.pdf.

The data that is transmitted in KeyEvent message are X11 keysym (note that it does not correspond to a keycode) and the X11/keysymdef.h file contains all the mapping from latin (1, 2, 4, 8, 9, ...), Arabic, Cyrillic, ... to their respective keysims. The RealVNC Java viewer has a class that can translate from ascii/unicode to keysyms (based on a 742 elements map). Note that the standard latin 1 character (a, b, ...) keysyms are the same as there ascii code.

I have to look more closely on Windows RealVNC server and Mac OS X ChickenOnVNC to see how they convert X11 keysyms to their own keycode.

In page 23 of the specification, it also raise up the problem of character that requires "shift" character to be print out correctly on some keyboard mapping and not on others. They propose to fake the additionnal key (for example a "shift"). Again I need to look at server implementation to see how they know (if they could) the combination to do to obtain the right character.

I wonder if we should do like VNC (transfer X11 keysyms and then perform X11 keysym to native keycode). Maybe we should be able to print the characters that cannot return valid keysym using XStringToKeysym() in current implementation. Again we should test if non latin character works with the OS native function we used (X11's XStringToKeysym() and Windows's VkKeyScan) for example with a cyrillic ones. If it does not work we should probably used a solution based on X11 keysym (at least it should work on Unix-based system).

What do you think ?

Regards,
--
Seb

Le 04/08/2010 09:08, Sebastien Vincent a �crit :

Hello all,

I am currently working on a desktop sharing feature (with remote control) with SIP (and later with XMPP when JIngle will be ready). The mouse interaction is working fine (move, click, dragged and wheel) but I face some problems with keyboards.

In the first implementation I use java.awt.Robot to do all the stuff (mouse and keyboards). The keyPress and keyRelease methods use a keycode and all available keycodes are located in KeyEvent classes such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be different from native keycode of the OS).
So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in the messages to control remote desktop. It works fine... except for some key such as @, /, ... that has no keycode returned by KeyEvent.getKeyCode().
And if keyboard mapping are different between server (desktop sharer) and client (remote desktop controller) some keycode may be map to different letter. Typically on a Mac OS X that act as server (french keyboard) and a Linux client, if I type "a" on Linux, "q" will be displayed on Mac OS X.
Another problem is that AltGr key (that is found on some european keyboards) is not properly generated. Other keys combinations or other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...

Now I try another solution. It is to sent directly the character typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) and I use a JNI to do the regeneration of keys. This solution can be used (I think) to display latin character and also cyrillic or other. For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I read somewhere it will not changed accross keyboard mapping) and regenerates key with Robot. For AltGr key we handle it in the JNI directly in the JNI code.

Technical details of native functions used:
- On Windows, use of VkKeyScan to find the Virtual key from a letter, MapVirtualKey to find the scancode and keybd_event to regenerate key press or release. AltGr is obtained by CTRL+ALT keys press/release;

- On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 keycode from a letter and XTestFakeKeyEvent to regenerate key press or release. For special character (like @, /, \, ...) a map is used with the X11 string representation (@ => "at", # => "numbersign", "altgr" => "ISO_Level3_Shift");

- Mac OS X, use of code from http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, CGEventCreateKeyboardEvent to create the key press/release event and CGPostEvent to regenerate key press or release.

Now some problems with this solution:
- Some character like "�", "�", ... on X11 and Mac OS X does not have a native keycode, Typically NoSymbol is returned from Unix's XStringToKeysym() (and thus JNI cannot regenerates key press/release) but Windows does.

- A problem related with keyboard mapping. Some letters may be generated with a key combination on a specific mapping ("/" can be obtained by SHIFT+: on a french keyboard) and others may not. Typically on a french Mac OS X keyboard mapping "#" is obtained by SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the character "#" will be sent to Mac OS X, the JNI will find the right key but as no SHIFT has been pressed, "@" will be displayed instead of "#".

- Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, backspace... produce a printable character and have to be process as if they do not.

Has someone another idea on how to handle keyboard interactions in a desktop sharing sessions or solutions that can work accross keyboard mapping ?

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Hi,

I just re-read MSDN information about VkKeyScan (that translate character to virtual key code) and I found that the return value tells also about the shift state (i.e. if shift, alt or control is pressed to obtain the exact character).

I will immediately implement this behavior in the Windows native code to see if any character sent is also regenerate correctly even with a different keyboard layout (typically a french Mac OS X ones). So if all work as expected, Windows users will be able to share their desktop with any computer/any keyboard layout.

Regards,

···

--
Seb

Le 06/08/2010 17:55, Sebastien Vincent a �crit :

Hi,

When receiving an RFB's KeyEvent message (with the X11 keysym), here basically the behavior of VNC servers:

- X11 based: generate keycode with XKeysymToKeycode();

- OSX: find directly the CGKeycode in a map of Keysym/CGKeycode correspondance. => US only so no character like �, �, ...

- Windows VNC: Look at some acute/circumflex/diaresis character keysym (�, �, �, �, ...) and generate two characters (� = XK_grave + XK_A), if this is an latin ascii character like a, b, c it is passed to VkScanCode to find the keycode and then regenerate via keybd_event. Nothing about cyrillic or arabic specific code.

The VNC Java Client converts the ascii/unicode directly to X11 keysym (see rfb/UnicodeToKeysym.java class).

For information, current implementation of the SIP Communicator's desktop sharing on Windows (server side aka sharer) works fine since all character from my keyboard are printed correctly (including the characters like �, �, ...). So basically a Linux/Windows controller _with_ same keyboard layout (or at least same character is obtained by one key press or same key combination for both layout) can interact perfectly with a Windows desktop.

As I said Linux/Mac desktop sharer (with french keyboard layout) could not display some acute/grave/circumflex/diaresis character. To solve this problem, we could do as VNC, that is to have a map of ascii character and key or key combination _but_ we should do it for all keyboard layouts to be complete. I am not sure there is a programmatic way to find the keycode combination (or not) to obtain a specific keys (A => SHIFT + A).
Having a map can also solve the problem of having different key combinations or not for same key (french Mac OS X keyboard layout and standard Windows/Linux ones) and Windows code should also have its own ascii characters/keycode(s)

What do you think ?

Regards,
--
Seb

Le 06/08/2010 09:30, Sebastien Vincent a �crit :

Hi,

Yesterday I study a little bit what is done in RFB protocol (in other word VNC). The specification is here http://www.realvnc.com/docs/rfbproto.pdf.

The data that is transmitted in KeyEvent message are X11 keysym (note that it does not correspond to a keycode) and the X11/keysymdef.h file contains all the mapping from latin (1, 2, 4, 8, 9, ...), Arabic, Cyrillic, ... to their respective keysims. The RealVNC Java viewer has a class that can translate from ascii/unicode to keysyms (based on a 742 elements map). Note that the standard latin 1 character (a, b, ...) keysyms are the same as there ascii code.

I have to look more closely on Windows RealVNC server and Mac OS X ChickenOnVNC to see how they convert X11 keysyms to their own keycode.

In page 23 of the specification, it also raise up the problem of character that requires "shift" character to be print out correctly on some keyboard mapping and not on others. They propose to fake the additionnal key (for example a "shift"). Again I need to look at server implementation to see how they know (if they could) the combination to do to obtain the right character.

I wonder if we should do like VNC (transfer X11 keysyms and then perform X11 keysym to native keycode). Maybe we should be able to print the characters that cannot return valid keysym using XStringToKeysym() in current implementation. Again we should test if non latin character works with the OS native function we used (X11's XStringToKeysym() and Windows's VkKeyScan) for example with a cyrillic ones. If it does not work we should probably used a solution based on X11 keysym (at least it should work on Unix-based system).

What do you think ?

Regards,
--
Seb

Le 04/08/2010 09:08, Sebastien Vincent a �crit :

Hello all,

I am currently working on a desktop sharing feature (with remote control) with SIP (and later with XMPP when JIngle will be ready). The mouse interaction is working fine (move, click, dragged and wheel) but I face some problems with keyboards.

In the first implementation I use java.awt.Robot to do all the stuff (mouse and keyboards). The keyPress and keyRelease methods use a keycode and all available keycodes are located in KeyEvent classes such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be different from native keycode of the OS).
So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in the messages to control remote desktop. It works fine... except for some key such as @, /, ... that has no keycode returned by KeyEvent.getKeyCode().
And if keyboard mapping are different between server (desktop sharer) and client (remote desktop controller) some keycode may be map to different letter. Typically on a Mac OS X that act as server (french keyboard) and a Linux client, if I type "a" on Linux, "q" will be displayed on Mac OS X.
Another problem is that AltGr key (that is found on some european keyboards) is not properly generated. Other keys combinations or other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...

Now I try another solution. It is to sent directly the character typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) and I use a JNI to do the regeneration of keys. This solution can be used (I think) to display latin character and also cyrillic or other. For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I read somewhere it will not changed accross keyboard mapping) and regenerates key with Robot. For AltGr key we handle it in the JNI directly in the JNI code.

Technical details of native functions used:
- On Windows, use of VkKeyScan to find the Virtual key from a letter, MapVirtualKey to find the scancode and keybd_event to regenerate key press or release. AltGr is obtained by CTRL+ALT keys press/release;

- On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 keycode from a letter and XTestFakeKeyEvent to regenerate key press or release. For special character (like @, /, \, ...) a map is used with the X11 string representation (@ => "at", # => "numbersign", "altgr" => "ISO_Level3_Shift");

- Mac OS X, use of code from http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, CGEventCreateKeyboardEvent to create the key press/release event and CGPostEvent to regenerate key press or release.

Now some problems with this solution:
- Some character like "�", "�", ... on X11 and Mac OS X does not have a native keycode, Typically NoSymbol is returned from Unix's XStringToKeysym() (and thus JNI cannot regenerates key press/release) but Windows does.

- A problem related with keyboard mapping. Some letters may be generated with a key combination on a specific mapping ("/" can be obtained by SHIFT+: on a french keyboard) and others may not. Typically on a french Mac OS X keyboard mapping "#" is obtained by SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the character "#" will be sent to Mac OS X, the JNI will find the right key but as no SHIFT has been pressed, "@" will be displayed instead of "#".

- Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, backspace... produce a printable character and have to be process as if they do not.

Has someone another idea on how to handle keyboard interactions in a desktop sharing sessions or solutions that can work accross keyboard mapping ?

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Hi,

After some tests with a Windows desktop sharer, it works fine even with some "obscure" key combinations like | ("pipe") which is obtained with SHIFT+ALT+L on MacBook and AltGr+" on Windows.

I will see if I can find a way to find modifiers keys for a particular character (as VkKeyScan does) on Mac OS X and X11.

···

--
Seb

Le 09/08/2010 10:02, Sebastien Vincent a �crit :

Hi,

I just re-read MSDN information about VkKeyScan (that translate character to virtual key code) and I found that the return value tells also about the shift state (i.e. if shift, alt or control is pressed to obtain the exact character).

I will immediately implement this behavior in the Windows native code to see if any character sent is also regenerate correctly even with a different keyboard layout (typically a french Mac OS X ones). So if all work as expected, Windows users will be able to share their desktop with any computer/any keyboard layout.

Regards,
--
Seb

Le 06/08/2010 17:55, Sebastien Vincent a �crit :

Hi,

When receiving an RFB's KeyEvent message (with the X11 keysym), here basically the behavior of VNC servers:

- X11 based: generate keycode with XKeysymToKeycode();

- OSX: find directly the CGKeycode in a map of Keysym/CGKeycode correspondance. => US only so no character like �, �, ...

- Windows VNC: Look at some acute/circumflex/diaresis character keysym (�, �, �, �, ...) and generate two characters (� = XK_grave + XK_A), if this is an latin ascii character like a, b, c it is passed to VkScanCode to find the keycode and then regenerate via keybd_event. Nothing about cyrillic or arabic specific code.

The VNC Java Client converts the ascii/unicode directly to X11 keysym (see rfb/UnicodeToKeysym.java class).

For information, current implementation of the SIP Communicator's desktop sharing on Windows (server side aka sharer) works fine since all character from my keyboard are printed correctly (including the characters like �, �, ...). So basically a Linux/Windows controller _with_ same keyboard layout (or at least same character is obtained by one key press or same key combination for both layout) can interact perfectly with a Windows desktop.

As I said Linux/Mac desktop sharer (with french keyboard layout) could not display some acute/grave/circumflex/diaresis character. To solve this problem, we could do as VNC, that is to have a map of ascii character and key or key combination _but_ we should do it for all keyboard layouts to be complete. I am not sure there is a programmatic way to find the keycode combination (or not) to obtain a specific keys (A => SHIFT + A).
Having a map can also solve the problem of having different key combinations or not for same key (french Mac OS X keyboard layout and standard Windows/Linux ones) and Windows code should also have its own ascii characters/keycode(s)

What do you think ?

Regards,
--
Seb

Le 06/08/2010 09:30, Sebastien Vincent a �crit :

Hi,

Yesterday I study a little bit what is done in RFB protocol (in other word VNC). The specification is here http://www.realvnc.com/docs/rfbproto.pdf.

The data that is transmitted in KeyEvent message are X11 keysym (note that it does not correspond to a keycode) and the X11/keysymdef.h file contains all the mapping from latin (1, 2, 4, 8, 9, ...), Arabic, Cyrillic, ... to their respective keysims. The RealVNC Java viewer has a class that can translate from ascii/unicode to keysyms (based on a 742 elements map). Note that the standard latin 1 character (a, b, ...) keysyms are the same as there ascii code.

I have to look more closely on Windows RealVNC server and Mac OS X ChickenOnVNC to see how they convert X11 keysyms to their own keycode.

In page 23 of the specification, it also raise up the problem of character that requires "shift" character to be print out correctly on some keyboard mapping and not on others. They propose to fake the additionnal key (for example a "shift"). Again I need to look at server implementation to see how they know (if they could) the combination to do to obtain the right character.

I wonder if we should do like VNC (transfer X11 keysyms and then perform X11 keysym to native keycode). Maybe we should be able to print the characters that cannot return valid keysym using XStringToKeysym() in current implementation. Again we should test if non latin character works with the OS native function we used (X11's XStringToKeysym() and Windows's VkKeyScan) for example with a cyrillic ones. If it does not work we should probably used a solution based on X11 keysym (at least it should work on Unix-based system).

What do you think ?

Regards,
--
Seb

Le 04/08/2010 09:08, Sebastien Vincent a �crit :

Hello all,

I am currently working on a desktop sharing feature (with remote control) with SIP (and later with XMPP when JIngle will be ready). The mouse interaction is working fine (move, click, dragged and wheel) but I face some problems with keyboards.

In the first implementation I use java.awt.Robot to do all the stuff (mouse and keyboards). The keyPress and keyRelease methods use a keycode and all available keycodes are located in KeyEvent classes such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be different from native keycode of the OS).
So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in the messages to control remote desktop. It works fine... except for some key such as @, /, ... that has no keycode returned by KeyEvent.getKeyCode().
And if keyboard mapping are different between server (desktop sharer) and client (remote desktop controller) some keycode may be map to different letter. Typically on a Mac OS X that act as server (french keyboard) and a Linux client, if I type "a" on Linux, "q" will be displayed on Mac OS X.
Another problem is that AltGr key (that is found on some european keyboards) is not properly generated. Other keys combinations or other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...

Now I try another solution. It is to sent directly the character typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) and I use a JNI to do the regeneration of keys. This solution can be used (I think) to display latin character and also cyrillic or other. For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I read somewhere it will not changed accross keyboard mapping) and regenerates key with Robot. For AltGr key we handle it in the JNI directly in the JNI code.

Technical details of native functions used:
- On Windows, use of VkKeyScan to find the Virtual key from a letter, MapVirtualKey to find the scancode and keybd_event to regenerate key press or release. AltGr is obtained by CTRL+ALT keys press/release;

- On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 keycode from a letter and XTestFakeKeyEvent to regenerate key press or release. For special character (like @, /, \, ...) a map is used with the X11 string representation (@ => "at", # => "numbersign", "altgr" => "ISO_Level3_Shift");

- Mac OS X, use of code from http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, CGEventCreateKeyboardEvent to create the key press/release event and CGPostEvent to regenerate key press or release.

Now some problems with this solution:
- Some character like "�", "�", ... on X11 and Mac OS X does not have a native keycode, Typically NoSymbol is returned from Unix's XStringToKeysym() (and thus JNI cannot regenerates key press/release) but Windows does.

- A problem related with keyboard mapping. Some letters may be generated with a key combination on a specific mapping ("/" can be obtained by SHIFT+: on a french keyboard) and others may not. Typically on a french Mac OS X keyboard mapping "#" is obtained by SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the character "#" will be sent to Mac OS X, the JNI will find the right key but as no SHIFT has been pressed, "@" will be displayed instead of "#".

- Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, backspace... produce a printable character and have to be process as if they do not.

Has someone another idea on how to handle keyboard interactions in a desktop sharing sessions or solutions that can work accross keyboard mapping ?

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#6

For information,

I finally manage to make the key combinations (with different keyboard layout) works with a Mac OS X (acting as desktop sharer) but unfortunately accentuated character (�, �, ...) are not recognized (and thus not printed).

···

--
Seb

Le 09/08/2010 13:45, Sebastien Vincent a �crit :

Hi,

After some tests with a Windows desktop sharer, it works fine even with some "obscure" key combinations like | ("pipe") which is obtained with SHIFT+ALT+L on MacBook and AltGr+" on Windows.

I will see if I can find a way to find modifiers keys for a particular character (as VkKeyScan does) on Mac OS X and X11.

--
Seb

Le 09/08/2010 10:02, Sebastien Vincent a �crit :

Hi,

I just re-read MSDN information about VkKeyScan (that translate character to virtual key code) and I found that the return value tells also about the shift state (i.e. if shift, alt or control is pressed to obtain the exact character).

I will immediately implement this behavior in the Windows native code to see if any character sent is also regenerate correctly even with a different keyboard layout (typically a french Mac OS X ones). So if all work as expected, Windows users will be able to share their desktop with any computer/any keyboard layout.

Regards,
--
Seb

Le 06/08/2010 17:55, Sebastien Vincent a �crit :

Hi,

When receiving an RFB's KeyEvent message (with the X11 keysym), here basically the behavior of VNC servers:

- X11 based: generate keycode with XKeysymToKeycode();

- OSX: find directly the CGKeycode in a map of Keysym/CGKeycode correspondance. => US only so no character like �, �, ...

- Windows VNC: Look at some acute/circumflex/diaresis character keysym (�, �, �, �, ...) and generate two characters (� = XK_grave + XK_A), if this is an latin ascii character like a, b, c it is passed to VkScanCode to find the keycode and then regenerate via keybd_event. Nothing about cyrillic or arabic specific code.

The VNC Java Client converts the ascii/unicode directly to X11 keysym (see rfb/UnicodeToKeysym.java class).

For information, current implementation of the SIP Communicator's desktop sharing on Windows (server side aka sharer) works fine since all character from my keyboard are printed correctly (including the characters like �, �, ...). So basically a Linux/Windows controller _with_ same keyboard layout (or at least same character is obtained by one key press or same key combination for both layout) can interact perfectly with a Windows desktop.

As I said Linux/Mac desktop sharer (with french keyboard layout) could not display some acute/grave/circumflex/diaresis character. To solve this problem, we could do as VNC, that is to have a map of ascii character and key or key combination _but_ we should do it for all keyboard layouts to be complete. I am not sure there is a programmatic way to find the keycode combination (or not) to obtain a specific keys (A => SHIFT + A).
Having a map can also solve the problem of having different key combinations or not for same key (french Mac OS X keyboard layout and standard Windows/Linux ones) and Windows code should also have its own ascii characters/keycode(s)

What do you think ?

Regards,
--
Seb

Le 06/08/2010 09:30, Sebastien Vincent a �crit :

Hi,

Yesterday I study a little bit what is done in RFB protocol (in other word VNC). The specification is here http://www.realvnc.com/docs/rfbproto.pdf.

The data that is transmitted in KeyEvent message are X11 keysym (note that it does not correspond to a keycode) and the X11/keysymdef.h file contains all the mapping from latin (1, 2, 4, 8, 9, ...), Arabic, Cyrillic, ... to their respective keysims. The RealVNC Java viewer has a class that can translate from ascii/unicode to keysyms (based on a 742 elements map). Note that the standard latin 1 character (a, b, ...) keysyms are the same as there ascii code.

I have to look more closely on Windows RealVNC server and Mac OS X ChickenOnVNC to see how they convert X11 keysyms to their own keycode.

In page 23 of the specification, it also raise up the problem of character that requires "shift" character to be print out correctly on some keyboard mapping and not on others. They propose to fake the additionnal key (for example a "shift"). Again I need to look at server implementation to see how they know (if they could) the combination to do to obtain the right character.

I wonder if we should do like VNC (transfer X11 keysyms and then perform X11 keysym to native keycode). Maybe we should be able to print the characters that cannot return valid keysym using XStringToKeysym() in current implementation. Again we should test if non latin character works with the OS native function we used (X11's XStringToKeysym() and Windows's VkKeyScan) for example with a cyrillic ones. If it does not work we should probably used a solution based on X11 keysym (at least it should work on Unix-based system).

What do you think ?

Regards,
--
Seb

Le 04/08/2010 09:08, Sebastien Vincent a �crit :

Hello all,

I am currently working on a desktop sharing feature (with remote control) with SIP (and later with XMPP when JIngle will be ready). The mouse interaction is working fine (move, click, dragged and wheel) but I face some problems with keyboards.

In the first implementation I use java.awt.Robot to do all the stuff (mouse and keyboards). The keyPress and keyRelease methods use a keycode and all available keycodes are located in KeyEvent classes such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be different from native keycode of the OS).
So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in the messages to control remote desktop. It works fine... except for some key such as @, /, ... that has no keycode returned by KeyEvent.getKeyCode().
And if keyboard mapping are different between server (desktop sharer) and client (remote desktop controller) some keycode may be map to different letter. Typically on a Mac OS X that act as server (french keyboard) and a Linux client, if I type "a" on Linux, "q" will be displayed on Mac OS X.
Another problem is that AltGr key (that is found on some european keyboards) is not properly generated. Other keys combinations or other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...

Now I try another solution. It is to sent directly the character typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) and I use a JNI to do the regeneration of keys. This solution can be used (I think) to display latin character and also cyrillic or other. For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I read somewhere it will not changed accross keyboard mapping) and regenerates key with Robot. For AltGr key we handle it in the JNI directly in the JNI code.

Technical details of native functions used:
- On Windows, use of VkKeyScan to find the Virtual key from a letter, MapVirtualKey to find the scancode and keybd_event to regenerate key press or release. AltGr is obtained by CTRL+ALT keys press/release;

- On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 keycode from a letter and XTestFakeKeyEvent to regenerate key press or release. For special character (like @, /, \, ...) a map is used with the X11 string representation (@ => "at", # => "numbersign", "altgr" => "ISO_Level3_Shift");

- Mac OS X, use of code from http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, CGEventCreateKeyboardEvent to create the key press/release event and CGPostEvent to regenerate key press or release.

Now some problems with this solution:
- Some character like "�", "�", ... on X11 and Mac OS X does not have a native keycode, Typically NoSymbol is returned from Unix's XStringToKeysym() (and thus JNI cannot regenerates key press/release) but Windows does.

- A problem related with keyboard mapping. Some letters may be generated with a key combination on a specific mapping ("/" can be obtained by SHIFT+: on a french keyboard) and others may not. Typically on a french Mac OS X keyboard mapping "#" is obtained by SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the character "#" will be sent to Mac OS X, the JNI will find the right key but as no SHIFT has been pressed, "@" will be displayed instead of "#".

- Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, backspace... produce a printable character and have to be process as if they do not.

Has someone another idea on how to handle keyboard interactions in a desktop sharing sessions or solutions that can work accross keyboard mapping ?

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net