I am currently working on a desktop sharing feature (with remote control) with SIP (and later with XMPP when JIngle will be ready). The mouse interaction is working fine (move, click, dragged and wheel) but I face some problems with keyboards.
In the first implementation I use java.awt.Robot to do all the stuff (mouse and keyboards). The keyPress and keyRelease methods use a keycode and all available keycodes are located in KeyEvent classes such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be different from native keycode of the OS).
So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in the messages to control remote desktop. It works fine... except for some key such as @, /, ... that has no keycode returned by KeyEvent.getKeyCode().
And if keyboard mapping are different between server (desktop sharer) and client (remote desktop controller) some keycode may be map to different letter. Typically on a Mac OS X that act as server (french keyboard) and a Linux client, if I type "a" on Linux, "q" will be displayed on Mac OS X.
Another problem is that AltGr key (that is found on some european keyboards) is not properly generated. Other keys combinations or other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...
Now I try another solution. It is to sent directly the character typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) and I use a JNI to do the regeneration of keys. This solution can be used (I think) to display latin character and also cyrillic or other. For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I read somewhere it will not changed accross keyboard mapping) and regenerates key with Robot. For AltGr key we handle it in the JNI directly in the JNI code.
Technical details of native functions used:
- On Windows, use of VkKeyScan to find the Virtual key from a letter, MapVirtualKey to find the scancode and keybd_event to regenerate key press or release. AltGr is obtained by CTRL+ALT keys press/release;
- On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 keycode from a letter and XTestFakeKeyEvent to regenerate key press or release. For special character (like @, /, \, ...) a map is used with the X11 string representation (@ => "at", # => "numbersign", "altgr" => "ISO_Level3_Shift");
- Mac OS X, use of code from http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, CGEventCreateKeyboardEvent to create the key press/release event and CGPostEvent to regenerate key press or release.
Now some problems with this solution:
- Some character like "�", "�", ... on X11 and Mac OS X does not have a native keycode, Typically NoSymbol is returned from Unix's XStringToKeysym() (and thus JNI cannot regenerates key press/release) but Windows does.
- A problem related with keyboard mapping. Some letters may be generated with a key combination on a specific mapping ("/" can be obtained by SHIFT+: on a french keyboard) and others may not. Typically on a french Mac OS X keyboard mapping "#" is obtained by SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the character "#" will be sent to Mac OS X, the JNI will find the right key but as no SHIFT has been pressed, "@" will be displayed instead of "#".
- Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, backspace... produce a printable character and have to be process as if they do not.
Has someone another idea on how to handle keyboard interactions in a desktop sharing sessions or solutions that can work accross keyboard mapping ?