I seem to have uncovered an issue with how emoticons are parsed (presumably package net.java.sip.communicator.impl.replacement.smiley)
If the string ';(' (without quotes) is a registered emoticon (and it is by default at net.java.sip.communicator.impl.replacement.smiley.Resources#51) then the output is a sad face as you'd expect.
If the user input is '<(' however the output is the character '<' followed by a sad face, which is not intentional behaviour.
This happens because the '<' character expands to '<', making the user input string '<(' which contains the ';('. Jitsi renders the '<' as a literal '<' and parses the remaining ';(' as an image.
The solution for this would be to prefilter the characters given to the replacement service, as it should presumably only replace decoded text, not pieces of markup such as entity references, html tags and so forth. Unfortunately I am not yet familiar enough with Jitsi to create a suitable patch).
I hope this helps diagnose the issue for someone more adept at understanding the replacement system.