Localization

[hershe]

Hi there. I'm one of the authors for an addon called lightwell buddy.

It's recently come to my attention that the addon doesn't work well outside of of english locales (big surprise eh?)... but not in ways I expected. I use the spellid's instead of spell names, and all that basic stuff.


Instead it's failing from similar, but more subtle pitfalls. For example, the addon is designed to allow players to enter the names of people they want to deliver personal messages to. I made this function to properly format the names, remove spaces, and if extra words were present, format them like a server name.

PHP Code:

function LWB:FormatName(name)

    local f = ""
    local c = 1 
    for w in string.gmatch(name, "%a+") do
        if w  then w = strupper(strsub(w,1,1))..strlower(strsub(w,2)) end
        if c == 1 and w then
            f = w
        elseif c == 2 and w then
            f = f.."-"..w
        elseif w then
            f = f.." "..w
        end
        c = c + 1
    end 
    return f

end 


Now, apparently this function is not treating ... what would you call them... "non-standard latin characters" like normal text, and is discarding them, or rather even splitting up the text between each non-standard character it finds, so håppy would become H-ppy. I can figure a way to work around this in this particular scenario I'm sure, but my main concern is I don't know where else I might be having issues. For example, do other locals even separate the name and server of each player with a hyphen? Does capitalization or lower case even exist in other locales, and if so, will it cause an issue treating foreign text to functions like strupper()? Are there resources available for me to look into to learn about the issues, and best practices to avoid them?

In general, I'm not too concerned (yet) with making every button, message and tooltip appear in the localized language. That seems easy enough to do if I really wanted... I just want to make sure it FUNCTIONS properly in different locales - without having the benefit of being able to test is there myself.

Thanks for your help.

-noob programmer.

[Haleth]

I'm quite sure that strupper() and strlower() shouldn't cause problems with other locales, but I'm not -entirely- certain.

What you could try to avoid a lengthy workaround is to simply let people enter the name of the character and the realm (optional) separately and use find/match with those input values on the player names. That way you won't run into hyphen or other misc issues.

[Rilgamon]

My guess would be utf8 encoding fails. Special letters like äöü are encoded and the encoding is longer than one char. Counting will fail in this cases eg in strsub (I think).
In my coords addon I have a function that shortens the displayed area name and if the length "hits" one of the special chars the encoding will be damaged and it can happen that it displays a different char or none at all.

[Vlad]

strsub(w,1,1) -> w:subutf8(1,1)
strsub(w,2) -> w:subutf8(2)

There are actually some functions not many know about, simply do:
/dump string
/dump math
/dump table
(e.g.) to get all the various functions you can call for strings, numbers and tables. For example it can say "rep=<function [strrep]>" meaning that string.rep("hello", ...) is the same as strrep("hello", ...) and you can even do it this way ("hello"):rep(...) where the dots are unfinished code. :P

Anyway, maybe this helps fix the issue.

[Ketho]

I just tested this for non-ascii chars (korean/chinese/japanese characters)

apparently it will cut out the first character or something ..

w => "你好吗" => "\228\189\160\229\165\189\229\144\151"
strupper(strsub(w,1,1))..strlower(strsub(w,2)) => "好吗" => "\229\165\189\229\144\151"

Originally Posted by Vladinator strsub(w,1,1) -> w:subutf8(1,1) strsub(w,2) -> w:subutf8(2) There are actually some functions not many know about, simply do: /dump string /dump math /dump table (e.g.) to get all the various functions you can call for strings, numbers and tables. For example it can say "rep=<function [strrep]>" meaning that string.rep("hello", ...) is the same as strrep("hello", ...) and you can even do it this way ("hello"):rep(...) where the dots are unfinished code. :P Anyway, maybe this helps fix the issue.

ehmm what does s:subutf8 do? I never heard of it, and I can't find it in the game either

/dump strsubutf8
/dump string.subutf8
/dump ("text").subutf8

Edit: I actually heard of the library before, but it slipped out of my mind

[Dridzt]

WoWAce -> UTF8 Library.

Also Library thread.

[Vlad]

It should be in the game, basically:

string.subutf8("string here", 1, 5)

would do the same as (for ascii):

string.sub("string here", 1,5)

Only that it should pay attention to utf-8 characters.

If you replace your parts where for example you use:
strupper(strsub(w,1,1))..strlower(strsub(w,2))
into this:
strupper(string.subutf8(w,1,1))..strlower(string.subutf8(w,2))
It would maybe work, as long Blizzard allows you to use the subutf8, it's there when I /dump string class so it should work as well.

[Phanx]

The only UTF8-aware string function built into WoW is strlenutf8, and they were even too lazy to put it in the string table, so it's only available as a global. If there are any other UTF8-related functions in there, they have been added by an addon (probably the UTF8 library Dridzt linked, which is used in a number of widely used addons, including Grid and RatingBuster).

[Vlad]

You are right Phanx, most likely added there by Ace libraries from my addons.

[hershe]

Thank you all for your help, I think I have it figured out :-)

Thanks Ketho, you are very right - strupper() was removing the non ascii characters - including ones lke ß á. I've eliminated strupper() in some other parts of the code, opting for strlower() since it doesn't seem to cause issued with these special characters. strsub() also does not appear to have an issue IF you exclude the second optional argument for where to end the string. If you include the second argument, it can cause parts of the string to be cut off if these non-ascii letters are in there. Which works fine for me because the only letter I'm concerned about is the first one... the remaining part of the word is captured with strsub(string,2), causing no issues.

Unfortunately I can't test the korean/chineese/japaneese text in my client, it just becomes a ? for me. Might be a mac thing :-P. But based on the test above, it should do fine with the strlower()

PHP Code:

function LWB:FormatName(name)

    local f = ""
    local c = 1 
    for w in string.gmatch(gsub(name,"-"," ",1), "%S+") do
        local charTest = strmatch(strsub(w,1,1),"%a")
        if w  then 
            if charTest == nil then
                w = strlower(w)
            else w = strupper(strsub(w,1,1))..strlower(strsub(w,2))
            end
        end
        if c == 1 and w then
            f = w
        elseif c == 2 and w then
            f = f.."-"..w
        elseif w then
            f = f.." "..w
        end
        c = c + 1
    end 
    return f

end 


[hershe]

By the way, on the subject of localization - is there any good way to speak to people to either provide translations, or at least tell me that the translations provided via google translate aren't just totally wrong? :-) Or is the task kinda impractical/unnecessary? I'd be on the eu.battle.net forums, among others if I could be, but I'm reluctant to buy new wow account just for that purpose :-P

I saw something on curseforge that would allow people to deliver translations for phrases, so I added some to see what would come of it. Not sure how well received that function is for volunteer translators, probably about as good as the raid-finder in game.