Home → Questions →

utf8 customer fields, first_name last_name

• Edit

Gabor Nagy

12 Jan, 2016 01:36 PM

Hi,

I would like to create new customers with utf8 characters in their name fields, I've tested it with: Gáborűáűéúpüóöóööööúúúú
and on CG admin it looks like this: GáborűáűéúpüÃ

How should I encode it to have the correct character encoding on CG amdin area?

Tried ut8_encode and urlencode, but both resulted a response exception with auxCode: firstName:stringLengthTooLong.

Thanks,
Gabor

1. Support Staff 1 Posted by Marc Guyer on 12 Jan, 2016 02:04 PM

   

   Hi Gabor -- That's odd. CG is known to work perfectly with UTF8 chars in every aspect of the system. We'll need to dig into this a little deeper together to figure out what's going on. Can you recreate what you've described in your account so I can see it? It appears that you've deleted the attempts you've referenced.

   • Edit

2. 2 Posted by Gabor Nagy on 12 Jan, 2016 02:10 PM

   

   Hi Marc,

   Of course. I have just created one.

   I'm using your PHP client library in a laravel system. File encodings are all utf8.

               $response = $this->cheddar->newCustomer([
                   'code' => 'CUST_' . $user['id'],
                   'firstName' => $user['first_name'],
                   'lastName' => $user['last_name'],
                   'email' => $user['email'],
                   'subscription' => [
                       'planCode' => 'FAKE'
                   ]
               ]);

   • Edit

3. Support Staff 3 Posted by Marc Guyer on 12 Jan, 2016 03:21 PM

   

   Ok, I see. This is a known issue. We have a discrepancy between our field validation and database field limitations. In short, we have our database field length limit set smaller to the field validation in the form. The team increased the string length limit on some form field validation but didn't yet accommodate for the increases at the database layer. The database silently truncates the string to fit in the table but that mangles UTF8 chars. I've attached this thread to our internal ticket that addresses this. That'll require a maintenance event to resolve. In the meantime, you can work around this problem by restricting the character length to 20 for first and last name. The company field, btw, has an effective length limit of 30 currently.

   • Edit

4. 4 Posted by Gabor Nagy on 12 Jan, 2016 04:27 PM

   

   Ok, thanks, it works with 20 character length.

   But just to be clear: I have tested around, and found that the byte length should be 20, not the character length.
   So "Gáborűáűűáéúőóö" is 14 character length, but it still became chunked and encoding messed up. But if I truncate it to "Gáborűáűűáéú" (which has 8 multibyte character and 4 singlebyte, that is 20 byte) it works :)

   In php you have to use mb_strcut instead of mb_substr.

   Thanks!

   • Edit

5. Support Staff 5 Posted by Marc Guyer on 12 Jan, 2016 04:52 PM

   

   When using PHP, we use the iconv functions with UTF8 encoding for multibyte support rather than the mb_strlen function. It's effectively the same in this case. The strings pass app validation anyway, right? The problem is with the restriction at the db layer, I suspect.

   Out of curiosity, I've tested this in the db and I cannot confirm your statement. The db restriction is to a char length, not byte length. When I test, I can insert the 20 char value without any trouble. I'm glad to have your help with this and pointing out what you're experiencing. Due to the discrepancy between our experiences, I think I need more info to get to the bottom of this.

   When you try the 20 char value, what doesn't work? Do you get a validation error or just the mangled chars in the display? Can you try again with the 20 chars and note the time and report back?

   • Edit

6. 6 Posted by Gabor Nagy on 13 Jan, 2016 08:36 AM

   

   Sorry, mb_strcut suggestion not meant for you and your code, but for other developers who are implementing the client :)

   Yesterday I got the mangled chars on admin area, without validation error.
   Today I just tested multiple times with <20, >20 and =20 character length and now I can't reproduce the mangled data, but it's always truncated to 20 byte length on your side, even if I try with more than 20 characters. I don't truncate it now.
   For example:
   "ííííííííííííííííííííí" => "ííííííííííí"
   "Gáborűáéúőóöüűáéúőóüöú" => "Gáborűáéúőóö"

   I have now more test users in my account between CET 2016-01-13 08:00 and 08:35.

   Thanks

   • Edit

7. 7 Posted by Gabor Nagy on 13 Jan, 2016 08:38 AM

   

   Sorry CET 09:00 - 09:35...

   • Edit

8. 8 Posted by Gabor Nagy on 13 Jan, 2016 09:15 AM

   

   Still getting mangled first_name with "Gáborűáűéúpüóöóööööú" input data.
   becomes "GáborűáűéúpüÃ"
   but input string has character length 20.

   • Edit

9. Support Staff 9 Posted by Marc Guyer on 10 May, 2016 12:29 PM

   

   Hi Gabor -- I know it's been a while but we were finally able to get a fix in place for this. It required a fairly substantial database table alter which was the primary holdup. We've also repaired any mangled values to the best of our ability. Thanks for the report and the additional help!

   • Edit

10. Marc Guyer closed this discussion on 10 May, 2016 12:29 PM.