[Proposal] Enhanced Translator

[MMRIZE]

:::
This is not a request but rather a proposal.
Recently, I rewrote Translator for my purpose, but if someone has interests, I’ll make a PR to the original repository to share.
Anyway, I need some help; test, proof of concept.

I’m just afraid that I was the only one who needs this. :)

I apologise in advance for a too-long article to read.
:::

MagicMirror² Enhanced Translator

This project is rewriting the global Translator object of MagicMirror to have enhanced features.

• dev repository : https://github.com/MMRIZE/MagicMirror_EnhancedTranslator
• dev branch : https://github.com/MMRIZE/MagicMirror_EnhancedTranslator/tree/enhanced-translator (original based 2.16-develop of MicMich/MagicMirror)
• demo branch(here) : https://github.com/MMRIZE/MagicMirror_EnhancedTranslator/tree/master

Motivation

For several years, the current translator class works for L10N/I18N of MM successfully. But there is still some lack of features. Especially in the below cases, we need more improvements.

• Maintenance of translation files.

  Whenever a new translation by the user is added or updated, translations.js(core) or .getTranslations()(module) should be updated and released officially to prevent the update-conflicts issue.

  By versioning up, some old translation might be obsoleted or needs to be updated for the change of terms used in MM.

  And also, it is hard to add a light-modified custom dictionary like de-au.json. It would probably be copied from the original de.json, and even though the difference might be very little, but entire de-au.json should be published and maintained.

• Locale-aware

  One xx.json cannot hold several locale-specific translations difference. US English and Australian English are noticeably different from each other from vocabulary to grammar. Some Australian users may prefer "G’day mate" instead of "Hello". Some British people might not be comfortable with the spelling of "Color".

  English users in Canada(en-CA) notate dates as 4 September 2021 or September 4, 2021, And French in Canada(fr-CA) notate date as 4 septembre 2021. and the usual numeric format is 2021-09-04. But in the US(en-US), people genrally use 09/04/2021. And in France(fr-FR) it is 04/09/21.

  German people prefer to write a number like 1.234.567,89, but in US, it will be 1,234,567.89. In India, it will be 12,34,567.89. In China, under ‘hanidec’ numbering format, it will be 一,二三四,五六七.八九.

  Generally, MM module developers have tried to solve these issues by providing specific additional L10N converting by config. But occasionally it makes another too-many-config-value issue.

  And for the date format, we are facing momentJS deprecation. (Well, luxon might be the excellent alternative, though.)

• For multi-lingual user (too-simple-fallback)

  Current fallback mechanism - primary language or ‘en’ is not enough. Some people use multi-languages in their country.

  Certain Serbian users can speak Serbian, Croatian, Romanian, Hungarian, Slovak, Czech, etc. Anyway English might not be his prefer language. With luck, some modules might provide hu.json. and some modules might provide hr.json. But he should select only one language for his Mirror. And whatever he chooses, other modules will display English. Of course, en must be the last-final-safe fallback. However, it will be convenient and user-friendly if MM provided more steps before the final solution.

  Also, there could be cases like this; sometimes fr-CA user needs fr-CA translation at first, then want to use familiar fr, then en-CA if available, at last reluctantly use standard en as final fallback.

• flexibility for natural language process

  translator is not a template engine. However, if it could handle Object and Array it would provide more abundant natural translation results. In some counties, the first name is preferred to Family name, and in some countries, vice versa. If translator can handle user: {firstName, familyName, gender, title} together instead of addressing each userFirstName, userFamilyName, userGender, userTitle, it could reduce the codes and make meaningful context.

  Customizable value-formatters would be a help for L10N/I18N also. A Module developer needs to support only general value and format, and then a translation provider could convert that value to another format with those formatter on dictionaries.

  For example; Instead of "Unread mail: 3", more natural sentences "There is a mail unread.", "There are 3 mails unread." or "Es gibt eine ungelesene Mail.", "Es sind 3 Mails ungelesen" could be possible with some formatters and language/locale information by translation provider, not by module developer himself.

  Finally, translation could be used as a light template engine with those features. It will give more freedom to the user who wants to customize the MM.

Those are the main reason what this project begins.

Improvements

• Multi-language fallback by prefer order and related-locales
• Translation dictionaries autoload
• Handling Object and Array as replacement variables
• custom value formatting

Usage

config and fallback order

// in `config.js`
language: "fr-CH",
languages: ["it", "de-CH"],
locale: "fr-CH",

• config.languages are proposed. You can define multi-languages to use by preferred order.

• For backward compatibility, config.language would still be used. config.language will be regarded as a primary language. In above example, this configuration is same with languages: ["fr-CH", "it", "de-CH"]. (translator itself doesn’t refer config.language but other modules might be using it.)

• For convenience, config.languages: "it, de-CH" would also be allowed.

• "en-CA" or "en-ca" would be suitable for language code. But don’t use "en_CA". Ithe t is not standard BCP-47 format.

• Each locale-like language code implies to refer its ancestor dictionaries. "fr-CH" will use fr-ch.json and fr.json. When translator cannot find the term in the fr-ch.json, it will try to seek from fr.json. Another case; zh-Hans-HK will try zh-hans-hk.json then zh-hans.json, then zh.json (and finally en.json, of course). But zh-hant.json or zh-tw.json will not be referred.

• So, in the above case, the translator will refer to dictionaries by order - fr-ch, fr, it, de-ch, de and the final implied fallback en.

• If the translator cannot find any term or dictionary in module’s /translations, it will try MM’s /translations(core). If nothing is matched in the end, null or runtime fallback message(from .translate()) would be returned.

• So you can extend your sub-dictionary easily. If you already have a complete dictionary - de.json, you can make de-au.json without copying whole terms. Just describe exclusive terms only being different with de.json. All other unmentioned terms in de-au.json will be referred from de.json automatically.

• locale would be needed. If not described, default would be used in translator, and it would generally infer your default system locale.

translation file and syntax

• For backward compatibility; same naming rule with current lower capitalized BCP-47 format. (e.g. en.json, en-ca.json)

  By the way, current MM’s translations files are not fully fitted for BCP-47. kr.json should be ko.json, etc…

• For developer; You don’t need to maintain translations.js or .getTranslations(). Needed translation files that exist in /translations directory will be loaded automatically by the user’s configuration.

• For backward compatibility; same syntax with current - "TERMS" : "DEFINITION" is still used. (e.g. { "SAY_HELLO" : "Hello, {userName}!" })

• (new) Nested Object/Array index is usable. (e.g. { "SAY_HELLO" : "Hello, {user.0.name} and {user.1.name}!" })

• (new) pre-defined or custom formatter is usable. Formatter symbol is @. (e.g. { "TIME_INSTANT" : "It's {now@myTime}" }). Variable now will be converted to specific format by definition of @myTime formatter defined in translation dictionary.

• (new) Definition of formatter could be added. Translation file provider can adjust options to format the variable replacements for his language/locale.

• Drawback; Dynamic loading translation files on runtime could spit out dev-console 404 error messages, because translator doesn’t know that every translations exist or not. Error messages make no harm and they are ignorable, but annoying anyway.

Example

// in MM-Something module
var translated = this.translate("TIME_INSTANT", { now: new Date() });
// => It's Monday, 9 in the morning.

/* modules/MM-Something/translations/en.json */
{
  "SAY_HELLO": "Hello, {user.name}!",
  "TIME_INSTANT": "It's {now@myTime}.",

  "@myTime": {
    "format": "DateTimeFormat",
    "options": { "dayPeriod": "short", "hour12": true, "weekday": "long", "hour": "numeric" }
  }
}

/* modules/MM-Something/translations/de.json */
{
  "SAY_HELLO": "Hallo, {user.name}!",
  "TIME_INSTANT": "Es ist {now@myTime}.",

  "@myTime": {
    "format": "DateTimeFormat",
    "options": { "dateStyle": "long", "timeStyle": "short" }
  }
}

With locale = "en-US" and language = "en", translated result will be It's Monday, 9 at night.

With locale = "de-DE" and language = "de", translated result will be Es ist 6. September 2021 um 21:09.

As it shows, module developer doesn’t need to preserve every available conversion result. Translator maker could format it by himself for his locale and language.

Translator spec.

(module).translate(key [, variablesObject][, fallbackMessage][, asObject])

• key {string} (required) terms identifier to translate.
• variableObject {object} (optional) replacement values as object.
• fallbackMessage {string} (optional) fallback message.
• asObject {boolean} (optional) return translated result as object instead string. When you need more info about translated result, set this to true.

Each optional values are omittable.

this.translate("SAY_HELLO");
this.translate("SAY_HELLO", true);
this.translate("SAY_HELLO", { name: "Tom" });
this.translate("SAY_HELLO", { name: "Tom" }, true);
this.translate("SAY_HELLO", "Hello, nobody");
this.translate("SAY_HELLO", "Hello, nobody", true);
this.translate("SAY_HELLO", { name: "Tom" }, "Hello, nobody");
this.translate("SAY_HELLO", { name: "Tom" }, "Hello, nobody", true);

Return value will become a translated result with the variables. When translations would be failed(not found terms in all the dictionaries, some error causing, invalid variables, etc.)

When asObject set as true, the return object would have these properties;

{
  key, // original seeking term
    variables, // replacement
    asObject, // return value as object or string
    language, // which language is used
    source, // translation template before replacement with variables
    translated, // final translated result
    criteria, // where the dictionary locates ('core' or each module)
    fallback, // fallback message from `.translate()`
    moduleName, // which module call this
    toString(); // toString method. return value will be same with `translated`
}

In general module developing, this.translate() might be waht all you need to know. But for more control of translator, you can use the below methods.

Translator.getLanguages()

It will return current array of language list used in Translator by seeking order. Usually it will be a combination mix of config.language and config.languages.

Translator.getLocale()

It will return current locale value. Usually it will be a BCP-47 regulated config.locale. When the user’s locale info is not proper, it will have default as a default value.

Translator.registerFormatter(formatName, formatFunc)

• formatName {string} (required) format identifier
• formatFunc {Function} (required) callback function to format value.
  • formatFunc will get a format object as a parameter when it called.
  • Format object will have this property.

{
  value,  // original value from replacement variables to format by this formatter
  locale, // If not described in dictionary, default locale value of translator will be used
  ...rest // all other values of user definition in dictionary
}

With this method, you can add or overwrite global formatter from your module.

Example

// in MM-Something module.
Translator.registerFormatter("TemperatureConverter", function ({ value, locale, options } = {}) {
  if (isNaN(value)) return value;
  if (locale === "en-US") {
    // // just for example of how to use locale.
    options.unit = "°F";
    options.convert = "c2f";
  }
  var unit = options.unit ? options.unit : "";
  if (options.convert) {
    if (options.convert.toLowerCase() === "c2f") return Math.round((value * 9 * 10) / 5) / 10 + 32 + unit;
    if (options.convert.toLowerCase() === "f2c") return Math.round(((value - 32) / 9) * 5 * 10) / 10 + unit;
  }
  return value + unit;
});

// ...

var translated = this.translate("CURRENT_TEMP", { temp: 22 });
// It will have 'It is 71.6°F.' as defined in dictionary.

/* in translation file */
{
  "CURRENT_TEMP": "It is {temp@myTemp}.",

  "@myTemp": {
    "format": "TemperatureConverter",
    "options": { "unit": "°F", "convert": "c2f" }
    /* "locale": "en-US" */
  }
}

Regardless of whatever original locale is, translation provider can set locale to be used frocely for fromatting in this translation when he sets locale.

Ready-made formats.

NumberFormat

Implementation of Intl.NumberFormat - more info

You can change the number values to various formats for the locale - currency, unit, conversion, separtor grouping, aproximation, etc. See above link.

• input variable type : number or calculable data (e.g. stringified number - “123”)
• using properties in translation : options, locale

Examples of what possible

• 123456.789 => “123.456,79 €”
• 123456.789 => “￥ 123,457”
• 123456.789 => “1,23,000”
• 3500 => “3,500 liters”
• -3500 => “-$3,500.00”
• 987654321 => “988M”

DateTimeFormat

Implementation of Intl.DateTimeFormat - more info

You can change the date values to various formats for the locale - various calendar/date/time/timezone and misc. parts.

• input variable type : Date object or date-like data (e.g. stringified date - “2021-08-19 12:34:56”)
• using properties in translation : options, locale

Examples of what possible

• 2021-01-23 01:23:45 => “23/01/2021” or “21/01/23 Mon.” or “1 at night”, etc. by locale and format option

RelativeTimeFormat

Implementation of Intl.RelativeTimeFormat - more info

You can convert period to relative humanized format by unit.

• input variable type : number or calculable data (e.g. stringified number - “123”) as period
• using properties in translation : options, locale, unit

Examples of what possible

• “2 hours ago”, “tomorrow”, “in 3 days”

RelativeTimeFormat needs base unit to calculate. AutoScaledRelativeTimeFormat will auto calculate unit for convenience. See below.

AutoScaledRelativeTimeFormat

Similar to RelativeTimeFormat but unit will be decided automatically. The period is calculated based on the current time.

• input variable type : Date object or date-like data (e.g. stringified date - “2021-08-19 12:34:56”)
• using properties in translation : options, locale

ListFormat

Implementation of Intl.ListFormat - more info

You can list items with language-sensitive list formatting.

• input variable type : array of string
• using properties in translation : options, locale

Examples of what possible

• [‘A’, ‘B’, ‘C’] => “A, B, and C”
• [‘A’, ‘B’, ‘C’] => “A, B und C”
• [‘A’, ‘B’, ‘C’] => “A, B oder C”
• [‘A’, ‘B’, ‘C’] => “A, B, C”
• [‘A’, ‘B’, ‘C’] => “A B C”

PluralRules

Implementation of Intl.PluralRules - more info

It enables plural-sensitive formatting and plural-related language rules.

• input variable type : number or calculable data (e.g. stringified number - “123”)
• using properties in translation : options, locale, rules

Examples of what possible

• “I have 1 ball.” / “I have 3 balls.”
• “1st, 2nd, 3rd, 4th, 11th, 12th, 13th, 21st, 22nd, 23rd”

/* in translation xx.json */
{
  "I_HAVE_BALL": "I have {count} ball{count@plural_postfix_s}.",
  "@plural_postfix_s": {
    "format": "PluralRules",
    "options": { "type": "ordinal" },
    "rules": {
      "one": "",
      "other": "s"
    }
  }
}

Select

Simple conditional value converter

• input variable type : string
• using properties in translation : rules

/* in translation xx.json */
{
  "STATUS": "Main job is {status@mySelect}.",
  "@mySelect": {
    "rules": {
      "ONGOING": "processing now",
      "STOP": "stopped",
      "FINISH": "completed",
      "other": "working"
    }
  }
}

When this formatter cannot find the matched rule with given value, "other" will be used.

this.translate("STATUS", { status: "ONGOING" });
// => 'Main job is processing now.'
this.translate("STATUS", { status: "PREPARED" });
// => 'Main job is working.'

Current dev status

• based on MM 2.16 dev (2.17 beta).

• All spec. described above is implemented.

• new Unit Test for translator is written and is passed.

  unit test : jest tests/unit/classes/translator_spec.js -i --forceExit

I haven’t made PR to main MM repository, because;

• (TODO) e2e test. > but I have no idea how to build it. I need a help.

• (TODO) Documentation > new trnaslator probably need a documentation for the users. I need a help too, because I’m not a native English user so my skill of writing is not enough.

• (TODO maybe) Sharing translator for node_helper

To this moment, I’m not sure whether this project has worthy to be included in main MM project or not. Will this be useful? I’m afraid that I’m the only one who needs these enhanced feature.

Thanks.

Seongnoh Yi (eouia0819@gmail.com)

[BKeyport]

I love this idea! Could we include other localization as well, such as week start (Sunday start, Monday Start, Saturday start), etc.

[MMRIZE]

@bkeyport
Thanks.
Intl spec is not completed yet perfectly. weekData by locale is STAGE3 now, waiting for STAGE4(release). It has not been implemented in all environments at this moment. (But V8 has that feature with --harmony_intl_locale_info. In the case of Chrome, it’s in developer trial, shipped since Chrome92. It will be released publicly soon.)

Anyway, it will be used like this after public release;

let he = new Intl.Locale("he")
he.weekInfo
// {firstDay: 7, weekendStart: 5, weekendEnd: 6, minimalDays: 1}

let enGB = new Intl.Locale("en-GB")
enGB.weekInfo
// {firstDay: 1, weekendStart: 6, weekendEnd: 7, minimalDays: 4}

At that time, we can use this feature for our Translator or calendar module also. We can deprecate momentJS or other dependencies.

[karsten13]

@mmrize I see no reason why it should not be a part of the core project.

But I don’t know if @MichMich is very present here, I tagged him now so he will hopefully read this thread because he has to decide if he wants it in the core project.

The e2e tests are a mess and there is already an issue to replace them (or at least a part of them).

[BKeyport]

@mmrize So, let me get this straight, we’re getting international support within vanilla JS?

[sdetweil]

@karsten13 someone would need to submit a PR for him to consider

[MMRIZE]

@bkeyport said in [Proposal] Enhanced Translator:

So, let me get this straight, we’re getting international support within vanilla JS?

Well, it’s not about a matter of using Vanilla JS or not. But briefly answered, “possible”.
The main point of this project is;
“Give more freedom to the translation providers and Reduce the burden from the module developers.”
This will not become the alternative of “momentJS”. The main algorithm for WHAT TO SHOW still should be done in the module. But the translation provider could have more freedom for HOW TO SHOW under specific language and locale.

[MMRIZE]

@karsten13
I’ll make a PR in a few days. I just need some assurances. :)

[sdetweil]

@mmrize there are no ‘assurances’

the team will review the submission.
may ask for changes, may reject it.

[BKeyport]

@mmrize Gotcha. Cool.