Home Company Services Portfolio Contact us nav spacer

I18N and MySite

A draft of a chapter in Raphael Ritz's MySite tutorial.

Definitions

Before we address this interesting topic we need to define our terms.

The two central concepts are internationalisation and localisation (i18n and l10n for short .. because 18 and 10 are how many letters were left out to make the abbreviation). Perhaps surprisingly, the concepts are somewhat independent: a site can be internationalised without being localised, and vice versa, although internationalisation supports localisation. A helpful dictionary definition is: Internationalisation abstracts out local details, localisation specifies those details for a particular locale. To sum up:

Internationalisation
Creating an infrastructure and guidelines that supports translation of the user interface and content.
Localisation
Providing translations and locale specific content, making use of the i18n infrastructure. Note that localised content need not necessarily imply translations. Can you think of occasions where it doesn't? [1]

Plone today is thoroughly internationalised, and products exist that help site authors to localise their sites.

Plone and Internationalisation

Internationalisation in Zope has a long history and many products , which I won't summarize here. Plone itself has used a number of products and approaches (such as Localizer and TranslationService), depending on what the state of the art was at the time ... you can find traces of this on the web. Currently, the standard approach is to use PlacelessTranslationService, I18NLayer, and I18NFolder.

Future plans for Plone include the unification of I18NLayer and I18NFolder, and LinguaPlone, which will provide a less intrusive method of associating content with translations than the current containment-based approach, and will probably replace the I18N* products.

For the purposes of this tutorial, we'll look at three topics, namely:

  • Translating Plone
  • MySite and translation
  • Translating Content

Plone is Internationalised

A lot of work has already gone into internationalising Plone. To experience this, go to the preferences page of your browser and change the selected language to another of Plone's supported languages.

Language preferences in Mozilla Language preferences in Mozilla

Language preferences in Internet Explorer Language preferences in Internet Explorer

Check plone.org for the list of supported languages

When you now visit your local Plone (or indeed most public Plone sites), you should see most of the user interface elements (all the text supplied by the system) in your language of choice. In our case, the exception would be any Events and their Deadlines that we added in the course of the MySite tutorial. For them, we would still see texts such as "Event Type" and "Upcoming Deadlines". Third party products are responsible for integrating with Plone i18n infrastructure, and providing their own translations.

If you want to translate Plone to a new language, you need to work with the Plone i18n Team . This team is responsible for assigning message IDs, quality assurance and translation conventions. Read about them on the web.

Visitors to your Plone site may or may not have their browser's Language preferences configured, depending on their computer supplier or IT staff, and they may not know what is determining their language choice. We'll see later how the browser negotiates which language to show.

Using the I18N infrastructure

Zope i18n uses gettext and PlacelessTranslationService.

gettext

Underlying all the Zope i18n work is the Gnu project's gettext system.

PlacelessTranslationService builds on gettext. Gettext supplies conventions for programs to support message catalogs and tools for parsing the catalogs. The gettext message catalogs in your Plone installation may be found in Products/<ProductName>/i18n directories on the filesystem.

PlacelessTranslationService

PlacelessTranslationService forms part of the Plone 2.0 distribution. If it's present in your Zope instance's Products directory, it shows up in the ZMI Control_Panel at /Control_Panel/TranslationService/manage_main after a restart. If you go there, you'll see the list of all the message catalogs found during Zope startup, and whether they were successfully parsed. PTS looks for message catalogs in directories called i18n. It searches for these directories in your Zope's INSTANCE_HOME, as well as in each product directory in $INSTANCE_HOME/Products . If you watch the Zope logfile when restarting, you'll see a lot of messages similar to this one: 

      ------
      2004-06-18T20:10:21 INFO(0) PlacelessTranslationService
      Initialized: ['portaltransport-cs.po', 'portaltransport-ru.po']
      from /<path to zope>/Products/PortalTransport/i18n

Translating Page Templates

The content of this section owes a lot (sometimes verbatim) to a good overview called i18n For Developers

We'll begin with a theoretical digression on the translation of Page Templates, and then start translating MySite.

Let's take a simple page template (the one from the overview linked above): 

      <html>
      <body>
        <p>Welcome to Plone.</p>
        <img src="plone.gif" alt="Plone Icon" />
        <p>There have been over 
          <span tal:content="here/download_count">100,000</span>
          downloads of Plone.</p>
        <p>Please visit <a href="about">About Plone</a>
          for more information.</p>
      </body>
      </html>

That template needs four different kinds of translation. They are:

  1. The text "Welcome to Plone.".
  2. The alt attribute "Plone Icon".
  3. The phrase "There have been over 100,000 downloads of Plone." The number 100,000 should be calculated dynamically, and in another language it may appear in a different position in the sentence.
  4. The phrase "Please visit About Plone for more information.". In this case, About Plone should remain a link, it should be translatable separately, and the position of the link name in the sentence may vary in other languages.

Apart from the translation work, we need to alert the PlacelessTranslationService that it needs to process this template. We do this by specifying a domain on the HTML element. For our product, we'll use a product-specific domain: 

      <html i18n:domain="mysite">

The value mysite specifies which message catalogs should be consulted during translation. If we were working on Plone templates, the value would be plone.

Case One: Welcome to Plone

We need to associate the text to translate with a message id that will be used to look up translations. If you leave the message id blank, the text to be translated will itself be used as the id (here: "Welcome to Plone.").

If this text is a part of Plone proper (in the plone domain), for which the i18n team takes responsibility, you need to give them a chance to assign a message id to each string to translate. Do this by using XXX for the id. Now, when the translation team use their tools to extract all the msgids from the page templates, it's obvious which ones need fixing. If this is a product of your own, you'll need to choose an id yourself. There are some guidelines for the naming of ids (from the Guidelines for translators [4]):

  • 'heading_': for <h> elements
  • 'description_': Explanatory text directly below
  • 'legend_': Used in <legend> elements
  • 'label_': For field labels, input labels, i.e. <label>, and for <a> elements
  • 'help_': Any text that provides help for form input.
  • 'box_': Content inside portlets.
  • 'listingheader_': For headers in tables (normally of class "listing").
  • 'date_': For date/time-related stuff. E.g. "Yesterday", "Last week".
  • 'text_': Messages that do not fit any other category, normally inside

  • 'batch_': for batch-related things - like "Displaying X to Y of Z total documents"

Exercise: When will you not choose an id yourself? [2]

Here, we're translating a simple phrase. To do this, we add an i18n:translate attribute to the P tag containing the translation. Let's pretend we're translating Plone, and use "XXX" as value for 'i18n:translate': 

        <p i18n:translate="XXX">Welcome to Plone.</p>

Ultimately, the translation team will choose a message identifer (msgid) to uniquely identify this phrase, replacing "XXX" with their id, and that will become the new value of the i18n:translate attribute.

Case Two: "Plone Icon"

To translate attributes, we use i18n:attributes instead of i18n:translate. The value of i18n:attributes is a list of all the attributes that require translation. This list may include the message ids; if it doesn't, the value of the attribute (here: "Plone Icon") will be used as the id.

For our example, the simplest possibility is: 

        <img src="plone.gif" alt="Plone Icon" i18n:attributes="alt" />

If we wanted to translate multiple attributes, we'd have: 

        <img src="plone.gif" alt="Plone Icon" title="Plone Icon Title"
            i18n:attributes="alt title">

If we also wanted to specify message ids, we'd have something like: 

        <img src="plone.gif" alt="Plone Icon" title="Plone Icon Title"
            i18n:attributes="alt mysite_plone_icon;
                             title mysite_plone_icon_title">

Note that the simple list of attributes is space-seperated, while the attribute-and-msgid list is semicolon-seperated.

Case Three: "There have been over 100,000 downloads of Plone."

As before, we use the i18n:translate attribute on the outer containing element (here, P) to mark the phrase for translation. In addition, we add i18n:name attributes on dynamic child elements that may change position. This gives us: 

        <p i18n:translate="XXX">There have been over 
          <span tal:content="here/download_count"
            i18n:name="count">100,000</span>
          downloads of Plone. </p>

When this is processed, the resulting message for translation will look like this: 

        msgid "XXX"
        msgstr "There have been ${count} downloads of Plone."

While msgids need to be unique within their i18n:domain (that's one reason for using naming conventions), the name only needs to be unique within the context of the enclosing tag.

Case Four: "Please visit About Plone for more information."

Here we are again translating a phrase that contains a subelement that may appear in different positions in other languanges. Furthermore, the subelement should itself be translated. To manage this, we use i18n:translate together with i18n:name. Let's pretend that this is part of our product, and assign msgids: 

        <p i18n:translate="mysite_more_plone_info">Please visit 
          <span i18n:name="about-plone">
            <a href="about"
               i18n:translate="mysite_about_plone">About Plone</a>
          </span> for more information.
        </p>

This results in two messages in the catalog, that may be translated individually: 

        msgid "mysite_more_plone_info"
        msgstr "Please visit ${about-plone} for more information."

        msgid "mysite_about_plone"
        msgstr "About Plone"

The final remaining possibility is that such a sub-element may also include an attribute to be translated. In this case, handle it as follows: 

        <p i18n:translate="mysite_more_plone_info">Please visit 
          <span i18n:name="about-plone">
            <a href="about" i18n:translate="mysite_about_plone"
               i18n:attributes="title" title="Go to About Page">
            About Plone</a>
          </span> for more information.
        </p>

This results in three messages in the catalog: the above two, and additionally: 

        msgid "Go to About Page"
        msgstr "Go to About Page"

Exercise: supply a msgid for the attribute. [3]

Translating MySite

OK, enough theory. Let's translate MySite and see what problems we encounter on the way!

Preparing the templates

Take a look at the MySite tempates in skins/my_templates. You'll notice that the preparatory work has already been done: the HTML elements have i18n:domain attributes, and there are i18n:translate attributes elsewhere in the templates. (You'll notice that MySite has very little content to translate.)

portlet_deadlines

The default domain for translations in this template is specified on the HTML element: 

        <html xmlns:tal="http://xml.zope.org/namespaces/tal"
              xmlns:metal="http://xml.zope.org/namespaces/metal"
              i18n:domain="mysite">

Since standard Plone doesn't mention deadlines anywhere, the first string for translation belongs in this domain: 

        <h5 i18n:translate="box_deadlines">Upcoming Deadlines</h5>

The next (and last!) string in this template is one that is common to most of the boxes in Plone. It already has a translation in the plone domain, so let's reference it there: 

        <a href=""
            class="portletMore"
            tal:attributes="href string:${portal_url}/events/deadlines"
            i18n:domain="plone"
            i18n:translate="box_morelink"> More...
        </a>

And that's it!

search_form

As mentioned in the solution to an earlier exercise ("Figure out in which respect the custom search template differs from Plone's original one"), the only change to MySite's search_form is where the list of types to search comes from. None of the text has changed. Therefore, the i18n:domain can be left at Plone: 

        <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
              lang="en"
              metal:use-macro="here/main_template/macros/master"
              i18n:domain="plone">

(Note to self: what's the role of the lang attributes here?)

Preparing the translation directory

PlacelessTranslationService needs an i18n directory, so let's create it: 

      MySite $ mkdir i18n

Now we need to put a message catalog in there. To get the strings for translation from our templates and into the catalog, we'll use i18ndude, a tool which is part of the plone-i18n project at SourceForge -- download it from there. (It requires Python 2.3, so check that you're up to date.) The tool includes help, but there's an i18ndude HowTo if you're looking for a shortcut. The only slightly tricky things are choosing the right domain for processing, and to tidy the resultant catalog. In our case, this little dance works: 

      MySite $ TEMPLATES=`find . -iregex '.*\..?pt'`
      MySite $ i18ndude rebuild-pot --pot i18n/mysite-tmp.pot \
               --create mysite $TEMPLATES 2> i18n/report.txt
      MySite $ i18ndude filter i18n/mysite-tmp.pot \ 
               ~/.../CMFPlone/i18n/plone.pot > i18n/mysite.pot

The first two lines are the same as the HowTo, but the last one is new. This is because the HowTo doesn't deal with templates that contain more than one i18n:domain, and i18ndude isn't clever enough to parse one domain at a time. That is, it generates a 'pot'-file putting all the messages in the mysite domain. We filter out the ones that are in the plone domain by comparing the generated 'pot'-file with the Plone master file (that already contains these msgids). This leaves us with i18n/mysite.pot containing a grand total of one (1) message (hey, it's the idea that counts!).

Translating code in Python files

In a well-designed Zope Product, the user interface should ideally be restricted to page templates in a skin layer. At the moment, however, there are some corners where text lingers in Python classes. One of these places is the labels and descriptions of widgets. One way of quickly finding widgets to translate is: 

      MySite $ grep -r  "Widget(" `find -name "*.py"`

In our case this yields Event.py and Deadline.py. Here are the widgets in 'Event.py': 

      StringField('event_type', vocabulary='EventTypes',
        widget=SelectionWidget(
            label='Event Type',
            description='The type of the event'),),
      ...
      UrlField("event_url",
        widget=UrlWidget(
            label="Homepage",
            description="Webpage of the event"),),

And here they are after internationalisation: 

      StringField('event_type', vocabulary='EventTypes',
        widget=SelectionWidget(
            label='Event Type',
            label_msgid='label_event_type',
            description='The type of the event',
            description_msgid='help_event_type',
            i18n_domain='mysite'),),
      ...
      UrlField("event_url",
        widget=UrlWidget(
            label="Homepage",
            label_msgid="label_homepage",
            description="Webpage of the event",
            description_msgid="help_homepage",
            i18n_domain='mysite'),),

What changed? And what motivated the domain choice? [5]

Here are the widgets in 'Deadline.py': 

      StringField('title',
        searchable=1,
        default='',
        accessor='Title',
        widget=StringWidget(visible={'edit':'hidden'}),
        ),
      StringField("deadline_type",
        vocabulary='DeadlineTypes',
        widget=SelectionWidget()
        ),

Take a close look at those. Are those all the fields you see when adding a deadline? Nooo .. we're still lacking the Date and the Comment fields. In the schema definition they're present as: 

      DateTimeField("date"),
      StringField("comment"),

Since no widget is specified, they get the default widgets for their fields. These are CalendarWidget and StringWidget (you need to climb a Jacob's ladder of imports and inheritance to find this out).

In order to get all these fields translated, add the i18n parameters: 

      StringField('title',
        searchable=1,
        default='',
        accessor='Title',
        widget=StringWidget(
            visible={'edit':'hidden'},
            label='Title',
            label_msgid='label_title',
            i18n_domain='plone',
            ),
        ),
      StringField("deadline_type",
        vocabulary='DeadlineTypes',
        widget=SelectionWidget(
            label='Deadline Type',
            label_msgid='label_deadline_type',
            description='Choose the type of deadline',
            description_msgid='help_deadline_type',
            i18n_domain='mysite',
            )
        ),
      DateTimeField("date",
        widget=CalendarWidget(
            label="Date",
            label_msgid="label_date",
            i18n_domain='plone',
            )),
      StringField("comment",
        widget=StringWidget(
            label="Comment",
            label_msgid="label_deadline_comment",
            description="Any further detail regarding the deadline",
            description_msgid="help_deadline_comment",
            i18n_domain='mysite',
            )),

Here's a quick question: why provide for the translation of the Deadline's Title, if it isn't visible? [6] Another question: why translate the Comment label? Surely Plone has already translated it? [7]

Note: this is fine, as far as it goes, but it does not take translation of vocabularies (e.g. vocabulary="DeadlineTypes",) into account.

Updating the i18n directory

We now have some more msgids to take care of. i18ndude won't find them in the *.py files, so we have to handle them manually. Let's follow Plone's lead, and put them into a manual.pot that we keep up to date ourselves. When you've done that, append contents of manual.pot to mysite.pot, to get a single complete master catalog.

Now, we need catalogs for our translations. I'm going to translate into Afrikaans, since that's what I know. First, ensure that the file exists, and then bring it into sync using 'i18ndude': 

      MySite $ touch i18n/mysite-af.po
      MySite $ i18ndude sync --pot i18n/mysite.pot i18n/mysite-af.po
      i18n/mysite-af.po: 10 added, 0 removed

Remember to edit all the required the metadata (such as Language-Code!) in the opening stanza of this file, and also in mysite.pot. Go ahead and translate the rest of the file as well. You'll find mysite-af.po in the MySite distribution, but create a new language that you know.

Note: Don't use single quotes in your message catalogs, as i18ndude ignores them. Wrong: 

      msgid 'help_deadline_type'
      msgstr 'Kies die soort keerdatum'

Right: 

      msgid "help_deadline_type"
      msgstr "Kies die soort keerdatum"

You need to restart Zope or refresh the catalog from the Zope Control_Panel to see the new translations.

Set your browser to the language of your translation, and you should see Events and Deadlines in your language of choice. [8]

Are we done yet?

Well, yes and no. Plone and Archetypes aren't going to do any more automatically, but we haven't translated everything. What have we missed? [9]

The Plone templates look up actions in the plone domain. Look at main_template and follow the white rabbit from there to e.g. global_contentviews, where you'll see: 

      <span tal:omit-tag="" i18n:translate="">
          <span tal:replace="action/name">dummy</span>
      </span>

As long as our actions are named the same as existing Plone actions, this code will translate them. If we have other actions that need to be translated, we'd have to add them to the plone domain. How do we do this? [10]

Although a product ideally would keep all user-visible text in Page Templates, and out of Python classes, this isn't always possible. And sure enough, our little MySite has a string squirreled away in Deadline.

Translating text in Python code

Here is the untranslated method: 

     class Deadline(BaseContent):
       [... snip ...]
       # provide a mainingful title
       def Title(self):
           """overwrite the default title field"""
           return "%s for '%s'" % (getattr(self, 'deadline_type', 'deadline'),
                                   self.aq_parent.getId(),
                                   )

Here it is translated: 

      def Title(self):
          """overwrite the default title field"""
          m = {'deadline_type': getattr(self, 'deadline_type', 'deadline'),
               'parent_id': self.aq_parent.getId(), }
          return self.translate('deadline_title',
                                mapping=m, domain='mysite')

This corresponds to the third use case under "Translating Page Templates" above. The corresponding Page Template snippet would be: 

      <span tal:omit-tag="" i18n:translate="deadline_title">
        <span i18n:name="deadline_type">Deadline</span> for
        <span i18n:name="parent_id">Event</span>
      </span>

What would the entry in the message catalog look like? [11]

Translating vocabularies

Both our Event and Deadline classes have fields with associated vocabularies that get their initial values from config.py. (They are event_type and deadline_type.) Archetypes does try to translate these vocabularies when rendering the widgets, but it always uses the plone domain (see the macros in Products/Archetypes/skins/archetypes/widgets/ ). Therefore, if we need to translate these vocabularies, we need to provide message catalog files with more msgids for the plone domain.

This is one way to do it:

  • Create Products/MySite/i18n/plone.pot and prepare it just as with manual.pot.
  • Create and generate plone-XX.pot (where XX is your language) using i18ndude.
  • Go ahead and translate.

Localising content

That's it for the translation of Products. The next step is to provide for the delivery of different content depending on the locale preferences of a user. To do this, we need current versions of the I18NLayer and I18NFolder products. The first is part of the Plone Collective project; get it at the I18NLayer download area The other one is provided by Ingeniweb. Get it from their I18NFolder homepage To install them, just unpack the tarballs in your Zope instance's Products folder and restart. Then, login to your Plone instance as a Manager user, visit the Add/Remove Products section of the Plone Control Panel, and install these products.

Although I18NLayer and I18NFolder have their limitations, you can build a fine internationalised site with them. Let's see, what are the limitations?

Folders and documents are treated differently
Most documents (i.e. content types that don't contain anything) can be translated using I18NLayer, but I18NFolder provides only a translatable version of Folder. Other containers (such as the MySite Event, which can contain Deadlines) cannot be translated.
Not all content types can be translated
You won't be able to translate your Member details, or issues in an issue tracker. How about bibliography entries? I'm not sure.
They aren't integrated into all aspects of Plone
In particular, the templates for searching, the news page, the news portlet, and other summary views will contain duplicates unless they're customised.

That said, let's create a translated area of our site.

  • Add an I18NFolder, and browse to it. Do you see a language dropdown just below the content view tabs? Err, right, you don't. To fix this, go have a look at the ReadMe for I18NFolder in the Plone Control Panel, fix it, and come back here. [12]
  • Now you should see a language dropdown. Unfortunately this dropdown contains all the languages known to Plone, which makes it unusable. To narrow it down, we need another product.

    Fetch it from the collective PloneLanguageTool download area and install as usual. Configure the portal_languages tool it provides by visiting it in the Zope Management Interface. In the Allowed Languages field, choose only the ones that you'll want to provide. Visit the new folder's language dropdown again, and you should see only the chosen languages.

In the translated folder, add some content. Content types that work well include Document, News Item, Event, Image and Photo. Topics can just about be accommodated, as long as they have no subtopics. (Don't add I18NLayer instances.)

When you've added a Document, and before trying to translate it, visit the folder in the ZMI (append manage_main to the URL). Just take note of the icon and satisfy yourself that you're browsing to a Document instance here.

That done, browse to it via the Plone UI and select a new language from the dropdown. Translate everything except the id field, which will have changed to the ISO language code of your target language. When you've saved the translation, go back to the folder in the ZMI, where you'll find that the icon has changed to the green I18NLayer icon. Browse to this item, and inside it you'll see not one but two Document instances, each named for the language code of the translation they provide.

This is how I18NLayer works its magic: it automatically wraps translated items in an I18NLayer container, and renders the correct one depending on the user's preferences.

[1] Here's a couple of examples:

  • Some people use a flag to indicate the locale a document is meant for (providing alternative images, not translations);
  • A business might display different information depending on the locale of a website visitor, e.g. listing German forums before forums in other languages (providing alternate information, not only translations).

[2] When you're re-using an existing Plone translation. Even though it requires quite intimate familiarity with Plone's message catalog, it can be a big win to use Plone's message ids within your product. There are many generic phrases (e.g. "Next ${number} items", msgid batch_next_x_items) that your product could use, and that are already maintained in more languages than you could manage on your own.

[3] Then you'll have: 

        <p i18n:translate="mysite_more_plone_info">Please visit 
          <span i18n:name="about-plone">
            <a href="about" i18n:translate="mysite_about_plone"
               i18n:attributes="title mysite_about_plone_title"
               title="Go to About Page">
            About Plone</a>
          </span> for more information.
        </p>

[4] This document recommends using product-prefixes: "There are also Product-specific prefixes, eg. the Product ZWiki has a heading, then the prefix would be zwiki_heading_edit_wiki_page. This prevents collision between Message IDs." When the product is in its own i18n domain, I don't believe it's necessary to use product-specific prefixes.

[5] We passed the parameters label_msgid, description_msgid and i18n_domain in both cases. Although standard Plone also provides an Event type as part of CMFCalendar, that type isn't an archetype, and therefore doesn't accommodate this kind of internationalisation. MySite's Event type belongs to its domain.

[6] It's only hidden on the edit form. It shows up on the view template.

[7] We could find "Comment" in the plone i18n domain, but we won't find a translation for "Any further detail regarding the deadline". Because we can only specify one i18n_domain for this field, we need to translate both.

[8] If you choose a language that isn't included among the 30-some provided with Plone, only Events and Deadlines will be translated and everything else will be in a fallback language, probably English.

[9] We've missed the action names (the actions = (...) parts), and the configuration text (EVENT_TYPES and DEADLINE_TYPES in config.py). We've also missed any text that might be returned from our Python code, if we're evil enough to have such text.

[10] Err, search me! Either hack the CMFPlone message catalogs, or supply another catalog for the plone domain. Probably the second one of these.

[11] For Afrikaans: 

    msgid "deadline_title"
    msgstr "${deadline_type} vir '${parent_id}'"

[12] At portal_skins/manage_propertiesForm, move up the I18NFolder layer so it looks like this: 

     i18n_layer
     I18NFolder
     i18n_layer_plone2
Document Actions